[ t h e f r a g g l e . c o m ]

I noticed an oddity the other day with a xen Domain0 host we have. There’s a cron scripted job that verifies the RPM database and the RPM’s that are installed on the system, for some reason this job failed, but kept the process open, and kept spinning around trying to do it’s job. Now, I really ought to have set up a “process count” check on the nagios monitoring we have here, but I didn’t have this at the time, so didn’t pick it up for a few days. Whilst this was all going on, the Domain0 got pretty busy and started having to use time on the other CPU’s as well as the main VCPU that wasn’t pinned to anything but the Domain0.

You can see this from the list below of the vcpu resources used by a xen server currently:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   -b-  1535018.3 0
Domain-0                           0     1     1   -b-  139549.6 1
Domain-0                           0     2     2   -b-  943651.0 2
Domain-0                           0     3     3   -b-   53883.4 3
Domain-0                           0     4     4   -b-  336268.9 4
Domain-0                           0     5     5   -b-   65240.1 5
Domain-0                           0     6     6   -b-   42854.6 6
Domain-0                           0     7     7   r–   67960.9 7
domain1                           4     0     2   r–  1791844.4 1-2
domain1                           4     1     1   r–  1619120.1 1-2
domain2                       5     0     3   -b-  511300.0 3-5
domain2                       5     1     3   -b-  456253.1 3-5
domain2                       5     2     5   -b-  456516.1 3-5
domain3                     6     0     6   -b-  166344.6 6-7
domain3                     6     1     7   -b-  137435.2 6-7

You’ll see Domain-0 which is the control domain, is pinned to all the other cpu’s that should only be used by the guests.

This isn’t ideal, and as a result you find that usually instead of a vmstat looking quite healthy and the “steal %” value that shows up being at 0, it’ll start to creep up. This means that the scheduler on the Domain0 side is interrupting the VCPU and requires CPU time from it, interrupting whatever is happening on the DomainU side.

There is a vcpu-pin action available within the xm command, which isn’t ideal to be used when you have the server live. What I found best, was to change the boot configuration for the Domain0 from the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

To the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5 dom0_max_vcpus=1
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

You’ll notice the option dom0_max_vcpus=1, this tells the Domain0 to pin to only one available VCPU, the one it’ll choose should be the first one.

You’ll see a difference in the vcpu-list afterwards like this:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   r–      54.0 0
domain1                           3     0     7   -b-       3.2 6-7
domain1                           3     1     6   -b-       3.0 6-7
domain2                          1     0     1   -b-      10.3 1-2
domain2                           1     1     2   -b-       2.9 1-2
domain3                          2     0     3   -b-       3.7 3-5
domain3                          2     1     4   -b-       2.5 3-5
domain3                          2     2     5   -b-       0.9 3-5

It’s worth noting that you can also limit this on the fly, by using the following command:

xm vcpu-pin Domain0 0 0

Which can be useful if you can’t get the down time for a box and it’s guests.

Most people won’t find this paticularly useful, but as I’m prone to forget things:

sed -e '/^$/d'

will delete blank newlines from whatever it’s input is.

Well reading popeys blog entry on how many servers he has, and what he uses them for made me feel a bit better than I did previously, about running more than one server of my own for personal use. I only have three servers, and a workstation and a laptop; maybe I don’t waste as much electricity as I thought I did

• etch.thefraggle.com – Debian etch, xen vps from bitfolk; general webserver for www.thefraggle.com, and master mail server.
• sarge.thefraggle.com – Debian etch, xen vps from bitfolk; run’s IRCd’s for blitzed.org and nixhelp.org and tertiary mail exchanger.
• beastie.thefraggle.com – FreeBSD-6.2-stable on an old p2 400mhz 128mb ram; used to run an ircd for nixhelp, and thefraggle.com website, but now has been retired to being a development machine and tertiary mail exchanger.
• laptop – centrino duo 1.7ghz 1gb ram; work laptop with winxp / debian etch for work stuff

There’s actually another box there, my dads p4 3ghz, that I have pretty much nicked off him for day to day internetting . I suppose the fact that I have three servers kind of means I am pretty geeky?

Would be interesting if anyone reading this also commented with what they use .

Another nifty thing I figured out a while ago, whilst having to upload a script written by one of the guys we support, in a windows editor, was how to check quickley and easily with cat(1)whether it had silly windows CR/LF’s :

cat -ev /path/to/file

Would output something like the following if the file had silly line endings…

This is some text written in windaz^M$
And another line edited in windaz, woo^M$

cat suffixes the end of line with a $ and shows any special characters that have been inserted, in our case here ^M is the windows special character for new line, so all you need to do is remove that, which is again pretty easy:

sed -i.bak -sed -e "s/r//g" /path/to/file

Should remove the windaz file endings from the file, copying the file with a .bak suffi as a backup.

So today I had to basically clone a hard disk from a failing disk, to a new disk Sun had sent for one of the v20z’s at work… Haven’t not had to mess about with partition tables and other such things outside of the installer in linux for some time (last time I did this kind of messing about was with freebsd which uses slices->partitions, unlike linux and it’s mad idea of partitions and extended partitions), so I had to take stock for a short while while I figured out the best way to go ahead. Here’s what I started with :

/dev/sda – main scsi hard disk at scsi id 0
/dev/sdb – new replacement disk at scsi id 1

Saving and restorring the partition table to the sdb was as easy as:

sfdisk -d /dev/sda > /tmp/sda-parttable.out

and then:

cat /tmp/sda-parttable.out | sdisk /dev/sdb

I’d never realised sfdisk could do easy things like that, which saved me some time having to input partition boundries by hand What next was a bit of a pain was getting files over to the new filesystems (obviously once I had created new filesystems on them and mounted them somewhere easy to use – /mnt/newroot). I had looked at possibly using cpio to do this, and considered dd.

It seemed the best in the end, to create tar archives of the data, so that if I made a mess of the new disk I could at least quickly extract the backups again without too much faff, to do this I simply archived up each partition using something along the lines of:

tar --preserve --one-file-system -vcf /mnt/newroot/root.tar /

this archived the root into /mnt/newroot/root.tar, but diddn’t traverse into new directories, –one-file-system is a pretty handy feature of tar and I’m sure it’ll come in useful again sometime in the future. Tomorrow I’ll update with how easy it was to install the new mbr to sdb and how removing the origional disk and moving sdb to sda’s slot, goes