monitoring solr in nagios

I was finding it difficult to find anything that fit well, to monitor the solr 3.4 instances at work. Because of this I wrote a little plugin for nrpe/nagios and put it up on github

It seems to work pretty well, and only took a short time to make, there’s also a ganglia plugin I wrote, too, I’ll post details of that later.

No tag for this post.

Xen single user mode boot…

Lost a xen Dom0 the other day temporarily, and when it came back up one of the DomU’s was having issues booting because the label had mysteriously disappeared from the volume.

There used to be the possibility to add ‘single’ or ‘single init=/bin/bash’ to the extras option in the DomU’s configuration file, however this doesn’t work anymore and probably hasn’t since pygrub introduction.

Quick way to force single user mode is to get yourself access to the grub console, using -c when creating the domain:

xm create -c domu

Which will give you the grub boot menu, which you can edit the boot line to your requirements.

No tag for this post.

Replacing a failed disk in an SVM root mirror

So SVM isn’t something I seem to have to fiddle with much as it just works ™ once it’s all setup, but what if you have a disk fail, and this is an older system using SVM to mirror root/boot devices? Well it’s pretty simple and can all be done online…

First you need to identify which disk has actually failed. In our case it’s c0t0d0.

You’ll need to make sure that you remove the stale state database replicas first using metadb:

metadb

The output above shows the slices that replicas have been created on, to remove the ones from our disk that has failed use the following:

metadb -d /dev/dsk/c0t0d0s7

Now a lot of the time the case might be that the machine you’re working on needs to stay up whilst you’re repairing it. Thankfully Solaris will allow us to remove the disk whilst the system is still running and replace it. My advice would be to simply un-configure the disk that has failed, because even if we make a mistake it might be possible to recover some data from the disk we’re removing, depending on how long you’ve left it in error for.

cfgadm -al

Will show us where the disk lives, identify the correct disk in the list (look at the disks attached from c0, you should be able to work out from there…)

cfgadm -c unconfigure c0::dsk/c0t0d0

Now that the disk is removed, we need to get the list of the failed submirrors:

metastat -c

Will give us the list of mirrors that are failed because of c0t0d0, next remove them from the mirrors they are a member of:

metadetach -f d0 d1
metadetach -f d10 d11
metadetach -f d35 d31
metadetach -f d40 d41
metadetach -f d50 d51
metadetach -f d60 d61

Once they’ve been cleared we need to clear the concats:

metaclear d1
metaclear d11
metaclear d31
metaclear d41
metaclear d51
metaclear d61

Now that’s cleaned up we can insert the new disk and format it:

#insert the new hard disk...
cfgadm -c configure c0::dsk/c0t0d0
#get the vtoc configuration from the good disk
prtvtoc /dev/rdsk/c1t0d0s2 > /tmp/format.out
#and write it to the new disk
fmthard -s /tmp/format.out /dev/rdsk/c0t0d0s2

Now that we’ve done that it’d be a good idea to re-create the database state replicas on the new disk:

metadb -a -c 2 /dev/dsk/c0t0d0s7

Once this is done we can move on to recreating the concats:

metainit d1 1 1 c0t0d0s0
metainit d11 1 1 c0t0d0s1
metainit d31 1 1 c0t0d0s3
metainit d41 1 1 c0t0d0s4
metainit d51 1 1 c0t0d0s5
metainit d61 1 1 c0t0d0s6

And then attach these to the mirrors they’re meant to be in:

metattach d0 d1
metattach d10 d11
metattach d35 d31
metattach d40 d41
metattach d50 d51
metattach d60 d61

As you attach the mirrors you’ll be able to start monitoring the progress of the mirror re-syncs, you can do this by using metastat:

metastat -c

Which will give you a short output of all the configured disks, and show the “Resync %” status of each mirror that is being re-synced.

A fairly easy task once you know all the steps, you just need to be careful that once you’ve removed the bad concats that if you need to reboot at all, you boot from the correct disk. Usually I setup devaliases in the obp called bootdiska (c0t0d0) and bootdiskb (c1t0d0) and the devalias ‘disk’ is usually just an alias to the same place as bootdiska. So, given c0t0d0 is the disk that has failed on us, make sure you boot from the secondary disk:

ok boot disk

If you’re not sure what was created, just issue:

ok devalias

to show the list of aliases configured, if none useful seem to be there, use:

ok probe-scsi-all

to give you a list of the disks that are available on the system, from there you should be able to figure out where the disk you want to boot from is.

iSCSI quick-start guide.

I had to configure an iSCSI target for testing purposes, the steps I followed are below.

This assumes that you have the tgtadm software installed, and that you are making use of LVM.

1. create the backing volume in lvm

we have one volume group, called vg01, if you want to see what volume groups you have available, issue:

$ vgs
VG   #PV #LV #SN Attr   VSize  VFree
vg01   1   6   0 wz--n- 501G    295G

To create your volume, issue something like this:

$ lvcreate -L50G vg01 -n iscsiback01

Logical volume “iscsiback01” created

2. set up the iSCSI target
2.1 create the iSCSI target

$ tgtadm --lld iscsi --op new --mode target --tid 1 -T test-file01:storagetest

To verify this was created successfully:

$ tgtadm --lld iscsi --op show --mode target

Target 1: test-file01:storagetest
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store: No backing store
Account information:
ACL information:

2.2  Add a LUN to the target

$ tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg01/iscsiback01

2.3 Allow access to the new target

$ tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Instead of using “ALL” you can specify addresses that you want to be allowed to access these LUNs.

3. Set up an iSCSI initiator

Lets assume that our iSCSI target server is sat on 192.168.10.1, on the initiator you need to issue the following to discover the targets on there

$ iscsiadm --mode discovery --type sendtargets --portal 192.168.10.1
192.168.10.1:3260,1 test-file01:storagetest

Once you’ve done this log into the target session

$ iscsiadm --mode node --targetname test-file01:storagetest --portal 192.168.10.1:3260 --login

Once you’ve done this, the first LUN should be added as a new sdX device, “fdisk -l” should show this new disk, if not, restart the iscsi service on the initiator and then perform a partprobe.

If you add a new LUN to the target, you can always refresh your iscsi session to force it to show up:

$ iscsiadm -m session -R

It’s worth noting that this tool won’t save any of this configuration to a config file. I resorted to entering the commands above into /etc/rc.local to make sure they were run on boot. You also need to make sure that iscsid and tgtadm are set to start on boot, if you’re using RHEL or clone of that, a simple:


$ chkconfig iscsid on
$ chkconfig tgtadm on

Will do this for you, if you’re using debian:


$ update-rc.d iscsid defaults
$ update-rc.d tgtadm defaults

Should do the same.

See http://stgt.berlios.de/ for more information about tgt

xen vcpu pinning defaults aren’t ideal

I noticed an oddity the other day with a xen Domain0 host we have. There’s a cron scripted job that verifies the RPM database and the RPM’s that are installed on the system, for some reason this job failed, but kept the process open, and kept spinning around trying to do it’s job. Now, I really ought to have set up a “process count” check on the nagios monitoring we have here, but I didn’t have this at the time, so didn’t pick it up for a few days. Whilst this was all going on, the Domain0 got pretty busy and started having to use time on the other CPU’s as well as the main VCPU that wasn’t pinned to anything but the Domain0.

You can see this from the list below of the vcpu resources used by a xen server currently:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   -b-  1535018.3 0
Domain-0                           0     1     1   -b-  139549.6 1
Domain-0                           0     2     2   -b-  943651.0 2
Domain-0                           0     3     3   -b-   53883.4 3
Domain-0                           0     4     4   -b-  336268.9 4
Domain-0                           0     5     5   -b-   65240.1 5
Domain-0                           0     6     6   -b-   42854.6 6
Domain-0                           0     7     7   r–   67960.9 7
domain1                           4     0     2   r–  1791844.4 1-2
domain1                           4     1     1   r–  1619120.1 1-2
domain2                       5     0     3   -b-  511300.0 3-5
domain2                       5     1     3   -b-  456253.1 3-5
domain2                       5     2     5   -b-  456516.1 3-5
domain3                     6     0     6   -b-  166344.6 6-7
domain3                     6     1     7   -b-  137435.2 6-7

You’ll see Domain-0 which is the control domain, is pinned to all the other cpu’s that should only be used by the guests.

This isn’t ideal, and as a result you find that usually instead of a vmstat looking quite healthy and the “steal %” value that shows up being at 0, it’ll start to creep up. This means that the scheduler on the Domain0 side is interrupting the VCPU and requires CPU time from it, interrupting whatever is happening on the DomainU side.

There is a vcpu-pin action available within the xm command, which isn’t ideal to be used when you have the server live. What I found best, was to change the boot configuration for the Domain0 from the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

To the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5 dom0_max_vcpus=1
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

You’ll notice the option dom0_max_vcpus=1, this tells the Domain0 to pin to only one available VCPU, the one it’ll choose should be the first one.

You’ll see a difference in the vcpu-list afterwards like this:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   r–      54.0 0
domain1                           3     0     7   -b-       3.2 6-7
domain1                           3     1     6   -b-       3.0 6-7
domain2                          1     0     1   -b-      10.3 1-2
domain2                           1     1     2   -b-       2.9 1-2
domain3                          2     0     3   -b-       3.7 3-5
domain3                          2     1     4   -b-       2.5 3-5
domain3                          2     2     5   -b-       0.9 3-5

It’s worth noting that you can also limit this on the fly, by using the following command:

xm vcpu-pin Domain0 0 0

Which can be useful if you can’t get the down time for a box and it’s guests.