[ t h e f r a g g l e . c o m ]

I was finding it difficult to find anything that fit well, to monitor the solr 3.4 instances at work. Because of this I wrote a little plugin for nrpe/nagios and put it up on github

It seems to work pretty well, and only took a short time to make, there’s also a ganglia plugin I wrote, too, I’ll post details of that later.

A few big things have happened since I last posted…

• I left Frankfurt, and moved to London
• I decided to lose weight
• I decided to take part in a new sport, Olympic Weightlifting

Moving to Frankfurt wasn’t such a great move after all

I really enjoyed my time there, Germany is such an amazing place to live. The people, the food, the beer, basically the entire lifestyle is great.

The job however didn’t quite turn out to what I was lead to believe it would be. Unfortunately it turned into a pretty basic operators role, with no real scope for project work or building up any new infrastructure, as this was all handled by another team.

It also turned out that they had plans to have everyone move into a shift pattern, covering 6am – 9pm, but critically including weekends as normal working days. I’ve worked similar shift patterns before, and it took over my entire life. I really didn’t want that to happen again so I made the difficult decision to move back to the UK.

Living in Germany made me want to get fit

Most of the people I came across living out there, had a healthy lifestyle, they ate and drank beer, sure, but the weekends were for biking or hiking or whatever activities you could think of doing in the forests. There’s so much beautiful countryside surrounding Frankfurt, and most people are loathe to ignore it.

I started to eat better again, and managed to go from weighing 110kg (240lbs) to around 75kg (165lbs), given I’m only 5’7″ weighing 110kg was really really bad for me.

But what should I do to stay thin, and fit?

Well, I decided to take up olympic weightlifting, I got a coach and started to learn the lifts, a couple of months in though, I got chatting to another lifter, who was doing his BWLA coaching course. He said he’d help me train and so I decided to train with him, instead. Unfortunately after a while trying to get hold of this coach was pretty difficult, so from November to December 2011, I was without a coach, making few gains in my lifts and making more and more technical mistakes.

So, come January, I decided to join Bethnal Green Weightlifting Club. So far this year I’ve added 15kg to each of my lifts and even competed in my first ever weightlifting competition!

I’m hoping to make a total that will qualify me for the English Senior championships next year, at the next London open in December.

Maybe I’ll manage to keep updating with more stuff, writing blog posts is time consuming though, so we’ll have to see!

Lost a xen Dom0 the other day temporarily, and when it came back up one of the DomU’s was having issues booting because the label had mysteriously disappeared from the volume.

There used to be the possibility to add ‘single’ or ‘single init=/bin/bash’ to the extras option in the DomU’s configuration file, however this doesn’t work anymore and probably hasn’t since pygrub introduction.

Quick way to force single user mode is to get yourself access to the grub console, using -c when creating the domain:

xm create -c domu

Which will give you the grub boot menu, which you can edit the boot line to your requirements.

So SVM isn’t something I seem to have to fiddle with much as it just works ™ once it’s all setup, but what if you have a disk fail, and this is an older system using SVM to mirror root/boot devices? Well it’s pretty simple and can all be done online…

First you need to identify which disk has actually failed. In our case it’s c0t0d0.

You’ll need to make sure that you remove the stale state database replicas first using metadb:

metadb

The output above shows the slices that replicas have been created on, to remove the ones from our disk that has failed use the following:

metadb -d /dev/dsk/c0t0d0s7

Now a lot of the time the case might be that the machine you’re working on needs to stay up whilst you’re repairing it. Thankfully Solaris will allow us to remove the disk whilst the system is still running and replace it. My advice would be to simply un-configure the disk that has failed, because even if we make a mistake it might be possible to recover some data from the disk we’re removing, depending on how long you’ve left it in error for.

cfgadm -al

Will show us where the disk lives, identify the correct disk in the list (look at the disks attached from c0, you should be able to work out from there…)

cfgadm -c unconfigure c0::dsk/c0t0d0

Now that the disk is removed, we need to get the list of the failed submirrors:

metastat -c

Will give us the list of mirrors that are failed because of c0t0d0, next remove them from the mirrors they are a member of:

metadetach -f d0 d1
metadetach -f d10 d11
metadetach -f d35 d31
metadetach -f d40 d41
metadetach -f d50 d51
metadetach -f d60 d61

Once they’ve been cleared we need to clear the concats:

metaclear d1
metaclear d11
metaclear d31
metaclear d41
metaclear d51
metaclear d61


Now that’s cleaned up we can insert the new disk and format it:

#insert the new hard disk...
cfgadm -c configure c0::dsk/c0t0d0
#get the vtoc configuration from the good disk
prtvtoc /dev/rdsk/c1t0d0s2 > /tmp/format.out
#and write it to the new disk
fmthard -s /tmp/format.out /dev/rdsk/c0t0d0s2


Now that we’ve done that it’d be a good idea to re-create the database state replicas on the new disk:

metadb -a -c 2 /dev/dsk/c0t0d0s7

Once this is done we can move on to recreating the concats:

metainit d1 1 1 c0t0d0s0
metainit d11 1 1 c0t0d0s1
metainit d31 1 1 c0t0d0s3
metainit d41 1 1 c0t0d0s4
metainit d51 1 1 c0t0d0s5
metainit d61 1 1 c0t0d0s6

And then attach these to the mirrors they’re meant to be in:

metattach d0 d1
metattach d10 d11
metattach d35 d31
metattach d40 d41
metattach d50 d51
metattach d60 d61

As you attach the mirrors you’ll be able to start monitoring the progress of the mirror re-syncs, you can do this by using metastat:

metastat -c

Which will give you a short output of all the configured disks, and show the “Resync %” status of each mirror that is being re-synced.

A fairly easy task once you know all the steps, you just need to be careful that once you’ve removed the bad concats that if you need to reboot at all, you boot from the correct disk. Usually I setup devaliases in the obp called bootdiska (c0t0d0) and bootdiskb (c1t0d0) and the devalias ‘disk’ is usually just an alias to the same place as bootdiska. So, given c0t0d0 is the disk that has failed on us, make sure you boot from the secondary disk:

ok boot disk

If you’re not sure what was created, just issue:

ok devalias

to show the list of aliases configured, if none useful seem to be there, use:

ok probe-scsi-all

to give you a list of the disks that are available on the system, from there you should be able to figure out where the disk you want to boot from is.

I had to configure an iSCSI target for testing purposes, the steps I followed are below.

This assumes that you have the tgtadm software installed, and that you are making use of LVM.

1. create the backing volume in lvm

we have one volume group, called vg01, if you want to see what volume groups you have available, issue:

$ vgs
VG   #PV #LV #SN Attr   VSize  VFree
vg01   1   6   0 wz--n- 501G    295G

To create your volume, issue something like this:

$ lvcreate -L50G vg01 -n iscsiback01

Logical volume “iscsiback01″ created

2. set up the iSCSI target

2.1 create the iSCSI target

$ tgtadm --lld iscsi --op new --mode target --tid 1 -T test-file01:storagetest

To verify this was created successfully:

$ tgtadm --lld iscsi --op show --mode target

Target 1: test-file01:storagetest
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store: No backing store
Account information:
ACL information:


2.2  Add a LUN to the target

$ tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg01/iscsiback01

2.3 Allow access to the new target

$ tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Instead of using “ALL” you can specify addresses that you want to be allowed to access these LUNs.

3. Set up an iSCSI initiator

Lets assume that our iSCSI target server is sat on 192.168.10.1, on the initiator you need to issue the following to discover the targets on there

$ iscsiadm --mode discovery --type sendtargets --portal 192.168.10.1
192.168.10.1:3260,1 test-file01:storagetest


Once you’ve done this log into the target session

$ iscsiadm --mode node --targetname test-file01:storagetest --portal 192.168.10.1:3260 --login

Once you’ve done this, the first LUN should be added as a new sdX device, “fdisk -l” should show this new disk, if not, restart the iscsi service on the initiator and then perform a partprobe.

If you add a new LUN to the target, you can always refresh your iscsi session to force it to show up:

$ iscsiadm -m session -R

It’s worth noting that this tool won’t save any of this configuration to a config file. I resorted to entering the commands above into /etc/rc.local to make sure they were run on boot. You also need to make sure that iscsid and tgtadm are set to start on boot, if you’re using RHEL or clone of that, a simple:


$ chkconfig iscsid on
$ chkconfig tgtadm on

Will do this for you, if you’re using debian:


$ update-rc.d iscsid defaults
$ update-rc.d tgtadm defaults

Should do the same.

See http://stgt.berlios.de/ for more information about tgt