First you need to identify which disk has actually failed. In our case it’s c0t0d0.

You’ll need to make sure that you remove the stale state database replicas first using metadb:

metadb

The output above shows the slices that replicas have been created on, to remove the ones from our disk that has failed use the following:

metadb -d /dev/dsk/c0t0d0s7

Now a lot of the time the case might be that the machine you’re working on needs to stay up whilst you’re repairing it. Thankfully Solaris will allow us to remove the disk whilst the system is still running and replace it. My advice would be to simply un-configure the disk that has failed, because even if we make a mistake it might be possible to recover some data from the disk we’re removing, depending on how long you’ve left it in error for.

cfgadm -al

Will show us where the disk lives, identify the correct disk in the list (look at the disks attached from c0, you should be able to work out from there…)

cfgadm -c unconfigure c0::dsk/c0t0d0

Now that the disk is removed, we need to get the list of the failed submirrors:

metastat -c

Will give us the list of mirrors that are failed because of c0t0d0, next remove them from the mirrors they are a member of:

metadetach -f d0 d1
metadetach -f d10 d11
metadetach -f d35 d31
metadetach -f d40 d41
metadetach -f d50 d51
metadetach -f d60 d61

Once they’ve been cleared we need to clear the concats:

metaclear d1
metaclear d11
metaclear d31
metaclear d41
metaclear d51
metaclear d61


Now that’s cleaned up we can insert the new disk and format it:

#insert the new hard disk...
cfgadm -c configure c0::dsk/c0t0d0
#get the vtoc configuration from the good disk
prtvtoc /dev/rdsk/c1t0d0s2 > /tmp/format.out
#and write it to the new disk
fmthard -s /tmp/format.out /dev/rdsk/c0t0d0s2


Now that we’ve done that it’d be a good idea to re-create the database state replicas on the new disk:

metadb -a -c 2 /dev/dsk/c0t0d0s7

Once this is done we can move on to recreating the concats:

metainit d1 1 1 c0t0d0s0
metainit d11 1 1 c0t0d0s1
metainit d31 1 1 c0t0d0s3
metainit d41 1 1 c0t0d0s4
metainit d51 1 1 c0t0d0s5
metainit d61 1 1 c0t0d0s6

And then attach these to the mirrors they’re meant to be in:

metattach d0 d1
metattach d10 d11
metattach d35 d31
metattach d40 d41
metattach d50 d51
metattach d60 d61

As you attach the mirrors you’ll be able to start monitoring the progress of the mirror re-syncs, you can do this by using metastat:

metastat -c

Which will give you a short output of all the configured disks, and show the “Resync %” status of each mirror that is being re-synced.

A fairly easy task once you know all the steps, you just need to be careful that once you’ve removed the bad concats that if you need to reboot at all, you boot from the correct disk. Usually I setup devaliases in the obp called bootdiska (c0t0d0) and bootdiskb (c1t0d0) and the devalias ‘disk’ is usually just an alias to the same place as bootdiska. So, given c0t0d0 is the disk that has failed on us, make sure you boot from the secondary disk:

ok boot disk

If you’re not sure what was created, just issue:

ok devalias

to show the list of aliases configured, if none useful seem to be there, use:

ok probe-scsi-all

to give you a list of the disks that are available on the system, from there you should be able to figure out where the disk you want to boot from is.

This assumes that you have the tgtadm software installed, and that you are making use of LVM.

1. create the backing volume in lvm

we have one volume group, called vg01, if you want to see what volume groups you have available, issue:

$ vgs
VG   #PV #LV #SN Attr   VSize  VFree
vg01   1   6   0 wz--n- 501G    295G

To create your volume, issue something like this:

$ lvcreate -L50G vg01 -n iscsiback01

Logical volume “iscsiback01″ created

2. set up the iSCSI target

2.1 create the iSCSI target

$ tgtadm --lld iscsi --op new --mode target --tid 1 -T test-file01:storagetest

To verify this was created successfully:

$ tgtadm --lld iscsi --op show --mode target

Target 1: test-file01:storagetest
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store: No backing store
Account information:
ACL information:


2.2  Add a LUN to the target

$ tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg01/iscsiback01

2.3 Allow access to the new target

$ tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Instead of using “ALL” you can specify addresses that you want to be allowed to access these LUNs.

3. Set up an iSCSI initiator

Lets assume that our iSCSI target server is sat on 192.168.10.1, on the initiator you need to issue the following to discover the targets on there

$ iscsiadm --mode discovery --type sendtargets --portal 192.168.10.1
192.168.10.1:3260,1 test-file01:storagetest


Once you’ve done this log into the target session

$ iscsiadm --mode node --targetname test-file01:storagetest --portal 192.168.10.1:3260 --login

Once you’ve done this, the first LUN should be added as a new sdX device, “fdisk -l” should show this new disk, if not, restart the iscsi service on the initiator and then perform a partprobe.

If you add a new LUN to the target, you can always refresh your iscsi session to force it to show up:

$ iscsiadm -m session -R

It’s worth noting that this tool won’t save any of this configuration to a config file. I resorted to entering the commands above into /etc/rc.local to make sure they were run on boot. You also need to make sure that iscsid and tgtadm are set to start on boot, if you’re using RHEL or clone of that, a simple:


$ chkconfig iscsid on
$ chkconfig tgtadm on

Will do this for you, if you’re using debian:


$ update-rc.d iscsid defaults
$ update-rc.d tgtadm defaults

Should do the same.

See http://stgt.berlios.de/ for more information about tgt

A server I have access too uses LDAP for user info and Kerberos5 for realm authentication. It was reported that this server wasn’t letting anyone login via ssh, and the only way that I was able to login, was via the console connection for the box (so essentially the only way to connect was locally).

I was able to prove that LDAP lookups were working, by simply id’ing on user accounts I knew to not exist locally which were stored in LDAP. I was also able to init a kerberos ticket when logged in, and login as ldap/krb5 users “locally”.

After a while of faffing about, enabling debug logging on sshd and so on, it dawned on me to check the disk space, thanks to an odd I/O moan in the sshd debug log. Low and behold! the partition where the kerberos key cache for ssh was completely full!

It goes to show that even simple checks like that which sometimes seem noddy, should always be done!