[ t h e f r a g g l e . c o m ]

I had to configure an iSCSI target for testing purposes, the steps I followed are below.

This assumes that you have the tgtadm software installed, and that you are making use of LVM.

1. create the backing volume in lvm

we have one volume group, called vg01, if you want to see what volume groups you have available, issue:

$ vgs
VG   #PV #LV #SN Attr   VSize  VFree
vg01   1   6   0 wz--n- 501G    295G

To create your volume, issue something like this:

$ lvcreate -L50G vg01 -n iscsiback01

Logical volume “iscsiback01″ created

2. set up the iSCSI target

2.1 create the iSCSI target

$ tgtadm --lld iscsi --op new --mode target --tid 1 -T test-file01:storagetest

To verify this was created successfully:

$ tgtadm --lld iscsi --op show --mode target

Target 1: test-file01:storagetest
System information:
Driver: iscsi
State: ready
I_T nexus information:
LUN information:
LUN: 0
Type: controller
SCSI ID: deadbeaf1:0
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store: No backing store
Account information:
ACL information:


2.2  Add a LUN to the target

$ tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg01/iscsiback01

2.3 Allow access to the new target

$ tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL

Instead of using “ALL” you can specify addresses that you want to be allowed to access these LUNs.

3. Set up an iSCSI initiator

Lets assume that our iSCSI target server is sat on 192.168.10.1, on the initiator you need to issue the following to discover the targets on there

$ iscsiadm --mode discovery --type sendtargets --portal 192.168.10.1
192.168.10.1:3260,1 test-file01:storagetest


Once you’ve done this log into the target session

$ iscsiadm --mode node --targetname test-file01:storagetest --portal 192.168.10.1:3260 --login

Once you’ve done this, the first LUN should be added as a new sdX device, “fdisk -l” should show this new disk, if not, restart the iscsi service on the initiator and then perform a partprobe.

If you add a new LUN to the target, you can always refresh your iscsi session to force it to show up:

$ iscsiadm -m session -R

It’s worth noting that this tool won’t save any of this configuration to a config file. I resorted to entering the commands above into /etc/rc.local to make sure they were run on boot. You also need to make sure that iscsid and tgtadm are set to start on boot, if you’re using RHEL or clone of that, a simple:


$ chkconfig iscsid on
$ chkconfig tgtadm on

Will do this for you, if you’re using debian:


$ update-rc.d iscsid defaults
$ update-rc.d tgtadm defaults

Should do the same.

See http://stgt.berlios.de/ for more information about tgt

I noticed an oddity the other day with a xen Domain0 host we have. There’s a cron scripted job that verifies the RPM database and the RPM’s that are installed on the system, for some reason this job failed, but kept the process open, and kept spinning around trying to do it’s job. Now, I really ought to have set up a “process count” check on the nagios monitoring we have here, but I didn’t have this at the time, so didn’t pick it up for a few days. Whilst this was all going on, the Domain0 got pretty busy and started having to use time on the other CPU’s as well as the main VCPU that wasn’t pinned to anything but the Domain0.

You can see this from the list below of the vcpu resources used by a xen server currently:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   -b-  1535018.3 0
Domain-0                           0     1     1   -b-  139549.6 1
Domain-0                           0     2     2   -b-  943651.0 2
Domain-0                           0     3     3   -b-   53883.4 3
Domain-0                           0     4     4   -b-  336268.9 4
Domain-0                           0     5     5   -b-   65240.1 5
Domain-0                           0     6     6   -b-   42854.6 6
Domain-0                           0     7     7   r–   67960.9 7
domain1                           4     0     2   r–  1791844.4 1-2
domain1                           4     1     1   r–  1619120.1 1-2
domain2                       5     0     3   -b-  511300.0 3-5
domain2                       5     1     3   -b-  456253.1 3-5
domain2                       5     2     5   -b-  456516.1 3-5
domain3                     6     0     6   -b-  166344.6 6-7
domain3                     6     1     7   -b-  137435.2 6-7

You’ll see Domain-0 which is the control domain, is pinned to all the other cpu’s that should only be used by the guests.

This isn’t ideal, and as a result you find that usually instead of a vmstat looking quite healthy and the “steal %” value that shows up being at 0, it’ll start to creep up. This means that the scheduler on the Domain0 side is interrupting the VCPU and requires CPU time from it, interrupting whatever is happening on the DomainU side.

There is a vcpu-pin action available within the xm command, which isn’t ideal to be used when you have the server live. What I found best, was to change the boot configuration for the Domain0 from the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

To the following:

title Enterprise Linux (2.6.18-128.el5xen)
root (hd0,0)
kernel /xen.gz-2.6.18-128.el5 dom0_max_vcpus=1
module /vmlinuz-2.6.18-128.el5xen ro root=/dev/vg01/root console=tty0 rhgb quiet
module /initrd-2.6.18-128.el5xen.img

You’ll notice the option dom0_max_vcpus=1, this tells the Domain0 to pin to only one available VCPU, the one it’ll choose should be the first one.

You’ll see a difference in the vcpu-list afterwards like this:

[root@somedomain0 ~]# xm vcpu-list
Name                              ID VCPUs   CPU State   Time(s) CPU Affinity
Domain-0                           0     0     0   r–      54.0 0
domain1                           3     0     7   -b-       3.2 6-7
domain1                           3     1     6   -b-       3.0 6-7
domain2                          1     0     1   -b-      10.3 1-2
domain2                           1     1     2   -b-       2.9 1-2
domain3                          2     0     3   -b-       3.7 3-5
domain3                          2     1     4   -b-       2.5 3-5
domain3                          2     2     5   -b-       0.9 3-5

It’s worth noting that you can also limit this on the fly, by using the following command:

xm vcpu-pin Domain0 0 0

Which can be useful if you can’t get the down time for a box and it’s guests.