I am not actually set up yet to take advantage of the multipath part of the equation… and I didn’t realize (because *none* of the doco sources I’ve been looking at bothered to say so) that the stock install comes with a block-iscsi script that does *not* do multipath, but apparently works ok.

Oops. :-}

I’ll come back to this, but probably not for a week or two. I’ll send you patches when I figure them out. Thanks for the help, though, sir.

Oh. Nope; just a typography problem.banner.

/appl/xen/cent/57 # set -vx
banner:/appl/xen/cent/57 # SCSI_DEV=”/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\“”
SCSI_DEV=”/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\“”
basename `/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0
+++ /usr/bin/readlink /dev/disk/by-path/ip-:-iscsi–lun-0
++ basename
basename: missing operand
Try `basename –help’ for more information.
+ SCSI_DEV=/dev/

Aha.

The symlinks in the /dev/disk/by-path directory on SUSE are *relative*; they’re “../../sda”. Is that what’s causing the problem?

SCSI_DEV=”/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\“”

MPATH_MAP=”/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk ‘{ print $1}’`”

Sep 28 15:18:26 banner kernel: [344763.765160] blkback: ring-ref 8, event-channel 70, protocol 1 (x86_32-abi)
Sep 28 15:18:26 banner kernel: [344763.767690] blkback: ring-ref 8, event-channel 70, protocol 1 (x86_32-abi)
Sep 28 15:18:26 banner logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/0/268441856
Sep 28 15:18:26 banner logger: /etc/xen/scripts/block-iscsi: add XENBUS_PATH=backend/vbd/0/268441856
Sep 28 15:18:26 banner block-iscsi: *** Beginning device add ***
Sep 28 15:18:26 banner block-iscsi: IQN: iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0
Sep 28 15:18:26 banner block-iscsi: 192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0 192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot1
Sep 28 15:18:26 banner block-iscsi: TARGET: 192.168.1.190:3260
Sep 28 15:18:26 banner block-iscsi: logging in to iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0 on 192.168.1.190:3260
Sep 28 15:18:31 banner kernel: [344769.234912] scsi6 : iSCSI Initiator over TCP/IP
Sep 28 15:18:32 banner kernel: [344769.491819] scsi 6:0:0:0: Direct-Access Openfile Virtual disk 0 PQ: 0 ANSI: 4
Sep 28 15:18:32 banner block-iscsi: Logging in to [iface: default, target: iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0, portal: 192.168.1.190,3260]
Sep 28 15:18:32 banner block-iscsi: Login to [iface: default, target: iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0, portal: 192.168.1.190,3260]: successful
Sep 28 15:18:32 banner kernel: [344769.492445] sd 6:0:0:0: Attached scsi generic sg2 type 0
Sep 28 15:18:32 banner kernel: [344769.494970] sd 6:0:0:0: [sda] 16777216 512-byte logical blocks: (8.58 GB/8.00 GiB)
Sep 28 15:18:32 banner kernel: [344769.495315] sd 6:0:0:0: [sda] Write Protect is off
Sep 28 15:18:32 banner kernel: [344769.495322] sd 6:0:0:0: [sda] Mode Sense: 77 00 00 08
Sep 28 15:18:32 banner kernel: [344769.497558] sd 6:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn’t support DPO or FUA
Sep 28 15:18:32 banner kernel: [344769.520537] sda: sda1 sda2 sda3
Sep 28 15:18:32 banner kernel: [344769.524294] sd 6:0:0:0: [sda] Attached SCSI disk
Sep 28 15:18:32 banner iscsid: connection2:0 is operational now
Sep 28 15:18:37 banner logger: /etc/xen/scripts/block: Writing backend/vbd/0/268441856/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/0/268441856/hotplug-status error to xenstore.
Sep 28 15:18:37 banner logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
Sep 28 15:18:37 banner logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/0/268441856
Sep 28 15:18:37 banner logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/0/268441856

The modified script:

banner:/appl/xen/cent/57 # cat /etc/xen/scripts/block-iscsi
#!/bin/bash
# block-iscsi – 2009 Keith Herron
#
# multipath enabled block-iscsi xen block script.
#
# Note: This script depends on a block-iscsi.conf file
# located in the same directory. This file contains
# an array of available iSCSI target IPs
#

dir=$(dirname “$0″)
. “$dir/block-common.sh”
. “$dir/block-iscsi.conf”

# Where do your node files live?
#NODES=/var/lib/iscsi
NODES=/etc/iscsi

# Log which mode we are in
logger -t block-iscsi “*** Beginning device $command ***”

# Fetch the iqn we specify in the domu config file
#
IQN=$(xenstore_read “$XENBUS_PATH/params”)
logger -t block-iscsi “IQN: ${IQN}”

# We define portal ip in order to support new luns which don’t yet have
# ${NODES}/node entrys yet, not dynamic but avoids manual discovery
#
for PORTAL in ${PORTALS[@]}; do
logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done

# Using the iscsi node directory we can determine the ip and port of
# our iscsi target on a lun by lun basis
#
IP=`ls ${NODES}/nodes/${IQN} | cut -d , -f 1`
PORT=`ls ${NODES}/nodes/${IQN} | cut -d , -f 2`

logger -t block-iscsi “TARGET: ${IP}:${PORT}”

# This is called by each command to determine which multipath map to use
#
function get_mpath_map {
# Re-run multipath to ensure that maps are up to date
#
multipath
sleep 2

# Now we determine which /dev/sd* device belongs to the iqn
#
SCSI_DEV=”/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\“”
logger -t block-iscsi “scsi device: ${SCSI_DEV}”

# And using the /dev/sd* device we can determine its corresponding multipath entry
#
MPATH_MAP=”/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk ‘{ print $1}’`”
logger -t block-iscsi “mpath device: ${MPATH_MAP}”
}

case $command in
add)
# Login to the target
logger -t block-iscsi “logging in to ${IQN} on ${IP}:${PORT}”
sleep 5
#FIXME needs more advanced race condition logic
iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} –login | logger -t block-iscsi
sleep 5
#FIXME needs more advanced race condition logic
get_mpath_map

if [ -a ${MPATH_MAP} ]; then
logger -t block-iscsi “${command}ing device: ${MPATH_MAP}”
write_dev ${MPATH_MAP}
fi
;;

remove)
get_mpath_map
if [ -a ${MPATH_MAP} ]; then
logger -t block-iscsi “flushing buffers on ${MPATH_MAP}”
blockdev –flushbufs ${MPATH_MAP}
logger -t block-iscsi “attempting logout of ${IQN} on ${IP}:${PORT}”
iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} –logout | logger -t block-iscsi
sleep 10
#FIXME needs more advanced race condition logic
fi
sleep 5
#FIXME needs more advanced race condition logic
;;
esac

/var/lib/iscsi doesn’t exist.

It’s /etc/iscsi/nodes in this build.

banner:/etc/xen/scripts # cd /etc/iscsi/nodes/
banner:/etc/iscsi/nodes # l
total 16
drw——- 4 root root 4096 Sep 28 15:09 ./
drwxr-xr-x 5 root root 4096 Sep 28 15:09 ../
drw——- 3 root root 4096 Sep 28 15:09 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0/
drw——- 3 root root 4096 Sep 28 15:09 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot1/

Let me go change that path and try again.

IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`

# block-iscsi.conf – 2009 Keith Herron
#
# Note: Config file for block-iscsi xen block script /etc/xen/scripts/block-iscsi

# Define iSCSI portal addresses here, necessary for discovery
PORTALS[0]=”192.168.1.190″

It isn’t executable, though; as the script also wasn’t. Since it’s picking up the portal, I assume that’s not the problem.

run manually:

banner:/etc/xen/scripts # iscsiadm -m discovery -t st -p 192.168.1.190
192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0
192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot1

and as I said: I can log in to the filer manually, and mount one of those — I did so to copy the local image over to the filer with dd, logged out again afterwards.

So I know the connection proper is working.

My open-iscsi:

open-iscsi-2.0.870-37.38.1.i586

which is the distro package on SuSE 11.4.

This snippet from the script:
# Login to the target
logger -t block-iscsi “logging in to ${IQN} on ${IP}:${PORT}”

Is yielding the following output:
Sep 27 23:39:41 banner block-iscsi: logging in to iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0 on :

As you can see, the $IP:$PORT is coming out blank (is /etc/xen/scripts/block-iscsi.conf in place?). The below section is where the IP and port are assigned, I would try running these commands by hand to ensure that they provide the expected output:

# We define portal ip in order to support new luns which don’t yet have
# /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery
#
for PORTAL in ${PORTALS[@]}; do
logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done

# Using the iscsi node directory we can determine the ip and port of
# our iscsi target on a lun by lun basis
#
IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`

logger -t block-iscsi “TARGET: ${IP}:${PORT}”

name=”centos57base”
vcpus=1
memory=1024
# disk=['tap:aio:/appl/xen/images/centos57base/centos57base.img,xvda,w']
disk=['iscsi:iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0,xvda,w']
vif=['ip=67.78.195.9,bridge=br0,mac=00:16:3E:FF:00:4E']
bootloader = ‘/usr/bin/pygrub’
extra=”(hd0,0)/boot/grub/menu.lst”

The tap image is the one I DDd over; it works when local; the IQN is what comes out of the discovery, copy/pasta.

The image is CentOS5.7, installed via HTTP from … OSU, I think. Some mirror.

Any ideas what might have caused this, oh person-who-knows-xen-well-enough-to-write-block-scripts?

Sep 27 23:39:41 banner kernel: [288438.536745] blkback: ring-ref 8, event-channel 70, protocol 1 (x86_32-abi)
Sep 27 23:39:41 banner kernel: [288438.539718] blkback: ring-ref 8, event-channel 70, protocol 1 (x86_32-abi)
Sep 27 23:39:41 banner logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/0/268441856
Sep 27 23:39:41 banner logger: /etc/xen/scripts/block-iscsi: add XENBUS_PATH=backend/vbd/0/268441856
Sep 27 23:39:41 banner block-iscsi: *** Beginning device add ***
Sep 27 23:39:41 banner block-iscsi: IQN: iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0
Sep 27 23:39:41 banner block-iscsi: 192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0 192.168.1.190:3260,1 iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot1
Sep 27 23:39:41 banner block-iscsi: TARGET: :
Sep 27 23:39:41 banner block-iscsi: logging in to iqn.2006-01.com.openfiler:zing_atlas.lv-vmroot0 on :
Sep 27 23:39:51 banner logger: /etc/xen/scripts/block: Writing backend/vbd/0/268441856/hotplug-error /etc/xen/scripts/block failed; error detected. backend/vbd/0/268441856/hotplug-status error to xenstore.
Sep 27 23:39:51 banner logger: /etc/xen/scripts/block: /etc/xen/scripts/block failed; error detected.
Sep 27 23:39:51 banner logger: /etc/xen/scripts/block: remove XENBUS_PATH=backend/vbd/0/268441856
Sep 27 23:39:51 banner logger: /etc/xen/scripts/xen-hotplug-cleanup: XENBUS_PATH=backend/vbd/0/268441856

I’d just copied a local diskimage file to an openfiler iSCSI share the same size; the entire subnet has access permissions on it — and the dom0 in particular, cause I mounted it by hand to (successfully) dd it over to the filer.