Xen block iSCSI script with multipath support

Tags: iSCSI, linux, Multipath, redhat, Xen

When connecting a server to a storage area network (SAN) its important to make certain that you’re hosts are prepared for the occasional blip in SAN connectivity. Device mapper multipath to the rescue! Multipath is an abstraction layer between you and the raw block devices which allows for multiple I/O paths or networks (IO multipathing) and gives you an increased level of control over what happens should a block device start reporting errors. Best of all its built right in to the modern Linux kernel.

I maintain a cluster of Xen servers that store VM images on an EqualLogic PS6000 Series iSCSI SAN as raw LUNs. Its super-stable and makes it very simple to manage, snapshot and replicate storage. The only drawback is EqualLogic’s limitation of 512 connections per storage pool. This means that for every LUN (read VM) created we consume a connection. Multiply this by the number of dom0s and you’ll quickly see that the available connections would get eaten up in no time. In order to step around this boundary I made some significant modifications the block-iscsi Xen block script I found on an e-mail thread. Sorry, I don’t remember where it came from and there are many variations floating around.

I’ve tested this script on RHEL5 running Xen 3.1.4, your mileage may vary but as always, I’d love to hear your feedback!

/etc/xen/scripts/block-iscsi

#!/bin/bash
# block-iscsi  -  2009 Keith Herron <[email protected]>
#
# multipath enabled block-iscsi xen block script.
#
# Note: This script depends on a block-iscsi.conf file
#       located in the same directory.  This file contains
#       an array of available iSCSI target IPs
#
 
dir=$(dirname "$0")
. "$dir/block-common.sh"
. "$dir/block-iscsi.conf"
 
# Log which mode we are in
logger -t block-iscsi "*** Beginning device $command ***"
 
# Fetch the iqn we specify in the domu config file
#
IQN=$(xenstore_read "$XENBUS_PATH/params")
logger -t block-iscsi "IQN: ${IQN}"
 
# We define portal ip in order to support new luns which don't yet have
# /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery 
#
for PORTAL in ${PORTALS[@]}; do
  logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target on a lun by lun basis
#
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
logger -t block-iscsi "TARGET: ${IP}:${PORT}"
 
# This is called by each command to determine which multipath map to use
#
function get_mpath_map {
   # Re-run multipath to ensure that maps are up to date
   #
   multipath
   sleep 2
 
   # Now we determine which /dev/sd* device belongs to the iqn
   #
   SCSI_DEV="/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\``"
   logger -t block-iscsi "scsi device: ${SCSI_DEV}"
 
   # And using the /dev/sd* device we can determine its corresponding multipath entry
   #
   MPATH_MAP="/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk '{ print $1}'`"
   logger -t block-iscsi "mpath device: ${MPATH_MAP}"
}
 
case $command in
   add)
      # Login to the target
      logger -t block-iscsi "logging in to ${IQN} on ${IP}:${PORT}"
      sleep 5
      #FIXME needs more advanced race condition logic
      iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --login | logger -t block-iscsi
      sleep 5
      #FIXME needs more advanced race condition logic
      get_mpath_map
 
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "${command}ing device: ${MPATH_MAP}"
         write_dev ${MPATH_MAP}
      fi
   ;;
 
   remove)
      get_mpath_map
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "flushing buffers on ${MPATH_MAP}"
         blockdev --flushbufs ${MPATH_MAP}
         logger -t block-iscsi "attempting logout of ${IQN} on ${IP}:${PORT}"
         iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --logout | logger -t block-iscsi
         sleep  10
         #FIXME needs more advanced race condition logic
      fi
      sleep 5
      #FIXME needs more advanced race condition logic
   ;;
esac

#!/bin/bash # block-iscsi - 2009 Keith Herron <[email protected]> # # multipath enabled block-iscsi xen block script. # # Note: This script depends on a block-iscsi.conf file # located in the same directory. This file contains # an array of available iSCSI target IPs # dir=$(dirname "$0") . "$dir/block-common.sh" . "$dir/block-iscsi.conf" # Log which mode we are in logger -t block-iscsi "*** Beginning device $command ***" # Fetch the iqn we specify in the domu config file # IQN=$(xenstore_read "$XENBUS_PATH/params") logger -t block-iscsi "IQN: ${IQN}" # We define portal ip in order to support new luns which don't yet have # /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery # for PORTAL in ${PORTALS[@]}; do logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL` done # Using the iscsi node directory we can determine the ip and port of # our iscsi target on a lun by lun basis # IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1` PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2` logger -t block-iscsi "TARGET: ${IP}:${PORT}" # This is called by each command to determine which multipath map to use # function get_mpath_map { # Re-run multipath to ensure that maps are up to date # multipath sleep 2 # Now we determine which /dev/sd* device belongs to the iqn # SCSI_DEV="/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\``" logger -t block-iscsi "scsi device: ${SCSI_DEV}" # And using the /dev/sd* device we can determine its corresponding multipath entry # MPATH_MAP="/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk '{ print $1}'`" logger -t block-iscsi "mpath device: ${MPATH_MAP}" } case $command in add) # Login to the target logger -t block-iscsi "logging in to ${IQN} on ${IP}:${PORT}" sleep 5 #FIXME needs more advanced race condition logic iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --login | logger -t block-iscsi sleep 5 #FIXME needs more advanced race condition logic get_mpath_map if [ -a ${MPATH_MAP} ]; then logger -t block-iscsi "${command}ing device: ${MPATH_MAP}" write_dev ${MPATH_MAP} fi ;; remove) get_mpath_map if [ -a ${MPATH_MAP} ]; then logger -t block-iscsi "flushing buffers on ${MPATH_MAP}" blockdev --flushbufs ${MPATH_MAP} logger -t block-iscsi "attempting logout of ${IQN} on ${IP}:${PORT}" iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --logout | logger -t block-iscsi sleep 10 #FIXME needs more advanced race condition logic fi sleep 5 #FIXME needs more advanced race condition logic ;; esac

/etc/xen/scripts/block-iscsi.conf

# block-iscsi.conf  -  2009 Keith Herron <[email protected]>
# 
# Note: Config file for block-iscsi xen block script /etc/xen/scripts/block-iscsi
 
# Define iSCSI portal addresses here, necessary for discovery
PORTALS[0]="10.241.34.100"

To make use of this script you’ll need to update your xen guest config file to specify “iscsi” in the disk line instead of “phy” or similar.

domU configuration example

#
disk = [ 'iscsi:iqn.2001-05.com.equallogic:0-8a0906-23fe93404-c82797962054a96d-examplehost,xvda,w' ];
#