Automounting iSCSI with Autofs

Tags: , , , , , , , , , , ,

I use SAN snapshots extensively to provide point-in-time recoverability for a number of different systems. This is a very reliable and efficient backup solution but it comes at the cost of additional complexity for individual file restoration. This is because the snapshot backups are of the entire volume and not each individual file. So when a user says “hey, could you restore file x from two days ago?” I have some work to do.

The Problem


Recovering files from a SAN snapshot involves a 10 step manual process.

Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the system
4. Perform an iSCSI discovery
5. Perform an iSCSI login to the snapshot volume
6. Mount the snapshot volume
7. Copy the backup file(s) into place
8. Un-mount the volume
9. Perform an iSCSI logout
10. Revoke access to the snapshot from the host

This manual process is tedious and error prone. Luckily many of these steps can be easily automated. Enter Autofs.

The Solution


Autofs is a toolkit which enables you to automatically mount file systems in accordance to a configuration file (called a map) as you traverse a certain directory structure. It is very flexible and supports external scripts, or “program maps” in autofs-speak. As you can see above, the majority of the steps are oriented toward getting the volume to the host and mounted. Only one of the steps actually pertains to the file(s) we need to recover so let’s focus on automating the rest.

This program map, named auto.iscsi, eliminates manual steps 4, 5, 6, and 8. An additional cron job could be added to automate step 9 by checking to see if the devices associated with the hosts current iSCSI sessions are still mounted.

New Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the snaphost browsing system
4. cd to the automounted directory using the volume IQN
5. Copy the backup file(s) into place
6. Revoke access to the snapshot from the host

So we’ve removed 4 steps from the process with the following automounter script. I’m also working to automate 1, 2 and 6, but more on that later.

[ad]

auto.iscsi program map


#!/bin/bash
#
# auto.iscsi - Automounter map for mounting iscsi devices by iqn.  
#              Keith Herron <[email protected]> 
#
# Invoke this program in /etc/auto.master with a line similar to:
# /iscsiauto  program:/etc/auto.iscsi
 
# IQN - This script expects to recieve an iscsi iqn as the only argument.
IQN="$1"
 
# OPTS - Filesystem mount options.
OPTS="-nocheck,ro,fstype=ext3"
 
# PART_NO - Partition number.  This script assumes only one fs per iqn.
PART_NO="1"
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target.  We use this later to figure out the block dev.
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
# If we're already logged in don't bother trying to log in.
if iscsiadm -m session | grep "$IQN"; then
  echo node exists
else
  iscsiadm -m node -T "$IQN" --login 1>&2
fi
 
# Let udev settle before we try to look up the block device.
sleep 1
 
# Figure out the block device using the udev symlink.
DEV_LINK=`readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
DEV_NODE=`basename $DEV_LINK`
DEV_FULL="/dev/$DEV_NODE"
 
# Return the magic to automount
echo "${OPTS} :${DEV_FULL}${PART_NO}"

How to use it


In the below example I am recovering a file called /etc/really_important_file back to a host using scp.

[root@snapbrowse ~]# cd /iscsiauto/iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# ls
bin  boot  dev  etc  home  lib  lost+found  media  misc  mnt  opt  proc  root  sbin  selinux  shells  srv  sys  tmp  usr  var
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# scp -pr etc/real_important_file [email protected]:/etc/

Known Limitations


  • This script assumes only one file system per iSCSI volume and that the volume has a partition table. This works well for my environment but your mileage may vary.
  • This script assumes that the file system is ext3. Again, works great for me. You could parse the output of the file command and map it to the appropriate file system mount argument if you needed to automount different file system types.
  • Autofs ignores the nocheck mount option and forces a fsck of the volume in many cases. For me this resulted in delays while I waited for gigabytes of file system data to be read from the network and parsed by fsck. I reviewed the source and there appears to be no supported mechanism to disable this check at runtime and then contacted the developers to see if having this feature added would be feasible. Until that reaches a resolution I have put a temporary workaround in place by simply moving fsck.ext3 aside and creating a symbolic link pointing /sbin/fsck.ext3 at /bin/true. This results in all calls to /sbin/fsck.ext3 always immediately returning success. Scary I know but luckily this is hosted on a virtual machine dedicated to automounting iSCSI volumes.
  • Next Steps


  • I’m in the process of automating snapshot manipulation on the SAN (EqualLogic PS Series) which will enable automated snapshot browsing without ever logging in to the SAN.
  • Build a directory hierarchy containing human-readable symbolic links to each IQN. Building a directory based hierarchy would enable us to potentially export snapshot browsing to the host itself via NFS.
  • [ad]