I use SAN snapshots extensively to provide point-in-time recoverability for a number of different systems. This is a very reliable and efficient backup solution but it comes at the cost of additional complexity for individual file restoration. This is because the snapshot backups are of the entire volume and not each individual file. So when a user says “hey, could you restore file x from two days ago?” I have some work to do.

The Problem

Recovering files from a SAN snapshot involves a 10 step manual process.

Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the system
4. Perform an iSCSI discovery
5. Perform an iSCSI login to the snapshot volume
6. Mount the snapshot volume
7. Copy the backup file(s) into place
8. Un-mount the volume
9. Perform an iSCSI logout
10. Revoke access to the snapshot from the host

This manual process is tedious and error prone. Luckily many of these steps can be easily automated. Enter Autofs.

The Solution

Autofs is a toolkit which enables you to automatically mount file systems in accordance to a configuration file (called a map) as you traverse a certain directory structure. It is very flexible and supports external scripts, or “program maps” in autofs-speak. As you can see above, the majority of the steps are oriented toward getting the volume to the host and mounted. Only one of the steps actually pertains to the file(s) we need to recover so let’s focus on automating the rest.

This program map, named auto.iscsi, eliminates manual steps 4, 5, 6, and 8. An additional cron job could be added to automate step 9 by checking to see if the devices associated with the hosts current iSCSI sessions are still mounted.

New Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the snaphost browsing system
4. cd to the automounted directory using the volume IQN
5. Copy the backup file(s) into place
6. Revoke access to the snapshot from the host

So we’ve removed 4 steps from the process with the following automounter script. I’m also working to automate 1, 2 and 6, but more on that later.

[ad]

auto.iscsi program map

#!/bin/bash
#
# auto.iscsi - Automounter map for mounting iscsi devices by iqn.  
#              Keith Herron <[email protected]> 
#
# Invoke this program in /etc/auto.master with a line similar to:
# /iscsiauto  program:/etc/auto.iscsi
 
# IQN - This script expects to recieve an iscsi iqn as the only argument.
IQN="$1"
 
# OPTS - Filesystem mount options.
OPTS="-nocheck,ro,fstype=ext3"
 
# PART_NO - Partition number.  This script assumes only one fs per iqn.
PART_NO="1"
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target.  We use this later to figure out the block dev.
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
# If we're already logged in don't bother trying to log in.
if iscsiadm -m session | grep "$IQN"; then
  echo node exists
else
  iscsiadm -m node -T "$IQN" --login 1>&2
fi
 
# Let udev settle before we try to look up the block device.
sleep 1
 
# Figure out the block device using the udev symlink.
DEV_LINK=`readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
DEV_NODE=`basename $DEV_LINK`
DEV_FULL="/dev/$DEV_NODE"
 
# Return the magic to automount
echo "${OPTS} :${DEV_FULL}${PART_NO}"

How to use it

In the below example I am recovering a file called /etc/really_important_file back to a host using scp.

[root@snapbrowse ~]# cd /iscsiauto/iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# ls
bin  boot  dev  etc  home  lib  lost+found  media  misc  mnt  opt  proc  root  sbin  selinux  shells  srv  sys  tmp  usr  var
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# scp -pr etc/real_important_file [email protected]:/etc/

Known Limitations

Next Steps

[ad]