Today, most web sites are powered by content management systems. And while this is great news for most users and content writers it can spell disaster for system administrators and webmasters. These systems can be incredibly complex and often lack robust security hardening and auditing features. In order to make up for some of these shortcomings (and to sleep just a little a bit better at night) we can implement simple yet effective security checks right in the web server using Apache’s mod_rewrite.

In this example we create a .htaccess file that acts as an IP access control list or firewall. Anyone who connects from an IP that is not in this list will be presented with a forbidden page. This is a good way to protect your site’s administrator interface, for example.

#.htaccess
RewriteEngine On
 
RewriteCond %{REMOTE_ADDR} !172.16.*
RewriteCond %{REMOTE_ADDR} !1.2.3.*
RewriteCond %{REMOTE_ADDR} !192.168.1.140
RewriteRule .* - [F]

As you can see, we use not statements to redirect everyone but authorized users to a forbidden page. This sort of protection is very easy to implement and maintain and provides you the ability to restrict individual components of your site at a very granular level.

Effectively backing up your virtual machines is a problem with a multitude of potential solutions. Many solutions are centered around making a copy of the full volume(s) upon which your virtual machine(s) reside. But what happens if you want to recover just a single file and not the entire VM? And is it possible to delegate restoration to end users? Not to mention the space that is wasted by storing many copies of the same data.

Here I will describe how to implement a backup scheme that I have found very robust, efficient and reliable. It enables automated backups of Xen virtual machines that each reside on individual LVM volumes using a utility called rsnapshot (which is based on rsync).

Note: This method additionally can be utilized to back LVM volumes that aren’t housing Xen virtual machines.

Goals

---

LVM Snapshots

---

LVM provides a convenient snapshot feature. This allows you to create an identical clone of a logical volume and store only the blocks that differ from it. It is an unintuitive action and the result is a mirror image of your virtual machine’s disk that you can mount, read and even write data to.

Rsnapshot backups

---

Rsnapshot is a very flexible backup application which provides robust automation facilities. It supports LVM, ssh, rsync as well as custom scripts and will manage the process of rotating your backup files according to a defined schedule and retention policy. For our purposes we will be using the built-in LVM snapshot support.

How it works

---

Each of my Xen virtual machines has own LVM volume per file system used by the system. For many systems this results in a single and somewhat large root file system housed on LVM. When rsnapshot is configured to back up one of these volumes it runs through the following steps:

Note: The use of LVM snapshots does introduce a write performance implication during the period that the snapshot exists. I have found this to be an acceptable tradeoff as most backup suites introduce some sort of performance effect while they are processing.

How to set it up

---

Assuming you already have a working LVM configuration (read more about LVM here), you’ll just need to install rsnapshot. It is available through dag for RedHat based platforms and available in the standard repositories for ubuntu and debian.

Once you have installed rsnapshot, defined your retention periods, and installed the corresponding cron jobs to call the backup process, you’re ready to add the LVM specifics to the rsnapshot configuration.

For example, I have a logical volume named “vm_insomniac-root” in a volume group named “vg0″. This represents the root filesystem of a virtual machine named “insomniac”.

[root@localhost .snapshot]# lvs | grep insomniac-root
   vm_insomniac-root vg0  -wi-ao  36.00G

To tell rsnapshot that I want to backup this LVM volume I add the following to /etc/rsnapshot.conf. Make sure this is tab delimited. Rsnapshot is unreasonably insistent upon tabs in its config file.

#/etc/rsnapshot.conf
backup  lvm://vg0/vm_insomniac-root/    insomniac/

My snapshot_root is set as “/.snapshot”, and as such the backups look like this:

[root@localhost .snapshot]# cd /.snapshot
[root@localhost .snapshot]# ls
daily.0/  daily.1/  daily.2/  daily.3/  daily.4/  daily.5/  daily.6/  lost+found/
[root@localhost .snapshot]# ls daily.0/ | grep insomniac
insomniac/
[root@localhost .snapshot]# ls daily.0/insomniac/
bin/  boot/  etc/  home/  lib/  lost+found/  media/  mnt/  opt/  root/  sbin/  selinux/  srv/  tmp/  usr/  var/

As you can see, I’m only set up to keep a weeks worth of daily backups. If you have the disk space you could opt for a much longer retention period.

Delegation to end-users

---

To provide users with a simple interface to access their backups I share each of their rsnapshot directories as read-only NFS exports. To only their server of course. This allows users to traverse their backups.

One significant benefit is that there is no backup client for them to install and learn how to use. Restoration is performed with standard unix commands like cp and rsync, saving much time and frustration.

Potential Improvements and follow-up

Time permitting, I would like to test the performance, reliability and storage saving of using a dedup file systems such as lessfs and SDFS. I think these have the potential to make this solution even more robust, especially from a cost perspective.

I’m a sysadmin by trade and as such I deal with RAID enabled servers on a daily basis. Today a server with a hardware RAID controller reported (when I say reported I actually mean lit a small red LED on the front of the machine) a bad disk, which is not uncommon. But when replacing the failed disk with a shiny new one suddenly both drives went red and the system crashed. Upon reboot the hardware controller says to me “sorry buddy, I don’t see any drives” and wouldn’t boot. Luckily I had a similar system sitting idle, so I tested the same disks in this server and they worked just fine. After cursing loudly at the RAID controller I started wondering if the pros of hardware raid really outweigh the cons for a general purpose, nothing special, 1 or 2U server that just needs disk mirroring.

Pros of Hardware RAID

Cons of Hardware RAID

Pros of Software RAID

Cons of Software RAID

So what do I make of all this?

I can certainly understand the argument of simple deployment and having a vendor to blame. But at the end of the day if you loose your data then it’s gone. No vendor will be able to bring it back. And what about manageability after initial deployment?

It seems to me that if you care about the integrity of your data and do not need ultra-intense IO performance then software RAID is a good choice. You’ll get the same core features as hardware with a standard management and monitoring suite. Additionally you are no longer bound to a specific piece of hardware. So in the event of a disk controller failure you could simply replace the controller (or the whole server) and the disks will remain usable. I also find it desirable that software raid will allow you to define the raid configuration during the installation process which when coupled with kickstart or a similar automated build process will allow you to ensure proper raid configuration and monitoring as soon as the server is brought online.

Considering these benefits, and my sour experiences with hardware RAID, I plan to deploy software RAID on a much wider scale from this point forward. But I’m curious…

Have others struggled with similar problems? How many people are running software RAID in an enterprise setting? And how many people have been similarly burned by software RAID?

Recently the hard drive in my mac mini running Mac OSX Leopard (10.5) failed. Luckily I had time machine backing it up to an external USB disk. Now, since I had to replace the drive and rebuild my system anyway I figured, why not upgrade to Snow Leopard? Planning to just pull what I needed off the backup drive manually I went ahead with the upgrade. There aren’t too many files on this machine that I depend on. Just some ssh keys, gpg keys and random documents scattered about here and there. So I upgraded, installed my apps and copied my files from the backup. Everything was going smoothly until I tried to actually write to one of the files I copied from the backup drive. This is when I started getting permission errors.

Here’s what happened when I tried to update my ssh known_hosts file:

airbag:~ keith$ echo foo > .ssh/known_hosts 
-bash: .ssh/known_hosts: Permission denied

Huh? But I own this file…dont I?

airbag:~ keith$ id
uid=501(keith) gid=20(staff) groups=20(staff),402(com.apple.sharepoint.group.1),204(_developer),100(_lpoperator),98(_lpadmin),81(_appserveradm),80(admin),79(_appserverusr),61(localaccounts),12(everyone),401(com.apple.access_screensharing)
 
airbag:~ keith$ ls -al .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

I do own it… And so began much head scratching and man page reading.

Well, as it turns out I forgot to look at the file ACLs…

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts
 0: group:everyone deny write,delete,append,writeattr,writeextattr,chown

Well no wonder, the ACL is set to deny write,delete,append,writeattr,writeextattr and chown from everyone! Let’s get rid of that.

airbag:~ keith$ sudo chmod -N .ssh/known_hosts 
Password:

That ought to do it. The -N flag says get rid of all the ACL info on the file. You could also update this to be just right for your user or group but I’d rather use only the standard unix permissions.

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

Seems to have removed all ACLs from the file. I wonder if we can write to it now…

airbag:~ keith$ echo foo >> .ssh/known_hosts 
airbag:~ keith$

And there you have it, the file is writable once again. Now its time to get some real work done!

While working on a new ubuntu karmic system I ran across the following locale error when attempting to perform a dist-upgrade. I was able to reproduce it by running ‘dpkg-reconfigure locales’ thinking that this might fix the error, sadly it did not.

Here’s the error I was seeing:

root@harrowdown:~# dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

If you boil the error message down a bit the problem becomes a bit more clear. It’s trying to say that the “en_US” locale could not be found:

perl: warning: Setting locale failed.
        LANG = "en_US"

To install the missing “en_US” locale run ‘locale-gen en_US’ which should yeild the following output.

root@harrowdown:~# locale-gen en_US
Generating locales...
  en_US.ISO-8859-1... done
Generation complete.

And as you can see the errors are no longer printed when I call dpkg-reconfigure (or anything else for that matter).

root@harrowdown:~# dpkg-reconfigure locales
Generating locales...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.ISO-8859-1... up-to-date
  en_US.UTF-8... up-to-date
  en_ZA.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.

Unison File Synchronizer is a tremendous utility. It keeps the contents of two directories in sync with one another. It’s fast, secure, lightweight and good at sorting out collisions but unfortunately it lacks the ability to generate alerts when errors occur. Instead it just silently makes note of the error in a log file.

Luckily we can watch Unison’s logs for error events and notify ourselves as soon as a match is found with a tool called Simple Event Correlator, or SEC.

By default, Unison writes log entries in a file named “unison.log” under the home directory of the user running the application. We’ll assume that unison is running as root and is logging to the default location of /root/unison.log.

To configure SEC we simply need to write a short rule that includes the regular expression that matches a unison error message and the action we wish to execute when a match is found. Here is a simple example rule to send e-mail alerts on unison file conflicts.

type=Single
ptype=RegExp
pattern=(^\[CONFLICT\]|^Fatal error)
desc=$0
action=pipe '' /usr/bin/tac /root/unison.log | grep '^UNISON' -m 1 -B 9999 | tac | /usr/bin/mailx -s "UNISON: $0" root

This will send an e-mail alert to root when a line starting with [CONFLICT] or Fatal Error is logged and include the most recent unison run log output in the body of the message.

To invoke sec use something to the effect of:

 sec -conf=./sec_unison.conf -input=/root/unison.log

Where -conf is the file containing of the above rule and -input is unison’s log file.

So far the simplest way find the last occurrence of a string and print everything below it is with a combination of grep and tac (it’s cat backwards, get it?!).

Say you want to find the last instance of foo and everything that came after it in a file:

 tac /var/log/logfile | grep 'foo' -m 1 -B 9999 | tac

This flips the logfile upside and pipes it to grep, who finds only the first occurrence and prints everything above it (well, 9999 lines), then flips that output right side up again. Works great but I wonder if there is a simpler way to do this.

When connecting a server to a storage area network (SAN) its important to make certain that you’re hosts are prepared for the occasional blip in SAN connectivity. Device mapper multipath to the rescue! Multipath is an abstraction layer between you and the raw block devices which allows for multiple I/O paths or networks (IO multipathing) and gives you an increased level of control over what happens should a block device start reporting errors. Best of all its built right in to the modern Linux kernel.

I maintain a cluster of Xen servers that store VM images on an EqualLogic PS6000 Series iSCSI SAN as raw LUNs. Its super-stable and makes it very simple to manage, snapshot and replicate storage. The only drawback is EqualLogic’s limitation of 512 connections per storage pool. This means that for every LUN (read VM) created we consume a connection. Multiply this by the number of dom0s and you’ll quickly see that the available connections would get eaten up in no time. In order to step around this boundary I made some significant modifications the block-iscsi Xen block script I found on an e-mail thread. Sorry, I don’t remember where it came from and there are many variations floating around.

I’ve tested this script on RHEL5 running Xen 3.1.4, your mileage may vary but as always, I’d love to hear your feedback!

/etc/xen/scripts/block-iscsi

#!/bin/bash
# block-iscsi  -  2009 Keith Herron <keith@backdrift.org>
#
# multipath enabled block-iscsi xen block script.
#
# Note: This script depends on a block-iscsi.conf file
#       located in the same directory.  This file contains
#       an array of available iSCSI target IPs
#
 
dir=$(dirname "$0")
. "$dir/block-common.sh"
. "$dir/block-iscsi.conf"
 
# Log which mode we are in
logger -t block-iscsi "*** Beginning device $command ***"
 
# Fetch the iqn we specify in the domu config file
#
IQN=$(xenstore_read "$XENBUS_PATH/params")
logger -t block-iscsi "IQN: ${IQN}"
 
# We define portal ip in order to support new luns which don't yet have
# /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery 
#
for PORTAL in ${PORTALS[@]}; do
  logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target on a lun by lun basis
#
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
logger -t block-iscsi "TARGET: ${IP}:${PORT}"
 
# This is called by each command to determine which multipath map to use
#
function get_mpath_map {
   # Re-run multipath to ensure that maps are up to date
   #
   multipath
   sleep 2
 
   # Now we determine which /dev/sd* device belongs to the iqn
   #
   SCSI_DEV="/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\``"
   logger -t block-iscsi "scsi device: ${SCSI_DEV}"
 
   # And using the /dev/sd* device we can determine its corresponding multipath entry
   #
   MPATH_MAP="/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk '{ print $1}'`"
   logger -t block-iscsi "mpath device: ${MPATH_MAP}"
}
 
case $command in
   add)
      # Login to the target
      logger -t block-iscsi "logging in to ${IQN} on ${IP}:${PORT}"
      sleep 5
      #FIXME needs more advanced race condition logic
      iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --login | logger -t block-iscsi
      sleep 5
      #FIXME needs more advanced race condition logic
      get_mpath_map
 
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "${command}ing device: ${MPATH_MAP}"
         write_dev ${MPATH_MAP}
      fi
   ;;
 
   remove)
      get_mpath_map
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "flushing buffers on ${MPATH_MAP}"
         blockdev --flushbufs ${MPATH_MAP}
         logger -t block-iscsi "attempting logout of ${IQN} on ${IP}:${PORT}"
         iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --logout | logger -t block-iscsi
         sleep  10
         #FIXME needs more advanced race condition logic
      fi
      sleep 5
      #FIXME needs more advanced race condition logic
   ;;
esac

/etc/xen/scripts/block-iscsi.conf

# block-iscsi.conf  -  2009 Keith Herron <keith@backdrift.org>
# 
# Note: Config file for block-iscsi xen block script /etc/xen/scripts/block-iscsi
 
# Define iSCSI portal addresses here, necessary for discovery
PORTALS[0]="10.241.34.100"

To make use of this script you’ll need to update your xen guest config file to specify “iscsi” in the disk line instead of “phy” or similar.

domU configuration example

#
disk = [ 'iscsi:iqn.2001-05.com.equallogic:0-8a0906-23fe93404-c82797962054a96d-examplehost,xvda,w' ];
#

This is a handy way to log user’s bash histories to syslog without making any modifications to the bash source code itself. Simply drop the following snippet into either the per-user or system-wide bash profile (~/.bash_profile and /etc/profile, respectively)

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${COMMAND}"
}
trap log2syslog DEBUG

This won’t guarantee you log consistency in the event of a compromised host but you’ll certainly have and increased ability to correlate events on your systems.

Currently backdrift.org is running on a Xen VM. In order to make this more affordable and better utilize the co-located hardware, we’re offering dedicated VMs for lease on the primary backdrift.org server.

$20 per month gets you:

2x AMD Opteron VCPUs @ 2.6GHz
1G RAM
100GB Disk (on underlying RAID-1)
Un-metered bandwidth on bonded T1s (3mbps burstable)

Its a great platform for most applications. We’re currently running Xen 3.3 with PvGrub enabled and out of band management accounts are available for remote console and reboot access to your VM.

Payments via PayPal.

To get started send mail to vps@backdrift.org.