Unison File Synchronizer is a tremendous utility. It keeps the contents of two directories in sync with one another. It’s fast, secure, lightweight and good at sorting out collisions but unfortunately it lacks the ability to generate alerts when errors occur. Instead it just silently makes note of the error in a log file.

Luckily we can watch Unison’s logs for error events and notify ourselves as soon as a match is found with a tool called Simple Event Correlator, or SEC.

By default, Unison writes log entries in a file named “unison.log” under the home directory of the user running the application. We’ll assume that unison is running as root and is logging to the default location of /root/unison.log.

To configure SEC we simply need to write a short rule that includes the regular expression that matches a unison error message and the action we wish to execute when a match is found. Here is a simple example rule to send e-mail alerts on unison file conflicts.

type=Single
ptype=RegExp
pattern=(^\[CONFLICT\]|^Fatal error)
desc=$0
action=pipe '' /usr/bin/tac /root/unison.log | grep '^UNISON' -m 1 -B 9999 | tac | /usr/bin/mailx -s "UNISON: $0" root

This will send an e-mail alert to root when a line starting with [CONFLICT] or Fatal Error is logged and include the most recent unison run log output in the body of the message.

To invoke sec use something to the effect of:

 sec -conf=./sec_unison.conf -input=/root/unison.log

Where -conf is the file containing of the above rule and -input is unison’s log file.

[ad]

So far the simplest way I’ve discovered to find the last occurrence of a string and print everything below it is with a combination of grep and tac (it’s cat backwards, get it?!).

Say you want to find the last instance of foo and everything that came after it in a file. You could run the following…

 tac /var/log/logfile | grep 'foo' -m 1 -B 9999 | tac

Which will flip the log file upside and pipe it through grep. Grep then finds only the first occurrence and prints everything above it (well, 9999 lines), then flips that output right side up again. Works great but I wonder if there is a simpler way to do this.

[ad]

When connecting a server to a storage area network (SAN) its important to make certain that you’re hosts are prepared for the occasional blip in SAN connectivity. Device mapper multipath to the rescue! Multipath is an abstraction layer between you and the raw block devices which allows for multiple I/O paths or networks (IO multipathing) and gives you an increased level of control over what happens should a block device start reporting errors. Best of all its built right in to the modern Linux kernel.

I maintain a cluster of Xen servers that store VM images on an EqualLogic PS6000 Series iSCSI SAN as raw LUNs. Its super-stable and makes it very simple to manage, snapshot and replicate storage. The only drawback is EqualLogic’s limitation of 512 connections per storage pool. This means that for every LUN (read VM) created we consume a connection. Multiply this by the number of dom0s and you’ll quickly see that the available connections would get eaten up in no time. In order to step around this boundary I made some significant modifications the block-iscsi Xen block script I found on an e-mail thread. Sorry, I don’t remember where it came from and there are many variations floating around.

I’ve tested this script on RHEL5 running Xen 3.1.4, your mileage may vary but as always, I’d love to hear your feedback!

/etc/xen/scripts/block-iscsi

#!/bin/bash
# block-iscsi  -  2009 Keith Herron <[email protected]>
#
# multipath enabled block-iscsi xen block script.
#
# Note: This script depends on a block-iscsi.conf file
#       located in the same directory.  This file contains
#       an array of available iSCSI target IPs
#
 
dir=$(dirname "$0")
. "$dir/block-common.sh"
. "$dir/block-iscsi.conf"
 
# Log which mode we are in
logger -t block-iscsi "*** Beginning device $command ***"
 
# Fetch the iqn we specify in the domu config file
#
IQN=$(xenstore_read "$XENBUS_PATH/params")
logger -t block-iscsi "IQN: ${IQN}"
 
# We define portal ip in order to support new luns which don't yet have
# /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery 
#
for PORTAL in ${PORTALS[@]}; do
  logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target on a lun by lun basis
#
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
logger -t block-iscsi "TARGET: ${IP}:${PORT}"
 
# This is called by each command to determine which multipath map to use
#
function get_mpath_map {
   # Re-run multipath to ensure that maps are up to date
   #
   multipath
   sleep 2
 
   # Now we determine which /dev/sd* device belongs to the iqn
   #
   SCSI_DEV="/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\``"
   logger -t block-iscsi "scsi device: ${SCSI_DEV}"
 
   # And using the /dev/sd* device we can determine its corresponding multipath entry
   #
   MPATH_MAP="/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk '{ print $1}'`"
   logger -t block-iscsi "mpath device: ${MPATH_MAP}"
}
 
case $command in
   add)
      # Login to the target
      logger -t block-iscsi "logging in to ${IQN} on ${IP}:${PORT}"
      sleep 5
      #FIXME needs more advanced race condition logic
      iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --login | logger -t block-iscsi
      sleep 5
      #FIXME needs more advanced race condition logic
      get_mpath_map
 
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "${command}ing device: ${MPATH_MAP}"
         write_dev ${MPATH_MAP}
      fi
   ;;
 
   remove)
      get_mpath_map
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "flushing buffers on ${MPATH_MAP}"
         blockdev --flushbufs ${MPATH_MAP}
         logger -t block-iscsi "attempting logout of ${IQN} on ${IP}:${PORT}"
         iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --logout | logger -t block-iscsi
         sleep  10
         #FIXME needs more advanced race condition logic
      fi
      sleep 5
      #FIXME needs more advanced race condition logic
   ;;
esac

/etc/xen/scripts/block-iscsi.conf

# block-iscsi.conf  -  2009 Keith Herron <[email protected]>
# 
# Note: Config file for block-iscsi xen block script /etc/xen/scripts/block-iscsi
 
# Define iSCSI portal addresses here, necessary for discovery
PORTALS[0]="10.241.34.100"

To make use of this script you’ll need to update your xen guest config file to specify “iscsi” in the disk line instead of “phy” or similar.

domU configuration example

#
disk = [ 'iscsi:iqn.2001-05.com.equallogic:0-8a0906-23fe93404-c82797962054a96d-examplehost,xvda,w' ];
#

Prior to Bash 4.1 logging to syslog required either 3rd party patches, wrappers or clever hacks to glean command history information and send it to syslog. Until bash 4.1 becomes available for the majority of distributions these workaround and hacks are still useful to some who wish to obtain syslog functionality without altering, upgrading and maintaining bash manually.

Trap Method

Drop the following snippet into either the per-user or system-wide bash profile (~/.bash_profile or /etc/profile, respectively)

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${COMMAND}"
}
trap log2syslog DEBUG

Read more about the trap method here

Prompt Method

This method logs by hacking the prompt command to call history and write to syslog.

PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -t "$USER[$$] $SSH_CONNECTION")'

You can read more about the prompt method here

Upgrade or Patch Bash

Bash version 4.1 support syslog natively, so if it’s available in your distribution, or you’re willing to hand compile this is likely your best bet.

bash-syslog patches are available to bring syslog features into bash 3.x

Currently backdrift.org is running on a Xen VM. In order to make this more affordable and better utilize the co-located hardware, we’re offering dedicated VMs for lease on the primary backdrift.org server.

$20 per month gets you:

2x AMD Opteron VCPUs @ 2.6GHz
1G RAM
100GB Disk (on underlying RAID-1)
Un-metered bandwidth on bonded T1s (3mbps burstable)

Its a great platform for most applications. We’re currently running Xen 3.3 with PvGrub enabled and out of band management accounts are available for remote console and reboot access to your VM.

Payments via PayPal.

To get started send mail to [email protected].

[ad]

This is a list of handy key combinations I use when working with the HP iLO on some DL145 servers I manage.

Login Banner

Lights-Out 100 Management
Copyright 2000-2004 QLogic Corporation
Copyright 2004 Hewlett-Packard Development Company, L.P.
Type 'HELP' for options.  Type 'QUIT' to return to console.

Help menu

Lights-Out> help
CLI Commands:
   C[hangePassword] <enter>
   P[ower] <ON, OFF> <enter>
   I[PMICommand] <ipmb command data> <enter>
   SEN[sorStatus] <enter>
   SD[RRead] <enter>
   SEL[Read] <enter>
   N[etwork setup] <enter>
   R[eset] <WARM, COLD> <enter> (Host Reset)
   V[ersion] <enter>
   H[ELP] <enter>
   Q[UIT] <enter>
Lights-Out>

Shortcuts, Hotkeys and Keystrokes

To enter the virtual serial console: ESC, Q
To return to the Lights-Out Console: ESC, (

[ad]

So if you’re like me you are probably somewhat lazy. I imagine most sysadmins are. Which must be why we put so much effort into scripting and automating repeated tasks. If you’ve managed a Xen VM before you are likely to notice that all your kernel images live outside the VM itself. This makes it rather difficult to patch and upgrade among other things, especially as the number and type of guests you support grows. Pvgrub can help solve this problem. It implements grub in a stub domain which in turn loads your guest domain. It works quite well and is rather easy to set up. Read on to learn more.

From the xen wiki: “PV-GRUB replaces PyGrub to boot domU images safely: it runs the regular grub inside the created domain itself and uses regular domU facilities to read the disk / fetch files from network etc.; it eventually loads the PV kernel and chain-boots it.”

This means that your domU guests can install and manage their own kernels as if they were running on regular hardware. Yum, apt-get and other package management software will let you patch your virtual machine’s kernel without the need to alter any Xen configuration. Here’s how it works.

Notes:

* Pvgrub requires Xen 3.3 or greater.
* You won’t actually need to install grub in the domU, it is implemented as a stub domain and provided with the Xen sources.

Steps:

1. Modify /etc/modprobe.conf in the guest.

We need to make sure that a couple of lines exist in /etc/modprobe.conf before we’re ready to install the kernel packages. We add these so that when the kernel install process goes to build a new initrd it will include the correct driver modules for your hardware, so we want to be certain that the Xen block and net drivers are included.

Note: If /etc/modprobe.conf doesn’t already exist you should create it.

#/etc/modprobe.conf
alias eth0 xennet
alias scsi_hostadapter xenblk

2. Update /etc/inittab and /etc/securetty in the guest.

Now we update /etc/inittab and /etc/securetty to make certain that we attach to and permit logins from the various xen consoles.

/etc/inittab

#/etc/inittab
hvc0:2345:respawn:/sbin/getty 38400 hvc0
xvc0:2345:respawn:/sbin/getty 38400 xvc0

/etc/securetty

#/etc/securetty
# hypervisor virtual consoles
hvc0
xvc0

3. Install a stub grub.conf in the guest so that the grub update scripts won’t fail.

Configuration file locations:

Depending on the Linux distribution you use you’re likely to see the grub config in either /boot/grub/grub.conf or /boot/grub/menu.lst. To keep things simple I usually use /boot/grub/grub.conf as a config file and symlink /boot/grub/menu.lst to it. This should ensure that scripts used to automatically update the grub config when a new kernel is installed will work correctly.

Reasonable stub for Debian Lenny

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.26-2-xen-686
   root (hd0)
   kernel /boot/vmlinuz-2.6.26-2-xen-686 console=hvc0 root=/dev/xvda1 ro
   initrd /boot/initrd.img-2.6.26-2-xen-686

Reasonable stub for CentOS/RHEL

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.18-128.4.1.el5xen
	root (hd0)
	kernel /boot/vmlinuz-2.6.18-128.4.1.el5xen root=LABEL=root ro
	initrd /boot/initrd-2.6.18-128.4.1.el5xen.img

4. Install kernel packages in the guest.

Debian

# apt-get install linux-image-2.6.26-1-686-bigmem

CentOS/RHEL

# yum install kernel-xen

5. Modify Xen guest config file in the dom0.

Now that the VM has been prepped for booting with pvgrub we need to update its configuration file to load the pvgrub kernel instead of a linux kernel.

Note: be certain to remove the old “root=” line, this seems to confuse pvgrub. Also, make certain that the “extra=” line points to the location of your grub config file inside the guest. My example does not use partitions, if you do use partitions in your guests you’ll need so use something like (hd0,0), etc.

name    = 'superchunk'
 
vcpus =2;
 
memory 	= 512;
maxmem  = 4096;
 
#pvgrub here
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0)/boot/grub/grub.conf"
#end pvgrub
 
disk = [ 'phy:/dev/vg0/vm_superchunk-root,xvda1,w', \
         'phy:/dev/vg0/vm_superchunk-swap,xvda2,w' ];
 
vif = [ 'bridge=eth1, vifname=superchunk, mac=B0:00:00:00:B1:35' ];

6. Shutdown and then create the domain.

My Xen guest is named superchunk. You should be familiar with shutdown and create at this point. Here’s how, just in case…

xm shutdown superchunk
#(wait until it has shut down)
xm create -c /path/to/superchunk-config

At this point you should see some kernel output and then either a grub menu or the grub command line. If you get the command line, no worries. This usually means that it couldn’t find a grub configuration file. Make sure that the “extra=” line of the Xen guest domU configuration file references the right location and that you have pointed grub at the correct disk, i.e. (hd0) vs. (hd0,0), etc…

Good luck! And feel free to drop me a line at [email protected], I always appreciate feedback and do my best to help.

So last night a RHEL3 machine at went down for patching and sadly it didn’t come back. Here’s what was sitting on the console when I got to the console. Nice… a kernel panic :/

pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2 umount
/initrd/proc failed: 2 Freeing ununsed kernel memory: 228k freed Kernel
panic: No init found. Try passing init= option to kernel.

So after spending a significant amount of time troubleshooting the file systems, rebuilding the initrd, trying different init=arguments and trying different kernels I took a good hard look at the error message. Its talking about /sysroot/initrd but I don’t see a directory named /initrd when I look at the root filesystem. So I tried this:

mkdir /initrd
reboot

And to my surprise the system starts happily booting along. All that over a little directory, sheesh. Turns out that this directory was removed by someone or something that thought it wasn’t needed, doh.

[ad]

I recently ran into an issue where my servers using LDAP logins became inaccessible on the console during a network outage. It turns out this was because the LDAP client was trying to reconnect to the server indefinitely which caused the login process to time out, locking me out of the machine. The fix was to set a few nss_reconnect_* entries in /etc/ldap.conf, which apparently are undocumented.

Here are the relevant lines of my /etc/ldap.conf:

timelimit 120
bind_timelimit 120
idle_timelimit 3600
 
# Added to permit console login during network outages.
nss_reconnect_tries 2
nss_reconnect_sleeptime 1
nss_reconnect_maxsleeptime 1
nss_reconnect_maxconntries 1

I was able to find some pretty terse documentation in the source which at least explains that these variables adjust the parameters of ldap connection retries. In the above configuration all servers are tried 2 times with a 1 second sleep between trys.

nss_reconnect_tries 5	      # no. of times to double the sleep time
nss_reconnect_sleeptime 4     # initial sleep value
nss_reconnect_maxsleeptime 64 # max sleep value to cap at
nss_reconnect_maxconntries 2  # how many tries before sleeping
 
# This leads to a delay of 124 seconds (4+8+16+32+64=124)
# per lookup if the server is not available.

[ad#adsense-inline1]

Overview

Xen 3+ supports memory “ballooning” which allows you to hot add and remove memory from a running system. It is a nice feature and has come in handy for me on many occasions.

Memory Hot Remove Example

Lets say I have a virtual machine named foo which I’ve given 1024 megs of ram. One day I notice that the system is using only 25% of this memory. I want to re-size it to 512 megs and free the rest of the memory for use by other domains but I can’t afford a downtime. No problem, Xen lets you resize memory on-the-fly.

Before we adjust the memory allocations lets see what the system looks like…

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      256     2 r-----  49678.8
foo                                        2     1024     1 -b----   2160.2

To resize we use the ‘xm mem-set’ command. For example, to re-size our 1024M domain to 512M execute the following.

# xm mem-set foo 512
# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      256     2 r-----  49681.0
foo                                        2      512     1 -b----   2160.3

Memory Hot Add Example

We can also use the xm-mem set command to hot-add memory to a running domain, there is one catch, however. Xen can only add memory up to the amount specified by the ‘maxmem’ configuration directive in your domU config file. If no ‘maxmem’ option is specified then the maximum memory value is set to the size of the ‘memory’ option.

This is an example config file for a domain named ‘foo’ that will boot with 1024M RAM and can grow up to 4096M.

#/etc/xen/configs/foo
 
name   = "foo"
memory = 1024 
maxmem = 4096
vcpus  = 1

When we boot this domain we’ll see that its initial memory size is 1024M.

# xm create /etc/xen/configs/foo
Using config file "/etc/xen/configs/foo".
Started domain foo
 
# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      256     2 r-----  49723.1
foo                                       11     1024     1 r-----      5.1

Now we can mem-set the domain up to the value of maxmem, in our case 4096M.

xm mem-set foo 4096

# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0      256     2 r-----  49725.6
foo                                       11     4096     1 -b----      8.8

Use Scenarios and Ideas

As you can see this feature provides for a great deal of flexibility in the management of your systems. I’ve found it especially helpful in dealing with live migration, you can easily shrink multiple hosts to make room for one more. You could also build a very powerful auto-scaling engine around the memory balloon and live migration features where domains can automatically re-size their memory as needed, migrating on and off of hosts according to changing demand.