Backdrift

Recently the hard drive in my mac mini running Mac OSX Leopard (10.5) failed. Luckily I had time machine backing it up to an external USB disk. Now, since I had to replace the drive and rebuild my system anyway I figured, why not upgrade to Snow Leopard? Planning to just pull what I needed off the backup drive manually I went ahead with the upgrade. There aren’t too many files on this machine that I depend on. Just some ssh keys, gpg keys and random documents scattered about here and there. So I upgraded, installed my apps and copied my files from the backup. Everything was going smoothly until I tried to actually write to one of the files I copied from the backup drive. This is when I started getting permission errors.

Here’s what happened when I tried to update my ssh known_hosts file:

airbag:~ keith$ echo foo > .ssh/known_hosts 
-bash: .ssh/known_hosts: Permission denied

Huh? But I own this file…dont I?

airbag:~ keith$ id
uid=501(keith) gid=20(staff) groups=20(staff),402(com.apple.sharepoint.group.1),204(_developer),100(_lpoperator),98(_lpadmin),81(_appserveradm),80(admin),79(_appserverusr),61(localaccounts),12(everyone),401(com.apple.access_screensharing)
 
airbag:~ keith$ ls -al .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

I do own it… And so began much head scratching and man page reading.

Well, as it turns out I forgot to look at the file ACLs…

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts
 0: group:everyone deny write,delete,append,writeattr,writeextattr,chown

Well no wonder, the ACL is set to deny write,delete,append,writeattr,writeextattr and chown from everyone! Let’s get rid of that.

airbag:~ keith$ sudo chmod -N .ssh/known_hosts 
Password:

That ought to do it. The -N flag says get rid of all the ACL info on the file. You could also update this to be just right for your user or group but I’d rather use only the standard unix permissions.

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

Seems to have removed all ACLs from the file. I wonder if we can write to it now…

airbag:~ keith$ echo foo >> .ssh/known_hosts 
airbag:~ keith$

And there you have it, the file is writable once again. Now its time to get some real work done!

While working on a new ubuntu karmic system I ran across the following locale error when attempting to perform a dist-upgrade. I was able to reproduce it by running ‘dpkg-reconfigure locales’ thinking that this might fix the error, sadly it did not.

Here’s the error I was seeing:

root@harrowdown:~# dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

If you boil the error message down a bit the problem becomes a bit more clear. It’s trying to say that the “en_US” locale could not be found:

perl: warning: Setting locale failed.
        LANG = "en_US"

To install the missing “en_US” locale run ‘locale-gen en_US’ which should yeild the following output.

root@harrowdown:~# locale-gen en_US
Generating locales...
  en_US.ISO-8859-1... done
Generation complete.

And as you can see the errors are no longer printed when I call dpkg-reconfigure (or anything else for that matter).

root@harrowdown:~# dpkg-reconfigure locales
Generating locales...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.ISO-8859-1... up-to-date
  en_US.UTF-8... up-to-date
  en_ZA.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.

Unison File Synchronizer is a tremendous utility. It keeps the contents of two directories in sync with one another. It’s fast, secure, lightweight and good at sorting out collisions but unfortunately it lacks the ability to generate alerts when errors occur. Instead it just silently makes note of the error in a log file.

Luckily we can watch Unison’s logs for error events and notify ourselves as soon as a match is found with a tool called Simple Event Correlator, or SEC.

By default, Unison writes log entries in a file named “unison.log” under the home directory of the user running the application. We’ll assume that unison is running as root and is logging to the default location of /root/unison.log.

To configure SEC we simply need to write a short rule that includes the regular expression that matches a unison error message and the action we wish to execute when a match is found. Here is a simple example rule to send e-mail alerts on unison file conflicts.

type=Single
ptype=RegExp
pattern=(^\[CONFLICT\]|^Fatal error)
desc=$0
action=pipe '' /usr/bin/tac /root/unison.log | grep '^UNISON' -m 1 -B 9999 | tac | /usr/bin/mailx -s "UNISON: $0" root

This will send an e-mail alert to root when a line starting with [CONFLICT] or Fatal Error is logged and include the most recent unison run log output in the body of the message.

To invoke sec use something to the effect of:

 sec -conf=./sec_unison.conf -input=/root/unison.log

Where -conf is the file containing of the above rule and -input is unison’s log file.

So far the simplest way find the last occurrence of a string and print everything below it is with a combination of grep and tac (it’s cat backwards, get it?!).

Say you want to find the last instance of foo and everything that came after it in a file:

 tac /var/log/logfile | grep 'foo' -m 1 -B 9999 | tac

This flips the logfile upside and pipes it to grep, who finds only the first occurrence and prints everything above it (well, 9999 lines), then flips that output right side up again. Works great but I wonder if there is a simpler way to do this.

When connecting a server to a storage area network (SAN) its important to make certain that you’re hosts are prepared for the occasional blip in SAN connectivity. Device mapper multipath to the rescue! Multipath is an abstraction layer between you and the raw block devices which allows for multiple I/O paths or networks (IO multipathing) and gives you an increased level of control over what happens should a block device start reporting errors. Best of all its built right in to the modern Linux kernel.

I maintain a cluster of Xen servers that store VM images on an EqualLogic PS6000 Series iSCSI SAN as raw LUNs. Its super-stable and makes it very simple to manage, snapshot and replicate storage. The only drawback is EqualLogic’s limitation of 512 connections per storage pool. This means that for every LUN (read VM) created we consume a connection. Multiply this by the number of dom0s and you’ll quickly see that the available connections would get eaten up in no time. In order to step around this boundary I made some significant modifications the block-iscsi Xen block script I found on an e-mail thread. Sorry, I don’t remember where it came from and there are many variations floating around.

I’ve tested this script on RHEL5 running Xen 3.1.4, your mileage may vary but as always, I’d love to hear your feedback!

/etc/xen/scripts/block-iscsi

#!/bin/bash
# block-iscsi  -  2009 Keith Herron <keith@backdrift.org>
#
# multipath enabled block-iscsi xen block script.
#
# Note: This script depends on a block-iscsi.conf file
#       located in the same directory.  This file contains
#       an array of available iSCSI target IPs
#
 
dir=$(dirname "$0")
. "$dir/block-common.sh"
. "$dir/block-iscsi.conf"
 
# Log which mode we are in
logger -t block-iscsi "*** Beginning device $command ***"
 
# Fetch the iqn we specify in the domu config file
#
IQN=$(xenstore_read "$XENBUS_PATH/params")
logger -t block-iscsi "IQN: ${IQN}"
 
# We define portal ip in order to support new luns which don't yet have
# /var/lib/iscsi/node entrys yet, not dynamic but avoids manual discovery 
#
for PORTAL in ${PORTALS[@]}; do
  logger -t block-iscsi `iscsiadm -m discovery -t st -p $PORTAL`
done
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target on a lun by lun basis
#
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
logger -t block-iscsi "TARGET: ${IP}:${PORT}"
 
# This is called by each command to determine which multipath map to use
#
function get_mpath_map {
   # Re-run multipath to ensure that maps are up to date
   #
   multipath
   sleep 2
 
   # Now we determine which /dev/sd* device belongs to the iqn
   #
   SCSI_DEV="/dev/`basename \`/usr/bin/readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0\``"
   logger -t block-iscsi "scsi device: ${SCSI_DEV}"
 
   # And using the /dev/sd* device we can determine its corresponding multipath entry
   #
   MPATH_MAP="/dev/mapper/`multipath -ll ${SCSI_DEV} | head -1 | awk '{ print $1}'`"
   logger -t block-iscsi "mpath device: ${MPATH_MAP}"
}
 
case $command in
   add)
      # Login to the target
      logger -t block-iscsi "logging in to ${IQN} on ${IP}:${PORT}"
      sleep 5
      #FIXME needs more advanced race condition logic
      iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --login | logger -t block-iscsi
      sleep 5
      #FIXME needs more advanced race condition logic
      get_mpath_map
 
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "${command}ing device: ${MPATH_MAP}"
         write_dev ${MPATH_MAP}
      fi
   ;;
 
   remove)
      get_mpath_map
      if [ -a ${MPATH_MAP} ]; then
         logger -t block-iscsi "flushing buffers on ${MPATH_MAP}"
         blockdev --flushbufs ${MPATH_MAP}
         logger -t block-iscsi "attempting logout of ${IQN} on ${IP}:${PORT}"
         iscsiadm -m node -T ${IQN} -p ${IP}:${PORT} --logout | logger -t block-iscsi
         sleep  10
         #FIXME needs more advanced race condition logic
      fi
      sleep 5
      #FIXME needs more advanced race condition logic
   ;;
esac

/etc/xen/scripts/block-iscsi.conf

# block-iscsi.conf  -  2009 Keith Herron <keith@backdrift.org>
# 
# Note: Config file for block-iscsi xen block script /etc/xen/scripts/block-iscsi
 
# Define iSCSI portal addresses here, necessary for discovery
PORTALS[0]="10.241.34.100"

To make use of this script you’ll need to update your xen guest config file to specify “iscsi” in the disk line instead of “phy” or similar.

domU configuration example

#
disk = [ 'iscsi:iqn.2001-05.com.equallogic:0-8a0906-23fe93404-c82797962054a96d-examplehost,xvda,w' ];
#

This is a handy way to log user’s bash histories to syslog without making any modifications to the bash source code itself. Simply drop the following snippet into either the per-user or system-wide bash profile (~/.bash_profile and /etc/profile, respectively)

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${COMMAND}"
}
trap log2syslog DEBUG

This won’t guarantee you log consistency in the event of a compromised host but you’ll certainly have and increased ability to correlate events on your systems.

Currently backdrift.org is running on a Xen VM. In order to make this more affordable and better utilize the co-located hardware, we’re offering dedicated VMs for lease on the primary backdrift.org server.

$20 per month gets you:

2x AMD Opteron VCPUs @ 2.6GHz
1G RAM
100GB Disk (on underlying RAID-1)
Un-metered bandwidth on bonded T1s (3mbps burstable)

Its a great platform for most applications. We’re currently running Xen 3.3 with PvGrub enabled and out of band management accounts are available for remote console and reboot access to your VM.

Payments via PayPal.

To get started send mail to vps@backdrift.org.

This is a list of handy key combinations I use when working with the HP iLO on some DL145 servers I manage.

Login Banner

Lights-Out 100 Management
Copyright 2000-2004 QLogic Corporation
Copyright 2004 Hewlett-Packard Development Company, L.P.
Type 'HELP' for options.  Type 'QUIT' to return to console.

Help menu

Lights-Out> help
CLI Commands:
   C[hangePassword] <enter>
   P[ower] <ON, OFF> <enter>
   I[PMICommand] <ipmb command data> <enter>
   SEN[sorStatus] <enter>
   SD[RRead] <enter>
   SEL[Read] <enter>
   N[etwork setup] <enter>
   R[eset] <WARM, COLD> <enter> (Host Reset)
   V[ersion] <enter>
   H[ELP] <enter>
   Q[UIT] <enter>
Lights-Out>

Shortcuts, Hotkeys and Keystrokes

To enter the virtual serial console: ESC, Q
To return to the Lights-Out Console: ESC, (

So if you’re like me you are probably somewhat lazy. I imagine most sysadmins are. Which must be why we put so much effort into scripting and automating repeated tasks. If you’ve managed a Xen VM before you are likely to notice that all your kernel images live outside the VM itself. This makes it rather difficult to patch and upgrade amond other things, especially as the number and type of guests you support grows. Pvgrub can help solve this problem. It implements grub in a stub domain which in turn loads your guest domain. It works quite well and is rather easy to set up. Read on to learn more.

From the xen wiki: “PV-GRUB replaces PyGrub to boot domU images safely: it runs the regular grub inside the created domain itself and uses regular domU facilities to read the disk / fetch files from network etc.; it eventually loads the PV kernel and chain-boots it.”

This means that your domU guests can install and manage their own kernels as if they were running on regular hardware. Yum, apt-get and other package management software will let you patch your virtual machine’s kernel without the need to alter any Xen configuration. Here’s how it works.

Notes:

* Pvgrub requires Xen 3.3 or greater.
* You won’t actually need to install grub in the domU, it is implemented as a stub domain and provided with the Xen sources.

Steps:

1. Modify /etc/modprobe.conf in the guest.

We need to make sure that a couple of lines exist in /etc/modprobe.conf before we’re ready to install the kernel packages. We add these so that when the kernel install process goes to build a new initrd it will include the correct driver modules for your hardware, so we want to be certain that the Xen block and net drivers are included.

Note: If /etc/modprobe.conf doesn’t already exist you should create it.

#/etc/modprobe.conf
alias eth0 xennet
alias scsi_hostadapter xenblk

2. Update /etc/inittab and /etc/securetty in the guest.

Now we update /etc/inittab and /etc/securetty to make certain that we attach to and permit logins from the various xen consoles.

/etc/inittab

#/etc/inittab
hvc0:2345:respawn:/sbin/getty 38400 hvc0
xvc0:2345:respawn:/sbin/getty 38400 xvc0

/etc/securetty

#/etc/securetty
# hypervisor virtual consoles
hvc0
xvc0

3. Install a stub grub.conf in the guest so that the grub update scripts won’t fail.

Configuration file locations:

Depending on the Linux distribution you use you’re likely to see the grub config in either /boot/grub/grub.conf or /boot/grub/menu.lst. To keep things simple I usually use /boot/grub/grub.conf as a config file and symlink /boot/grub/menu.lst to it. This should ensure that scripts used to automatically update the grub config when a new kernel is installed will work correctly.

Reasonable stub for Debian Lenny

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.26-2-xen-686
   root (hd0)
   kernel /boot/vmlinuz-2.6.26-2-xen-686 console=hvc0 root=/dev/xvda1 ro
   initrd /boot/initrd.img-2.6.26-2-xen-686

Reasonable stub for CentOS/RHEL

#/boot/grub/grub.conf
default 0
timeout 5
 
title vmlinuz-2.6.18-128.4.1.el5xen
	root (hd0)
	kernel /boot/vmlinuz-2.6.18-128.4.1.el5xen root=LABEL=root ro
	initrd /boot/initrd-2.6.18-128.4.1.el5xen.img

4. Install kernel packages in the guest.

Debian

# apt-get install linux-image-2.6.26-1-686-bigmem

CentOS/RHEL

# yum install kernel-xen

5. Modify Xen guest config file in the dom0.

Now that the VM has been prepped for booting with pvgrub we need to update its configuration file to load the pvgrub kernel instead of a linux kernel.

Note: be certain to remove the old “root=” line, this seems to confuse pvgrub. Also, make certain that the “extra=” line points to the location of your grub config file inside the guest. My example does not use partitions, if you do use partitions in your guests you’ll need so use something like (hd0,0), etc.

name    = 'superchunk'
 
vcpus =2;
 
memory 	= 512;
maxmem  = 4096;
 
#pvgrub here
kernel = "/usr/lib/xen/boot/pv-grub-x86_32.gz"
extra = "(hd0)/boot/grub/grub.conf"
#end pvgrub
 
disk = [ 'phy:/dev/vg0/vm_superchunk-root,xvda1,w', \
         'phy:/dev/vg0/vm_superchunk-swap,xvda2,w' ];
 
vif = [ 'bridge=eth1, vifname=superchunk, mac=B0:00:00:00:B1:35' ];

6. Shutdown and then create the domain.

My Xen guest is named superchunk. You should be familiar with shutdown and create at this point. Here’s how, just in case…

xm shutdown superchunk
#(wait until it has shut down)
xm create -c /path/to/superchunk-config

At this point you should see some kernel output and then either a grub menu or the grub command line. If you get the command line, no worries. This usually means that it couldn’t find a grub configuration file. Make sure that the “extra=” line of the Xen guest domU configuration file references the right location and that you have pointed grub at the correct disk, i.e. (hd0) vs. (hd0,0), etc…

Good luck! And feel free to drop me a line at soundsoldier@backdrift.org, I always appreciate feedback and do my best to help.

So last night a RHEL3 machine at went down for patching and sadly it didn’t come back. Here’s what was sitting on the console when I got to the console. Nice… a kernel panic :/

pivotroot: pivot_root(/sysroot,/sysroot/initrd) failed: 2 umount
/initrd/proc failed: 2 Freeing ununsed kernel memory: 228k freed Kernel
panic: No init found. Try passing init= option to kernel.

So after spending a significant amount of time troubleshooting the file systems, rebuilding the initrd, trying different init=arguments and trying different kernels I took a good hard look at the error message. Its talking about /sysroot/initrd but I don’t see a directory named /initrd when I look at the root filesystem. So I tried this:

mkdir /initrd
reboot

And to my surprise the system starts happily booting along. All that over a little directory, sheesh. Turns out that this directory was removed by someone or something that thought it wasn’t needed, doh.