If you have been a Linux sysadmin for a long time you probably consider ext3 a fairly inflexible file system. Fortunately, enhancements incorporated in the not so distant past have raised the bar in terms of functionality. It is possible now to perform online resizing with an ext3 formatted file system and if your system is using LVM online grow is as simple as two commands. Here is how to perform an online grow of a file system that is backed by a LVM volume.

Growing the LVM volume

[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                      9.9G  7.5G  1.9G  80% /var/lib/mysql
 
[root@host ~]# lvextend -L +1G /dev/vg0/vol_mysql
  Extending logical volume vol_mysql to 11.00 GB
  Logical volume vol_mysql successfully resized

Online resizing the ext3 file system

[root@host ~]# resize2fs /dev/vg0/vol_mysql 
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg0/vol_mysql is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/vg0/vol_mysql to 2883584 (4k) blocks.
The filesystem on /dev/vg0/vol_mysql is now 2883584 blocks long.
 
[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                       11G  7.5G  2.9G  73% /var/lib/mysql

As you can see we increased the file system from 10G to 11G which took us from 80% utilization down to a more comfortable 73%.

Establishing a VPN connection is usually a pain. They time out and close and, at least for me, are never up and running when you need them. I use VPN primarily to ssh into systems and because openssh is one of the most flexible pieces of software on the planet there is naturally a way to automate the VPN connection. Here we will use the ProxyCommand ssh config setting to establish a vpn session behind the scenes when we ssh to a machine.

What is a ssh ProxyCommand?

The ssh_config man page tells us that… “ProxyCommand specifies the command to use to connect to the server. The command string extends to the end of the line, and is executed with /bin/sh. In the command string, ‘%h’ will be substituted by the host name to connect and ‘%p’ by the port. The command can be basically anything, and should read from its standard input and write to its standard output. It should eventually connect an sshd(8) server running on some machine, or execute sshd -i somewhere. Host key management will be done using the HostName of the host being connected (defaulting to the name typed by the user). Setting the command to none disables this option entirely. Note that CheckHostIP is not available for connects with a proxy command.”

In other words, we can run an arbitrary command before ssh establishes its connection. Perfect! The only drawback is that ssh expects this command to be able to carry the ssh connection. Not a problem, we can script that.

[ad]

Tying it all together

This script is very easy, and only supports a single VPNC session at a time, but it is a good start for you to extend upon. This belongs in the .ssh/config file located inside your home directory. If the file or directory (or both) don’t already exist just create them.

Normally I would connect to the VPN manually using a VPNC configuration profile named ‘myvpn’ (which is located in /etc/vpnc/myvpn.conf).

~# vpnc myvpn
VPNC started in background (pid: 4579)...

To automate this, we will place the VPNC command into the ProxyCommand string as well as a small test condition to determine if a session is already running. Additionally, we need to include the nc command string. This is so that ssh can pass data through to the host after the command has been run. Finally, I have redirected stdout for VPNC to stderr so that it is printed to the terminal while being used as a ProxyCommand.

#~/.ssh/config
Host 192.168.10.*
  ProxyCommand bash -c 'if ! pgrep vpnc; then vpnc myvpn 1>&2; fi; nc -w 60 %h %p'

You can specify the host using an IP range (as demonstrated above) or via host names. I find IP addresses easier simply because I can wildcard the whole VPN range.

Here is what a ssh connection looks like with the updated .ssh/config

# gw2:~# ssh [email protected]
VPNC started in background (pid: 5091)...
Last login: Thu Jul 15 12:40:19 2010 from 172.16.1.11
[root@host ~]#

And upon subsequent connections while the VPN is established…

~# ssh [email protected]
root@192.168.10.110 password: 
Last login: Thu Jul 15 12:40:48 2010 from 172.16.1.11
[root@host ~]#

As you can see there is no output from vpnc, this means that the existing VPN session is being used.

There is much more you can do from here to further automate the process and implement more robust session detection and concurrency. Happy hacking!

Overview

Ping is an extremely useful utility for troubleshooting network connectivity. However, on today’s networks, icmp pings are often discarded. And with the prevalence of firewalls and load balancers it is often necessary to test the availability and response time of a particular port anyway. In this article we will use the hping utility to perform a tcp ping (ping a tcp port).

Running the TCP Ping

Hping is a network packet crafting utility. It is extremely flexible. This article only scratches the surface of what it can do.

In this example we will tcp ping a web server on tcp port 80 (-p 80) by sending it a SYN (-S) packet.

$ hping -S -p 80 google.com
HPING google.com (eth0 66.249.92.104): S set, 40 headers + 0 data bytes
len=44 ip=66.249.92.104 ttl=47 id=10442 sport=80 flags=SA seq=0 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=40838 sport=80 flags=SA seq=1 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=64607 sport=80 flags=SA seq=2 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=10443 sport=80 flags=SA seq=3 win=5720 rtt=97.7 ms
^C
--- google.com hping statistic ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 97.7/97.7/97.7 ms

Look familiar? The output is formatted very similarly to ping. In fact hping can perform traditional icmp echo pings as well by using the “-1” switch.

This utility has been immensely helpful in troubleshooting and documenting performance problems and testing the capabilities of firewalls. It’s also very useful for detecting when sshd comes back up after a system reboot (hint: -p 22).

More information

The hping utility is available on a wide variety of unix and windows operating systems.

Hping on Mac OSX is available using homebrew: http://brewformulas.org/Hping

Hping on RedHat compatible Linux distributions is available from EPEL

Hping on debian/ubuntu is available through apt.

Hping on windows is available from the hping download site http://www.hping.org/download.html

Have you ever dealt with a system that wasn’t partitioned properly when it was built and now it has gone into production? You’ll probably be hard pressed to find the time and patience to rebuild the system any time soon. Luckily there is a way to step around many of the limitations of a poorly partitioned system using bind mounts.

Bind mounts are quite simple. Instead of mounting a device (with a file system) on a particular path you are mounting one path into another path.

For example: Let’s say you have a small /var but a very large /opt partition and you need additional space for your growing log files.

First, shut down the services writing to log files, then…

mv /var/log /opt/var_log
mkdir /var/log
mount -o bind /opt/var_log /var/log

You will now see this reflected when running the mount command:

# mount | grep var
/opt/var_log on /var/log type none (rw,bind)

At this point you are ready to restart the previously stopped services.

If you want this to persist across reboots, you’ll just need to update your /etc/fstab with the bind mount as well.

# /etc/fstab
/opt/var_log              /var/log                 none    bind    0 0

And there you have it! Its not beautiful, but it will help you keep the lights on until you can get a long-term fix in place.

More details about bind mounts

From the man page on ‘mount’.

       The bind mounts.
              Since Linux 2.4.0 it is possible to remount part of the file hierarchy somewhere else. The call is
                     mount --bind olddir newdir
              or shortoption
                     mount -B olddir newdir
              or fstab entry is:
                     /olddir /newdir none bind
 
              After this call the same contents is accessible in two places.  One can also remount a single file (on a single file).
 
              This call attaches only (part of) a single filesystem, not possible submounts. The entire file hierarchy including submounts is attached a  sec-
              ond place using
                     mount --rbind olddir newdir
              or shortoption
                     mount -R olddir newdir
 
              Note that the filesystem mount options will remain the same as those on the original mount point, and cannot be changed by passing the -o option
              along with --bind/--rbind. The mount options can be changed by a separate remount command, for example:
 
                     mount --bind olddir newdir
                     mount -o remount,ro newdir

I use SAN snapshots extensively to provide point-in-time recoverability for a number of different systems. This is a very reliable and efficient backup solution but it comes at the cost of additional complexity for individual file restoration. This is because the snapshot backups are of the entire volume and not each individual file. So when a user says “hey, could you restore file x from two days ago?” I have some work to do.

The Problem

Recovering files from a SAN snapshot involves a 10 step manual process.

Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the system
4. Perform an iSCSI discovery
5. Perform an iSCSI login to the snapshot volume
6. Mount the snapshot volume
7. Copy the backup file(s) into place
8. Un-mount the volume
9. Perform an iSCSI logout
10. Revoke access to the snapshot from the host

This manual process is tedious and error prone. Luckily many of these steps can be easily automated. Enter Autofs.

The Solution

Autofs is a toolkit which enables you to automatically mount file systems in accordance to a configuration file (called a map) as you traverse a certain directory structure. It is very flexible and supports external scripts, or “program maps” in autofs-speak. As you can see above, the majority of the steps are oriented toward getting the volume to the host and mounted. Only one of the steps actually pertains to the file(s) we need to recover so let’s focus on automating the rest.

This program map, named auto.iscsi, eliminates manual steps 4, 5, 6, and 8. An additional cron job could be added to automate step 9 by checking to see if the devices associated with the hosts current iSCSI sessions are still mounted.

New Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the snaphost browsing system
4. cd to the automounted directory using the volume IQN
5. Copy the backup file(s) into place
6. Revoke access to the snapshot from the host

So we’ve removed 4 steps from the process with the following automounter script. I’m also working to automate 1, 2 and 6, but more on that later.

[ad]

auto.iscsi program map

#!/bin/bash
#
# auto.iscsi - Automounter map for mounting iscsi devices by iqn.  
#              Keith Herron <[email protected]> 
#
# Invoke this program in /etc/auto.master with a line similar to:
# /iscsiauto  program:/etc/auto.iscsi
 
# IQN - This script expects to recieve an iscsi iqn as the only argument.
IQN="$1"
 
# OPTS - Filesystem mount options.
OPTS="-nocheck,ro,fstype=ext3"
 
# PART_NO - Partition number.  This script assumes only one fs per iqn.
PART_NO="1"
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target.  We use this later to figure out the block dev.
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
# If we're already logged in don't bother trying to log in.
if iscsiadm -m session | grep "$IQN"; then
  echo node exists
else
  iscsiadm -m node -T "$IQN" --login 1>&2
fi
 
# Let udev settle before we try to look up the block device.
sleep 1
 
# Figure out the block device using the udev symlink.
DEV_LINK=`readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
DEV_NODE=`basename $DEV_LINK`
DEV_FULL="/dev/$DEV_NODE"
 
# Return the magic to automount
echo "${OPTS} :${DEV_FULL}${PART_NO}"

How to use it

In the below example I am recovering a file called /etc/really_important_file back to a host using scp.

[root@snapbrowse ~]# cd /iscsiauto/iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# ls
bin  boot  dev  etc  home  lib  lost+found  media  misc  mnt  opt  proc  root  sbin  selinux  shells  srv  sys  tmp  usr  var
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# scp -pr etc/real_important_file [email protected]:/etc/

Known Limitations

Next Steps

[ad]

Today, most web sites are powered by content management systems. And while this is great news for most users and content writers it can spell disaster for system administrators and webmasters. These systems can be incredibly complex and often lack robust security hardening and auditing features. In order to make up for some of these shortcomings (and to sleep just a little a bit better at night) we can implement simple yet effective security checks right in the web server using Apache’s mod_rewrite.

In this example we create a .htaccess file that acts as an IP access control list or firewall. Anyone who connects from an IP that is not in this list will be presented with a forbidden page. This is a good way to protect your site’s administrator interface, for example.

#.htaccess
RewriteEngine On
 
RewriteCond %{REMOTE_ADDR} !172.16.*
RewriteCond %{REMOTE_ADDR} !1.2.3.*
RewriteCond %{REMOTE_ADDR} !192.168.1.140
RewriteRule .* - [F]

As you can see, we use not statements to redirect everyone but authorized users to a forbidden page. This sort of protection is very easy to implement and maintain and provides you the ability to restrict individual components of your site at a very granular level.

[ad]

Effectively backing up your virtual machines is a problem with a multitude of potential solutions. Many xen backup solutions are centered around making a copy of the full volume(s) upon which your virtual machine(s) reside. But what happens if you want to recover just a single file and not the entire VM? And is it possible to delegate restoration to end users? Not to mention the space that is wasted by storing many copies of the same data.

Here I will describe how to implement a xen backup scheme that I have found very robust, efficient and reliable. It enables automated backups of Xen virtual machines that each reside on individual LVM volumes using a utility called rsnapshot (which is based on rsync).

Note: This method additionally can be utilized to back LVM volumes that aren’t housing Xen virtual machines.

Goals

LVM Snapshots

LVM provides a convenient snapshot feature. This allows you to create an identical clone of a logical volume and store only the blocks that differ from it. It is an unintuitive action and the result is a mirror image of your virtual machine’s disk that you can mount, read and even write data to.

Rsnapshot backups

Rsnapshot is a very flexible backup application which provides robust automation facilities. It supports LVM, ssh, rsync as well as custom scripts and will manage the process of rotating your backup files according to a defined schedule and retention policy. For our purposes we will be using the built-in LVM snapshot support.

Xen Backup – How it works

Each of my Xen virtual machines has own LVM volume per file system used by the system. For many systems this results in a single and somewhat large root file system housed on LVM. When rsnapshot is configured to back up one of these volumes it runs through the following steps:

Note: The use of LVM snapshots does introduce a write performance implication during the period that the snapshot exists. I have found this to be an acceptable tradeoff as most backup suites introduce some sort of performance effect while they are processing.

How to set it up

Assuming you already have a working LVM configuration (read more about LVM here), you’ll just need to install rsnapshot. It is available through dag for RedHat based platforms and available in the standard repositories for ubuntu and debian.

Once you have installed rsnapshot, defined your retention periods, and installed the corresponding cron jobs to call the backup process, you’re ready to add the LVM specifics to the rsnapshot configuration.

For example, I have a logical volume named “vm_insomniac-root” in a volume group named “vg0”. This represents the root filesystem of a virtual machine named “insomniac”.

[root@localhost .snapshot]# lvs | grep insomniac-root
   vm_insomniac-root vg0  -wi-ao  36.00G

To tell rsnapshot that I want to backup this LVM volume I add the following to /etc/rsnapshot.conf. Make sure this is tab delimited. Rsnapshot is unreasonably insistent upon tabs in its config file.

#/etc/rsnapshot.conf
linux_lvm_cmd_lvcreate	/sbin/lvcreate
linux_lvm_cmd_lvremove	/sbin/lvremove
linux_lvm_snapshotsize	5G
linux_lvm_snapshotname	rsnapshot-tmp
linux_lvm_vgpath	/dev
linux_lvm_mountpath	/mnt/rsnapshot-tmp
linux_lvm_cmd_mount	/bin/mount
linux_lvm_cmd_umount	/bin/umount
 
backup  lvm://vg0/vm_insomniac-root/    insomniac/

My snapshot_root is set as “/.snapshot”, and as such the backups look like this:

[root@localhost .snapshot]# cd /.snapshot
[root@localhost .snapshot]# ls
daily.0/  daily.1/  daily.2/  daily.3/  daily.4/  daily.5/  daily.6/  lost+found/
[root@localhost .snapshot]# ls daily.0/ | grep insomniac
insomniac/
[root@localhost .snapshot]# ls daily.0/insomniac/
bin/  boot/  etc/  home/  lib/  lost+found/  media/  mnt/  opt/  root/  sbin/  selinux/  srv/  tmp/  usr/  var/

As you can see, I’m only set up to keep a weeks worth of daily backups. If you have the disk space you could opt for a much longer retention period.

Delegation to end-users

To provide users with a simple interface to access their backups I share each of their rsnapshot directories as read-only NFS exports. To only their server of course. This allows users to traverse their backups.

One significant benefit is that there is no backup client for them to install and learn how to use. Restoration is performed with standard unix commands like cp and rsync, saving much time and frustration.

Potential Improvements and follow-up

Time permitting, I would like to test the performance, reliability and storage saving of using a dedup file systems such as lessfs and SDFS. I think these have the potential to make this solution even more robust, especially from a cost perspective.

I’m a sysadmin by trade and as such I deal with RAID enabled servers on a daily basis. Today a server with a hardware RAID controller reported (when I say reported I actually mean lit a small red LED on the front of the machine) a bad disk, which is not uncommon. But when replacing the failed disk with a shiny new one suddenly both drives went red and the system crashed. Upon reboot the hardware controller says to me “sorry buddy, I don’t see any drives” and wouldn’t boot. Luckily I had a similar system sitting idle, so I tested the same disks in this server and they worked just fine. After cursing loudly at the RAID controller I started wondering if the pros of hardware raid really outweigh the cons for a general purpose, nothing special, 1 or 2U server that just needs disk mirroring.

[ad]

Pros of Hardware RAID

Cons of Hardware RAID

Pros of Software RAID

Cons of Software RAID

[ad]

So what do I make of all this?

I can certainly understand the argument of simple deployment and having a vendor to blame. But at the end of the day if you loose your data then it’s gone. No vendor will be able to bring it back. And what about manageability after initial deployment?

It seems to me that if you care about the integrity of your data and do not need ultra-intense IO performance then software RAID is a good choice. You’ll get the same core features as hardware with a standard management and monitoring suite. Additionally you are no longer bound to a specific piece of hardware. So in the event of a disk controller failure you could simply replace the controller (or the whole server) and the disks will remain usable. I also find it desirable that software raid will allow you to define the raid configuration during the installation process which when coupled with kickstart or a similar automated build process will allow you to ensure proper raid configuration and monitoring as soon as the server is brought online.

Considering these benefits, and my sour experiences with hardware RAID, I plan to deploy software RAID on a much wider scale from this point forward. But I’m curious…

Have others struggled with similar problems? How many people are running software RAID in an enterprise setting? And how many people have been similarly burned by software RAID?

Recently the hard drive in my mac mini running Mac OSX Leopard (10.5) failed. Luckily I had time machine backing it up to an external USB disk. Now, since I had to replace the drive and rebuild my system anyway I figured, why not upgrade to Snow Leopard? Planning to just pull what I needed off the backup drive manually I went ahead with the upgrade. There aren’t too many files on this machine that I depend on. Just some ssh keys, gpg keys and random documents scattered about here and there. So I upgraded, installed my apps and copied my files from the backup. Everything was going smoothly until I tried to actually write to one of the files I copied from the backup drive. This is when I started getting permission errors.

Here’s what happened when I tried to update my ssh known_hosts file:

airbag:~ keith$ echo foo > .ssh/known_hosts 
-bash: .ssh/known_hosts: Permission denied

Huh? But I own this file…dont I?

airbag:~ keith$ id
uid=501(keith) gid=20(staff) groups=20(staff),402(com.apple.sharepoint.group.1),204(_developer),100(_lpoperator),98(_lpadmin),81(_appserveradm),80(admin),79(_appserverusr),61(localaccounts),12(everyone),401(com.apple.access_screensharing)
 
airbag:~ keith$ ls -al .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

I do own it… And so began much head scratching and man page reading.

Well, as it turns out I forgot to look at the file ACLs…

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts
 0: group:everyone deny write,delete,append,writeattr,writeextattr,chown

Well no wonder, the ACL is set to deny write,delete,append,writeattr,writeextattr and chown from everyone! Let’s get rid of that.

airbag:~ keith$ sudo chmod -N .ssh/known_hosts 
Password:

That ought to do it. The -N flag says get rid of all the ACL info on the file. You could also update this to be just right for your user or group but I’d rather use only the standard unix permissions.

airbag:~ keith$ ls -le .ssh/known_hosts 
-rw-r--r--@ 1 keith  502  56140 Mar 25  2009 .ssh/known_hosts

Seems to have removed all ACLs from the file. I wonder if we can write to it now…

airbag:~ keith$ echo foo >> .ssh/known_hosts 
airbag:~ keith$

And there you have it, the file is writable once again. Now its time to get some real work done!

While working on a new ubuntu karmic system I ran across the following locale error when attempting to perform a dist-upgrade. I was able to reproduce it by running ‘dpkg-reconfigure locales’ thinking that this might fix the error, sadly it did not.

Here’s the error I was seeing:

root@harrowdown:~# dpkg-reconfigure locales
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
        LANGUAGE = (unset),
        LC_ALL = (unset),
        LANG = "en_US"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

If you boil the error message down a bit the problem becomes a bit more clear. It’s trying to say that the “en_US” locale could not be found:

perl: warning: Setting locale failed.
        LANG = "en_US"

To install the missing “en_US” locale run ‘locale-gen en_US’ which should yeild the following output.

root@harrowdown:~# locale-gen en_US
Generating locales...
  en_US.ISO-8859-1... done
Generation complete.

And as you can see the errors are no longer printed when I call dpkg-reconfigure (or anything else for that matter).

root@harrowdown:~# dpkg-reconfigure locales
Generating locales...
  en_AG.UTF-8... done
  en_AU.UTF-8... done
  en_BW.UTF-8... done
  en_CA.UTF-8... done
  en_DK.UTF-8... done
  en_GB.UTF-8... done
  en_HK.UTF-8... done
  en_IE.UTF-8... done
  en_IN.UTF-8... done
  en_NG.UTF-8... done
  en_NZ.UTF-8... done
  en_PH.UTF-8... done
  en_SG.UTF-8... done
  en_US.ISO-8859-1... up-to-date
  en_US.UTF-8... up-to-date
  en_ZA.UTF-8... done
  en_ZW.UTF-8... done
Generation complete.

[ad]