If you’re like me then you enjoy time away from the desk. For IT professionals, however, it is not unusual to be interrupted during your personal time with a question or problem from a panicking co-worker or user. It used to be the case that you’d have to drop what you were doing and run to a computer to look into an issue but now that smart phones (iPhones specifically) are so widely available it is easy to troubleshoot a problem remotely from your hand held device. That is, if you have the right apps installed!

Here are the 5 apps that I rely on daily as an IT pro.

Network Utility – IP Troubleshooting Utility

Stuck in an area away from your computer? But still need to check your website or server status? With Network Utility you can check your website or server status from anywhere that you have internet or cell phone reception.

Features:

* Ping (ICMP Echo)
* TCP/IP Port Scan
* GeoIP lookup
* Geotargeting with Google Maps
* Whois Query
* IP Address Information

Screenshots:

[Show as slideshow]

iSource – Source Viewer and DNS Swiss Army Knife

iSource Browser is a source-aware web browser for the iPhone. It is a fully functional web browser with the following additional features.

* HTML source
* WHOIS lookup
* DIG lookup
* HOST lookup
* Emailing of debug info
* Transparent debug console

Screenshots:

[Show as slideshow]

Domain Scout – Domain Availability

Domain scout is a handy utility to perform domain name availability queries quickly from your iPhone.

Features:

* Supports all domains, including global country-codes.
* Customizable top-level domain buttons, to save typing.
* Smart-parsing of top WHOIS formats for convenient viewing.
* Searches are always private & safe from domain front-running.

Screenshots:

[Show as slideshow]

IRC 999 – IRC Client

IRC is an excellent way to get another set of eyes on the problem you’re experiencing. IRC999 provides a fully functional IRC client in your pocket for free. It was a little bit unintuitive to get set up initially, but once up and running it works great!

Screenshots:

[Show as slideshow]

iRdesktop – Remote Desktop Client

iRdesktop is a free Remote Desktop Client for Windows Terminal Services (Remote Desktop Services), capable of natively using the Remote Desktop Protocol (RDP) in order to view and control your Windows Desktop using your iPhone and iPod Touch.

Screenshots:

[Show as slideshow]

Now you are probably wondering where the SSH client is. Unfortunately I haven’t come across a free SSH client for the iPhone. However there are a number of paid clients that you can read more about here.

I hope you find these apps as useful as I do!

Time and time again I run into an instance where I have a file on host 1 and a file on host 2 but host 1 can’t connect directly to host 2. Typically this requires me to copy the file locally to my desktop, then copy it again to the destination host. Thanks to the fact that ssh can tunnel standard input, however, there is a simple way to effectively scp directly from remote host to remote host using your machine as an intermediary. Here’s how.

SCP files using tar and ssh pipes

root@desktop ~ $ ssh host1 'tar -c ./foo' | ssh host2 'tar -x'

As you can see, we are simply redirecting the output of ‘tar -c ./foo’ on host1 to ‘tar -x’ on host2 through ssh. If you want to copy the file to a location other that your home directory on the remote server you can use something similar to the following.

root@desktop ~ $ ssh host1 'tar -c ./foo' | ssh host2 'cd /tmp; tar -x'

All we need to do is execute a cd command prior to our tar command to change the target of the copy.

I hope this helps! If you have a different way to do this leave a commend, I would love to hear it!

If you have been a Linux sysadmin for a long time you probably consider ext3 a fairly inflexible file system. Fortunately, enhancements incorporated in the not so distant past have raised the bar in terms of functionality. It is possible now to perform online resizing with an ext3 formatted file system and if your system is using LVM online grow is as simple as two commands. Here is how to perform an online grow of a file system that is backed by a LVM volume.

Growing the LVM volume

[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                      9.9G  7.5G  1.9G  80% /var/lib/mysql
 
[root@host ~]# lvextend -L +1G /dev/vg0/vol_mysql
  Extending logical volume vol_mysql to 11.00 GB
  Logical volume vol_mysql successfully resized

Online resizing the ext3 file system

[root@host ~]# resize2fs /dev/vg0/vol_mysql 
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg0/vol_mysql is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/vg0/vol_mysql to 2883584 (4k) blocks.
The filesystem on /dev/vg0/vol_mysql is now 2883584 blocks long.
 
[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                       11G  7.5G  2.9G  73% /var/lib/mysql

As you can see we increased the file system from 10G to 11G which took us from 80% utilization down to a more comfortable 73%.

Establishing a VPN connection is usually a pain. They time out and close and, at least for me, are never up and running when you need them. I use VPN primarily to ssh into systems and because openssh is one of the most flexible pieces of software on the planet there is naturally a way to automate the VPN connection. Here we will use the ProxyCommand ssh config setting to establish a vpn session behind the scenes when we ssh to a machine.

What is a ssh ProxyCommand?

The ssh_config man page tells us that… “ProxyCommand specifies the command to use to connect to the server. The command string extends to the end of the line, and is executed with /bin/sh. In the command string, ‘%h’ will be substituted by the host name to connect and ‘%p’ by the port. The command can be basically anything, and should read from its standard input and write to its standard output. It should eventually connect an sshd(8) server running on some machine, or execute sshd -i somewhere. Host key management will be done using the HostName of the host being connected (defaulting to the name typed by the user). Setting the command to none disables this option entirely. Note that CheckHostIP is not available for connects with a proxy command.”

In other words, we can run an arbitrary command before ssh establishes its connection. Perfect! The only drawback is that ssh expects this command to be able to carry the ssh connection. Not a problem, we can script that.

[ad]

Tying it all together

This script is very easy, and only supports a single VPNC session at a time, but it is a good start for you to extend upon. This belongs in the .ssh/config file located inside your home directory. If the file or directory (or both) don’t already exist just create them.

Normally I would connect to the VPN manually using a VPNC configuration profile named ‘myvpn’ (which is located in /etc/vpnc/myvpn.conf).

~# vpnc myvpn
VPNC started in background (pid: 4579)...

To automate this, we will place the VPNC command into the ProxyCommand string as well as a small test condition to determine if a session is already running. Additionally, we need to include the nc command string. This is so that ssh can pass data through to the host after the command has been run. Finally, I have redirected stdout for VPNC to stderr so that it is printed to the terminal while being used as a ProxyCommand.

#~/.ssh/config
Host 192.168.10.*
  ProxyCommand bash -c 'if ! pgrep vpnc; then vpnc myvpn 1>&2; fi; nc -w 60 %h %p'

You can specify the host using an IP range (as demonstrated above) or via host names. I find IP addresses easier simply because I can wildcard the whole VPN range.

Here is what a ssh connection looks like with the updated .ssh/config

# gw2:~# ssh root@192.168.10.110
VPNC started in background (pid: 5091)...
Last login: Thu Jul 15 12:40:19 2010 from 172.16.1.11
[root@host ~]#

And upon subsequent connections while the VPN is established…

~# ssh root@192.168.10.110
root@192.168.10.110 password: 
Last login: Thu Jul 15 12:40:48 2010 from 172.16.1.11
[root@host ~]#

As you can see there is no output from vpnc, this means that the existing VPN session is being used.

There is much more you can do from here to further automate the process and implement more robust session detection and concurrency. Happy hacking!

Ping is an extremely useful utility for troubleshooting network connectivity. However, on today’s networks, pings are often discarded. And with the prevalence of firewalls and load balancers it is often necessary to test the availability and response time of a particular port anyway. The hping utility allows us to do exactly this, and more.

Hping is a packet crafting utility. It is extremely flexible and here we will only touch the surface of what it can do.

In this example we will ping a webserver on tcp port 80 (-p 80) by sending it a SYN (-S) packet.

$ hping -S -p 80 google.com
HPING google.com (eth0 66.249.92.104): S set, 40 headers + 0 data bytes
len=44 ip=66.249.92.104 ttl=47 id=10442 sport=80 flags=SA seq=0 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=40838 sport=80 flags=SA seq=1 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=64607 sport=80 flags=SA seq=2 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=10443 sport=80 flags=SA seq=3 win=5720 rtt=97.7 ms
^C
--- google.com hping statistic ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 97.7/97.7/97.7 ms

Look familiar? The output is formatted very similarly to ping. In fact hping can perform traditional icmp echo pings as well by using the “-1″ switch.

This utility has been immensely helpful in troubleshooting and documenting performance problems and testing the capabilities of firewalls. It’s also very useful for detecting when sshd comes back up after a system reboot (hint: -p 22).

Have you ever dealt with a system that wasn’t partitioned properly when it was built and now it has gone into production? You’ll probably be hard pressed to find the time and patience to rebuild the system any time soon. Luckily there is a way to step around many of the limitations of a poorly partitioned system. How? With bind mounts.

Bind mounting is quite simple. Instead of mounting a block device (formatted with a file system) into a particular path you are mounting one path into another path.

For example: Let’s say you have a small /var but a very large /opt partition and you need additional space for your growing log files.

First, shut down the services writing to log files, then…

mv /var/log /opt/var_log
mkdir /var/log
mount -o bind /opt/var_log /var/log

You will now see this reflected when running the mount command:

# mount | grep var
/opt/var_log on /var/log type none (rw,bind)

At this point you are ready to restart the previously stopped services.

If you want this to persist across reboots, you’ll just need to update your /etc/fstab with the bind mount as well.

# /etc/fstab
/opt/var_log              /var/log                 none    bind    0 0

And there you have it! Its not beautiful, but it will help you keep the lights on until you can get a long-term fix in place.

I use SAN snapshots extensively to provide point-in-time recoverability for a number of different systems. This is a very reliable and efficient backup solution but it comes at the cost of additional complexity for individual file restoration. This is because the snapshot backups are of the entire volume and not each individual file. So when a user says “hey, could you restore file x from two days ago?” I have some work to do.

The Problem

Recovering files from a SAN snapshot involves a 10 step manual process.

Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the system
4. Perform an iSCSI discovery
5. Perform an iSCSI login to the snapshot volume
6. Mount the snapshot volume
7. Copy the backup file(s) into place
8. Un-mount the volume
9. Perform an iSCSI logout
10. Revoke access to the snapshot from the host

This manual process is tedious and error prone. Luckily many of these steps can be easily automated. Enter Autofs.

The Solution

Autofs is a toolkit which enables you to automatically mount file systems in accordance to a configuration file (called a map) as you traverse a certain directory structure. It is very flexible and supports external scripts, or “program maps” in autofs-speak. As you can see above, the majority of the steps are oriented toward getting the volume to the host and mounted. Only one of the steps actually pertains to the file(s) we need to recover so let’s focus on automating the rest.

This program map, named auto.iscsi, eliminates manual steps 4, 5, 6, and 8. An additional cron job could be added to automate step 9 by checking to see if the devices associated with the hosts current iSCSI sessions are still mounted.

New Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the snaphost browsing system
4. cd to the automounted directory using the volume IQN
5. Copy the backup file(s) into place
6. Revoke access to the snapshot from the host

So we’ve removed 4 steps from the process with the following automounter script. I’m also working to automate 1, 2 and 6, but more on that later.

[ad]

auto.iscsi program map

#!/bin/bash
#
# auto.iscsi - Automounter map for mounting iscsi devices by iqn.  
#              Keith Herron <keith@backdrift.org> 
#
# Invoke this program in /etc/auto.master with a line similar to:
# /iscsiauto  program:/etc/auto.iscsi
 
# IQN - This script expects to recieve an iscsi iqn as the only argument.
IQN="$1"
 
# OPTS - Filesystem mount options.
OPTS="-nocheck,ro,fstype=ext3"
 
# PART_NO - Partition number.  This script assumes only one fs per iqn.
PART_NO="1"
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target.  We use this later to figure out the block dev.
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
# If we're already logged in don't bother trying to log in.
if iscsiadm -m session | grep "$IQN"; then
  echo node exists
else
  iscsiadm -m node -T "$IQN" --login 1>&2
fi
 
# Let udev settle before we try to look up the block device.
sleep 1
 
# Figure out the block device using the udev symlink.
DEV_LINK=`readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
DEV_NODE=`basename $DEV_LINK`
DEV_FULL="/dev/$DEV_NODE"
 
# Return the magic to automount
echo "${OPTS} :${DEV_FULL}${PART_NO}"

How to use it

In the below example I am recovering a file called /etc/really_important_file back to a host using scp.

[root@snapbrowse ~]# cd /iscsiauto/iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# ls
bin  boot  dev  etc  home  lib  lost+found  media  misc  mnt  opt  proc  root  sbin  selinux  shells  srv  sys  tmp  usr  var
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# scp -pr etc/real_important_file root@host.recovering.to:/etc/

Known Limitations

Next Steps

[ad]

Today, most web sites are powered by content management systems. And while this is great news for most users and content writers it can spell disaster for system administrators and webmasters. These systems can be incredibly complex and often lack robust security hardening and auditing features. In order to make up for some of these shortcomings (and to sleep just a little a bit better at night) we can implement simple yet effective security checks right in the web server using Apache’s mod_rewrite.

In this example we create a .htaccess file that acts as an IP access control list or firewall. Anyone who connects from an IP that is not in this list will be presented with a forbidden page. This is a good way to protect your site’s administrator interface, for example.

#.htaccess
RewriteEngine On
 
RewriteCond %{REMOTE_ADDR} !172.16.*
RewriteCond %{REMOTE_ADDR} !1.2.3.*
RewriteCond %{REMOTE_ADDR} !192.168.1.140
RewriteRule .* - [F]

As you can see, we use not statements to redirect everyone but authorized users to a forbidden page. This sort of protection is very easy to implement and maintain and provides you the ability to restrict individual components of your site at a very granular level.

[ad]

Effectively backing up your virtual machines is a problem with a multitude of potential solutions. Many solutions are centered around making a copy of the full volume(s) upon which your virtual machine(s) reside. But what happens if you want to recover just a single file and not the entire VM? And is it possible to delegate restoration to end users? Not to mention the space that is wasted by storing many copies of the same data.

Here I will describe how to implement a backup scheme that I have found very robust, efficient and reliable. It enables automated backups of Xen virtual machines that each reside on individual LVM volumes using a utility called rsnapshot (which is based on rsync).

Note: This method additionally can be utilized to back LVM volumes that aren’t housing Xen virtual machines.

Goals

---

LVM Snapshots

---

LVM provides a convenient snapshot feature. This allows you to create an identical clone of a logical volume and store only the blocks that differ from it. It is an unintuitive action and the result is a mirror image of your virtual machine’s disk that you can mount, read and even write data to.

Rsnapshot backups

---

Rsnapshot is a very flexible backup application which provides robust automation facilities. It supports LVM, ssh, rsync as well as custom scripts and will manage the process of rotating your backup files according to a defined schedule and retention policy. For our purposes we will be using the built-in LVM snapshot support.

How it works

---

Each of my Xen virtual machines has own LVM volume per file system used by the system. For many systems this results in a single and somewhat large root file system housed on LVM. When rsnapshot is configured to back up one of these volumes it runs through the following steps:

Note: The use of LVM snapshots does introduce a write performance implication during the period that the snapshot exists. I have found this to be an acceptable tradeoff as most backup suites introduce some sort of performance effect while they are processing.

How to set it up

---

Assuming you already have a working LVM configuration (read more about LVM here), you’ll just need to install rsnapshot. It is available through dag for RedHat based platforms and available in the standard repositories for ubuntu and debian.

Once you have installed rsnapshot, defined your retention periods, and installed the corresponding cron jobs to call the backup process, you’re ready to add the LVM specifics to the rsnapshot configuration.

For example, I have a logical volume named “vm_insomniac-root” in a volume group named “vg0″. This represents the root filesystem of a virtual machine named “insomniac”.

[root@localhost .snapshot]# lvs | grep insomniac-root
   vm_insomniac-root vg0  -wi-ao  36.00G

To tell rsnapshot that I want to backup this LVM volume I add the following to /etc/rsnapshot.conf. Make sure this is tab delimited. Rsnapshot is unreasonably insistent upon tabs in its config file.

#/etc/rsnapshot.conf
backup  lvm://vg0/vm_insomniac-root/    insomniac/

My snapshot_root is set as “/.snapshot”, and as such the backups look like this:

[root@localhost .snapshot]# cd /.snapshot
[root@localhost .snapshot]# ls
daily.0/  daily.1/  daily.2/  daily.3/  daily.4/  daily.5/  daily.6/  lost+found/
[root@localhost .snapshot]# ls daily.0/ | grep insomniac
insomniac/
[root@localhost .snapshot]# ls daily.0/insomniac/
bin/  boot/  etc/  home/  lib/  lost+found/  media/  mnt/  opt/  root/  sbin/  selinux/  srv/  tmp/  usr/  var/

As you can see, I’m only set up to keep a weeks worth of daily backups. If you have the disk space you could opt for a much longer retention period.

Delegation to end-users

---

To provide users with a simple interface to access their backups I share each of their rsnapshot directories as read-only NFS exports. To only their server of course. This allows users to traverse their backups.

One significant benefit is that there is no backup client for them to install and learn how to use. Restoration is performed with standard unix commands like cp and rsync, saving much time and frustration.

Potential Improvements and follow-up

Time permitting, I would like to test the performance, reliability and storage saving of using a dedup file systems such as lessfs and SDFS. I think these have the potential to make this solution even more robust, especially from a cost perspective.

I’m a sysadmin by trade and as such I deal with RAID enabled servers on a daily basis. Today a server with a hardware RAID controller reported (when I say reported I actually mean lit a small red LED on the front of the machine) a bad disk, which is not uncommon. But when replacing the failed disk with a shiny new one suddenly both drives went red and the system crashed. Upon reboot the hardware controller says to me “sorry buddy, I don’t see any drives” and wouldn’t boot. Luckily I had a similar system sitting idle, so I tested the same disks in this server and they worked just fine. After cursing loudly at the RAID controller I started wondering if the pros of hardware raid really outweigh the cons for a general purpose, nothing special, 1 or 2U server that just needs disk mirroring.

[ad]

Pros of Hardware RAID

Cons of Hardware RAID

Pros of Software RAID

Cons of Software RAID

[ad]

So what do I make of all this?

I can certainly understand the argument of simple deployment and having a vendor to blame. But at the end of the day if you loose your data then it’s gone. No vendor will be able to bring it back. And what about manageability after initial deployment?

It seems to me that if you care about the integrity of your data and do not need ultra-intense IO performance then software RAID is a good choice. You’ll get the same core features as hardware with a standard management and monitoring suite. Additionally you are no longer bound to a specific piece of hardware. So in the event of a disk controller failure you could simply replace the controller (or the whole server) and the disks will remain usable. I also find it desirable that software raid will allow you to define the raid configuration during the installation process which when coupled with kickstart or a similar automated build process will allow you to ensure proper raid configuration and monitoring as soon as the server is brought online.

Considering these benefits, and my sour experiences with hardware RAID, I plan to deploy software RAID on a much wider scale from this point forward. But I’m curious…

Have others struggled with similar problems? How many people are running software RAID in an enterprise setting? And how many people have been similarly burned by software RAID?