I came across this odd puppet error while setting up a puppet client on a host running in amazon EC2 and it took me a bit of head scratching to figure it out.

(/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': undefined method `closed?' for nil:NilClass
 
(/File[/var/lib/puppet/lib]) Failed to retrieve current state of resource: undefined method `closed?' for nil:NilClass Could not retrieve file metadata for puppet://example.com/plugins: undefined method `closed?' for nil:NilClass
 
Could not retrieve catalog from remote server: undefined method `closed?' for nil:NilClass

The problem was that the address I used in my puppet.conf file as the puppet server (example.com) did not match the host name that puppet had in the server CA. A quick update to puppet.conf and everything was working as expected!

This is the most complete list of ports used by NetApp filers that I have been able to find to date. It is from the /etc/services file that NetApp provides within their firmware that they have updated to show only services consumed or served by a NetApp filer.

This is taken from Data ONTAP firmware Release 7.2.6.1.

#/vol/vol0/etc/services 
#
# Network services, Internet style
#
 
ftp-data        20/tcp
ftp             21/tcp
ssh             22/tcp
telnet          23/tcp
smtp            25/tcp
time            37/tcp           # Time Service
time            37/udp           # Time Service
domain          53/udp           # DNS
domain          53/tcp           # DNS
portmap         111/udp
portmap         111/tcp
dhcps           67/udp           # DHCP server
dhcpc           68/udp           # DHCP client
tftp            69/udp
http            80/tcp
kerberos        88/udp           # Kerberos 5 
kerberos        88/tcp           # Kerberos 5 
nntp            119/tcp         
ntp             123/tcp          # Network Time Protocol
ntp             123/udp          # Network Time Protocol
netbios-name    137/udp          # NetBIOS nameserver 
netbios-dg      138/udp          # NetBIOS datagram service 
netbios-ssn     139/tcp          # NetBIOS service session 
snmp            161/udp
ldap            389/tcp          # LDAP session
https           443/tcp          # SecureAdmin/SSL
cifs-tcp        445/tcp          # CIFS over TCP with NetBIOS framing
kpasswd         464/tcp          # Filer does not listen on this port;
                                 # used as Domain Controller destination port 
                                 # for Kerberos passwd set/change operations
shell           514/tcp
syslog          514/udp
route           520/udp
ldap-ssl        636/tcp          # LDAP over SSL
kerberos-sec    750/udp          # For compatibility with older "750" clients
kerberos-sec    750/tcp          # For compatibility with older "750" clients
nfsd            2049/udp
nfsd            2049/tcp
nrv             2050/tcp         # NetApp Remote Volume protocol, used in 
                                 # FlexCache and Restore-On-Demand.
 
iscsi-target    3260/tcp
 
nlockmgr        4045/tcp         # NLM
nlockmgr        4045/udp
mountd          4046/tcp         # NFS mountd protocol
mountd          4046/udp
status          4047/tcp
status          4047/udp
pcnfsd          4048/tcp         # PCNFS protocol
pcnfsd          4048/udp
rquotad         4049/udp
 
ndmp           10000/tcp
sm-ics         10565/tcp         # Snapmirror Multipath 
snapmirror     10566/tcp         
sm-sync-block  10567/tcp         # Snapmirror Sync Block Data
sm-sync-trans  10568/tcp         # Snapmirror Sync Transaction Data
sm-sync-ctrl   10569/tcp         # Snapmirror Sync Control Data
nbu-nearstore  10571/tcp         # NetBackup - Nearstore
sm-ics-test    10670/tcp         # INTERNAL USE: Snapmirror Multipath Test
ndmp-local     32243/tcp         # Internal connection inside NetApp box

There are probably a million and one individual pieces of software and websites you can use to generate a randomized password string. But the truth of the matter is that, if you have a UNIX machine, you don’t need them at all! Here’s how to generate a randomized password using widely available UNIX commands.

Random Data

UNIX is really, really good at generating random output. In fact, there is a device dedicated specifically to this cause. Meet /dev/random. We will be using this device as the source of our random password.

What about ASCII?

/dev/random provides us with some *really* random output. If you were to use a section of this random output in its raw form you would likely run into characters that are hard if not impossible to enter with your keyboard. To address this we will use uuencode to convert the raw output into a more human readable base64 version.

Putting it all together

Using dd we can take a small slice of randomness and pipe it into uuencode. The second to last line will be our randomized password.

Note: you may need to install the ’sharutils’ package onto your system if uuencode isn’t installed by default.

$ dd if=/dev/random bs=1 count=12 | uuencode -m -
begin-base64 644 -
12+0 records in
12+0 records out
12 bytes transferred in 0.000165 secs (72734 bytes/sec)
KJ1yeC4MtSg5QQCY
====

“dd if=/dev/random bs=1 count=12 ” outputs 12 (count=12) bytes (bs=1) of random data (if=/dev/random).

“| uuencode -m -” This reads the input from the previous command (pipe and trailing -) and encodes it into base64 (-m)

And there you have it, your shiny new random password!

If you’re like me then you enjoy time away from the desk. For IT professionals, however, it is not unusual to be interrupted during your personal time with a question or problem from a panicking co-worker or user. It used to be the case that you’d have to drop what you were doing and run to a computer to look into an issue but now that smart phones (iPhones specifically) are so widely available it is easy to troubleshoot a problem remotely from your hand held device. That is, if you have the right apps installed!

Here are the 5 apps that I rely on daily as an IT pro.

Network Utility – IP Troubleshooting Utility

Stuck in an area away from your computer? But still need to check your website or server status? With Network Utility you can check your website or server status from anywhere that you have internet or cell phone reception.

Features:

* Ping (ICMP Echo)
* TCP/IP Port Scan
* GeoIP lookup
* Geotargeting with Google Maps
* Whois Query
* IP Address Information

Screenshots:

iSource – Source Viewer and DNS Swiss Army Knife

iSource Browser is a source-aware web browser for the iPhone. It is a fully functional web browser with the following additional features.

* HTML source
* WHOIS lookup
* DIG lookup
* HOST lookup
* Emailing of debug info
* Transparent debug console

Screenshots:

Domain Scout – Domain Availability

Domain scout is a handy utility to perform domain name availability queries quickly from your iPhone.

Features:

* Supports all domains, including global country-codes.
* Customizable top-level domain buttons, to save typing.
* Smart-parsing of top WHOIS formats for convenient viewing.
* Searches are always private & safe from domain front-running.

Screenshots:

IRC 999 – IRC Client

IRC is an excellent way to get another set of eyes on the problem you’re experiencing. IRC999 provides a fully functional IRC client in your pocket for free. It was a little bit unintuitive to get set up initially, but once up and running it works great!

Screenshots:

iRdesktop – Remote Desktop Client

iRdesktop is a free Remote Desktop Client for Windows Terminal Services (Remote Desktop Services), capable of natively using the Remote Desktop Protocol (RDP) in order to view and control your Windows Desktop using your iPhone and iPod Touch.

Screenshots:

Now you are probably wondering where the SSH client is. Unfortunately I haven’t come across a free SSH client for the iPhone. However there are a number of paid clients that you can read more about here.

I hope you find these apps as useful as I do!

Time and time again I run into an instance where I have a file on host 1 and a file on host 2 but host 1 can’t connect directly to host 2. Typically this requires me to copy the file locally to my desktop, then copy it again to the destination host. Thanks to the fact that ssh can tunnel standard input, however, there is a simple way to effectively scp directly from remote host to remote host using your machine as an intermediary. Here’s how.

SCP files using tar and ssh pipes

root@desktop ~ $ ssh host1 'tar -c ./foo' | ssh host2 'tar -x'

As you can see, we are simply redirecting the output of ‘tar -c ./foo’ on host1 to ‘tar -x’ on host2 through ssh. If you want to copy the file to a location other that your home directory on the remote server you can use something similar to the following.

root@desktop ~ $ ssh host1 'tar -c ./foo' | ssh host2 'cd /tmp; tar -x'

All we need to do is execute a cd command prior to our tar command to change the target of the copy.

I hope this helps! If you have a different way to do this leave a commend, I would love to hear it!

If you have been a Linux sysadmin for a long time you probably consider ext3 a fairly inflexible file system. Fortunately, enhancements incorporated in the not so distant past have raised the bar in terms of functionality. It is possible now to perform online resizing with an ext3 formatted file system and if your system is using LVM online grow is as simple as two commands. Here is how to perform an online grow of a file system that is backed by a LVM volume.

Growing the LVM volume

[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                      9.9G  7.5G  1.9G  80% /var/lib/mysql
 
[root@host ~]# lvextend -L +1G /dev/vg0/vol_mysql
  Extending logical volume vol_mysql to 11.00 GB
  Logical volume vol_mysql successfully resized

Online resizing the ext3 file system

[root@host ~]# resize2fs /dev/vg0/vol_mysql 
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg0/vol_mysql is mounted on /var/lib/mysql; on-line resizing required
Performing an on-line resize of /dev/vg0/vol_mysql to 2883584 (4k) blocks.
The filesystem on /dev/vg0/vol_mysql is now 2883584 blocks long.
 
[root@host ~]# df -h | grep mysql
/dev/mapper/vg0-vol_mysql
                       11G  7.5G  2.9G  73% /var/lib/mysql

As you can see we increased the file system from 10G to 11G which took us from 80% utilization down to a more comfortable 73%.

Establishing a VPN connection is usually a pain. They time out and close and, at least for me, are never up and running when you need them. I use VPN primarily to ssh into systems and because openssh is one of the most flexible pieces of software on the planet there is naturally a way to automate the VPN connection. Here we will use the ProxyCommand ssh config setting to establish a vpn session behind the scenes when we ssh to a machine.

What is a ssh ProxyCommand?

The ssh_config man page tells us that… “ProxyCommand specifies the command to use to connect to the server. The command string extends to the end of the line, and is executed with /bin/sh. In the command string, ‘%h’ will be substituted by the host name to connect and ‘%p’ by the port. The command can be basically anything, and should read from its standard input and write to its standard output. It should eventually connect an sshd(8) server running on some machine, or execute sshd -i somewhere. Host key management will be done using the HostName of the host being connected (defaulting to the name typed by the user). Setting the command to none disables this option entirely. Note that CheckHostIP is not available for connects with a proxy command.”

In other words, we can run an arbitrary command before ssh establishes its connection. Perfect! The only drawback is that ssh expects this command to be able to carry the ssh connection. Not a problem, we can script that.

Tying it all together

This script is very easy, and only supports a single VPNC session at a time, but it is a good start for you to extend upon. This belongs in the .ssh/config file located inside your home directory. If the file or directory (or both) don’t already exist just create them.

Normally I would connect to the VPN manually using a VPNC configuration profile named ‘myvpn’ (which is located in /etc/vpnc/myvpn.conf).

~# vpnc myvpn
VPNC started in background (pid: 4579)...

To automate this, we will place the VPNC command into the ProxyCommand string as well as a small test condition to determine if a session is already running. Additionally, we need to include the nc command string. This is so that ssh can pass data through to the host after the command has been run. Finally, I have redirected stdout for VPNC to stderr so that it is printed to the terminal while being used as a ProxyCommand.

#~/.ssh/config
Host 192.168.10.*
  ProxyCommand bash -c 'if ! pgrep vpnc; then vpnc myvpn 1>&2; fi; nc -w 60 %h %p'

You can specify the host using an IP range (as demonstrated above) or via host names. I find IP addresses easier simply because I can wildcard the whole VPN range.

Here is what a ssh connection looks like with the updated .ssh/config

# gw2:~# ssh root@192.168.10.110
VPNC started in background (pid: 5091)...
Last login: Thu Jul 15 12:40:19 2010 from 172.16.1.11
[root@host ~]#

And upon subsequent connections while the VPN is established…

~# ssh root@192.168.10.110
root@192.168.10.110 password: 
Last login: Thu Jul 15 12:40:48 2010 from 172.16.1.11
[root@host ~]#

As you can see there is no output from vpnc, this means that the existing VPN session is being used.

There is much more you can do from here to further automate the process and implement more robust session detection and concurrency. Happy hacking!

Ping is an extremely useful utility for troubleshooting network connectivity. However, on today’s networks, pings are often discarded. And with the prevalence of firewalls and load balancers it is often necessary to test the availability and response time of a particular port anyway. The hping utility allows us to do exactly this, and more.

Hping is a packet crafting utility. It is extremely flexible and here we will only touch the surface of what it can do.

In this example we will ping a webserver on tcp port 80 (-p 80) by sending it a SYN (-S) packet.

$ hping -S -p 80 google.com
HPING google.com (eth0 66.249.92.104): S set, 40 headers + 0 data bytes
len=44 ip=66.249.92.104 ttl=47 id=10442 sport=80 flags=SA seq=0 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=40838 sport=80 flags=SA seq=1 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=64607 sport=80 flags=SA seq=2 win=5720 rtt=97.7 ms
len=44 ip=66.249.92.104 ttl=47 id=10443 sport=80 flags=SA seq=3 win=5720 rtt=97.7 ms
^C
--- google.com hping statistic ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 97.7/97.7/97.7 ms

Look familiar? The output is formatted very similarly to ping. In fact hping can perform traditional icmp echo pings as well by using the “-1″ switch.

This utility has been immensely helpful in troubleshooting and documenting performance problems and testing the capabilities of firewalls. It’s also very useful for detecting when sshd comes back up after a system reboot (hint: -p 22).

Have you ever dealt with a system that wasn’t partitioned properly when it was built and now it has gone into production? You’ll probably be hard pressed to find the time and patience to rebuild the system any time soon. Luckily there is a way to step around many of the limitations of a poorly partitioned system. How? With bind mounts.

Bind mounting is quite simple. Instead of mounting a block device (formatted with a file system) into a particular path you are mounting one path into another path.

For example: Let’s say you have a small /var but a very large /opt partition and you need additional space for your growing log files.

First, shut down the services writing to log files, then…

mv /var/log /opt/var_log
mkdir /var/log
mount -o bind /opt/var_log /var/log

You will now see this reflected when running the mount command:

# mount | grep var
/opt/var_log on /var/log type none (rw,bind)

At this point you are ready to restart the previously stopped services.

If you want this to persist across reboots, you’ll just need to update your /etc/fstab with the bind mount as well.

# /etc/fstab
/opt/var_log              /var/log                 none    bind    0 0

And there you have it! Its not beautiful, but it will help you keep the lights on until you can get a long-term fix in place.

I use SAN snapshots extensively to provide point-in-time recoverability for a number of different systems. This is a very reliable and efficient backup solution but it comes at the cost of additional complexity for individual file restoration. This is because the snapshot backups are of the entire volume and not each individual file. So when a user says “hey, could you restore file x from two days ago?” I have some work to do.

The Problem

Recovering files from a SAN snapshot involves a 10 step manual process.

Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the system
4. Perform an iSCSI discovery
5. Perform an iSCSI login to the snapshot volume
6. Mount the snapshot volume
7. Copy the backup file(s) into place
8. Un-mount the volume
9. Perform an iSCSI logout
10. Revoke access to the snapshot from the host

This manual process is tedious and error prone. Luckily many of these steps can be easily automated. Enter Autofs.

The Solution

Autofs is a toolkit which enables you to automatically mount file systems in accordance to a configuration file (called a map) as you traverse a certain directory structure. It is very flexible and supports external scripts, or “program maps” in autofs-speak. As you can see above, the majority of the steps are oriented toward getting the volume to the host and mounted. Only one of the steps actually pertains to the file(s) we need to recover so let’s focus on automating the rest.

This program map, named auto.iscsi, eliminates manual steps 4, 5, 6, and 8. An additional cron job could be added to automate step 9 by checking to see if the devices associated with the hosts current iSCSI sessions are still mounted.

New Workflow

1. Log in to the SAN
2. Present the snapshot to the appropriate system
3. Log in to the snaphost browsing system
4. cd to the automounted directory using the volume IQN
5. Copy the backup file(s) into place
6. Revoke access to the snapshot from the host

So we’ve removed 4 steps from the process with the following automounter script. I’m also working to automate 1, 2 and 6, but more on that later.

auto.iscsi program map

#!/bin/bash
#
# auto.iscsi - Automounter map for mounting iscsi devices by iqn.  
#              Keith Herron <keith@backdrift.org> 
#
# Invoke this program in /etc/auto.master with a line similar to:
# /iscsiauto  program:/etc/auto.iscsi
 
# IQN - This script expects to recieve an iscsi iqn as the only argument.
IQN="$1"
 
# OPTS - Filesystem mount options.
OPTS="-nocheck,ro,fstype=ext3"
 
# PART_NO - Partition number.  This script assumes only one fs per iqn.
PART_NO="1"
 
# Using the iscsi node directory we can determine the ip and port of 
# our iscsi target.  We use this later to figure out the block dev.
  IP=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 1`
PORT=`ls /var/lib/iscsi/nodes/${IQN} | cut -d , -f 2`
 
# If we're already logged in don't bother trying to log in.
if iscsiadm -m session | grep "$IQN"; then
  echo node exists
else
  iscsiadm -m node -T "$IQN" --login 1>&2
fi
 
# Let udev settle before we try to look up the block device.
sleep 1
 
# Figure out the block device using the udev symlink.
DEV_LINK=`readlink /dev/disk/by-path/ip-${IP}:${PORT}-iscsi-${IQN}-lun-0`
DEV_NODE=`basename $DEV_LINK`
DEV_FULL="/dev/$DEV_NODE"
 
# Return the magic to automount
echo "${OPTS} :${DEV_FULL}${PART_NO}"

How to use it

In the below example I am recovering a file called /etc/really_important_file back to a host using scp.

[root@snapbrowse ~]# cd /iscsiauto/iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# ls
bin  boot  dev  etc  home  lib  lost+found  media  misc  mnt  opt  proc  root  sbin  selinux  shells  srv  sys  tmp  usr  var
[root@snapbrowse iqn.2001-05.com.equallogic:0-8a0906-281bb3004-d7500056a1b4bde8-vm-beta-2010-05-03-04:00:01.50612]# scp -pr etc/real_important_file root@host.recovering.to:/etc/

Known Limitations

Next Steps