Xen Backup – Simple Backups With LVM and Rsnapshot

Tags: , , , , , , , , , ,


Effectively backing up your virtual machines is a problem with a multitude of potential solutions. Many xen backup solutions are centered around making a copy of the full volume(s) upon which your virtual machine(s) reside. But what happens if you want to recover just a single file and not the entire VM? And is it possible to delegate restoration to end users? Not to mention the space that is wasted by storing many copies of the same data.

Here I will describe how to implement a xen backup scheme that I have found very robust, efficient and reliable. It enables automated backups of Xen virtual machines that each reside on individual LVM volumes using a utility called rsnapshot (which is based on rsync).

Note: This method additionally can be utilized to back LVM volumes that aren’t housing Xen virtual machines.

Goals


  • Open Source
  • Simple
  • Reliable
  • Secure
  • Flexible
  • Maintainable
  • Efficient – Runs reasonably quickly and doesn’t store multiple copies of the same file
  • Not Dependent on software in the VM
  • Able to delegate restore operations to end-users
  • LVM Snapshots


    LVM provides a convenient snapshot feature. This allows you to create an identical clone of a logical volume and store only the blocks that differ from it. It is an unintuitive action and the result is a mirror image of your virtual machine’s disk that you can mount, read and even write data to.

    Rsnapshot backups


    Rsnapshot is a very flexible backup application which provides robust automation facilities. It supports LVM, ssh, rsync as well as custom scripts and will manage the process of rotating your backup files according to a defined schedule and retention policy. For our purposes we will be using the built-in LVM snapshot support.

    Xen Backup – How it works


    Each of my Xen virtual machines has own LVM volume per file system used by the system. For many systems this results in a single and somewhat large root file system housed on LVM. When rsnapshot is configured to back up one of these volumes it runs through the following steps:

  • Creates a temporary snapshot of the LVM volume
  • Mounts the created snapshot in a temporary directory
  • Rotates the backup directory, making room for this backup
  • Rsyncs the contents of the LVM snapshot into the backup location. Hardlinks are used here if the file has been unchanged since the last backup.
  • unmounts the temporary LVM snapshot
  • Removes the temporary LVM snapshot
  • Note: The use of LVM snapshots does introduce a write performance implication during the period that the snapshot exists. I have found this to be an acceptable tradeoff as most backup suites introduce some sort of performance effect while they are processing.

    How to set it up


    Assuming you already have a working LVM configuration (read more about LVM here), you’ll just need to install rsnapshot. It is available through dag for RedHat based platforms and available in the standard repositories for ubuntu and debian.

    Once you have installed rsnapshot, defined your retention periods, and installed the corresponding cron jobs to call the backup process, you’re ready to add the LVM specifics to the rsnapshot configuration.

    For example, I have a logical volume named “vm_insomniac-root” in a volume group named “vg0”. This represents the root filesystem of a virtual machine named “insomniac”.

    [root@localhost .snapshot]# lvs | grep insomniac-root
       vm_insomniac-root vg0  -wi-ao  36.00G

    To tell rsnapshot that I want to backup this LVM volume I add the following to /etc/rsnapshot.conf. Make sure this is tab delimited. Rsnapshot is unreasonably insistent upon tabs in its config file.

    #/etc/rsnapshot.conf
    linux_lvm_cmd_lvcreate	/sbin/lvcreate
    linux_lvm_cmd_lvremove	/sbin/lvremove
    linux_lvm_snapshotsize	5G
    linux_lvm_snapshotname	rsnapshot-tmp
    linux_lvm_vgpath	/dev
    linux_lvm_mountpath	/mnt/rsnapshot-tmp
    linux_lvm_cmd_mount	/bin/mount
    linux_lvm_cmd_umount	/bin/umount
     
    backup  lvm://vg0/vm_insomniac-root/    insomniac/

    My snapshot_root is set as “/.snapshot”, and as such the backups look like this:

    [root@localhost .snapshot]# cd /.snapshot
    [root@localhost .snapshot]# ls
    daily.0/  daily.1/  daily.2/  daily.3/  daily.4/  daily.5/  daily.6/  lost+found/
    [root@localhost .snapshot]# ls daily.0/ | grep insomniac
    insomniac/
    [root@localhost .snapshot]# ls daily.0/insomniac/
    bin/  boot/  etc/  home/  lib/  lost+found/  media/  mnt/  opt/  root/  sbin/  selinux/  srv/  tmp/  usr/  var/

    As you can see, I’m only set up to keep a weeks worth of daily backups. If you have the disk space you could opt for a much longer retention period.

    Delegation to end-users


    To provide users with a simple interface to access their backups I share each of their rsnapshot directories as read-only NFS exports. To only their server of course. This allows users to traverse their backups.

    One significant benefit is that there is no backup client for them to install and learn how to use. Restoration is performed with standard unix commands like cp and rsync, saving much time and frustration.

    Potential Improvements and follow-up


    Time permitting, I would like to test the performance, reliability and storage saving of using a dedup file systems such as lessfs and SDFS. I think these have the potential to make this solution even more robust, especially from a cost perspective.

    10 Responses to “Xen Backup – Simple Backups With LVM and Rsnapshot”

    1. Harrie Kohler Says:

      Hi, installed all without any problem, but when testing rsnapshot I receive an error “Cannot handle \” and “linux_lvm_cmd_lvcreate not defined in /etc/rsnapshot.conf”.
      Any idea?

      I run XenServer 5.5 on DELL PE R805, XenCenter 5.6
      I noticed that the lvs returns a crazy listing:
      “VHD-566647e8-313c-4c0b-bfe4-42b4e826db94 VG_XenStorage-504bbe39-d0d5-9b6e-fe03-806fc005a593 -wi— 100.20G”
      These names are created by Xen and when renaming then to more convient names, the connection to the volume is lost.

      [Reply]

    2. Harrie Kohler Says:

      Hi,
      I backtraced the problem to the commands necessary for lvm manupilation. These weren’t available in the default config file of rsnapshot. I searched the snapshot.org site and found these settings in the manual pages.
      I added following lines:
      linux_lvm_cmd_lvcreate
      linux_lvm_cmd_lvremove
      linux_lvm_cmd_mount
      linux_lvm_cmd_umount
      linux_lvm_snapshotsize
      linux_lvm_snapshotname
      linux_lvm_vgpath
      linux_lvm_mountpath
      The configtest is now OK.

      [Reply]

    3. Harrie Kohler Says:

      Hi,
      Testing the rsnapshot.config is OK. Also testing with “rsnapshot -t daily” returns no errors.
      Running “rsnapshot daily” fails with the message “Invalid path for Logical Volume.
      (see the post from 4:50am for the lvs listing on my XenServer)
      The origin name should include the volume group”
      I don’t where to find this, because Xen uses “virtual paths”.
      Do you have any idea how to resolve this issue?
      Thx.

      [Reply]

      keith Reply:

      Hi Harrie, You should be able to use the string starting with VHD- as your logical volume and the string beginning with VG_ as your volume group within the rsnapshot config. Something along the lines of this within your rsnapshot.conf, I think.

      backup lvm://VG_XenStorage-504bbe39-d0d5-9b6e-fe03-806fc005a593/VHD-566647e8-313c-4c0b-bfe4-42b4e826db94/ destination-folder/

      Thanks!
      –Keith

      [Reply]

    4. Robert Says:

      What would you do about backing up a windows virtual? At the moment we make an image of the snapshot, and scp that across to a remote machine for backup but its very inefficient and need a better way.

      [Reply]

    5. Web Hosting Says:

      Hi Keith,

      Very nice tool this rsnapshot indeed and I will give it a try… Now what I was wondering is if rsnapshot also allows you to fully restore your whole VM let’s say after you have lost the VM. And if yes how do you proceed?

      Thanks!

      [Reply]

    6. Debian 6: Backup Xen with LVM and Rsnapshot | Linux Sysadmin Blog Says:

      […] This is a draft of my installation process which is mainly based from this blog:  Efficient Xen Backups Using LVM and Rsnapshot. […]

    7. Felix Says:

      Hi there,
      thanks for the explanation and manual. Just to let you know, the above link “define your detention periods” http://rsnapshot.org/howto/1.2/rsnapshot-HOWTO.en.html#modifying_the_config_file is broken. Could you fix this or someone post the renewed link? It seems that rsnapshot has reorganised its documentation

      [Reply]

    8. Felix Says:

      Hi there!
      I used the instructions above to backup a virtual machine on a lvm drive. This works perfectly. Now I want to test the restore or the whole drive from the backup. Can anyone give me a hint on how to do that? Thanks!
      Kind regards,
      Felix

      [Reply]

      Ilya Reply:

      Try to use xen-tools to install new image from previously backuped tar archive for example. I think it must working out.
      http://manpages.ubuntu.com/manpages/precise/man8/xen-create-image.8.html
      set installation method to tar. Also intrested in best practices of restoring LVM snapshots in Xen =)

      [Reply]

    Join the Conversation