ec2-consistent-snapshot With Mongo

I setup MongoDB on my Amazon EC2 instance knowing full well that it would have to be backed up at some point. I also knew that by using XFS, I could take advantage of filesystem freezing in a similar fashion to LVM snapshots. I had remembered reading about backups on XFS with MySQL being done with ec2-consistent-snapshot. As with any piece of open source software, it just took a little tweaking to make it do what I wanted it to do.

Out of the box, ec2-consistent-snapshot works great for freezing an XFS filesystem with MySQL because it not only stops the server, but handles potential replication issues. By following the steps outlined here by 10gen, I just made a few slight adjustments to the core ec2-consistent snapshot script to allow for MongoDB support. In fact, it supports locking and fsyncing immediately prior to freezing and backup. I have been using this script in production for a while now and it seems to work without issue for me.

In the usual spirit of social coding, I have added the script to Github: https://github.com/elubow/ec2-consistent-snapshot.

Running it is just this:

ec2-consistent-snapshot                                    \
--mongo                                                    \
--xfs-filesystem /data                                     \
--region us-east-1                                         \
--description "RAID snapshot $(date +'%Y-%m-%d %H:%M:%S')" \
vol-VOL1 vol-VOL2 vol-VOL3 vol-VOL4 vol-VOL5 vol-VOL6 vol-VOL7 vol-VOL8

The options used here (for reference) are telling ec2-consistent-snapshot to use –mongo, on the –xfs-filesystem /data, in the us-east-1 –region (note that it’s just the region and not the availability zone within that region), to be backed up with the listed –description of the specified volumes. You can even throw a –mongo-stop in there to have Mongo stopped before the file system freeze and then restarted after the volumes have been backed up. Don’t forget that you need to set your Amazon keys in you environment variables (AMAZON_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for your key and secret respectively).

I attempted to keep the usage style consistent with Eric Hammond’s original version, just add Mongo support for it.

Note: I also mentioned this on the mailing list. But given the amount of messages that fly around on the list daily, some folks may have missed it.

References:

  • Matt

    When installing this version ec2-consistent-snapshot, is it reasonable to assume that doing an install of the original version using the package and then replacing the old version of the PERL script with this one will work as expected? Are there any other packages required that are not included in the original?

    Thanks!

  • http://eric.lubow.org Eric Lubow

    That’s correct. But there are some additional Perl modules that you’ll need to install to get it to work if you are using CentOS as opposed to Ubuntu.

  • David P. Nesbitt

    Here is a blog post I wrote on getting ec2-consistent-snapshot up and working for MySQL on CentOS:

    http://pojo-mojo.blogspot.com/2011/03/ec2-consistent-snapshot-for-centos.html

    I simply needed to do the following additional steps for MongoDB support:

    cpan> install MongoDB

    cpan> install MongoDB::Admin

    This seemed to work for me.  (see my comment above)

  • David P. Nesbitt

    What is the right way to restore one of the snapshots?  I am not very experienced with RAID10 on EBS, so I am not quite sure what original steps I need to repeat.

  • http://twitter.com/imcocoy rodney quillo
  • http://www.robbenwellwiki.com Rob Benwell

    You can create an image of a Windows instance via the AWS Management
    Console but I don’t know how to do it on Windows system. Maybe a
    scheduled task on Windows can make a daily snapshot.