Automated snapshots of EBS-backed EC2 instance running Ubuntu

Based on Jonik's concept, I created a python script using boto. You provide it a list of volumes to snapshot, and how many trailing snapshots to keep for each volume:

# Define the snapshots manage. We'll snapshot the specified volume ID, and only keep the X newest ones.
snapshots = [("vol-XXXXXXXX", 30), ("vol-YYYYYYYY", 180)]

import boto.ec2
auth = {"aws_access_key_id": "YOURACCESSKEY", "aws_secret_access_key": "YOURSECRETKEY"}
ec2 = boto.ec2.connect_to_region("YOURREGIONNAME", **auth)
description = "automated backup"
for volume, num_trailing in snapshots:
  snaps = ec2.get_all_snapshots(filters={"volume-id": volume, "description": description})
  print "%s: Creating new snapshot. %s automated snapshots currently exist." % (volume, len(snaps))
  ec2.create_snapshot(volume, description)
  purgeable = sorted(snaps, key=lambda x: x.start_time)[:-num_trailing]
  print "Deleting snapshots for %s > %s: %s" % (volume, num_trailing, purgeable)
  for snap in purgeable:
    ec2.delete_snapshot(snap.id)

I set this up as Jenkins job (via the Python plugin), configured to run daily. If you are using IAM to manage credentials, note that this will require in ec2 policies: DescribeRegions, DescribeVolumes, CreateSnapshot, DeleteSnapshot, DescribeSnapshots, CreateTags (because of boto's implementation).

Okay, for what it's worth, here's what I did. I hope my feeble scripts encourage people to post better solutions!

I wrote two simple bash scripts and automated them using cron. (For now I run these on a local server, as I think (?) it's not recommended to put AWS's certificates in the instances/AMIs/EBSs themselves.)

To create a new snapshot:

# ESB volume associated with the instance we want to back up:
EBS_VOL_ID=vol-xxxxyyyy

ec2-create-snapshot --region eu-west-1 -K pk.pem -C cert.pem -d "Automated backup" $EBS_VOL_ID 

To prune all except latest snapshot:

EBS_VOL_ID=vol-xxxxyyyy

ec2-describe-snapshots --region eu-west-1 -K pk.pem -C cert.pem  | grep "Automated backup" | grep "$EBS_VOL_ID" | awk '{ print $5 "\t" $2 }' | sort > .snapshots

latest_id=$(tail -n1 .snapshots | awk '{ print $2 }')

cat .snapshots | awk '{ print $2 }' > .snapshot_ids
for i in $(cat .snapshot_ids) 
do
    if [ "$i" != "$latest_id" ]
    then
        echo "Deleting snapshot $i"
        ec2-delete-snapshot --region eu-west-1 -K pk.pem -C cert.pem $i
    fi
done

(This parses appropriate snapshot information from ec2-describe-snapshots output and creates a temp file with [timestamp tab snapshot-id] entries (e.g. 2011-06-01T10:24:36+0000 snap-60507609) where the newest snapshot is on the last line.)

Notes:

• Put your X509 certificate and private key in some place where the scripts can find them.
• You must explicitly specify --region with all commands. Otherwise e.g. ec2-create-snapshot would fail with volume ID being unknown. (YMMV if you use the default region "us-east-1".)
• I used a snapshot description ("Automated backup") as a marker to avoid the prune script deleting other snapshots of the volume in question (e.g. snapshots related to AMIs).

Disclaimer: This became partly an exercise in Bash/Unix programming for me, especially the prune script. I readily admit you'd most likely get a much clearer result with e.g. Python, when you need logic like "do something for all but the last item in a list". And even with Bash you could probably do this more elegantly (for instance, you don't really need temp files). So, please feel free to post other solutions!

If you're open to external utilities, check out Skeddly.

Disclosure: I'm the CEO of Eleven41 Software, the company behind Skeddly.

I expanded on the idea of Jonik's script to allow multiple snapshots to be retained. The code is too long to fit in a comment so I'm adding a new answer. This code assumes all the right environment variables have been set up for the CLI tools. Also, this defaults to taking a snapshot of the current instance.

# Look up our instance ID using the magic URL
INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

# The number of previous backups we want to keep
N=3

# get list of locally attached volumes via EC2 API:
VOLUME_LIST=$(ec2-describe-volumes | grep ${INSTANCE_ID} | awk '{ print $2 }')
DATE=$(date '+%Y-%m-%d-%H%M%S')

sync

# actually creating the snapshots
for VOLUME in $(echo $VOLUME_LIST); do
    echo "Processing volume $VOLUME"
    SNAPSHOT_LIST=$(ec2-describe-snapshots | grep completed | grep "Automatic snapshot" | grep $VOLUME | awk '{print $5 "\t" $2}' | sort | head "--lines=-$N" | awk '{print $2}')
    ec2-create-snapshot $VOLUME -d "Automatic snapshot on $DATE"
    for SNAPSHOT in $(echo $SNAPSHOT_LIST); do
        ec2-delete-snapshot $SNAPSHOT
    done
done

I have written a script in PHP that will automate EBS Snapshots and delete old ones. It will even email the results of the snapshots to you. You must configure the AWS PHP SDK and PHPMailer for email functionality, but both of these steps are pretty easy. Then you just run the script every night with CRON or Windows Scheduled Tasks. Detailed instructions and code can be found on my blog:

http://www.caleblloyd.com/software/automatically-take-ebs-snapshots-and-delete-old-ones-with-php-script/