Home / server / Questions / 462918
Accepted
Temnovit
Asked: 2013-01-05 02:02:09 +0800 CST

Strategy for backing up a very large website (~40GB)

  • 772

I have a website which works with lots and lots of text files - now they take up about 40 GB of data and increasing over time. I need to make a full daily backup. My current strategy is to make a password protected archive and store it in the dropbox with this command:

tar cfz - /var/www/mysite | openssl enc -aes-256-cbc -e -k "b@ckupPassword" > /home/user/Dropbox/server_backups/sources/2013_01_04_0500_mysite_source_encrypted.tgz

It works, but making an archive takes about 14 hours and consumes a lot of IO - and it will be worse as the amount of data increases.

What is the proper strategy of backing up such large amount of files?

backup

3 Answers

  1. Best Answer

    I would use rsync, provided I have space enough.

    This example has a full backup and keeps a week of incremental ones.

    #!/bin/bash

DAY=$(date +%A)

if [ -e /bkp/incr/$DAY ] ; then
  rm -fr /bkp/incr/$DAY
fi

ionice -c 3 rsync -a --delete --inplace --backup --backup-dir=/bkp/incr/$DAY /var/www/mysite/ /bkp/full/ &> /dev/null

    The script can be complicated to store offsite, on dropbox folders and so on.

    If you really need to use tar, you can keep track of the modified files

    tar zcvf - --listed-incremental=/var/log/mysite.tarlog /var/www/mysite | openssl enc  ...

    If you want a full backup, delete /var/log/mysite.tarlog

    • 3

  2. As suggested above rsync seems to be the best way to backup the whole site. Yet i would suggest you to implement some sort of replicated filesystem, something like a simple GlusterFS volume with replication.

    Replication is not a backup, but it can help you to reduce the I/O impact of backups and eventually provide you with a solid base to expand your website later into a cluster.

    • 1

  3. It is better to use an incremental backup mechanism in this case.

    Using rsync you can take incremental backup.

    Click on me for more details about using rsync

    Click on me to read more about incremental backup

    • 0