Home / server / Questions / 110350
Accepted
Will
Asked: 2010-02-07 21:04:55 +0800 CST

cron script for scheduled backups

  • 772

i am looking for a cron script which can do some backups. im mostly a web developer so am kind of newbish with bash/cron stuff.

i want cron will copy all the files or directories in a "filelist.txt" into a mounted network storage drive once per day. ideally it will work on all 3 of my boxes, 1 fedora 1 ubuntu and 1 sunos (just with diff filenames in each)

the tricky part is: i only want to save backups of the last 7 days, and delete any older than that

backup linux bash cron

4 Answers

  1. Best Answer

    I'm sure some people can improve greatly on this but it may get you started.

    The commands you need are probably find for scanning and determining the list of files to backup, and then tar, cpio, or zip for archiving depending on your preferences.

    I will modify your spec a bit to scan directories but not individual files.

    Let's suppose /etc/backupdirlist.txt contains a list of directories to scan and /media/backup is the mount point of the backup media (already mounted) and /media/backup/backups is an already existing directory into which you want the backups and filelists placed.

    The seven day requirement is not difficult to meet if you use the day of the week as the destination file name in the backup process and write over the old ones.

    You need something about like this

    (note: untested, may have bugs, apologies if it does something stupid, GNU LGPL License and standard disclaimer applies)

    I suggest going over this one command at a time, and examining the variables with echo $VAR or the files with less to see what it does. Also read the man pages for find and cpio. 

    #!/bin/bash
DIRLIST=/etc/backupdirlist.txt
DAYOFWEEK=`date +%a`
FILELISTDEST=/media/backup/backups/$DAYOFWEEK.filelist
BACKUPDEST=/media/backup/backups/$DAYOFWEEK.backup.tgz
TIMELIMIT=-1  # says save everything modified less than or equal to one day ago
echo >$FILELISTDEST   # to erase last weeks list
for DIR in `cat $DIRLIST` 
 do find $DIR -mtime $TIMELIMIT >>$FILELISTDEST  # >> appends the lists
done
cat $FILELISTDEST | cpio --create --format=ustar | gzip -9 -c >$BACKUPDEST

    I haven't explained how to put this script into cron, I'll leave that to you or another participant. Make sure it works and does what you want as a script, first, before automating it.

    On the different unix boxes there can be variations in manufacturer's versions of standard utilities like find or cpio. The best way around this is to install the GNU versions on all platforms. Linux already has the GNU versions.

    Hope this helps!

    • 2

  2. I would recommend that you consider rsync with its --files-from=/path/to/filelist option.

    In the mean time here are some improvements to ~drpaulbrewer's script:

    #!/bin/bash
dirlist=/etc/backupdirlist.txt
dayofweek=$(date +%a)
backupdest=/media/backup/backups/${dayofweek}.backup.tgz
while read -r dir 
do
    find "$dir" -print0
done < "$dirlist" |
    cpio --null --create --format=ustar | gzip -9 -c > "$backupdest"

    This eliminates the need for a temporary file and is able to handle filenames with spaces.

    I've changed the variable names to lowercase. Even though these don't conflict with ones that Bash uses itself, I prefer not to use all-caps variable names out of habit to avoid name collisions. You can use any of various mixed-case formats if you prefer. You should always quote variable names that contain file or directory names.

    I've eliminated the time limit since if something isn't changed it never gets backed up at all or if it's changed then backed up then idle for seven days, the backup gets overwritten.

    Also, remember the first rule of backups: It's not backed up if you haven't successfully tested a restore.

    Add a line in your crontab (using crontab -e) similar to this:

    0  0  *  *  *  /path/to/backupscript > /dev/null 2>&1
    • 2

  3. I found this answer useful but don't have enough reputation to upvote. I used a combination of the other answers posted by Dennis Williamson and user33903 (https://serverfault.com/a/110367/149956 and https://serverfault.com/a/110356/149956) and came up with the script below to use on one of my servers:

    Note: I didn't include the backup user group details and my date string in the backupdestfile variable is "+%Y-%m-%d".

    #!/bin/bash
filelist=/etc/backup/filelist.txt
dirlist=/etc/backup/dirlist.txt
backupdestdir=/backup
backupdestfile=$backupdestdir/backup-$(date "+%Y-%m-%d").tar.gz
tmpfile=$(mktemp)
packagedump=/etc/backup/installed-packages.txt
servicedump=/etc/backup/service-config.txt
backupusergroup=not.telling

# generate list of installed packages
dpkg-query -W > $packagedump

# dump current service configuration
sysv-rc-conf --list > $servicedump

# generate list of files to backup, kept in $tmpfile
echo $tmpfile >> $tmpfile
echo $packagedump >> $tmpfile
echo $servicedump >> $tmpfile
cat $filelist >> $tmpfile
cat $dirlist | while read dir
do
    find "$dir" >> $tmpfile
done

# generate the backup archive
cat $tmpfile | cpio --create --format=ustar | gzip -9 -c > $backupdestfile

# cleanup
rm $tmpfile
chown $backupusergroup -R $backupdestdir
chmod og-rwx -R $backupdestdir
    • 1

  4. It's very good to save backups on Amazon S3.

    For backups I recommend use duplicity and DT-S3-Backup bash script.

    DT-S3-Backup was designed to automate and simplify the remote backup process using duplicity and Amazon S3. After the script is configured, you can easily backup, restore, verify and clean without having to remember lots of different command options.

    • 0