Home / server / Questions / 73971
Accepted
DennyHalim.com
Asked: 2009-10-14 00:29:36 +0800 CST

fastest backup for large files

  • 772

what is the fastest way to duplicate a folder contain few files with few hundred megs size and other few megs small files? (say, /var/lib/mysql) cp or tar or rsync or...??

backup

7 Answers

  1. Best Answer

    rsync in general will be faster than cp or tar, because rsync only transfers those files which have changed, and with --partial it will only transfer the parts of a file that changed.

    Having said that, rsync works much better if you know what you're backupping and can arrange things so that rsync doesn't have to do as much work. For example, rotated log files in /var/log work much better if you rotate them to a filename with a date in it, instead of .0, .1, .2, etc..

    One more note: In your question you mention /var/lib/mysql. Using rsync to backup that is a pretty bad idea, use mysqldump to get a reliable backup. If that backup is too large to transfer frequently, use mysql replication and backup from a slave. (You still should do a full backup on the master periodically though, replication can fail too).

    • 4

  2. Depends on a few factors. For a large number of small files it is generally better to do tar. If it is a small number of large files, cp is ok in most instances.

    If you have a list of files but only a small portion of them change, rsync is more efficient.

    • 1

  3. It depends on what you are backing up. You can't just backup live database with rsync, you'll end you with a corrupted file. For databases, like mysql, you need to set cron, for example, to run 'mysqldump' and then rsync that file. Also, make sure you append 'date' with the file name so, you know when this file was backed up and have several backup copies just in case.

    • 1

  4. Ah, the magic word backup

    I can't say much about relative speeds of the methods you mention, but I think rsync's ability to backup only files which have changed, makes it the best method.

    Especially if most of the files don't need to be copied.

    • 0

  5. If you have quite a few small files, running a tar over them will increase the transfer speed considerably.

    If the files are mostly text (which isn't the case of mysql binaries or databases), consider zipping the tar (bzip2 and the like). Otherwise you won't get anything from the compression so just leave the tar alone.

    • 0

  6. Assuming that you already have an older copy of the files and you only need to refresh the copy, then rsync is much faster than cp or tar because it only transfers the bytes that have changed. A 100MB file with a few changed records will update in a few seconds. I have found it to be perfect for database files but I have had less success with binary files.

    But if the purpose of the copy is for off-site backup, nothing comes near to rdiff-backup. rdiff-backup, like rsync, only transfers changes within files, but it also maintains the incremental changes so that you can restore any file from any earlier backup date.

    And, as for the SQL databases, rdiff-backup checks for changes made during the backup and fixes them automatically, so IMHO it should be safe. Unless, of course the SQL server holds data in memory without writing it to disk - then it will not be safe.

    • 0

  7. SAN Snapshot is the fastest way, but probably not the answer you were looking for.

    • 0