Home / server / Questions / 263009
Accepted
Stan
Asked: 2011-04-25 17:18:38 +0800 CST

rsync to import/export delta-transfer or incremental copy result

  • 772

My scenario is to distribute several GB files from a staging server to tens of identical destination servers. Since rsync can do delta-transfer or incremental copy, I was wondering when rsync file to 1st dest server, can rsync export the delta-transfer result? And use import that result for rsync'ing to the rest of dest servers. In this case, staging server can save CPU time on re-calculating delta-transfer. Is this possible?

Number of files: 160
Number of files transferred: 49
Total file size: 2993222827 bytes = 2854 MB
Total transferred file size: 1847285024 bytes =1761 MB
Literal data: 69543644 bytes = 66 MB
Matched data: 1777741380 bytes = 1695 MB
File list size: 3088
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1256178
Total bytes received: 70240527 = 67 MB

sent 1256178 bytes  received 70240527 bytes  182157.21 bytes/sec =
0.17 MB/s total size is 2993222827 speedup is 41.87
file-transfer rsync

1 Answers

  1. Best Answer

    Yes. This is what rsync's batch mode is for. From the man page:

    Batch mode can be used to apply the same set of updates to many identical systems. Suppose one has a tree which is replicated on a number of hosts. Now suppose some changes have been made to this source tree and those changes need to be propagated to the other hosts. In order to do this using batch mode, rsync is run with the write-batch option to apply the changes made to the source tree to one of the destination trees. The write-batch option causes the rsync client to store in a "batch file" all the information needed to repeat this operation against other, identical destination trees.

    Generating the batch file once saves having to perform the file status, checksum, and data block generation more than once when updating multiple destination trees. Multicast transport protocols can be used to transfer the batch update files in parallel to many hosts at once, instead of sending the same data to every host individually.

    To apply the recorded changes to another destination tree, run rsync with the read-batch option, specifying the name of the same batch file, and the destination tree. Rsync updates the destination tree using the information stored in the batch file.

    An example:

    $ rsync --write-batch=foo -a host:/source/dir/ /adest/dir/
$ scp foo* remote:
$ ssh remote ./foo.sh /bdest/dir/
    • 7