Home / server / Questions / 762524
Accepted
Gregory MOUSSAT
Asked: 2016-03-09 16:37:32 +0800 CST

rsync hangs on one big file

  • 772

Each night I copy several virtual disks with rsync from one Linux Debian computer to another Linux Debian.
Most files are raw images with "holes" into them: some parts were never written to, so remain unallocated on disk.

rsync hangs on one file, always the same. The hang occurs after 50 Gb transfered, and each time. I'm not sure if this is always at the exact same point, but ls -sh displays 50 Gb.
This is a 800 Gb file containing 151 Gb (so 649 Gb are unallocated). Some other virtual disks have similar figures, and rsync works well on them.

I have the exact same behaviour if I use rsync to update the file locally, without any network involvement (with --no-whole-file, this is a requirement, see later).

Once rsync is stalled, it uses one CPU core to 100% and zero disk activity on the receiving side (this is a pull request, so rsync is ran from this side) and zero CPU and zero disk on the sending side.
I let it ran during several hours.
Ctrl+c immediately stops rsync.
When ran to copy locally, once stalled I also have one CPU core to 100% and zero disk activity.

The only exception I found is when I rsync this file to a new location (ie the destination file does not exist). So I suspect the problem is related to the comparison between old and new data.

I use --inplace to limit writes to the destination disk, because snapshots are then used after each backup. So this option is a requirement, and rsync too, except if I found a tool able to update only changed parts of these files.

linux

2 Answers

  1. Best Answer

    rsync is known to have this kind of problem on huge files because of an inefficient algorithm for the internal hashes buffer.

    You have to use the --block-size option with a big value. But current versions doesn't allow to use more than 128 kB. Some webpages tell to use --block-size=10485760 --protocol=29 but in my case it is refused by rsync.

    • use version 29 or older
    • or recompile after modification of the MAX_BLOCK_SIZE constant
    • 7

  2. Take a look at the bug reports against rsync. Maybe file a new one. You can find those from https://rsync.samba.org/bugzilla.html

    Whether here or in a bug report, you should provide more specifics. What version of rsync is on each end, what's your OS, and exactly what command are you using? Are the files and/or rsync transfer compressed? With what zlib version?

    https://bugzilla.samba.org/show_bug.cgi?id=10518 and maybe https://bugzilla.samba.org/show_bug.cgi?id=10372 might be relevant. Also see http://www.anchor.com.au/blog/2013/08/out-tridging-tridge/

    • 5