Home / server / Questions / 40927
Accepted
pQd
Asked: 2009-07-16 00:44:13 +0800 CST

rsync of >4GB files

  • 772

silly one,

do you have any problems with rsync'ing large [ >4GB ] files under modern linux? [ 32bit, 64bit, large file support turned on ]? i've done some tests on my own between 2 64bit boxes and didn't have any problems transferring 6-10GB files. to make test thorough i altered files, run rsync again, checked md5... - all seems ok.

but after i saw this bug report i got a bit worried. i did some searching but have not found any confirmation of the problem.

thanks for your thoughts!

edit: file system: ext3, reiserfs

backup files rsync limitations

5 Answers

  1. Depends on the filesystem that you're using. I've run into trouble with FAT32 filesystems. I had a 200GB portable harddrive (formatted as FAT32) and was trying to copy a DVD .iso onto it. It didn't work cause you can't have files greater than 4.somethingGB in FAT32

    • 4

  2. Nope, I throw around 5-10GB VM images using rsync all the time, never seen a problem.

    • 3

  3. The largest file I have recently rsynced was 180GB which was in a a set of directories containing 15TB ( I have written a set of scripts which can do the sync in parallel, I can move the data at about 3TB an hour ... )

    • 3

  4. No, we synchronize two 30TB data sets (made of files ranging from 4 to 20GB) daily for months with rsync, no problem.

    • 2
  5. Best Answer

    The error report that you've linked to does not seem to be a 4GB+ filesize related error. 429796854 bytes is just shy of 410Mb, and it seems to be a transport error rather than an rsync one. If

    I would suspect that the transport connection (presumably SSH) has dropped, perhaps due to an inactivity timeout because the CPU spent so long between sends because it had to do something like checksum a very large file, and this is the reason that rsync reports a broken pipe.

    I'm sure I've used rsync successfully on files over 4Gb in the past with 32bit clients and servers, and at least once where more then 4Gb was actually transferred rather than just considered for transfer.

    • 2