Home / server / Questions / 208494
In Process
Andrew Fashion
Asked: 2010-12-03 11:14:35 +0800 CST

rsync is just hanging, building file list

  • 772
[root@centos /]# rsync -av --exclude thumbs /storage [email protected]:/storage
[email protected]'s password: 
building file list ...

I've been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

Is it normal for rsync to be sitting this long calculating the files/directories?

rsync

5 Answers

  1. rsync 2.x does build a full file list up front.

    Add the -P option and you'll see a progress indicator.

    If you think it really is hanging, in another terminal find the pid of rsync and then

    1. Run strace -p PID and see what it's doing. (Hit ^C to stop.)

    2. Run ps -o wchan PID to see where it is in the kernel.

    • 27

  2. You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

    Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

    • 16

  3. You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

    One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

    • 3

  4. Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

    • 1

  5. The hang after the message "building file list..." can be caused by MTU mismatch, e.g. if you've set up the network interfaces with MTU 9000 but not the switch in between, small packets (like connecting to the rsync server) will pass through but not bigger ones (like sending the file list).

    • 1