Home / server / Questions / 146377
Accepted
mark
Asked: 2010-05-29 11:38:30 +0800 CST

Why is my rsync so slow compared to pure cp or even scp?

  • 772

I'm transfering the files from Linux to Windows 7 via a mounted share (the share is mounted from Windows on Linux).. I'm copying lots of data (i.e. nearly a TB) from the old to the new machine within my LAN. I'm unfortunate enough already that I only have 100MBit. Naturally I blindly used rsync but already wondered after a day why it feels so slow. Enabling the progress meter showed my a transfer rate of about 2MB/s .

So I took a reasonable big file (800MB) and tracked the transfer timing (1):

cp : 05:33
scp (2): 06:33
rsync : 21:51

1) I deleted the files between each run
2) scp via localhost to the same Linux machine directly onto the share; completely useless but provided a progress meter

The tests were as simple as

(cp|scp|rsync) <source> <destination>

No special arguments except host/port for scp. I even tried the -W switch for rsync but cancelled after ten minutes. rsync is 3.0.3 running on Lenny. To be able to interrupt the copy process anytime and resume lead me to rsync, but now I think I seriously need to reconsider this requirement.

How's such a big difference possible?

Update/Solved:

Thanks to rschuler I was able to solve the matter: use the rsync daemon instead of smb mounting for efficiency reasons. The said DeltaCopy works, but one has to watch out for a few things

  • it's a nice GUI wrapper but if something is going wrong, it's good to know how to fix it. Seems I entered my user credentials, under which to run the rsync service, wrong the first time but the GUI didn't allow me to set a new one. I figured out it was running as a service and was able to set the proper credentials there
  • Needed to manually add the port to the firewall to allow connections
  • Personal taste: ensure the shares are either password protected or if without, maybe ensure the service does not automatically start with windows; just in case
  • The wrapped rsync binary is not a native Windows port but builds on cygwin. However, the contained cygwin DLL does not handle UTF8 properly and mangled non-ascii characters. Get a fixed DLL from http://www.okisoft.co.jp/esc/utf8-cygwin/ .

After that, the transfer rated jumped up from 2MB/s to ~8MB/s ; absolutely fantastic!

performance rsync

1 Answers

  1. Best Answer

    (the share is mounted from Windows on Linux)

    That is your problem. rsync is doing rolling checksums over the destination. The windows share. You are pulling all the data over the network to compute the checksums. (Possibly more than once).

    What you need to do is have rsync running on both machines. That way only the differences (and checksums) will be transferred over the network cable. DeltaCopy is a windowized rsync. It has good enough docs to get you going.

    See the up voted answer to this question for a better explanation why I think you might be misusing rsync.

    • 19