Home / server / Questions / 590230
Accepted
dyasny
Asked: 2014-04-21 10:02:06 +0800 CST

Maximizing rsync performance and throughput - directly-connected gigabit servers

  • 772

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the secondary every night using rsync over ssh. Monitoring the traffic, I see throughput of ~2MBps, which is much less than I'd expect from a gigabit port. I've set the MTU to 9000 on both sides, but that didn't seem to change anything.

Is there a recommended set of settings and optimizations that would take me to the maximum available throughput? Moreover, since I am using rsync over ssh (or potentially just NFS) to copy millions of files (~6Tb of small files - a huge Zimbra mailstore), the optimizations I am looking for might need to be more specific for my particular use case.

I am using ext4 on both sides, if that matters

Thanks

EDIT: I've used the following rsync options with pretty much similar results:

rsync -rtvu --delete source_folder/ destination_folder/

rsync -avHK --delete --backup --backup-dir=$BACKUPDIR source_folder/ destination_folder/

Currently, I'm looking at the same level of bad performance when using cp to an NFS export, over the same direct cable link.

EDIT2: after finishing the sync, I could run iperf and found performance was around 990Mbits/sec, the slowness was due to the actual dataset in use.

linux

3 Answers

  1. Best Answer

    The file count and SSH encryption overhead are likely the biggest barriers. You're not going to see wire-speed on a transfer like this.

    Options to improve include:

    • Using rsync+SSH with a less costly encryption algorithm (e.g. -e "ssh -c arcfour")
    • Eliminating encryption entirely over the SSH transport with something like HPN-SSH.
    • Block-based transfers. Snapshots, dd, ZFS snapshot send/receive, etc.
    • If this is a one-time or infrequent transfer, using tar, netcat (nc), mbuffer or some combination.
    • Check your CentOS tuned-adm settings.
    • Removing the atime from your filesystem mounts. Examining other filesystem mount options.
    • NIC send/receive buffers.
    • Tuning your rsync command. Would -W, the whole-files option make sense here? Is compression enabled?
    • Optimize your storage subsystem for the type of transfers (SSDs, spindle-count, RAID controller cache.)
    • 26

  2. As you probably know copying a lot of little files (eg mailboxes using MailDir format or similar) is definitely not the best option to take advantage of high bandwith interfaces. SSH is probably not the best transport protocol for that either. I would try using tar to create a tarball on the source host prior to send it to you secondary host. 

    tar c /var/mail | ssh root@secondary-host 'tar x -C /var/backups'

    If you need incremental backup you may want to try the -g options of tar. If you still need to maximize throuput, try using netcat instead of ssh.

    • 4

  3. Try teasing apart the contributing factors:

    • CPU (e.g. dd of /dev/zero piped through loopback)
    • disk I/O (e.g. dd of a large file piped to cat > /dev/null [piped to prevent short-circuiting])
    • physical network I/O (e.g. dd piped to the other machine)
    • etc.

    and testing them independently.

    I've had some bad experiences with Broadcom drivers, so my first suggestion is to test the usable network bandwidth with: dd if=/dev/zero bs=1m count=10k | rsh backup_host cat \> /dev/null

    • 1