Home / server / Questions / 628782
Accepted
Pete
Asked: 2014-09-16 23:46:17 +0800 CST

Prevent Rsync and SCP from causing page timeouts on LAMP server

  • 772

I'm trying to run Rsync to move static files from our Dedicated server to our new cloud server.

The following command works but when it runs it is stopping any web pages being served

rsync -azP --delete -e 'ssh -p 1234' 
    /var/www/vhosts/example.com/httpdocs/images/20* 
    user@server:/var/www/vhosts/example.com/httpdocs/images/.

So I've tried using nice and ionice to lower the CPI and I/O priority, and using a different cipher over SSH - I'd found posts suggesting this will lower the server load:

nice -n19 ionice -c2 -n7 rsync -azP --delete --rsh="ssh -c arcfour" -e 'ssh -p 1234' 
    /var/www/vhosts/example.com/httpdocs/images/20* 
    user@server:/var/www/vhosts/example.com/httpdocs/images/.

But this does not make a difference, as soon as the script runs pages on http://example.com stop serving, and once I use CTRL+C to kill the Rsync the command the pages start serving again.

I could put this in a crontab job to run when site traffic is low, but I'd like to be able to sync the servers through the day.

Is there a way of getting Rsync to stop blocking apache or is there another way I can transfer the data?

I've also tried making a tar.gz archive and transferring via scp, but the file is 2.5GB and I just get the same problem.

We will be moving all our static content onto Amazon S3 - but that's a much larger job and I'm hoping I can just get the static content moved across in the meantime to kill off the old machine

linux

2 Answers

  1. Best Answer

    I would not expect that Apache would stop serving during transfer. I can think of four things that could slow down Apache.

    • Bandwidth saturation. Use --bwlimit when starting rsync.
    • CPU (although rsync shouldn't be a problem). Use top to monitor. Use nice or renice to reduce priority.
    • Disk IO. Use ionice with idle priority.
    • Memory. Consider dropping the number of Apache child processes to release some memory. top may show some issues.

    For the initial transfer, I would consider using tar to transfer the content. You can then use rsync to pick up any changes.

    As has already been noted, consider remounting with noatime or relatime as updating the atime can put a lot of load on the disks.

    I would consider using sar to monitor the system to see which resource is the bottleneck. It can monitor all your resources. Capture the data or output to file so you can examine it a leisure.

    • 1

  2. Get faster disks. Rsync do not block apache. Shortage of io operations do.

    • 0