Home / server / Questions / 936928
Accepted
Bert
Asked: 2018-10-24 23:45:12 +0800 CST

Advices for 1.5m files syncing with rsync

  • 772

I have a folder with about 1.5 million files and I need them to be copied to a secondary server as often as possible. What would you recommend because clearly standard rsync commands do not make it because just sending the incremental file list (while using rsync -Pcr options) takes about 30 minutes... and the file transfer about 10 seconds. How would you solve this issue fellow SysAdmins?

I currently use the following command with a 30 minute interval:

rsync -Pcr /var/primary/storage /var/secondary/ --log-file=/tmp/rsync.log

Note: The /var/primary/storage is a folder mounted to the secondary server via NFS

UPDATE:

I've also tried now the following command:

cp -aur /var/primary/storage /var/secondary/

But it is odd for me that the command asks me if I allow the overwriting of a destination file that has the exact same modified date as the source file. Isn't the update option supposed to watch it and copy by default if a source file is newer (or modified) than the destination file?

backup

2 Answers

  1. I think the best bet is actually monitoring or logging which files have changed. Even though this could be solved using a SDS, since you're using rsync I suggest you're using a normal filesystem.

    Therefore, I think running inotifywait as a deamon is the best bet for you:

    sudo inotifywait --daemon --outfile /tmp/fschanges --recursive /path/to/watch \
  --event modify --event move --event create --event delete --event close_write

    When embedding this in a simple bash script that 1) reads the file, 2) pushes the changes and 3) clears the contents when done, you should get a pretty 'instant' solution:

    #!/bin/bash
logfile="/tmp/fschanges"
path="/home/roy/Downloads"

# Stop deamon upon exit
trap "killall inotifywait" EXIT

# Init deamon
inotifywait --daemon --outfile $logfile --recursive $path \
      --event modify --event move --event create --event delete --event close_write &

while true; do
  # Loop trough file
  while read change; do
    srcpath=$(echo "$change" | cut -d' ' -f1)
    action=$(echo "$change" | cut -d' ' -f2)
    file=$(echo "$change" | cut -d' ' -f3)

    if [[ $action == *"DELETE"* ]]; then
      echo "DELETE: rsync $srcpath$file"
    elif [[ $action == *"CREATE"* ]]; then
      echo "CREATE: rsync $srcpath$file"
    elif [[ $action == *"WRITE"* ]]; then
      echo "WRITE: rsync $srcptah$file"
    fi
  done < $logfile

  # Clear the log
  > $logfile

  # Allow for cooldown
  sleep 1
done

    Off-course the script above needs to be properly completed, but you get the general idea.

    • 0
  2. Best Answer

    I've found the solution, however it is a bit complicated and needs some thinking. If you don't want to read it, then TLDR: No NFS, no issue and it is better to give than get

    So I've tried to sync the files from FileServer-1 to FileServer-2. To ease up the stress on FileServer-1, I was thinking I could execute the rsync command on FileServer-2 and copy the files from FileServer-1. I've reversed it and ran the rsync command from the FileServer-1 to copy the folders incremental filelist to FileServer-2.Magic happened, it was faster. Still unacceptable, but faster.

    After a while I had to unmount an NFS folder from both FileServers. By accident I ran the rsync command from FileServer-1 to copy the files from local machine to FileServer-2. Now the real miracle happened because it was done in about 2 minutes.

    I've deleted about ~600MB of data from FileServer-2 and started rsync again just to see how much time does rsync need to copy all the files (that is about 10.000 files). It took less than 5 minutes !!!!

    Ever since I have the folders unmounted (that have nothing to do with the files I'm trying to rsync) rsync started working like a beast.

    Anyone has an explanation how come Rsync and NFS heates eachother like this? Or this is just a very unique phenomenon on my systems? Using CentOS 7, I've forgot to tell that.

    • 0