Home / server / Questions / 262411
Accepted
Stann
Asked: 2011-04-22 16:14:03 +0800 CST

rsync to NAS copies everything every time

  • 772

I run Ubuntu Linux and use backup script that copies things over using rsync:

rsync -rc /home/user/source /media/nas/destination

or

rsync -r /home/user/source /media/nas/destination

However - it takes approximately 2.5 hours every day (i run this every day), to get it done. There is approximately 76 Gb of data over 1Gbit lan. So - basically - rsync just doesn't skip existing files in destination - it just copies everything over and over again.

I assume that the problem lies in NAS (d-link 321) and that it's a samba share that doesn't support unix persmission system - right? So this way rsync can't distinguish old files from new files - right? I've tried rsync with -c parametr (skip based on checksum, not mod-time & size) but still no dice, still takes 2.5 hours.

Can anybody suggest the way of syncing appx 70-80 Gb of data that doesn't ivolve "dumb" copying of everything every time?

Thanks:)

UPDATE

well. no luck. I've erased destination directory and and ran this twice:

rsync -r --times /home/user/source /media/nas/destination

It still took 2.5 hours both times. so it copies everything twice.

mmm. I went deeper to investigate this problem. And it looks like rsync doesn't preserve times. I've ran stat command on some files from source and destination - and Access, Modify, Change times aren't preserved on destination. I have no clue as to why does this happen... Any other suggestions? P.s. I relatively new to Linux - sio it can be something really dumb and silly:)

UPDATE 2 Resolved: problem as I realized stemmed from me mounting shares incorrectly. I've been mounting them like this (in /etc/fstab):

//192.168.10.199/Volume_1 /media/Volume_1 cifs   guest,rw,iocharset=utf8,file_mode=0777,dir_mode=0777 0 0

And this is what worked:

//192.168.10.199/Volume_2 /media/Volume_2 cifs    
username=sam,password=some_password,uid=developer,gid=developer,_netdev 0 0

Rsync preserves times now and only takes 3 minutes!

backup linux

6 Answers

  1. You need to add -t (or --times) to your rsync command-line, so that rsync will preserve file modification times at the destination. That will allow subsequent rsync runs to skip all files with identical sizes and modification times. Also, don't bother with the -c option, unless you know for certain that your destination does not (or cannot) maintain accurate file modification times, for some reason.

    • 9

  2. You very likely want to be using the -u flag, which tells rsync not to send files that exist in the destination and that either have newer modified times in the destination or have equal modified times and sizes in both places.

    • 5
  3. Best Answer

    Why don't you try:

    rsync -a /home/user/source /media/nas/destination

    -a is convenience option specifically for archiving. It works for me all the time.

    Here is what man page says:

    -a, --archive archive mode; same as -rlptgoD (no -H)

    • 5

  4. In addition to using the -a flag, you should also check the man page for other options. The problem with using a simple mirror backup strategy, is that any corrupted file will be copied over and corrupt your backup as well. One of the rsync options is to backup a copy of the changed files on the destination side, which is much safer.

    I highly recommend to take a look at BackupPC as well. It uses rsync and has Debian/Ubuntu packages. It will handle incrementals properly, collating identical files, and compressing everything to save space.

    • 1

  5. Sometimes, when the NAS has a different time format on the files, it can be useful to add -u or --update which has the effect of skip files that are newer on the receiver.

    • 1

  6. I have used this for backup to NAS successfully for a couple of years

    rsync -av --delete /source-dir /target-dir and rsync -avn --delete /source-dir /target-dir -n option means 'show me what would be done, but don't do any changes' --delete delete files that are in the target-dir but not in the source-dir since I want the backup to be a backup

    • 0