rsync changing already existing files

By default rsync will read the entire file on both source and destination, to verify that they are identical. This does not consume network bandwidth, as it will only be comparing a hash value. But it does spend time reading from the disk.

In one usage scenario, I found this to be terribly inefficient because the source files were only being appended to. I used the --size-only, which worked well for me.

There is a few other options, which look like they may be applicable, --append and --append-verify, but I haven't tested those myself.

It does not look like you have a directory with a lot of small files, so the time to read the directory listing from disk and stat each file, shouldn't be much of a problem.

I added the --ignore-existing command and it looks like it won't change anything and only download new files.

rsync -vzr --ignore-existing -e

Edit: When there are new files it still takes longer each cycle.

I think adding -t to your argument list will help.

To verify this you could add --itemize-changes to the arguments (without -t). If I understood you correctly, this would show the T-flag in every line

man 1 rspec:

A t means the modification time is different and is being updated to the sender’s value (requires --times). An alternate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without --times and when a symlink is changed and the receiver can’t set its time. (Note: when using an rsync 3.0.0 client, you might see the s flag combined with t instead of the proper T flag for this time-setting failure.)

After this add -t to the command (keep --itemize-changes) and you will receive a t-flag on every line. In the next run the list will only contain the new files.

This is my example run:

krissi@host ~/tmp/rsync % l *
dst:
total 0

src:
total 0
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 bar
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 foo
-rw-r--r-- 1 krissi users 0 Jul 13 18:19 later
krissi@host ~/tmp/rsync % rsync -vzr --itemize-changes src/ dst/
sending incremental file list
>f+++++++++ bar
>f+++++++++ foo
>f+++++++++ later

sent 174 bytes  received 69 bytes  486.00 bytes/sec
total size is 0  speedup is 0.00
krissi@host ~/tmp/rsync % l *
dst:
total 0
-rw-r--r-- 1 krissi users 0 Jul 13 18:21 bar
-rw-r--r-- 1 krissi users 0 Jul 13 18:21 foo
-rw-r--r-- 1 krissi users 0 Jul 13 18:21 later

src:
total 0
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 bar
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 foo
-rw-r--r-- 1 krissi users 0 Jul 13 18:19 later
krissi@host ~/tmp/rsync % rsync -vzr --itemize-changes src/ dst/
sending incremental file list
>f..T...... bar
>f..T...... foo
>f..T...... later

sent 174 bytes  received 69 bytes  486.00 bytes/sec
total size is 0  speedup is 0.00
krissi@host ~/tmp/rsync % rsync -vzr --itemize-changes src/ dst/
sending incremental file list
>f..T...... bar
>f..T...... foo
>f..T...... later

sent 174 bytes  received 69 bytes  486.00 bytes/sec
total size is 0  speedup is 0.00
krissi@host ~/tmp/rsync % rsync -vzrt --itemize-changes src/ dst/
sending incremental file list
.d..t...... ./
>f..t...... bar
>f..t...... foo
>f..t...... later

sent 177 bytes  received 72 bytes  498.00 bytes/sec
total size is 0  speedup is 0.00
krissi@host ~/tmp/rsync % rsync -vzrt --itemize-changes src/ dst/
sending incremental file list

sent 66 bytes  received 12 bytes  156.00 bytes/sec
total size is 0  speedup is 0.00
krissi@host ~/tmp/rsync % l *
dst:
total 0
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 bar
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 foo
-rw-r--r-- 1 krissi users 0 Jul 13 18:19 later

src:
total 0
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 bar
-rw-r--r-- 1 krissi users 0 Jul 13 18:05 foo
-rw-r--r-- 1 krissi users 0 Jul 13 18:19 later

why do you say it takes longer each time? how is that possible?

maybe it's the program generating the files that is touching them?

try with --checksum: skip based on checksum, not mod-time & size, see if that changes anything (i wouldn't keep this option because it reads every file from the disk every time, too expensive, i'm only suggesting it to find the problem.)

(and maybe try to debug with the -t option, that preserves modification times)