Showing total progress in rsync: is it possible?

There is now an official way to do this in rsync (version 3.1.0 protocol version 31, tested with Ubuntu Trusty 14.04).

#> ./rsync -a --info=progress2 /usr .
    305,002,533  80%   65.69MB/s    0:00:01  xfr#1653, ir-chk=1593/3594)

I tried with my /usr folder because I wanted this feature for transferring whole filesystems, and /usr seemed to be a good representative sample.

The --info=progress2 gives a nice overall percentage, even if it's just a partial value. In fact, my /usr folder is more than 6 gigs:

#> du -sh /usr
6,6G    /usr/

and rsync took a lot of time to scan it all. So almost all the time the percentage I've seen was about 90% completed, but nonetheless it's comforting to see that something is being copied :)

References:

• https://stackoverflow.com/a/7272339/1396334
• https://download.samba.org/pub/rsync/NEWS#3.1.0

The following applies to rsync version 3.0.0 and above. The options described below were introduced in that release on March 1st, 2008.

Along with --info=progress2 you can also use --no-inc-recursive option (or its shorter --no-i-r alias) to disable incremental recursion.

This will build the entire file list at the beginning, rather than incrementally discovering more files as the transfer goes on. Since it will know all files before starting, it will give a better report of the overall progress. This applies to the number of files - it does not report any progress based on file sizes.

This involves a trade-off. Building the entire file list ahead of time is more memory-costly and it can significantly delay the start of the actual transfer. As you would expect, the more files there are, the longer the delay will be and the more memory it will require.

The following is from the rsync manual (source - http://rsync.samba.org/ftp/rsync/rsync.html ):

-r, --recursive

This tells rsync to copy directories recursively. See also --dirs (-d). Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.

Incremental recursion can be disabled using the --no-inc-recursive option or its shorter --no-i-r alias.

See also https://rsync.samba.org for specific version differences (scroll down and check out the Release News links).

You can with 'pv' (apt-get install pv with Debian and ubuntu). I recommend to monitor the number of files transferred, since the amount of data transferred is not correlated to the size of files but on the delta between the source and destination. And counting files will count the same progress for one big delta and another one with small delta. Which means that in any case the ETA estimation might be far off. The size-based ETA only works if your destination is empty, in this case delta == size of source.

The general idea is to emit one line per file 'transferred' from rsync, and count those lines with 'pv':

rsync -ai /source remote:/dest | pv -les [number of files] >/dev/null

I tend to backup whole filesystems (for several reasons), in this case you can use the much cheaper df to get the number of files (rather than du or find wich will traverse your source hierarchy another time after rsync did it). The -x option appears to make sure rsync stays on the same source filesystem (and does not follow other inner mounts):

rsync -aix /source remote:/dest | pv -les $(df -i /source | perl -ane 'print $F[2] if $F[5] =~ m:^/:') >/dev/null

If you want to count files in /source in a general way, use find /source|wc -l (warning again: might be slow and heavy on I/O).

danakim is correct. There are no trivial ways to add a total progress indicator.

The reason for this is that when rsync looks at a list of files to sync, it doesn't know ahead of time which files will need to change. If you are doing delta transfers, the deltas themselves have to be calculated ahead of time to give a total picture of the work that needs to be done.

In other words, the easiest way to calculate how much work there is to be done is to actually do it.

For long transfers, I'm happy with running du -s on both sides. Even watch -n1 du -s, if I feel really anxious.

watch executes a command (du -s here) periodically (every 1 second here) and shows the output fullscreen.

Basically no. You can only show progress per-file with the --progress flag, but that is about it.

I am guessing you can write a wrapper around it or use any of the patches that you already find but you must ask yourself if it is really worth it, do you actually need a total progress for rsync?

I also searched for how to show the total progress with rsync and I've found a useful answser from this post: https://stackoverflow.com/questions/7157973/monitoring-rsync-progress

Basicly, you can use --info=progress2 in the dev version of rsync 3.1.0. Here is what the doc has said:

There is also a --info=progress2 option that outputs statistics based on the whole transfer, rather than individual files. Use this flag without outputting a filename (e.g. avoid -v or specify --info=name0 if you want to see how the transfer is doing without scrolling the screen with a lot of names. (You don't need to specify the --progress option in order to use --info=progress2.)

I used the answer from zerodeux and wrote my own little bash script:

#!/bin/bash

RSYNC="ionice -c3 rsync"
# don't use --progress
RSYNC_ARGS="-vrltD --delete --stats --human-readable"
SOURCES="/dir1 /dir2 /file3"
TARGET="storage::storage"

echo "Executing dry-run to see how many files must be transferred..."
TODO=$(${RSYNC} --dry-run ${RSYNC_ARGS} ${SOURCES} ${TARGET}|grep "^Number of files transferred"|awk '{print $5}')

${RSYNC} ${RSYNC_ARGS} ${SOURCES} ${TARGET} | pv -l -e -p -s "$TODO"

Use

lsof -ad3-999 -c rsync

To see what files rsync currently has open (will show filesize) rsync copies into a hidden file locally

If you don’t have the latest rsync (e.g., OS X has 2.6.9) and can’t use --info=progress2, here’s another alternative to save yourself from pages of scrolling text on progress:

rsync -aPh <source> <destination> | xargs -L1 printf "\33[2K\rTransferring: %s"

That will print out, on one line, the name of the latest file being transferred:

Transferring: the-latest.file