Home / server / Questions / 52725
Accepted
rjmunro
Asked: 2009-08-12 05:28:09 +0800 CST

rsync to compressed archive of file

  • 772

I take backups of a server with rsync, but those backups are quite large, and I'd like to compress them. Is there a way to have some sort of wrapper around rsync to gunzip the file in the backup, rsync the changes from the live file, then re-gzip the file as soon as it is copied?

I.e. if the source files on the live are foo, bar and baz, the backup has foo.gz, bar.gz and baz.gz.

To restate: I want compressed files at one end and uncompressed files at the other end. I don't want to compress before rsyncing, because even with --rsyncable, it makes rsync less efficient. I know about the -z option to rsync. I don't have space on the backup machine to store all the files uncompressed.

backup rsync gzip

4 Answers

  1. I'm not sure I understand what you're trying to do, but you could create gzipped backups on the main server making sure that they're readily rsyncable and not bother decompressing anything in transit.

    gzip(1)

--rsyncable
       While compressing, synchronize the output occasionally based on
       the  input.   This  increases  size by less than 1 percent most
       cases, but means that the rsync(1) program can much more  effi‐
       ciently  synchronize  files  compressed with this flag.  gunzip
       cannot tell the difference between a  compressed  file  created
       with this option, and one created without it.
    • 11
  2. Best Answer

    I think that rsyncing to a fuseCompress mounted filesystem might be the best option. The project's history specifically mentions them making improvements to rsync's performance with it.

    http://code.google.com/p/fusecompress/

    https://github.com/tex/fusecompress/

    • 7

  3. You can write a wrapper to rsync. If you 'pull' data, there's nothing special to do. If you want to 'push' data from the original machines to the backup server it's more delicate to make it work:

    First you have to be sure the wrapper is invoked with the same name, path and parameters as the original rsync, at least for the user under which you run the backups.

    Second, be sure not to write anything to stdout, because it would mangle the communication channel.

    And third, be sure to pass all the parameters you get, just modifying as appropriate for the newly uncompressed destination.

    • -1

  4. He's probably looking for data transmission compression via rsync, not file compression at the other end.

    Check out: http://jimmyg.org/blog/2007/rsync-basics.html

    Compression parameters from the man page:

       -z, --compress              compress file data during the transfer

        --compress-level=NUM    explicitly set compression level

    By default ssh does its own compression to an extent. I don't know how much better rsync's is over native ssh. If the majority of your files are already compressed into gzip files, there probably wont be much that rsync's compression mechanism can do for you. You can only squeeze a file so much.

    • -2