Home / server / Questions / 209888
Accepted
Jean-Philippe Pellet
Asked: 2010-12-08 02:09:57 +0800 CST

How can I use rsync to duplicate a directory tree, creating hardlinks to files?

  • 772

From time to time, I have to perform several large migration changes on data files on my server, and I'm looking for a good way to do this. I was thinking about using rsync to duplicate my directory structure starting at the root data folder, creating hard links to all original the files (some of them are rather big), and I can overwrite in the destination tree only the files that need migrating. In the end, I can safely switch from the old files to the new files with two mv operations.

However, I can't seem to get rsync to do this. I tried

rsync -a --link-dest=$DATA $DATA $DATA/../upgrade_tmp

but instead of creating hard links to files, rsync copies them entirely. Is there a problem using the same source and link-dest directory?

mac-osx rsync hardlink

8 Answers

  1. Best Answer

    rsync is a powerful tool, but it is, unfortunately, strangely picky about some of its pathnames.

    If $DATA is an absolute path (i.e. it begins with a /), then the correct command-line to use is:

    rsync -a --link-dest=$DATA $DATA/ $DATA/../upgrade_tmp

    [Now, just a brief aside about rsync's strangeness. Note the trailing / added to the source argument. This tells rsync to work with the contents of the source directory, rather than with the source directory itself. (I'm assuming that $DATA doesn't already contain a trailing /.) In this case, we want to work with the contents, so we add the trailing /.]

    If, on the other hand, $DATA is a relative path (i.e. it does not begin with a /), then Sean R's comment about --link-dest is bang on: The link-dest path is interpreted relative to the destination path, so you would use the following:

    rsync -a --link-dest=../`basename $DATA` $DATA/ $DATA/../upgrade_tmp

    EDIT

    One final note, it turns out that the second rsync command-line I gave should work regardless of whether $DATA is an absolute path, since basename doesn't care whether a path is absolute or relative.

    • 27

  2. What you want is "cp -al":

    cp -al $DATA/ $DATA/../upgrade_tmp/
    • -a recurses like rsync -a
    • -l will hard link files instead of copying them.
    • 18

  3. Turns out it is more difficult to do this with rsync than with other tools. The correct answer for rsync is Steven Monai's, but the easiest way to do this is to use either cp -al or pax -rwl on systems where -l is not a valid option for cp:

    pax -rwl $DATA $DATA/../upgrade_tmp

    or

    cp -al $DATA/ $DATA/../upgrade_tmp/
    • 9

  4. The --link-dest option in rsync is relative to the destination directory, not the current directory. So what you want is:

    rsync -a --link-dest=../`basename $DATA` $DATA $DATA/../upgrade_tmp
    • 7

  5. It works for me:

    $ rsync --hard-links --recursive --link-dest=/local user@host:/remote/ /local

    I use rsync version 3.1.0.

    From man:

    --hard-links

    Tells rsync to look for hard-linked files in the transfer, without this option, hard-linked files in the transfer are treated as though they were separate files.

    --link-dest=DIR

    Unchanged files are hard linked from DIR to the destination directory. The files must be identical in all preserved attributes (e.g. permissions, possibly ownership) in order for the files to be linked together

    • 4

  6. Can try following link http://www.lessfs.com/wordpress/ it is work on COW (copy on write) that will save time and space

    • 2

  7. First create the directories only on the destination:

    rsync -av --include '*/' --exclude '*' /source/ /destination/

    Then hard link the files only:

    cd /source
find . -type f -exec ln -v {} /destination/{} \;
    • 2

  8. Use the Option -H to preserve Hardlinks and read the manpage.

    • 1