Home / server / Questions / 39534
Accepted
Adam Matan
Asked: 2009-07-13 02:55:46 +0800 CST

Best way to compare (diff) a full directory structure?

  • 772

What's the best way to compare directory structures?

I have a backup utility which uses rsync. I want to tell the exact differences (in terms of file sizes and last-changed dates) between the source and the backup.

Something like:

Local file                   Remote file                         Compare
/home/udi/1.txt (date)(size)   /home/udi/1.txt (date)(size)     EQUAL
/home/udi/2.txt (date)(size)   /home/udi/2.txt (date)(size)     DIFFERENT

Of course, the tool can be ready-made or an idea for a python script.

Many thanks!

Udi

filesystems files rsync

9 Answers

  1. Best Answer

    The tool your looking for is rdiff. It works like combining rsync and diff. It creates a patch file which you can compare, or distribute.

    • 19

  2. Some people want to compare filesystems for different reasons, so I'll write here what I wanted and how I did it.

    I wanted:

    • To compare the same filesystem with itself, i.e., snapshot, make changes, snapshot, compare.
    • A list of what files were added or removed, didn't care about inner file changes.

    What I did:

    First snapshot (before.sh script):

    find / -xdev | sort > fs-before.txt

    Second snapshot (after.sh script):

    find / -xdev | sort > fs-after.txt

    To compare them (diff.sh script):

    diff -daU 0 fs-before.txt fs-after.txt | grep -vE '^(@@|\+\+\+|---)'

    The good part is that this uses pretty much default system binaries. Having it compare based on content could be done passing find an -exec parameter that echoed the file path and an MD5 after that.

    • 13

  3. if you don't feel like installing another tool...

    for host in host1 host2
do
  ssh $host ' 
  cd /dir &&
  find . |
  while
    read line
  do
    ls -l "$line"
  done ' | sort  > /tmp/temp.$host.$$
done
diff /tmp/temp.*.$$ | less
echo "don't forget to clean up the temp files!"

    And yes, it could be done with find and exec or find and xargs just as easily as find in a for loop. And, also, you can pretty up the output of diff so it says things like "this file is on host1 but not host2" or some such but at that point you may as well just install the tools everyone else is talking about...

    • 9

  4. I've used dirdiff in the past to compare directory structures. It only works on local dirs so you will have to sshfs-mount your other directories.

    The good thing is that you can see visually if the files are equal or not and which one is newer or older. And it supports up to 5 directories. You can also see differencies and copy files from one to the other.

    • 5

  5. diff -r actually works quite well. If you just want to know if the files differ, not the actual contents of the differences, then do diff -qr

    • 5

  6. From rsync man page:

    -n, --dry-run
This  makes rsync perform a trial run that doesn’t make any changes (and produces mostly
the same output as a real run).  It is most commonly used in combination  with  the  -v,
--verbose  and/or -i, --itemize-changes options to see what an rsync command is going to
do before one actually runs it.

    May be this will help.

    • 4

  7. I would use Meld for that.

    • 3

  8. I didn't really want to install anything new, so my solution is a bit crude, but quick and effective.

    I wanted to compare all files have been copied and their file mode, ownership and SELinux security context remain unchnaged.

    So I ran:

    rsync -aX /orig /copy/

    and then compare like this:

    cd /orig && ls -lZR > /tmp/diff_orig
cd /copy && ls -lZR > /tmp/diff_copy
vimdiff /tmp/diff_*

    If the contents of both files remain collapsed then they are identical. If not, the diff is shown.

    Update parameters of ls to reflect the attributes you need to check. YMMV

    • 3

  9. Besides the tools already mentioned on windows you could use Total Commander or WinSCP, both have very comfortable functions to compare (and sync) directories.

    • 2