Home / server / Questions / 501994
Accepted
Christian
Asked: 2013-04-24 11:31:48 +0800 CST

Daemon to create hardlinks on Linux file server which finds identical files

  • 772

I have a Linux server where I only store new files or rename directories and never edit files. It contains backups from other Linux servers.

Due to certain circumstances there are quite some duplicate files, often with different names.

Is there any free Linux tool which periodically scans the filesystem and has a database with filenames, sizes and maybe sha1sums and then identifies duplicates and replaces them with hardlinks?

linux

2 Answers

  1. Best Answer

    some tools taken from https://unix.stackexchange.com/questions/3037/is-there-an-easy-way-to-replace-duplicate-files-with-hardlinks

    • trimtrees.pl
    • fduples -L
    • findup -m (from fslint)
    • rdfind -makehardlinks

    You could run one of them in a cron job.

    • 2

  2. You can use a deduplicating filesystem. There are two main choices in Linux - btrfs and zfs.

    With btrfs the drawback would be that it is still not marked as stable and has no fsck.

    ZFS is not in the Linux kernel due to licensing issues but there is a kernel module with support for most Linux distributions. Also ZFS sports some kind of online-fsck with the scrub feature. You can have a look at the supported distros on zfsonlinux.org

    Both have compression, deduplication and snapshotting features without the need of any additional userspace daemons - making them ideal for backup solutions.

    • 1