Home / server / Questions / 129969
In Process
BCS
Asked: 2010-04-07 18:57:57 +0800 CST

Is there a way to create a copy-on-write copy of a directory?

  • 772

I'm thinking of a situation where I would have something that creates a copy of a directory, tweaks a few files, and then does some processing on the result. This wold be done fairly often, maybe a few dozen times a day. (The exact use case is testing patch submissions; dupe the code, patch it, build/test/report/etc.)

What I'm looking for could be done by creating a new directory structure and populating it with hard links from the origonal. However this only works if all the tools you use delete and recreate files rather than edit them in place.

Is there a way to have the file system do copy-on-write for a file?

Note: I'm aware that many FSs use COW at a block level (all updates are done via writes to new blocks) but this is not what I want.

filesystems hardlink

5 Answers

  1. If you have your choice of platform for your fileserver, I'd go with a recent OpenSolaris build and use the deduplication feature of ZFS. This way copies of files would take up no additional space, and even common segments between files would not be replicated.

    You can read all about ZFS deduplication at this post on Jeff Bonwick's Blog.

    • 8

  2. Most linux virtualization solutions offer file-level COW among virtualized OSes (copied from the WP):

    • YES:
      • Docker, Linux-VServer, lmctfy, LXC, Singularity, Virtuozzo, Sandboxie, systemd-nspawn, Solaris Containers(ZFS), FreeBSD jail(ZFS), OpenVZ
    • NO:
      • chroot, sysjail, WPARs, iCore Virtual Accounts, Turbo
    • 0

  3. I find this question while searching a better solution instead of my current, which is like: cp -al SOURCEDIRNAME copy.SOURCEDIRNAME or

    find SOURCEDIRNAME -type d -exec mkdir -p copy.{} ';'
find SOURCEDIRNAME -type f -exec ln {} copy.{} ';'

    At this point i have a copy of the structure with a bunch of hardlinks. This works relative fast, and don't need one time more disk space, cause we do not rly copy all files. Now i can do some ''stuff'' (like patch or build) on the copy directory. If they do not modify the original files, only rename them and/or create new files. After the work, i have some results and can remove the copy directory and start over.

    If you use more complicated tools and dont trust them (or you know they try to overwrite existing files) you may need cowdancer.

    • 0

  4. Try use one of UniouFS, AUFS or Overlay Filesystem (depends on which supported in Your distro)

    For example OverlayFS used by docker as one of drivers to merge layers

    Overlay Filesystem

    Overlaying mainly involves directories. If a given name appears in both upper and lower filesystems and refers to a non-directory in either, then the lower object is hidden - the name refers only to the upper object.

    Where both upper and lower objects are directories, a merged directory is formed.

    At mount time, the two directories given as mount options “lowerdir” and “upperdir” are combined into a merged directory:

    mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,workdir=/work /merged

    The “workdir” needs to be an empty directory on the same filesystem as upperdir.

    • 0

  5. May be SVN or other versioning tool (git, cvs)?

    • -2