Home / server / Questions / 24067
In Process
Thomas Vander Stichele
Asked: 2009-06-12 05:24:40 +0800 CST

Recursive move utility on Unix?

  • 772

Sometimes I have two trees that used to have the same content, but have grown out of sync (because I moved disks around or whatever). A good example is a tree where I mirror upstream packages from Fedora.

I want to merge those two trees again by moving all of the files from tree1 into tree2.

Usually I do this with:

rsync -arv tree1/* tree2

Then delete tree1.

However, this takes an awful lot of time and disk space, and it would be much easier to be able to do:

mv -r tree1/* tree2

In other words, a recursive move. It would be faster because first of all it would not even copy, just move the inodes, and second I wouldn't need a delete at the end.

Does this exist ?

As a test case, consider the following sequence of commands:

$ mkdir -p a/b
$ touch a/b/c1
$ rsync -arv a/ a2
sending incremental file list
created directory
./
b/
b/c1
b/c2

sent 173 bytes  received 57 bytes  460.00 bytes/sec
total size is 0  speedup is 0.00
$ touch a/b/c2

What command would now have the effect of moving a/b/c2 to a2/b/c2 and then deleting the a subtree (since everything in it is already in the destination tree) ?

unix recursive

11 Answers

  1. Per the mv(1) manpage from gnu's mv:

    -u, --update move only when the SOURCE file is newer than the destination file or when the destination file is missing

    • 2

  2. the proposed mv -uf dir1/* dir2/ move the (sub)directories, not each file. you might try to use find

    cd dir1
find . -type d -exec mkdir -p dir2/"{}" \;
find . -type f -exec mv -uf "{}" dir2/"{}" \;

    or something similar

    • 2

  3. Does not

    mv -uf tree1/* tree2/

    work?

    • 1

  4. Midnight Commander (mc) is also nice for this kind of stuff. Tag files with CTRL-t, press F6, and when it asks to overwrite destination files, choose Update if you want to overwrite older files.

    • 1

  5. You could use "cp -l & rm" for in-device moving:

    cp -alv --backup=numbered tree1/* tree2 &&
rm -rf tree1/
    • -l of cp to use hard links instead of copying, (this also prevents cross-device operations)
    • --backup=numbered of cp for backing up the existing files in target directory

    And be careful about this two issues:

    • use && to prevent your uncopied data to be removed, if you accidentally run it on cross-device targets. (in corss-device case cp exits with a status of "1", at least for GNU coreutils)
    • files starting with "." in tree1, you will lost them if there is any.
    • 1

  6. Javier's answer with find works well, except that it doesn't remove the original directories. Add at the end:

    rmdir $(find . -type d  |grep -v ^\.$)
    • 1

  7. err

    mv dir1/* dir2/

    or simply

    rsync -arv --remove-source-files  tree1/* tree2

    should be enough, you will probably run into trouble at some point when too many entries are in dir1.

    find sourcedir -maxdepth 1 -exec echo mv {} targetdir/ \;

    should be a nice options

    find sourcedir -maxdepth 1 -print0 |xargs -0 -I _ echo mv _ targetdir
find sourcedir -maxdepth1 -exec mv {} targetdir/ +

    are both not really necessary because mv just takes 2 options (source target), so you will have to live with the multitude of processes in that case.

    • 0 
  8. cd /tree1
mv * /tree2

    This won't move hidden files or folders, but your original example wouldn't either.

    • 0

  9. I think mv doesn't do what you think it does.

    A unix filesystem has 3 components:

    • directory entries
    • inodes
    • blocks

    A directory entry points to an inode.

    The inode has the metatadata about the file (is it a file, a directory, a named pipe? Who owns it? What are the permissions? What blocks does that inode use?

    Blocks are the things that actually contain the contents of the file.

    So -- when you "mv" a file, all you're really doing is unlinking the first directory entry and relinking it somewhere else.

    snoopy -> inode 333 
woodstock -> inode 333

    No data ever gets duplicated / copied. You create the link snoopy, then you create the link woodstock, and then you delete the link snoopy. (things are a bit different with directories because typically you can't make hardlinked directories, but even so the "link" name just changes).

    What if you're moving from one filesystem to another? In the old days, mv would just thrown an error and make it explicit that you can't move a file from one filesystem to another. These days, it seems like mv silently copies the data then deletes the original.

    In the old days, because you couldn't move data from one file system to another, you got in the habit of using idioms like

    tar -cf - . | (cd /new/location && tar -xf -)

    then you delete the old data. Part of the reason for using tar was that in the old days, cp would destroy metadata like "this is a symbolic link" and "that's a hard link" and instead you'd just get new copies of that file as regular files. Even still, you need to give "cp" flags to tell it to preserve that sort of structure.

    There is no way to avoid "moving" lots of data if it is from one filesystem to another. It doesn't matter if you're using fancy new move or rsync or tar or cpio.

    But, if you keep all the data in the same filesystem, this:

    mv /filesystem-1/big/directory* /filesystem-1/big2/

    that will be extremely fast because it is just changing the directory entries and not actually moving any real data.

    There are other issues at play, such as what should you do if there is already a file / directory in the new location as well as the source location?

    • 0

  10. None of the answers in this thread fit my usecase, so I came up with one on my own as a shell script.

    At the heart of it is this function:

    recurse() {
    if [ ! -d "$1" ] || [ ! -e "$2" ]; then
        mv -u "$1" "$2" || exit
        return
    fi
    for entry in "$1/"* "$1/."[!.]* "$1/.."?*; do
        if [ -e "$entry" ]; then
            recurse "$entry" "$2/${entry##"$1/"}"
        fi
    done
}

    Which you could invoke like so (for the OP's usecase):

    recurse a/b/c2 a2/b/c2

    If source is a file or a directory for which the destination does not exist yet, it simply moves it with mv -u. If source is a directory for which the destination already exists, it iterates through its contents and recursively performs the same check followed by move|recurse with each of its members.

    I used -u because I only needed old files freshened and newer files untouched, but you could replace it with -f for an unconditional move or -i for an interactive one. Or not change anything and simply rm -rf your source after the script is done moving things.

    • 0