Home / server / Questions / 192590
Accepted
Xeoncross
Asked: 2010-10-20 10:09:30 +0800 CST

Recursively archive small files in document tree on linux?

  • 772

Given a base directory (like /home/user) is there a command that could be run which would create an archive back up all text files (i.e. files less than 100kb) in that directory and it's children? I know you can tar/gz a directory - but what about excluding files also?

The idea is that most photos, videos, and other large files would be ignored while all important hand-typed documents could easily be backed up quickly when moving around projects and servers.

UPDATE

Using skinp's awesome code I was able to backup a small amount of the files. However, as DerfK pointed out - there is a limit to the size of arguments you can pass to commands. With that in mind I was able to write out the contents of find hoping that I could use something to read the contents to tar and bypass this limit. The other other option seems to be a sh script that could ad each file to the archive at a time.

find /home/username -type f -size -100k > list.of.small.files.txt

Which rendered out a 6MB file.

linux files archive

3 Answers

  1. Best Answer

    Looking at the other answers posted here so far, I'm concerned that the uses I see of xargs and find -exec {} are erroneous. If and when the file list grows long enough that tar -c is executed by xargs more than once, the tar file populated to that point will be overwritten. Thus, only the files from the last invokation of tar will end up in the tarfile.

    Here's a one-liner that should always work, regardless of the total number of files, and regardless of whether filenames contain embedded newlines:

    find /home/user -type f -size -100k -print0 | tar -c -z --null --files-from=- -f backup.tgz

    The find command generates a list of null-terminated file names to backup, and the tar command reads that list from the pipe and creates the tarfile backup.tgz.

    • 5

  2. I would use the power of find:

    find /home/user -type f -size -100k -exec tar cvzf backup.tgz {} \+

    type: specifies you want a file not a directory
    size: with the number preceeded by -, means we want less than 100k
    exec: execute the tar with {} being the files found, \+ means end of the exec

    You could also use xargs:

    find /home/user -type f -size -100k | xargs tar cvzf backup.tgz

    Update:

    tar as a command to append a file to an existing tar archive (maybe even not existing, it works for me).

    Here's a simple example script doing this:

    find . -type f -size -100k -print > filelist
for i in `cat filelist`
do
    tar --append $i --file=backup.tar
done
gzip backup.tar

    Obviously, this script is highly ineficient... It only append one file at a time, launching the tar command as many time as there is files.
    It would be good to script it in a way that it append say 1000 files each pass...

    • 3

  3. Try 

    find /home/user -type f -size -100k -print0 | xargs -0 tar cvzf tarfile.tar.gz

    which will be safe for files with spaces in the name too.

    • 3