Home / server / Questions / 701525
In Process
UpTheCreek
Asked: 2015-06-25 23:39:38 +0800 CST

How to execute a command for each file in a deep directory structure, recursively?

  • 772

I have a deep directory structure, with a large number of files (about 1M). I'd like to execute a command against each. (In my case the files are pngs which I'd like to run optipng against.)

I've tried this:

find . -name *.png | xargs sudo optipng -o2

but am getting the error argument list too long.

I'm assuming that xargs should be able to handle this, and that there must be a problem with my syntax.

linux

3 Answers

  1. Do the exec directly with find rather than via xargs.

    sudo find . -name '*.png' -exec optipng -o {} +
    • 7

  2. Using sudo as command to execute by xargs isn't working, as you see. Try it the other way around:

    find . -name '*.png' | sudo xargs optipng -o2

    The difference is that your version creates a command like 

     sudo optipng -o2 file1 file2 file3 file4 ......

    and sudo can't handle so many parameters.

    Doing it the other way around executes sudo once, which in turn starts xargs which then generates a command like 

    optipng -o2 file1 file2 file3 file4 ......

    and if optipng can handle many files as parameters, it should work.

    If that still doesn't work, you will have to use -exec instead like in Geraint's answer, but this should be your last resort.

    • 4

  3. Yet another shot at this:

    find . -name '*.png' | sudo xargs -n x -P y optipng -o2

    Which will start one optipng for x files and y such processes in parallel (might help you get things done faster if you have some cores to spare). Having x greater than 10 probably won't have great impact on performance.

    That will help keep the argument list short. So you could probably move sudo to where you had it, but that's just another command that would have to be run for every batch.

    • 2