Home / ubuntu / Questions / 1257199
Accepted
An Ignorant Wanderer
Asked: 2020-07-08 14:11:24 +0800 CST

How can I run commands in batches?

  • 772

I have the following script:

command file1 &
command file2 &
command file3 &
command file4 &
command file5 &
command file6 &
# SOME OTHER COMMAND 
command file7 &
command file8 &
command file9 &
command file10 &
command file11 &
command file12 &

At the place of "SOME OTHER COMMAND", I would like a command to halt the execution of the next commands until all of the processes prior to it are done so that I'm effectively running commands in "batches". How can I do that?

bash process

3 Answers

  1. Best Answer

    You should be able to use the bash shell's built in wait command. From man bash (emphasis mine):

           wait [-n] [n ...]
              Wait for each specified child process and return its termination
              status.  Each n may be a process ID or a job specification; if a
              job  spec  is  given,  all  processes in that job's pipeline are
              waited for.  If n is not given, all currently active child  pro‐
              cesses are waited for, and the return status is zero.  If the -n
              option is supplied, wait waits for  any  job  to  terminate  and
              returns  its exit status.  If n specifies a non-existent process
              or job, the return status is 127.  Otherwise, the return  status
              is the exit status of the last process or job waited for.

    So

    command file1 &
command file2 &
command file3 &
command file4 &
command file5 &
command file6 &

wait

command file7 &
command file8 &
command file9 &
command file10 &
command file11 &
command file12 &
    • 2

  2. If you want to get a bit fancy, a bash script

    batch() {
    for arg; do
        theCommand "$arg" &
    done
    wait
}

batch file{1..6}
batch file{7..12}
    • 2

  3. xargs is a great option when you're running the same command with different arguments.

    echo file{1..32} | xargs -P 32 -n 1 command

    The -P 32 tells xargs to run at most 32 processes at a time. The -n 1 means pass at most one argument to the command (without it, its likely to send all the files to a single command, and thus get no parallel execution).

    So, in this case, set -P to your batch size. The other option (and benefit), is that you can set -P to the amount of parallelism you want to actually use. While the & approach will launch a process for each command, xargs will limit the number of processes to what you set. This means you can target parallelism to the number of cores you have, or use a smaller number to prevent a large batch process from bringing your machine to a crawl.

    Oh, and of course, replace command with your actual command.

    • 0