Home / server / Questions / 259491
Accepted
kagali-san
Asked: 2011-04-14 19:29:22 +0800 CST

shell script: run a batch of N commands in parallel, wait for all to finish, run next N

  • 772

Task: run blocks consisting of 3-5 commands (in parallel/background). Example block:

dd if=/dev/urandom of=/mnt/1/x bs=1024 count=1024000000 &
dd if=/dev/urandom of=/mnt/2/x bs=1024 count=1024000000 &
dd if=/dev/urandom of=/mnt/3/x bs=1024 count=1024000000 &

When it's done, next block should run. I suppose, this can be done via lock files:

task1.sh:

real_task1 real_param1 ; rm /var/lock/myscript/task1.lock

task2.sh:

real_task2 real_param1 ; rm /var/lock/myscript/task2.lock

...

taskgen.sh:

# loop
 # while directory isn't empty - wait...
 gen_tasks.pl # build task files from some queue
 for i in 1 2 3; do touch /var/lock/myscript/task$i.lock ; done
 ./task1.sh &
 ./task2.sh &
 ./task3.sh &
 # if task1.sh doesn't exits then exit, else loop waits for files to be deleted

A number of methods to check if the directory is empty can be found here, don't sure which to use;

Question: any better way to implement this?

P.S. Possible status reporting method:

 command && report_good_state.sh taskid ;  report_state_done.sh taskid; rm /var/lock/myscript/taskN.lock
bash

5 Answers

  1. Perhaps some variation on this?

    while true
do
  ./task1.sh&
  pid1=$!
  ./task2.sh&
  pid2=$!
  ./task3.sh&
  pid3=$!
  wait $pid1
  wait $pid2
  wait $pid3
done
    • 9
  2. Best Answer

    This is exactly what gnu parallel is designed for, so I strongly recommend you use it. In particular, look at running it as a semaphore:

    for i in {1..4}
do
  echo running $i
  sem -j3 df dd if=/dev/urandom of=/mnt/$i/x bs=1024 count=1024000000 ";" echo done
done
# sem --wait waits until all jobs are done.
sem --wait
    • 8

  3. Do you have any particular reason not to use something like GNU parallel? If you must use bash, then consider methods like those described in this blog post (wait and named pipes are helpful here).

    • 5

  4. "wait" waits for all background jobs to complete. Sample:

    sleep 30 & sleep 40 & sleep 120 & wait

    It waits till all commands are completed, i.e. at least 120 seconds for this example.

    Hope this helps.

    • 3

  5. We tried to use the GNU sem utility, as described by Phil Hollenback above, but found it too heavy-weight (300+ instances crippled the machine). I looked around for similar tools to implement a light-weight counting semaphore but couldn't find anything suitable.

    So I implemented one myself using flock, it's called semaphoric.

    • 0