Is there a way I can queue long running tasks?

The shell is perfect for that.

1. Start the first program.
2. Press Ctrl-z to suspend the program.
3. Start the next program.
4. Repeat steps 2 and 3 to add more programs to the queue. All the programs will be in Suspended mode.

5. Now run the queue like this:

   while fg; do :; done
   

   You may not suspend this while loop or exit bash until the queue is done.

If you will need to log out (e.g. the programs will run for many days) then consider running the above steps in screen.

Not very elegant, but quick and dirty:

process1 &
while kill -0 $! ; do sleep 1; done && process2 &
while kill -0 $! ; do sleep 1; done && process3 &

You would need to substitute actual PIDs for $! if you've run intervening background jobs.

The answer to your general question is: Absolutely. In the early days of computing, batch processing was the only way to do anything, and even when multi-user interactive systems were invented, batch-processing capability was the norm for large jobs. And it's still commonly done today in medium and large-scale environments using systems like Sun Grid Engine or Torque.

However, that's probably overkill for what you need. You could set up a more lightweight system to run scripts in a serial queue, but I don't think that approach is particularly well-suited to your specific task. Presuming parallel copies to different drives are acceptable, I think I'd attack it like this:

1. Create directory structure corresponding to your your target drives:

   ~/copysystem/drive1/ ~/copysystem/drive2/ ~/copysystem/drive3/

2. Install Incron.

3. Set up an incrontab entry for each of these directories, which runs your copy script automatically on IN_MOVED_TO.

4. Make your script either a) kill any previous instances of the same script when it starts or b) use a mkdir-based lockfile and block until the lock is cleared.

Then, all you need to do is move files to the various ~/copysystem/drive# directories, and they're all copied magically to your destination.

Especially in case of 4a, you probably want to use rsync -aP to copy your files, so that you can restart partial transfers from the middle. (Possibly in combination with --remove-sent-files, if you want to get rid of the originals.)

If you want to skip the complication of using incron, you can still take advantage of making your scripts block on a lock file. That works something like this:

#!/bin/bash
LOCKDIR="/var/run/copysystem/copysystem.lock"

while ! mkdir $LOCKDIR ; do
  echo "waiting for lock"
  sleep 5
done
trap rmdir $LOCKDIR EXIT

rsync commands go here....

This works because mkdir is an atomic operation — if it succeeds, you know the directory didn't exist. That's important, because if you use something like ! -f && touch, there's a race condition. (Same with scanning the process table for rsync commands, or the like.)

If this is something you'll be doing regularly, then its worth setting up some sort of scheduler to manage it for you.

I like the elegance of Aleksandr's solution - and it addresses all the points you originally raised - but I can see it does have limitations.

I've previously used the BSD lpd as a method for queueing jobs - the 'printer driver' is just a shell script so its easy to adapt to different tasks (in my case that meant managing 4 modems for polling data, sending faxes, SMS and other stuff).

What you want is a non-interactive command queue. Good news! I wrote one for you.

enqueue_cmd:

#!/usr/bin/perl -w

# Enqueue a job

use strict;

use Fcntl qw(:flock);

my $JOB_QUEUE_FILE = '/var/tmp/job_queue';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;

my $jq;
open $jq, ">> $JOB_QUEUE_FILE" or die "!open $JOB_QUEUE_FILE: $!";

my $locked = 0;

LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
    $lock_tries++)
{
    if (flock $jq, LOCK_EX) {
            $locked = 1;
            last LOCK_ATTEMPT;
    }

    sleep $LOCK_SLEEP;
}

$locked or die "could not lock $JOB_QUEUE_FILE";

for (@ARGV) {
    print $jq "$_\n";
}

close $jq;

dequeue_cmd:

#!/usr/bin/perl -w

# Dequeue a jobs and run them

use strict;

use Fcntl qw(:seek :flock);
use FileHandle;

my $QUEUE_FILE = '/var/tmp/job_queue';
my $OUTPUT_FILE = '/var/tmp/job_out';
my $LOCK_TRIES = 5;
my $LOCK_SLEEP = 1;
my $JOB_SLEEP = 1;

my $locked;

my $jo;
open $jo, ">> $OUTPUT_FILE" or die "!open $OUTPUT_FILE: $!";
$jo->autoflush(1);

my $jq;
open $jq, "+< $QUEUE_FILE" or die "!open $QUEUE_FILE: $!";

my @jobs = ( );
my $job;

JOB: while (1) {

    if (-s $QUEUE_FILE == 0) {
            sleep $JOB_SLEEP;
            next JOB;
    }

    $locked = 0    

    LOCK_ATTEMPT: for (my $lock_tries = 0; $lock_tries < $LOCK_TRIES;
            $lock_tries++)
    {
            if (flock $jq, LOCK_EX) {
                    $locked = 1;
                    last LOCK_ATTEMPT;
            }

            sleep $LOCK_SLEEP;
    }
    $locked or die "could not lock $QUEUE_FILE";

    seek $jq, 0, SEEK_SET or die "could not seek to start of file";

    push @jobs, <$jq>;

    truncate $jq, 0 or die "could not truncate $QUEUE_FILE";
    seek $jq, 0, SEEK_SET or die "could not seek to start of $QUEUE_FILE";

    flock $jq, LOCK_UN;

    for $job (@jobs) {
            chomp $job;
            print $jo "## executing $job\n";
            print $jo `$job`;
    }

    sleep $JOB_SLEEP;
}

First, run nohup ./dequeue_cmd &, Then add your commands like so:

./enqueue_cmd "echo 'hello world'" "sleep 5" "date"
./enqueue_cmd "ls /var/tmp"

The output appears in /var/tmp/job_out:

tail -F /var/tmp/job_out
## executing echo 'hello world'
hello world
## executing sleep 5
## executing date
Fri Dec 10 16:35:43 PST 2010
## executing ls /var/tmp
ff
job_out
job_queue
ss