I'm connecting to a Linux machine through SSH, and I'm trying to run a heavy bash script that makes filesystem operations. It's expected to keep running for hours, but I cannot leave the SSH session open because of internet connections issues I have.
I doubt that running the script with the background operator, the ampersand (&
), will do the trick, because I tried it and later found that process was not completed. How can I logout and keep the process running?
The best method is to start the process in a terminal multiplexer. Alternatively you can make the process not receive the HUP signal.
A terminal multiplexer provides "virtual" terminals which run independent from the "real" terminal (actually all terminals today are "virtual" but that is another topic for another day). The virtual terminal will keep running even if your real terminal is closed with your ssh session.
All processes started from the virtual terminal will keep running with that virtual terminal. When you reconnect to the server you can reconnect to the virtual terminal and everything will be as if nothing happened, other than the time which passed.
Two popular terminal multiplexers are screen and tmux.
Screen has a steep learning curve. Here is a good tutorial with diagrams explaining the concept: http://www.ibm.com/developerworks/aix/library/au-gnu_screen/
The HUP signal (or SIGHUP) is sent by the terminal to all its child processes when the terminal is closed. The common action upon receiving SIGHUP is to terminate. Thus when your ssh session gets disconnected all your processes will terminate. To avoid this you can make your processes not receive SIGHUP.
Two easy methods to do so are
nohup
anddisown
.For more information about how
nohup
anddisown
works read this question and answer: https://unix.stackexchange.com/questions/3886/difference-between-nohup-disown-andNote: although the processes will keep running you can no longer interact with them because they are no longer attached to any terminal. This method is mainly useful for long running batch processes which, once started, no longer need any user input.
There are a few ways to do this, but the one I find most useful is to use GNU Screen.
After you ssh in, run
screen
. This will start another shell running within screen. Run your command, then do a Ctrl-a d.This will "disconnect" you from the screen session. At this point, you can log out or do anything else you'd like.
When you want to re-connect to the screen session, just run
screen -RD
from the shell prompt (as the same use user that created the session).In
bash
, thedisown
keyword is perfectly suited to this. First, run your process in the background (either use&
, or^Z
then typebg
):By typing
jobs
you can see that the process is still owned by the shell:If you were to log out at this point, the background task would also get killed. However, if you run
disown
, bash detaches the job and allows it to continue running:You can confirm this:
You can even combine the
&
anddisown
on the same line, like:This is better than running
nohup
in my opinion because it doesn't leavenohup.out
files littered all over your filesystem. Also,nohup
must be run before you run the command —disown
can be used if you only decide later on that you want to background and detach the task.The tool nohup, available on most Linux boxes will do this.
Just to be thorough, I'll point out tmux, which has the same basic idea as screen:
It is, however, approximately infinitely easier to search for on Google.
Screen is overkill for just keeping processes running when you logout.
Try dtach:
Here's a way to daemonize any shell process, no external programs needed:
When you then close your session, the job will continue to run as evidenced by the output.txt file (which has buffering so it takes a while to show non-zero). Don't forget to kill your job after testing.
So all you need to do is close stdin and background the job. To be really good, first
cd /
so you don't hold on to a mount.This works even in simple sh under Solaris.
byobu
on Ubuntu is a nice front-end to screen. By pressing Ctrl-? you get a list of all the keyboard shortcuts. It adds a status bar that can be useful for watching CPU load, disk space, etc. Overall it provides an experience that I would describe as a terminal based VNC connection.nohup
allows starting a job in the background with its output routed to a log file, which can always be redirected to /dev/null if not required.The
at
command can be useful for this kind of situation. For example, type:And you can then enter a command or series of commands that will be run. The results should be e-mailed to you, if e-mail is set up correctly on the machine.
Instead of
now
, you can specify a time optionally with a date, or an time expression likenow + 15 minutes
. Seeman at
for more details.