Home / ubuntu / Questions / 1169211
In Process
ironstorm
Asked: 2019-08-29 12:28:09 +0800 CST

How can I fix nohup, byobu, tmux, screen, etc to not die when I disconnect from SSH?

  • 772

I'm running KDE Neon which is Ubuntu 18.04 Bionic (LTS) with the latest KDE desktop packages on top of it.

Since upgrading to this from Xenial I have lost the ability for processes to survive SSH session disconnects... specifically nohup, tmux, screen, byobu and all their children are killed when disconnecting.

Here's a simple test to demonstrate the problem

# connect over ssh
cd /tmp
nohup watch date &
ps -ef | grep watch
# disconnect and reconnect
ps -ef | grep watch # process is gone

I tried reverting tmux and byobu to older versions before discovering nohup was also effected. This leads me to believe that it's either SSHd config or systemd.

Apparently systemd decided to change it's default behaviour to nuke all users processes when a user disconnects some time ago. I tried to revert the behaviour by editing /etc/systemd/logind.conf and setting KillUserProcesses=no and then rebooting (service systemd-logind restart didn't work). It didn't work... I'm at my wits end.

Help!

ssh systemd systemd-logind tmux nohup

2 Answers

  1. I just had exactly the same issue, coincidentally also with Ubuntu 18.04 with the KDE Neon packages installed after the fact.

    It turns out that systemd is terminating all of your user processes when your session ends, for example when you terminate your SSH login because you think that you're just going to re-attach later with tmux. ;)

    To fix, do the following steps:

    1. Make the following changes to /etc/systemd/logind.conf:

      KillUserProcesses=no
KillExcludeUsers=root YOUR_USER_NAME

      and restart systemd-logind with:

      sudo systemctl restart systemd-logind

    2. Enable lingering for your account with:

      loginctl enable-linger YOU_USER_NAME

    I initially only had the lingering enabled and the KillUserProcesses=no, but my tmux still got killed. Only after I modified KillExcludeUsers (thanks to https://askubuntu.com/a/1097134/59971 ) could I continue my neural network training in peace.

    • 4

  2. You want to use the setsid command to make it a session leader.

    nohup means "no hang up" which for the most part just disconnects the various standard input/output streams, but that process is still "owned" by the bash process which is in turn owned by the SSH process. When you disconnect from SSH, all those child processes go with it.

    By using setsid you re-parent the process outside of the scope of SSH.

    • 0