Home / server / Questions / 283129
Accepted
Chris Lercher
Asked: 2011-06-23 10:12:49 +0800 CST

Why do consoles sometimes hang forever when SSH connection breaks?

  • 772

I've seen this with so many consoles (on Linux, Mac, ...), and with lots of different machines in many different networks. I can never pinpoint the exact reason, why this happens: All you have to do is log in to a machine via SSH. If the connection breaks for some reason (for simplicity, let's say the network cable was pulled), then sometimes the console just hangs forever - at other times, it just exits fine to the parent shell.

It's so annoying when this happens (e.g. you lose the command history.) Is there maybe a secret keyboard shortcut which can force an exit (Ctrl-C or Ctrl-D don't work)? And what's the reason for this random "bug" across all the implementations anyway?

connection networking ssh console

6 Answers

  1. Best Answer

    There is a "secret" keyboard shortcut to force an exit :~) From the frozen session, hit these keys in order: Enter~. The tilde (only after a newline) is recognized as an escape sequence by the ssh client, and the period tells the client to terminate it's business without further ado.

    The long-hang behavior on communication issues is not a bug, the SSH session is hanging out hoping the other side will come back. If the network breaks, sometimes even days later you can get an SSH session back. Of course you can specifically tell it to give up and die with the sequence above. There are also various things you can do such as setting keep-alive timeouts in your client so that if it doesn't have an active link for a certain amount of time it shuts off on it's own, but the default behavior is to stay as connected as possible!

    Edit: Another useful application of this interrupt key is to get the attention of the local ssh client and background it to get back to your local shell for a minute —say to get something from your history— then forground it to keep working remotely. Enter~ Ctrl+Z to send the ssh client to the background job queue of your local shell, then fg as normal to get it back.

    Edit: When dealing with nested SSH sessions, you can add multiple tilde characters to only break out of one of the SSH sessions in the chain, but retain the others. For example, if you're nested in 3 levels, (i.e. you ssh from local->Machine1->Machine2->Machine3), Enter~. will get you back to your local session, Enter~~. will leave you in Machine1, and Enter~~~. will leave you in Machine2. This works for other escape sequences as well, such as moving the ssh session to background temporarily. The above works for any level of nesting, by just adding more tilde's.

    Finally, you can use Enter~? to print a help menu of available escape commands.

    TL;DR - the supported escape commands are Supported escape sequences:

     ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)
    • 215

  2. SSH offers a keep-alive facility. Add the following to your local ~/.ssh/config (create if it doesn't exist):

    ServerAliveInterval 15
ServerAliveCount 3

    This setting will establish a keep-alive signal sent every 15 seconds through the secure tunnel. After three consecutive failures, the SSH client will exit.

    Note that on some systems (including macOS 10.14) instead it needs to be:

    ServerAliveInterval 15
ServerAliveCountMax 3

    Taken from this answer on ask.ubuntu: https://askubuntu.com/a/29967/30266

    • 18

  3. The fact that it hangs is a function of TCP, not SSH. The application has no way of knowing that the TCP session/connection has been severed unless TCP informs the application using the connection that the connection no longer exists. From the perspective of each host, the TCP session is still in the Established state and there's nothing to say that a long idle (no data flowing) session is not valid other than a RST or a lack of response to a TCP keepalive packet (which is not universally implemented). That doesn't appear to be a bug in SSH to me, I'd expect that behavior.

    • 10

  4. If the connection is properly hung any magic keystroke will not get through as the connection is already hung before you issue the key presses. You can tell the client to terminate but this won't affect the history kept (or not) at the server end.

    While this doesn't actually answer the question it may help reduce the effects of a connection hang: when I'm working remotely (and even usually when I'm not) I run through screen (with or without the byobu wrapper depending on its availability) so that if there is any sort of connection drop my session, with all its history, is preserved and available in the state I left it when I reconnect.

    • 5

  5. If you are on a windows machine and using PuTTY, you can go to Connection tab on the left and set Seconds between keep alive for this. eg: 15 secs.

    • 1

  6. In my case problem was in large MTU size. You can change MTU on router if you using NAT, but I change MTU on server:

    sudo /sbin/ifconfig eth0 mtu 1036
sudo /etc/init.d/networking restart

    On Windows you can also increase this key:

    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
"TcpMaxDataRetransmissions"=dword:00000010
    • 0