I was testing how an Ubuntu 16.04 machine handles low memory conditions and had several suspended jobs that that used up almost all the RAM on the server. I was still able to run normal bash commands when I left work last night. In the morning (without anyone doing anything to the server) all commands (ps, free, ls, etc.) gave the following error:
-bash: fork: Cannot allocate memory
I was eventually able to run jobs and kill %1 to recover the machine. It was lucky that I kept an SSH session open, because further SSH connections failed (the client simply didn't show a command prompt after authentication).
The fact that regular userland programs can effectively bring down the entire server to the point where, if there wasn't an existing SSH session, it would have to be hard-rebooted, is highly concerning! My understanding was that Linux (or any OS, really) should kill processes to free up RAM before the entire system gets to such an unusable state. How do I make it do that?
0 Answers