It happens pretty often to me when I am compiling software in the background and suddenly everything starts to slow down and eventually freeze up [if I do nothing], as I have run out of both RAM and swap space.
This question assumes that I have enough time and resources to open up Gnome Terminal, search through my history, and execute one sudo
command.
What command can save me from having to do a hard reboot, or any reboot at all?
In my experience Firefox and Chrome use more RAM than my first 7 computers combined. Probably more than that but I'm getting away from my point. The very first thing you should do is close your browser. A command?
I've tied the most popular browsers together into one command there but obviously if you're running something else (or know you aren't using one of these) just modify the command. The
killall -9 ...
is the important bit. People do get iffy aboutSIGKILL
(signal number 9) but browsers are extremely resilient. More than that, terminating slowly viaSIGTERM
will mean the browser does a load of cleanup rubbish —which requires a burst of additional RAM— and that's something you can't afford in this situation.If you can't get that into an already-running terminal or an Alt+F2 dialogue, consider switching to a TTY. Control + Alt + F2 will get you to TTY2 which should allow you to login (though it might be slow) and should even let you use something like
htop
to debug the issue. I don't think I've ever run out of RAM to the point I couldn't gethtop
up.The long term solution involves either buying more RAM, renting it via a remote computer, or not doing what you're currently doing. I'll leave the intricate economic arguments up to you but generally speaking, RAM is cheap to buy, but if you only need a burst amount, a VPS server billed per minute, or hour is a fine choice.
On a system with the Magic System Request Key enabled, pressing Alt + System Request + f (if not marked on your keyboard, System Request is often on the Print Screen key) will manually invoke the kernel's out of memory killer (oomkiller), which tries to pick the worst offending process for memory usage and kill it. You can do this if you have perhaps less time than you've described and the system is just about to start (or maybe has already started) thrashing - in which case you probably don't care exactly what gets killed, just that you end up with a usable system. Sometimes this can end up killing X, but most of the time these days it's a lot better at picking a bad process than it used to be.
Contrary to other answers, I suggest that you disable swap while you are doing this. While swap keeps your system running in a predictable manner, and is often used to increase the throughput of applications accessing the disk (by evicting unused pages to allow room for the disk cache), in this case it sounds like your system is being slowed down to unusable levels because too much actively used memory is being forcibly evicted to swap.
I would recommend disabling swap altogether while doing this task, so that the out-of-memory killer will act as soon as the RAM fills up.
Alternative solutions:
zswap
in the kernel. This compresses pages before they are sent to swap, which may provide just enough wiggle room to speed your machine up. On the other hand, it could just end up being a hindrance with the extra compression/decompression it does.tcc
), at the expense of a slight runtime performance hit to the compiled product. (This is usually acceptable if you're doing this for development/debugging purposes.)You can use the following command (repeatedly if needed) to kill the process using the most RAM on your system:
With:
ps -eo pid --no-headers --sort=-%mem
: display the process ids of all running processes, sorted by memory usagehead -1
: only keep the first line (process using the most memory)xargs kill -9
: kill the processEdit after Dmitry's accurate comment:
This is a quick and dirty solution that should be executed when there are no sensitive tasks running (tasks that you don't want to
kill -9
).Before running your resource consuming commands, you could also use the setrlimit(2) system call, probably with the
ulimit
builtin of your bash shell (or thelimit
builtin in zsh) notably with-v
forRLIMIT_AS
. Then too big virtual address space consumption (e.g. with mmap(2) or sbrk(2) used by malloc(3)) will fail (with errno(3) beingENOMEM
).Then they (i.e. the hungry processes in your shell, after you typed
ulimit
) would be terminated before freezing your system.Read also Linux Ate My RAM and consider disabling memory overcommitment (by running the command
echo 0 > /proc/sys/vm/overcommit_memory
as root, see proc(5)...).In that case, something like "killall -9 make" (or whatever you are using to manage your compilation, if not make). This will stop the compilation proceeding further, will SIGHUP all the compiler processes launched from it (hopefully causing them to stop as well) and, as a bonus, doesn't need sudo assuming you're compiling as the same user you're logged in as. And since it kills the actual cause of your problem instead of your web browser, X session or some process at random, it won't interfere with whatever else you were doing on the system at the time.
Create some more swap for yourself.
The following will add 8G of swap:
It will still be slow (you are swapping) but you shouldn't actually run out. Modern versions of Linux can swap to files. About the only use for a swap partition these days is for hibernating your laptop.
One way to get a chunk of free RAM on a short notice is to use zram, which creates a compressed RAM disk and swaps there. With any half-decent CPU, this is much faster than regular swap, and the compression rates are pretty high with many modern RAM hogs like web browsers.
Assuming you have zram installed and configured, all you have to do to is run
Another things that one could do is to free up memory page cache via this command:
From kernel.org documentation (emphasis added):
sudo swapoff -a
will disable the swap, making the kernel automatically kill the process with the highest score if the system runs out of memory. I use this if I know I'll be running something RAM-heavy that I'd rather kill if it goes out of control than let it go into swap and get stuck forever. Usesudo swapon -a
to re-enable it afterwards.Later, you may want to take a look at your swap settings. Sounds like your swap is on the same disk as the root partition, which would slow down your system when you hit swap, so avoid that if you can. Also, in my opinion, modern systems often get configured with too much swap. 32GiB RAM usually means 32GiB swap is allocated by default, as if you really want to put 32GiB into your swap space.