How do I prevent Linux from freezing when out of memory?

I'll bet that the system didn't actually "freeze" (in the sense that the kernel hung), but rather was just very unresponsive. Chances are it was just swapping very hard, causing interactive performance and system throughput to drop like a stone.

You could turn off swap, but that just changes the problem from poor performance to OOM-killed processes (and all the fun that causes), along with decreased performance due to less available disk cache.

Alternately, you could use per-process resource limits (commonly referred to as rlimit and/or ulimit) to remove the possibility of a single process taking a ridiculous amount of memory and causing swapping, but that just pushes you into entertaining territory with processes that die at inconvenient moments because they wanted a little more memory than the system was willing to give them.

If you knew you were going to do something that was likely to cause massive memory usage, you could probably write a wrapper program that did an mlockall() and then exec'd your shell; that'd keep it in memory, and would be the closest thing to "keep a responsive core" you're likely to get (because it's not that the CPU is being overutilised that is the problem).

Personally, I subscribe to the "don't do stupid things" method of resource control. If you've got root, you can do all sorts of damage to a system, and so doing anything that you don't know the likely results of is a risky business.

As mentioned above in comment by Tronic, it is possible to call OOM-killer (out of memory killer) directly by the keyboard combination SysRq-F.

SysRq key is usually combined within PrtSc key on keyboards.

OOM-killer kills some process(-es) and system becomes responsive again. Direct acces to OOM-killer may not be enabled by default, plz checkout this question to findout how to check its status and/or enable it.

PS: This helped me a lot. I agree with opinion that this is the most useful advise about that problem if it caused by Chrome or whatever memory greedy software. But you need to keep in mind that OOM-killer could kill some really important process, use it carefully.

This is a bug known since 2007 - see System freeze on high memory usage.

In this situation, Windows displays a dialog warning the user to close one or more applications.

You can use a daemon like earlyoom that checks Swap and available RAM, you can configure how much memory you want to be available, both RAM and SWAP, then if that treshold happens it kills the largest memory eater, that normally is the guilty eater, you can also have an exception list if you wish so.

If you feel like recompiling the kernel, you could try the patch from the EDIT section of this question: https://stackoverflow.com/q/52067753/10239615
It does not evict the Active(file) pages during high memory pressure and thus it allows OOM-killer to trigger almost instantly because the kernel no longer needs to spend minutes of constant from-disk re-reading of every process's executable code pages causing a frozen OS.

This is something particularly difficult to prevent. It's because the kernel starts swapping. One solution is to turn swap off. When the system runs out of memory, rather than start swapping, the kernel will kill some processes; usually it picks up the correct process to kill, but it is anyway better to kill a random process than to have an unresponsive system.

This can be a particularly good solution for servers, because servers often have enough RAM and when they start to use swap space it means something's wrong anyway. However, desktops usually need the swap space, so I think there's no good solution for desktops. I often turn swap space off in servers, especially when there is suspicion of a memory leak.