The problem I think is somewhat similar to this thread.
It doesn't matter if I have swap enabled or disabled, whenever the real used RAM amount starts going close to maximum and there is almost no space left for disk cache, system becomes totally unresponsive.
Disk is spinning wildly, and sometimes after long waits 10-30 minutes it will unfreeze, and sometimes not (or I run out of patience). Sometimes if I act quickly I can manage to slowly open console and kill some of ram eating applications like browser, and the system unfreezes almost instantly.
Because of this problem I almost never see anything in the swap, only sometimes there are some few MB there, and then soon after this problem appears. My not so educated guess would be that it is connected somehow to the disk cache being too greedy, or memory management too lenient, so when the memory is needed it is not freed quickly enough and starves the system.
Problem can be achieved really fast if working with large files (500MB+) which are loaded in disk cache and apparently afterwards system is unable to unload them fast enough.
Any help or Ideas will be greatly appreciated.
For now I have to live in constant fear, when doing something computer can just freeze and I usually have to restart It, if it is really running out of ram I would much more like it to just kill some of userspace applications, like browser (preferably if I could somehow mark which to kill first)
Although the mystery is why doesn't swap save me in this situation.
UPDATE: It didn't hang for some time, but now I got several occurrences again. I am now keeping ram monitor on my screen at all times and when the hang happened it still showed ~30% free (Used by disk cache probably). Additional symptoms: If at the time I am watching video (VLC player) the sound stops first, after a few seconds the image stops. While the sound has stopped I still have some control over PC, but when the image stops I cannot even move the mouse anymore, so I restarted it after some waiting. Btw, this didn't happen when I started to watch the video but some time in (20min) and I didn't actively do anything else at the time, even though browser and oowrite were open on the second screen the whole time. Basically something just decides to happen at one point and hangs the system.
As per request in the comments I ran dmesg right after the hang. I didn't notice anything weird, but didn't know for what to look, so here it is:
https://docs.google.com/document/d/1iQih0Ee2DwsGd3VuQZu0bPbg0JGjSOCRZhu0B05CMYs/edit?hl=en_US&authkey=CPzF7bcC