The problem I'm experiencing is quite simple: while the memory used by applications is quite small (~2GB), at the same time Task manager reports that the server has basically all memory in use (16GB). During this it starts swapping, performance goes through floor and latency goes through the roof.
I diagnosed this back to an application that opens many files using mapped memory access (probably in some not-quite-standard way, as it's written in Delphi). RamMap from Sysinternals reports that those 14GB are used for "Mapped File", in Active state. Thing is, this memory is not actively used, those applications are in stand-by mode, with no clients attached. As idle as they can be.
To bring memory usage back to normal levels I just need to use the "Empty→Empty System Working Set" option in RamMap. This fixes the issue of no free memory for as long as there is no new activity (reading from data files, etc.). Then the memory usage goes up and performance goes down again.
The solution would be to write a script to do the same thing RamMap does every few minutes. That's where I'm stuck. I found EmptyWorkingSet WinAPI function and found PowerShell script to run it on all processes in the system, but it does reduce used memory by at most 200-300MB not few GB RamMap is able to.
How to run RamMap Empty System Working Set option regularly?
Problem was caused by application using FILE_FLAG_RANDOM_ACCESS. I fixed the application.
If someone posts a generic workaround I'll reassign the correct answer to his answer.