Is there a way to reserve a specific amount of memory for a specific service/process?
I would like to guarantee that OpenSSH always has enough memory available to it that it can accept a new SSH connection in case the server's memory fills up.
Thanks.
Generally speaking you don't "reserve" memory on *nix systems - you ensure that there is enough physical RAM + Swap space to handle what your server is doing, and you let the VM subsystem handle it from there (though as dmckee pointed out in Linux you have the "helpful" OOM killer which will kill the "biggest memory user" when you run out of RAM -- the logic of the aforementioned "helpful" subsystem is suspect at best).
You should try to figure out why your system is running out of memory (if that's actually the problem): If you fix the underlying problem you won't need to reserve RAM.