When a server uses memory more than available RAM, the system will shut down the virtual machine. Then, it is only possible to boot from outside (VPS control panel, e.g. vePortal or SolusVM). However, it should be possible to plan a reboot before possible shut down.
What is the best practical method to check the used memory, and reboot the system upon reaching e.g. 90% of the allowed RAM? Is there a common program or script to do so? I am using Debian/Ubuntu.
Nagios could do it, but Monit is pretty much designed for exactly this problem and can be configured to restart services or the whole machine in low memory situations.
I use it on several servers to check for run away processes using excessive memory and restart them. I find it a lot easier to configure than nagios actions.
It's also useful for checking if services have failed and restarting them.
The relevant part of monitrc would look like:
Nice and simple :)
Instead of cutting the branches, you should cut the root of problem. Try to analyze that which process or activity eats up all the RAM. and fix that activity, instead of rebooting server everything when RAM files up 90%. you can use nagis for this purpose, that if RAM gets 90% fill, call an Event handler which will shutdown or reboots the server.
This is not advisable to reboot your VPS using script.
If your server goes out of memory at each boot time then your server never been up.
So, I will suggest you to use VPS management Control Panel for this.