I recently made the move to The Cloud from a bare metal server for personal use and I couldn't be happier. Except for one thing: My server crashes about once a week.
Rackspace has been really great and extremely helpful and I wish to stay with them and recommend them to others. But with this issue I have been told that the server is just running out of memory and I need to look into it. While I'm fine with that and this does seem like my issue, I've never heard of a Linux machine crashing from running out of memory. In my experience it would, worst case scenario, run really slowly or the kernel would start killing processes.
Some Details:
- Running Gentoo Linux (up to date)
- 512MB RAM, 1G Swap
- Services installed & running:
- 2 Apache 2 procs (1 minimal, for serving static/cached resources and proxying; 2nd has cgi, mod_perl and mod_jk)
- 2 Tomcat Instances (1 has 2 apps I made, the other is just for Nexus)
- Mysql
- When the crash happens:
- server can be pinged
- ssh connection hangs indefinitely
- console will allow username but password prompt never comes up. after 60 seconds it times out and I'm presented with another login prompt. repeat.
- services (http, tomcat, mysql) cannot be contacted; The connection will hang and not report 404, server not found, etc.
- Softboot will not work, hard boot needs to be performed often more than once (even with a 30 min wait between reboot attempts)
I've run this same setup on servers throughout the years but with only 256MB RAM. The only difference here is I'm on a virtual machine.
My question could take one of two forms: Has anyone had a similar problem with Rackspace cloud or other cloud hosts? (and if so, what was the solution?) or; What's a good way to track down my issue? I setup a cron that outputs uptime
and free
to a file every minute so I can examine it after the next crash but that seems hacky.
Full Disclosure: I'm a software developer by trade so that's where most of my experience is, but I have about 15 years experience using Linux for desktop and servers both for personal and professional use.
Another great way to track your memory usage would be to install
sar
on your Linux box. If you're using Debian it can be listed assysstat
. Usingsar -r
will give you a picture of what's going on in your memory and network statistics.