Home / server / Questions / 152079
Accepted
Sampson
Asked: 2010-06-18 04:42:20 +0800 CST

Server goes offline. What to look for?

  • 772

I'm using a new virtual server through GoDaddy, and this morning I received a call from the powers that be informing me our website was offline. After confirming this, I requested a power cycle through our GoDaddy control panel, and within a minute or two the server was back online. I made the call, and reported the news that we're back up.

Of course, a couple minutes later we're down again. I tried connecting through PuTTy, and it takes forever to prompt me for a username, and each successive prompt takes a long time to come up. I'm using CentOS. So my questions are:

  1. How can I determine the cause?
  2. What types of things can I do to prevent this in the future?

One interesting, and perhaps relevant, observation is that yesterday our bandwidth consumption was about 20% greater than our top figures from the past month.

centos troubleshooting

3 Answers

  1. Best Answer

    Perhaps the server is either being slashdotted or has a DOS attack against it.

    What's likely happening, is that Apache is using up way too much RAM, and swapping like crazy. Once it starts swapping, it's in a spiral of death (Since every new request after it starts swapping with take an exponentially longer time to complete). The only way to save it is to bounce Apache before it gets too deep into swap.

    To fix that problem, you need to tune Apache not to launch too many threads/worker processes. See the documentation on it. Either that, or go to a lighter weight web server (something like Lighttpd or Nginx).

    I'd recommend setting up a network monitor as well. I personally use Nagios and Munin to monitor all of my servers/services. Nagios provides me with alerts and warnings when resources become low or processes/servers go off line. Munin records historical information (so if you know it went down 20 minutes ago, you can see what changed up to the point it went down). You NEED both style monitoring systems if you want to effectively manage a production server (IMHO at least). That way you don't need to worry about relying on your host for anything other than the service...

    • 1

  2. At my web server we config monit it will send a SMS to admin monit&

    tracking

    • 0

  3. Ask the host for a printout of CPU and IO acitivty for your server, so you can see how much server side resources were used. Especially as you can not log in once it goes down - the VM side will have nice graphs about what it CAN see: network, disc IO and CPU.

    • 0