Pete's questions

Question

I am suspicious of an unexplained 1600% increase in traffic and massive slow-down that lasted about 10 minutes. I'm not sure if it was an attempted DoS attack, dictionary login attack, etc. Regardless, what actions should I take to monitor my server (which logs should I look at, what tools should I use, etc.) to make sure nothing nefarious happened? What steps should I take during future slowdowns such as these? Is there a standard way to have the server alert me during such a surge in traffic?

All the gory details:

One of my clients reported an unresponsive website (Ruby on Rails via Apache, Mongrel, and mongrel_cluster on a CentOS 5 box.) around 1:00 today.

I was in full troubleshooting mode when I got the email at 1:15. It was indeed exceptionally slow to ssh and load web pages, but ping output looked fine (78 ms), and traceroute from my workstation in Denver showed slow times on a particular hop mid-way from Dallas to the server in Phoenix (1611.978 ms 195.539 ms). 5 minutes later, the website was responsive & traceroute was now routing through San Jose to Phoenix. I couldn't find anything obviously wrong on my end--the system load looked quite reasonable (0.05 0.07 0.09) and I assumed it was just a networking problem somewhere. Just to be safe, I rebooted the machine anwyay.

Several hours later, I logged to Google Analytics to see how things looked for the day. I had a huge spike in hits: Usually this site averages 6 visits/hour, but at 1:00 I got 130 (a 1600% increase)! Nearly all of these hits appear to come from 101 different hosts spread across the world. Each visitor was on the website for 0 seconds and each visit was direct (i.e. it's not like the web page got slashdotted) and each visit was a bounce.

Ever since about 1:30, things are running smooth and I'm back to the average 6 visits per hour.

Disclaimer:

I am a code developer (not a sysadmin) who must maintain web servers for machines that run the code that I write.

I have followed this CentOS 5.2 Mongrel Cluster tutorial for starting mongrel_cluster at reboot on CentOS 5.2. However, I have to manually start mongrel_cluster at reboot.

Since that didn't work, I followed the advice in Set up Mongrel as a service and start it automatically on Centos 5.2. The key bits are:

chkconfig -add mongrel_cluster
chkconfig –level 345 mongrel_cluster on

Again... This sidn't work: I still have to manually start mongrel_cluster at boot. Finally, I tried adding "@reboot ..." to my crontab as so:

RAILS_ROOT=/path/to/root
# Restart Ferret Drb server on host reboot:
@reboot ${RAILS_ROOT}/script/ferret_server --root=${RAILS_ROOT} -e production start

# Restart rails apps (via Mongrel) on host reboot:
@reboot /etc/init.d/mongrel_cluster start

but again... I have to manually restart both Mongrel and Ferret!

Finally, I have followed the How to launch DRb server on reboot (linux) instructions, but I have to manually launch the Ferret DRb at reboot...

This is my first sysadmin gig, so I'm not even sure which logfile(s) to be looking at... If you need more information to help, please help me out & let me know where to look!

I have a CentOS 5.2 box running my production web server (Apache 2 + Mongrel_cluster) for a Ruby on Rails project. The machine is hosted by GoDaddy.com and is a "virtual private server". The system periodically reboots on its own (maybe once every 6 months). I have two questions:

1. How can I determine why my machine rebooted?
2. Is there a good way to automagically notify me (i.e. via e-mail) when the system reboots?