I'm seriously in need of help. My sites are now nearly impossible to use because of massive loads on my server. I'm already a month late on my mortgage and this really isn't helping my situation. I've been working on fixing this intermittent load problem for months (never this bad).
I'm suspecting some kind of attack since I'm under DDOS attack a lot! I've been trying to figure out what is causing the load but I'm afraid I just don't have the experience or knowledge to understand all the data I've been looking at. I don't even know where to begin or how to test for the large array of attacks out there.
Here's some data you might find useful...
Server: Xeon X3220 Quad Core 2.4 GHz - Linux, FreeBSD 500 GB HD and 8 Gig of Ram. Runs Centos release 5.7 Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_qos/9.74
Warning: All sites are softcore adult sites - mostly fantasy art like elves and amazons.
1) Sites may run fine for weeks or just days at less than 10 load then start jumping to 40-80 load - no idea why. Same sites, same mods, same amount of traffic - just WHAM!
2) I get an email almost every day that says: "Large Number of Failed Login Attempts from IP (different each time)". My webhost (who almost never helps me) told me it was a udp flood or something.
3) I've changed the port for MySQL from the default. If I ever put it back to the default - I get Loads of over 100 from what must be a constant mysql port flood.
4) I've reconfigured MYSQL. Link: http://www.deadlyamazons.com/logs/mycnf.txt
5) I have 3 Joomla Jomsocial networks. I've spent a couple weeks turning all the mods/plugins off, waiting a day and then turning them back on the next day or later if there isn't any change (there hasn't been). For example, on Thursday I'll turn off videos, on Friday I'll turn off chat.. etc and nothing changes the load appreciably.
6) Joomla info: All SEF turned off - sh404sef completely disabled and removed. Components: Joomla 1.5.22, Jomsocial 2.0.5, Kunena 1/31/2011, HWDMediashare 11/22/2010 and JBolo Chat 2.7.3, Comet Chat or Envolve Chat. Page Compression is on, Cache is on 15 mins.
Please click on this forum to see links to all my reports: http://forum.joomla.org/viewtopic.php?f=433&t=706035&p=2777500#p2777500
4/9/12 - Added this part:
Hi guys I'm back with some more info about my poor server. The server is currently limping along with a load of between 20 and 60 averaging about 30.
I'll add an incentive to solving my problem: $100 via Paypal for an answer that solves the 'load' problem without the suggestion of buying 1 or more extra servers. Again, these sites worked fine with even higher traffic on a lower powered server.
I just recompiled apache 2.22 adding eaccelerator and zend-optimizer - no change. The other mod I included was QOS which keeps the # of connections at a lower level. I've had QOS working for awhile.
Suggestions and Requests:
Yes I did turn off the port to MySQL I should've mentioned that.
Traffic stats:
March bandwidth: 579.19G
KBytes Mar 2012: 3,194,134,948 | Dec 2011: 3,504,864,832
Visits Mar 2012: 920,619 | Dec 2011: 727,843
Pages Mar 2012: 10,231,430 | Dec 2011: 10,830,700
Files Mar 2012: 89,218,232 | Dec 2011: 102,862,958
Hits Mar 2012: 106,515,577 | Dec 2011: 120,884,007
Videos of Top -C during high load: Here are AVIs of 2 'Top -c's that I took when the server was running between 30 and 40 load.
Download 1.5 minute / 30M clip: http://www.mediafire.com/?yk3b5xota7l7s30
Download 30 second / 10M clip: http://www.mediafire.com/?4c2t37i8gmd189w
Videos of MySQL Processlist during high load Here are AVIs of 2 'Show Processlist' inside CPanel when the server was running between 60-30 load.
Download 2 minute / 40M clip: http://www.mediafire.com/?ymmfe8599bx11ho
Download 30 second / 10M clip: http://www.mediafire.com/?e675p3p1f0l65jt
DStat stats: 4 Sceeencaps taken in a row... Links:
http://www.deadlyamazons.com/logs/dstat01.jpg
http://www.deadlyamazons.com/logs/dstat02.jpg
http://www.deadlyamazons.com/logs/dstat03.jpg
http://www.deadlyamazons.com/logs/dstat04.jpg
Stats placed on Joomla Board that couldn't be seen earlier:
netstat -alntp | grep :80 | wc -l (1586)
netstat -n | grep :80 | grep SYN |wc -l (30)
netstat -anp |grep .tcp\|udp. | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n (nothing)
netstat -alntp | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n - Report: http://www.deadlyamazons.com/logs/netstat_alntp_awk_print_5_2.txt
netstat -alntp | grep :80 - http://www.deadlyamazons.com/logs/netstat_alntp_grep_80_2.txt
Top (command line) http://www.deadlyamazons.com/logs/top01_cli.jpg Top 2 (command line) http://www.deadlyamazons.com/logs/top02_cli.jpg Top (WHM) http://www.deadlyamazons.com/logs/top01_whm.jpg Top 2 (WHM) http://www.deadlyamazons.com/logs/top02_whm.jpg IOStat (command line) http://www.deadlyamazons.com/logs/iostat.jpg Daily Process Log (WHM) http://www.deadlyamazons.com/logs/daily_process_log.jpg Process Trace MYSQL (txt) (HUGE!) http://www.deadlyamazons.com/logs/trace_mysql.txt Process Trace MYSQL (rtf) http://www.deadlyamazons.com/logs/trace_mysql.rtf Process Trace sxyamzn (txt) http://www.deadlyamazons.com/logs/sexyamazonscom_indexphp.txt Process Trace sxyamzn (rtf) http://www.deadlyamazons.com/logs/sexyamazonscom_indexphp.rtf Process Trace sleepps (txt) http://www.deadlyamazons.com/logs/sleeppeepscom_indexphp.txt Process Trace sleepps (rtf) http://www.deadlyamazons.com/logs/sleeppeepscom_indexphp.rtf
Any help would be appreciated.