I recently moved my site to a new server (Apache 2, PHP5, MySQL5). The site is an Invision based forum. Every few posts / topics it just hangs. The data has been written because if you stop and reload, the post / thread is there. I thought it was a write issue initially, but nope. The data is written but the page load never completes. It doesn't leave the page where the data has been input.
What's the best way to troubleshoot this issue? The only thing I have done recently is reduce my MySQL timeouts, but I can't see that being an issue as the values are still big enough and there are no mentions of timeouts in the MySQL log. For the record there is nothing in PHP's error log either.
EDIT: I checked my server-status. It all looked ok, but I have a suspicion I was hitting my ServerLimit, so I doubled that. Also enabled my Keepalives. Will keep an eye on it.
EDIT 2: It's now been a few days and this is still occuring. I have more info though;
- Apache is throwing seg faults, but enabling core dumps does not produce them.
- I have tried disabling the modules in apache but it just stops things from working.
- I fear it may actually be DNS related. If I watch Live Headers in Firefox, absolutely nothing happens during this 'hanging' period. After that, the responses come back fairly promptly.
UPDATE (05/04): I built the latest versions of Apache and PHP from source, no luck. I then removed those and used the remi repo to update all my packages to the latest stable. Segfaults seem to have stopped, but the hanging is continuing.
ini
's are at:
- www.skylinesaustralia.com/php.ini
- www.skylinesaustralia.com/my.cnf
- www.skylinesaustralia.com/httpd.conf
UPDATE - SOLVED! - The issue was having a gigantic query cache size in MySQL. It was 2GB, changing it to 64M sorted it.
There's your problem. Apache isn't hanging, your browser is hanging when apache's child dies and doesn't complete the request.
LogLevel debug may give you a bit more information (though, a ton of information if this is a busy production system).
My first guess would be suhosin/mod_security or something that is compiled against the wrong php api or possibly using shared libraries that have changed slightly. I'd doublecheck those and if you can isolate the different php extensions/apache modules that are running. Are you running apcc, eaccelerator or any opcode cache? Zend? Source Guardian? ioncube? Are you using the right version of each of these? Latest updates? Zend 3.3.9, ioncube had a recent update as well and can cause segfaults depending on the version of Zend you're running.
Bind something like strace or gdb to all of the apache worker processes, and reload pages in the browser one breaks. Then you'll get to see more info about what it's doing when it's crashing.
When you say that you've enabled coredumps, did you also enable them to be large enough to handle an Apache coredump? It's likely to be fairly large, so even setting them to unlimited-size temporarily might help.
Here is a few tips considering the high numbers of possibilities and low amount of details.
If nothing of this work, please post content of php.ini, httpd.conf and my.cnf.
If you use firefox install the Firebug plugin. Enable it by clicking the bug icon in the lower right hand corner of the browser and then set "Enable" in the Net tab. Load up a page of your forum and check for errors.
There's a good chance it's a serious issue with the servers OS or hardware.
I'd try to do a verify of the os (Debian based linuxes use
debsums
, not sure for anything else).Also try a run of
memtestx86+
which should confirm that there's no massive memory issue, they can show up only under load.I seen similar issues related to DNS, any chance you'd post the site URL?
EDIT: Just saw you mentioned that already, run the site through http://www.intodns.com/ it could also be related to the servers hostname. Like I said if you don't mind post the domain it may be easier.
Try stopping Apache, then restart it with: strace /path/to/httpd -X (only one worker will be started, and Apache won't detach from the console.) Reproduce the hang and inspect the output of strace. That should localize what part of the system is crashing.
Try reducing the query cache size in MySql.