We have moved one of our sites to a new server, which once in a while completely hangs. At first this happened very rarely, maybe once per month, but now it's starting to happen every day. By hangs, I mean that all the requests time out, and trying apache2ctl status, or apache2ctl graceful in a shell results in apache2ctl waiting indefinitely. Only issuing a apache2ctl restart brings it out of this state. I can't seem to find anything in the apache error logs, the access logs indicate that apache is indeed not serving requests.
Currently the server is mostly serving a single site, a rapidshare-like file sharing portal, serving around 1TB data. It is running ubuntu lucid, has 8GB of ram and a quad core intel xeon cpu, with apache-mpm-itk and PHP 5.2.11.
It has munin running on it, and at the times when apache is hanging, it is showing a very high number of threads (650+). The usual number of threads is around ~200, and it goes to 650 in about 1 hour of time. Running ps at this time reveals that it is indeed apache who has a lot of children running. CPU load/memory usage doesn't seem to be a problem, at the moment it's using only 1.1GB of RAM, and load is 0.05.
I have been told that Lighttpd, or nginx would have been a much better solution for this kind of site. I'm willing to switch, but I'd rather try and make it work with this setup. (the old server was running apache too)
Was mpm-itk a bad idea? I mostly chose it for the security of having each virtualhost run under a different user. It's running on the default configuration pulled from the package manager, which is bad I know, but I'm not sure which values I should tune.
I'm posting some values that might be relevant, please ask for anything else if needed:
Timeout 300
KeepAlive on
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
</IfModule>
So, what values should I be looking at? Is there anything particular that I should grep the logs for?