netstat shows that there are 153 connections are on status CLOSE_WAIT. The connections never gets closed. So overtime the server is filled with these connections which fills RAM and now the websites are not loading.
netstat shows many like the following:
tcp 160 0 my_server_name:http my_server_name:51584 CLOSE_WAIT
tcp 160 0 my_server_name:http my_server_name:51586 CLOSE_WAIT
tcp 0 0 my_server_name:http my_server_name:50827 CLOSE_WAIT
tcp 0 0 my_server_name:http my_server_name:50830 CLOSE_WAIT
tcp 312 0 my_server_ip.static.:http rate-limited-proxy-72:61249 CLOSE_WAIT
tcp 382 0 my_server_ip.static.:http b3090792.crawl.yahoo.:58663 CLOSE_WAIT
tcp 382 0 my_server_ip.static.:http b3090792.crawl.yahoo.:34655 CLOSE_WAIT
tcp 382 0 my_server_ip.static.:http b3090792.crawl.yahoo.:56681 CLOSE_WAIT
tcp 382 0 my_server_ip.static.:http b3090792.crawl.yahoo.:40829 CLOSE_WAIT
tcp 576 0 my_server_ip.static.:http b3090792.crawl.yahoo.:38278 CLOSE_WAIT
tcp 47 0 my_server_ip.static.:http 203.200.5.143.ill-bgl:49379 CLOSE_WAIT
If I look at the appache error_log, before the CLOSE_WAIT situation comes there are lines like the following
[warn] child process 15670 still did not exit, sending a SIGTERM
[error] child process 15670 still did not exit, sending a SIGKILL
[notice] child pid 3511 exit signal Segmentation fault (11)
My setup Apache 2.2.3 RAM 1024 MB (burst 2048 MB) CentOS release 5.3 (Final) running 2 WPMU 2.9.2 installations