I have tried to fix this but don't know enough, apparently. When I run top command I see that its filled with httpd processes. When this happens my site stops loading. It doesn't crash it just spins and spins. I am using prefork MPM. This looks like:
<IfModule prefork.c>
StartServers 1
MinSpareServers 5
#MinSpareServers 1
MaxSpareServers 10
#MaxSpareServers 5
ServerLimit 25
#ServerLimit 50
MaxClients 25
#MaxClients 50
#MaxRequestsPerChild 10000
MaxRequestsPerChild 100
</IfModule>
This problem started with the defaults which are commented out. I then tried to make changes thinking that it would help if I lower the MaxClients because I have about 1gb ram available. I thought the default MaxClients was to high because the average memory usage per httpd process is 55mb. Here is a screenshot of top, keep in mind this doesn't show all the httpd processes.
top - 09:48:27 up 42 days, 3:13, 1 user, load average: 1.09, 1.09, 1.09
Tasks: 35 total, 3 running, 32 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.7%us, 0.5%sy, 0.0%ni, 84.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2097152k total, 1134676k used, 962476k free, 0k buffers
Swap: 0k total, 0k used, 0k free, 0k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1649 root 17 0 315m 122m 14m S 0.0 6.0 1:51.67 java
27755 mysql 18 0 204m 71m 5744 S 4.0 3.5 806:09.08 mysqld
23603 apache 15 0 76404 51m 5900 S 0.0 2.5 0:20.21 httpd
20408 apache 15 0 77124 51m 5252 S 0.0 2.5 0:25.57 httpd
28026 apache 16 0 76788 50m 5240 R 97.7 2.5 0:05.15 httpd
24497 apache 16 0 76272 50m 5484 S 0.0 2.5 0:06.55 httpd
27724 apache 15 0 76380 50m 5268 S 0.0 2.5 0:04.81 httpd
21561 apache 15 0 75672 50m 5532 S 0.0 2.4 0:28.54 httpd
26537 apache 16 0 75116 49m 5488 R 19.9 2.4 0:10.09 httpd
28027 apache 15 0 75080 49m 5216 S 0.0 2.4 0:03.07 httpd
24498 apache 15 0 74180 49m 5656 S 0.0 2.4 0:12.33 httpd
21934 apache 15 0 74568 48m 5256 S 0.0 2.4 0:27.48 httpd
24484 apache 15 0 74152 48m 5260 S 0.0 2.4 0:17.70 httpd
28012 apache 15 0 73248 47m 5208 S 0.0 2.3 0:02.34 httpd
13428 root 18 0 37032 15m 8456 S 0.0 0.8 0:00.21 httpd
What am I doing wrong?
Looks to me like you have a code issue, which you're trying to control through the MPM parameters.
Some of those parameters are counter productive, specilaly MaxClients and MaxRequestPerChild, they won't help you lower the load, on the contrary they'll get it higher.
Also remember that top is showing the virtual memory allocation, in no case that means that each apache is really using 50mb of memory, to get more accurate statistics you can activate mod_status by adding this to your config
Just to give you an example, here's my config on a regular apache server under heavy load
What worries me most is that single httpd instance running at 97% CPU, this shows that the code running on apache is doing something quite nasty, I would rather get that investigated because you're just trying to control the situation (wisely) by lowering your prefork parameters but the only way to solve the issue is to tackle the main problem.
Good luck! Apache code issues are always a pain :/