Recently moved to a new server with CentOS 5.4 and is 64 bit as opposed to our old 32-bit. It's got dual processors now and a bit higher memory.
I realized though, that apache has been heavily eating away at the memory and sometimes it would cause the system to swap.
The current setup has Nginx 1.0.8 in front to serve static content and Apache 2.2 to serve dynamic content (PHP). I also have APC installed. What could be causing this memory usage spike? I tried tuning the config file for apache but it would work if I turn certain directives to a absurdly low number. Before on the old machine I was able to use the default settings with no more than ~260MB of memory usage for all apache processes:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
But now the new machine is using this with high memory usage, it can get as high has 600MB+:
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 20
MaxRequestsPerChild 500
</IfModule>
APC settings are as follows:
extension = apc.so
apc.enabled=1
apc.shm_segments=1
apc.optimization=0
apc.shm_size=64M
apc.ttl=0
apc.user_ttl=7200
apc.num_files_hint=1024
apc.mmap_file_mask=/tmp/apc.XXXXXX
apc.enable_cli=1
apc.cache_by_default=1
Any ideas why this is happening?
Edit: Some more process info
top - 21:18:45 up 10 days, 6:20, 4 users, load average: 0.00, 0.04, 0.06
Tasks: 103 total, 2 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.0%sy, 0.0%ni, 98.8%id, 0.2%wa, 0.0%hi, 0.7%si, 0.0%st
Mem: 1541280k total, 1479804k used, 61476k free, 99812k buffers
Swap: 10482404k total, 25548k used, 10456856k free, 725180k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5613 jenkins 17 0 1144m 184m 13m S 0.7 12.3 2:05.94 java
25306 apache 15 0 541m 125m 69m S 0.0 8.3 0:10.44 httpd
25308 apache 15 0 533m 118m 67m S 0.0 7.9 0:11.01 httpd
25356 apache 16 0 524m 112m 69m S 0.0 7.5 0:08.84 httpd
25305 apache 15 0 510m 103m 69m S 0.0 6.9 0:11.58 httpd
25309 apache 16 0 498m 92m 69m S 0.0 6.1 0:10.35 httpd
25307 apache 15 0 499m 90m 67m S 0.0 6.0 0:09.59 httpd
9759 mysql 15 0 428m 83m 6920 S 0.0 5.6 11:10.01 mysqld
16121 apache 15 0 494m 68m 48m S 0.0 4.6 0:05.50 httpd
2945 user 15 0 659m 18m 6364 S 0.0 1.2 0:00.53 node
25303 root 15 0 425m 12m 6176 S 0.0 0.8 0:00.07 httpd
2944 user 15 0 636m 10m 2288 S 0.0 0.7 0:00.11 node
[user@server~]# ps o rss,command -u apache
RSS COMMAND
70236 /usr/sbin/httpd
105860 /usr/sbin/httpd
128096 /usr/sbin/httpd
92888 /usr/sbin/httpd
121288 /usr/sbin/httpd
94768 /usr/sbin/httpd
115644 /usr/sbin/httpd
If you see the memory usage increasing over time, it might be a memory leak somewhere. You already tried reducing the "MaxRequestsPerChild" parameter - try tuning it down even more within a controlled environment to see if the total memory stops growing. If it does, you are probably seeing either your PHP code, the PHP interpreter or possibly the PHP cache module leaking memory.
If you see memory leaking, you could try some of the suggestions given for this question on how to debug a memory leak with Apache/PHP.