I have a LAMP system with CentOS 6.4, Apache 2.2.15, MySQL 5.5.32 and PHP 5.3.26 with APC 3.1.10 using a Drupal CMS (https) on a virtual machine with 4 CPUs and 6 GB RAM.
My problem is the CPU usage of the apache processes which will be up to 100% if I do a webpage crawler with 10 concurrent requests. What can I do to have less CPU usage?
Any ideas? Here are my top result and config files with a mod_status server-status page:
top - 20:37:23 up 14 days, 22:52, 2 users, load average: 8.54, 6.45, 3.22
Tasks: 180 total, 8 running, 171 sleeping, 0 stopped, 1 zombie
Cpu(s): 93.5%us, 6.2%sy, 0.0%ni, 0.0%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 8061508k total, 7634052k used, 427456k free, 146132k buffers
Swap: 8208376k total, 35116k used, 8173260k free, 5019896k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13155 apache 20 0 472m 95m 41m R 69.8 1.2 0:04.75 httpd
13144 apache 20 0 467m 88m 40m R 61.8 1.1 0:03.49 httpd
13141 apache 20 0 477m 97m 39m R 52.9 1.2 0:06.69 httpd
13162 apache 20 0 449m 71m 41m R 51.5 0.9 0:04.53 httpd
13145 apache 20 0 473m 93m 39m R 30.6 1.2 0:02.13 httpd
13136 apache 20 0 477m 98m 40m R 28.9 1.3 0:04.93 httpd
13158 apache 20 0 459m 79m 38m S 28.9 1.0 0:00.87 httpd
13160 apache 20 0 467m 84m 36m R 24.9 1.1 0:00.75 httpd
3496 mysql 20 0 2454m 374m 5072 S 19.6 4.8 31:10.75 mysqld
13120 apache 20 0 460m 81m 40m S 15.3 1.0 0:05.63 httpd
13148 apache 20 0 462m 82m 39m R 12.0 1.1 0:01.79 httpd
I use this in my httpd.conf:
Timeout 60
KeepAlive On
MaxKeepAliveRequests 128
KeepAliveTimeout 10
<IfModule prefork.c>
StartServers 16
MinSpareServers 10
MaxSpareServers 40
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
And this in my MySQL my.cnf (other is default):
max_allowed_packet = 64M
query_cache_size = 128M
query_cache_type = 1
query_cache_limit= 1M
slow-query-log-file=/var/log/mysqld/log-slow-queries.log // nothing logged -> no slow queries
table_cache = 1K
table_definition_cache = 4K
open_files_limit = 3K
thread_cache_size = 3
mod_statusshows me this if the server has nothing to do:
Parent Server Generation: 0
Server uptime: 23 seconds
Total accesses: 10 - Total Traffic: 49 kB
CPU Usage: u.04 s0 cu0 cs0 - .174% CPU load
.435 requests/sec - 2181 B/second - 5017 B/request
1 requests currently being processed, 15 idle workers
W_______________................................................
................................................................
................................................................
................................................................
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 14223 10/10/10 W 0.04 0 0 49.1 0.05 0.05 IP www.test.com GET /server-status HTTP/1.1
And with one website crawler (~10 concurrent requests):
Server uptime: 5 minutes 46 seconds
Total accesses: 1883 - Total Traffic: 65.5 MB
CPU Usage: u365.13 s17.79 cu0 cs0 - 111% CPU load
5.44 requests/sec - 193.8 kB/second - 35.6 kB/request
15 requests currently being processed, 10 idle workers
WWWW_WWKW___W_W__WW__K_WW.......................................
................................................................
................................................................
................................................................
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 14223 90/207/207 W 2.07 0 0 902.0 1.61 1.61 IP www.test.com GET /server-status HTTP/1.1
1-0 14224 11/70/70 W 18.87 47 0 279.7 0.84 0.84 IP www.test.com GET /site1
2-0 14225 0/19/19 W 9.66 1 0 0.0 0.95 0.95 IP www.test.com GET /site2
3-0 14226 1/84/84 W 13.94 36 0 0.0 2.14 2.14 IP www.test.com GET /site3.pdf
4-0 14227 0/153/153 _ 27.46 4 81 0.0 3.56 3.56 IP www.test.com GET /site4pdf
5-0 14228 12/78/78 W 17.59 1 0 275.8 2.67 2.67 IP www.test.com GET /site5.pdf
6-0 14229 10/90/90 W 14.62 0 0 611.4 3.85 3.85 IP www.test.com GET /site6
7-0 14230 4/53/53 K 34.12 0 141 9 81.9 2.48 IP www.test.com GET /site7
8-0 14231 1/114/114 W 12.39 27 0 0.0 2.29 2.29 IP www.test.com GET /site8.pdf
9-0 14232 0/106/106 _ 9.68 2 64 0.0 6.33 6.33 IP www.test.com GET /site9
10-0 14233 0/39/39 _ 15.66 5 1376 0.0 2.68 2.68 IP www.test.com GET /site10
11-0 14234 0/36/36 _ 11.31 5 128 0.0 2.78 2.78 IP www.test.com GET /site11
12-0 14235 1/63/63 W 10.58 7 0 70.9 3.59 3.59 IP www.test.com GET /site12.pdf
13-0 14236 0/100/100 _ 22.76 1 20 0.0 2.76 2.76 IP www.test.com GET /site13
14-0 14237 0/63/63 W 4.34 37 0 0.0 0.58 0.58 IP www.test.com GET /site14
15-0 14238 0/12/12 _ 1.75 0 29 0.0 1.85 1.85 IP www.test.com GET /site15
16-0 14283 0/43/43 _ 14.94 1 119 0.0 2.25 2.25 IP www.test.com GET /site16
17-0 14286 8/80/80 W 6.79 0 0 222.7 5.98 5.98 IP www.test.com GET /site17.pdf
18-0 14288 3/111/111 W 21.87 0 0 60.5 2.35 2.35 IP www.test.com GET /site18.pdf
19-0 14289 0/75/75 _ 18.81 5 31 0.0 3.06 3.06 IP www.test.com GET /site19
20-0 14290 0/50/50 _ 16.15 1 259 0.0 2.16 2.16 IP www.test.com GET /site20.pdf
21-0 14292 6/88/88 K 23.71 0 0 32.8 2.79 2.79 IP www.test.com GET /site21
22-0 14293 0/92/92 _ 30.82 1 45 0.0 3.78 3.78 IP www.test.com GET /site22
23-0 14294 11/37/37 W 12.85 27 0 774.2 1.48 1.48 IP www.test.com GET /site23.pdf
24-0 14302 16/20/20 W 10.18 37 0 557.3 0.68 0.68 IP www.test.com GET /site24.pdf
At the end of the crawling I have up to Srv 46-0.
Swappping? This I did after the crawling has finished:
# swapon -s
Filename Type Size Used Priority
/dev/am-1 partition 8208376 35172 -1
# free
total used free shared buffers cached
Mem: 8061508 7287776 773732 0 147220 4806004
-/+ buffers/cache: 2334552 5726956
Swap: 8208376 35172 8173204
Looking at some of this you seem to be swapping. Swapping is VERY BAD and causes large amounts of CPU to be consummed doing I/O swapping processes in and out. You might consider adding more memory....or reducing the number apache servers that startup.
Use New Relic and look into the web transactions of your application.
New Relic now contains Drupal specific instrumentation as well (Modules, Hooks, Views).
Sorry if this sounds like a commercial but if you're running Drupal, it's really a no-brainer for the wealth of information it provides. I'm not affiliated with New Relic in anyway, just a happy customer.
If you want to do it the hard way profile your code (e.g. oprofile)
HTH
P.S.
You could bump up your thread_cache_size to a more reasonable number (e.g. 32) but I doubt it'll make a significant difference.