Apache HTTPD configuration for high load [duplicate]

Good day,

I want to be able to serve at least 7000 (preferably 10k) concurrent requests from my Apache Httpd. I have configured my httpd.conf with MPM worker with the following setup

ServerLimit          330
StartServers          25
ThreadsPerChild       25
MaxClients          7500

My Apache Httpd server does nothing except do a reverse proxy against a cluster of java app servers and some (disk) caching on some static items like HTML/CSS/JS.

While trying to pound my system using JMeter (with about 1k concurrent requests), in a couple of minutes, my server starts to crash but it shows no additional information.

My /etc/security/limits.conf is configured to

*                soft    nofile          10000
*                hard    nofile          30000
apache           soft    nproc            8192
apache           hard    nproc           12288
*                soft    stack             512
*                hard    stack            1024

Any ideas how I can reach 7k (or preferably 10k) concurrent requests

[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Mon Apr 09 21:47:42 2012] [alert] Child 15139 returned a Fatal error... Apache is exiting!
[Mon Apr 09 21:47:42 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Mon Apr 09 21:47:46 2012] [warn] child process 14004 still did not exit, sending a SIGTERM

EDIT (Additional Info):

This is me trying to see how many active httpd processes that I have while the test is running (each command is about 1second apart - i.e. me pressing the up key on my keyboard and then pressing enter)

[franz@webserver ~]$ ps -ef | grep httpd | wc -l
5
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
5
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
8
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
9
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
10
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
11
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
11
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
13
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
13
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
17
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
17
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
25
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
25
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
41
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
41
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
39
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
37
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
37
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
36
[franz@webserver ~]$ ps -ef | grep httpd | wc -l
1
[franz@webserver ~]$ ps -ef | grep httpd | wc -l

Update: If I add ThreadStackTrace, it does not crash anymore: ServerLimit 330 StartServers 25 ThreadsPerChild 25 MaxClients 7500 ThreadStackSize 512

Note: I had stack entries in my limits.conf before adding ThreadStackSize, but I guess that wasn't enough. I had to add both the stack entries in limits.conf and ThreadStackSize in apache httpd.cofn.

However, it still cannot process 7500 concurrent requests. When I do a ps -ef | grep httpd | wc -l, the highest that I see is 42 only (and since one of that process is the grep command, that means it's about 41 apache httpd processes). But I've configured my apache to reach up to 330.

So to see if my apache configuration can really go beyond 40 processes, I tried modifying my StartServers into 50.

ServerLimit          330
StartServers          50
ThreadsPerChild       25
MaxClients          7500
ThreadStackSize      512

The result is the following:

[Wed Apr 11 03:33:40 2012] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Apr 11 03:33:40 2012] [notice] Digest: generating secret for digest authentication ...
[Wed Apr 11 03:33:40 2012] [notice] Digest: done
[Wed Apr 11 03:33:40 2012] [warn] pid file /etc/httpd/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 48
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Wed Apr 11 03:33:40 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0-fips configured -- resuming normal operations
[Wed Apr 11 03:33:40 2012] [alert] Child 55663 returned a Fatal error... Apache is exiting!

So I guess the main problem is that I cannot spawn more than 40 processes.

Update2: Since I cannot seem to be able to increase the number of processes, I tried increasing the number of threads. I can go only as far as 30 ThreadsPerChild (from the original ThreadsPerChild). After that, Apache is not able to start because it cannot seem to create worker thread.

Anybody out there has some ideas how to reach 10k concurrent requests with Apache HTTPd?

Why this is not a duplicate: Pardon, I am not sure why this was marked as duplicate. If the answer to my question is in Can you help me with my capacity planning?, can you kindly highlight how & where? That question is generic in nature while mine is very specific and a more "show me how to specifically reach c10k with apache httpd or point out what's wrong with my setup".

Thanks,

Franz