Questions[prefork](server)

How do I change from the prefork to worker mpm on Redhat Linux 64bit.

/bin/httpd -l
Compiled in modules:
  core.c
  mod_access.c
  mod_auth.c
  mod_include.c
  mod_log_config.c
  mod_env.c
  mod_setenvif.c
  mod_ssl.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_cgi.c
  mod_negotiation.c
  mod_dir.c
  mod_imap.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_so.c

bin/httpd -V
Server version: Apache/2.0.55
Server built:   Mar  7 2006 00:10:58
Server's Module Magic Number: 20020903:11
Architecture:   32-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D HTTPD_ROOT="/opt/httpd/apache-2.0.55"
-D SUEXEC_BIN="/opt/httpd/apache-2.0.55/bin/suexec"
-D DEFAULT_PIDLOG="logs/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="logs/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

I have a LAMP server that can be serving ~400 requests per second at times, the server configuration is:

CPU: Intel Quad Core Xeon X3430 (4 x 2.40 GHz, 8MB Cache)
RAM: 16 GB REG ECC DDR3
HD: 500 GB Enterprise Grade SATA II
OS: CentOS 64 Bit (Latest Stable)

Apache2 prefork config:

StartServers      128
MinSpareServers   16
MaxSpareServers   64
ServerLimit     1500
MaxClients      1500
MaxRequestsPerChild  10000

At peak times the server load is very high, cpu utilization around 90% and load average around 130

I'm not sure it's server hardware limit or my prefork config have something wrong - What could be the problem?

I have tried to fix this but don't know enough, apparently. When I run top command I see that its filled with httpd processes. When this happens my site stops loading. It doesn't crash it just spins and spins. I am using prefork MPM. This looks like:

<IfModule prefork.c>
StartServers       1
MinSpareServers    5
#MinSpareServers    1
MaxSpareServers    10
#MaxSpareServers    5
ServerLimit        25
#ServerLimit        50
MaxClients        25
#MaxClients        50
#MaxRequestsPerChild  10000
MaxRequestsPerChild  100
</IfModule>

This problem started with the defaults which are commented out. I then tried to make changes thinking that it would help if I lower the MaxClients because I have about 1gb ram available. I thought the default MaxClients was to high because the average memory usage per httpd process is 55mb. Here is a screenshot of top, keep in mind this doesn't show all the httpd processes.

top - 09:48:27 up 42 days,  3:13,  1 user,  load average: 1.09, 1.09, 1.09
Tasks:  35 total,   3 running,  32 sleeping,   0 stopped,   0 zombie
Cpu(s): 14.7%us,  0.5%sy,  0.0%ni, 84.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2097152k total,  1134676k used,   962476k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                        
 1649 root      17   0  315m 122m  14m S  0.0  6.0   1:51.67 java                                                                                            
27755 mysql     18   0  204m  71m 5744 S  4.0  3.5 806:09.08 mysqld                                                                                          
23603 apache    15   0 76404  51m 5900 S  0.0  2.5   0:20.21 httpd                                                                                           
20408 apache    15   0 77124  51m 5252 S  0.0  2.5   0:25.57 httpd                                                                                           
28026 apache    16   0 76788  50m 5240 R 97.7  2.5   0:05.15 httpd                                                                                           
24497 apache    16   0 76272  50m 5484 S  0.0  2.5   0:06.55 httpd                                                                                           
27724 apache    15   0 76380  50m 5268 S  0.0  2.5   0:04.81 httpd                                                                                           
21561 apache    15   0 75672  50m 5532 S  0.0  2.4   0:28.54 httpd                                                                                           
26537 apache    16   0 75116  49m 5488 R 19.9  2.4   0:10.09 httpd                                                                                           
28027 apache    15   0 75080  49m 5216 S  0.0  2.4   0:03.07 httpd                                                                                           
24498 apache    15   0 74180  49m 5656 S  0.0  2.4   0:12.33 httpd                                                                                           
21934 apache    15   0 74568  48m 5256 S  0.0  2.4   0:27.48 httpd                                                                                           
24484 apache    15   0 74152  48m 5260 S  0.0  2.4   0:17.70 httpd                                                                                           
28012 apache    15   0 73248  47m 5208 S  0.0  2.3   0:02.34 httpd                                                                                           
13428 root      18   0 37032  15m 8456 S  0.0  0.8   0:00.21 httpd

What am I doing wrong?

I have a cluster of webservers running on apache-php (with prefork). I'm planning to move from prefork to worker (fcgi). The problem is that I cannot move all servers at once so for a short period of times (matter of days) few of them will be on fcgi and some on modphp. I tested today if the sessions are going to work between these two kind of servers (a test server with fcgi and a real with mod_php) and looks like not, I'm getting logged out when my proxy (basic nginx) is redirecting me from a ws with fcgi to one with mod_php. The application that is hosted on these servers is a magento with few modifications (nothing on the core). If I take the proxy out, everything works perfect, I can change from a node to another and the session is the same.

Is there anything in nginx that can cause this? Below is a part of the config of nginx:

 proxy_ignore_headers "Cache-Control" "Expires";

    proxy_cache_valid  200 302 60m;

    proxy_redirect     off;
    proxy_set_header   Host             $host;
    proxy_set_header   X-Real-IP        $remote_addr;
    proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_max_temp_file_size 0;

    client_max_body_size       100m;
    client_body_buffer_size    128k;

    proxy_connect_timeout      600000;
    proxy_send_timeout         600000;
    proxy_read_timeout         600000;

    proxy_buffer_size          4k;
    proxy_buffers              4 32k;
    proxy_busy_buffers_size    64k;
    proxy_temp_file_write_size 64k;
    proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;

I am examining a high-volume Apache prefork site and noticing that many log entries such as:

CustomLog '|/foo/bar/cronolog -foo -bar'
RewriteLog '|/foo/bar/cronolog -bar -foo'

When apache logs the request or the rewrite is cronolog spawned for EACH request or is the pipe opened and written-to for the lifetime duration of the preforked apache process?

(This was asked on stackoverflow but I'll remove from there as more applicable here I think.)