Apache - drawback to high number of httpd processes always running

What your trying to avoid is incoming requests causing new processes to be forked at a rate higher than older processes can service the older connections. This risk is causing a situation where the system keeps creating new processes that consume the available memory and starts to increase the frequency in which the operating system uses swap. Which in turn causes a large increase in disk I/O where the system is just swapping pages from physical memory to virtual memory (on disk), and vice versa, without any real benefit to the workload.

The general formula to start with is:

(Total Memory - Operating System Memory - database memory) / Size Per Apache process.

The formula is only part of the equation. The more you give the system and MySQL memory, the more caching of the file system they do for you and avoid hitting disk which is very expensive. If the database is not on the same system then this is less of a issue.

The other scenario that happens if you do not tune Apache properly and the frequency of swapping increases is the users start hit stop and reload, artificially increasing the load on the server. You can control the MaxRequestWorkers setting so that your server does not spawn so many children that it starts swapping. It seems straightforward in that you just determine the size of your average Apache process, by looking at your process list via a tool such as top, or smem and then divide this into your total available memory, while leaving sufficient memory for other processes you will get a more accurate picture.

The other, key parameter in this scenario is ServerLimit. If ServerLimit is set to a value much higher than necessary, extra, unused shared memory will be allocated. If both ServerLimit and MaxRequestWorkers are set to high. Apache httpd performance and overall general stability of Apache httpd may start to become an issue.

I'm concerned about your testing methodology. It appears you're just spawning a whole bunch of processes then looking at their memory usage. The problem is that when they're first spawned, they're sharing a maximal amount of memory, and don't have any local, per-process data stored.

When you start actually using them, you'll find that those processes start to need more memory. That is true even in the absence of memory leaks, but if you're using any extensions that are leaky, the problem gets real bad, real quick.

I'd recommend doing a lot of realistic load testing, under a variety of conditions, before leaving a configuration like this to run without close supervision.

There is no issue with having a high number of spare httpd processes always running. It's actually a good idea.

The main point is : if you want httpd to reach that number of process, you'd better reach if right from the start and be sure it works. In other words, don't auto-scale, simply pre-scale. Less surprise.

The seconday point is : if you don't need to share ressources (mostly RAM in this case) or adapt to different services running at different times on your server, there is no point in changing the number of process over time. Allocate a RAM budget to your httpd and spawn the instances you need to handle that many concurrent requests.

15 years ago people would share lots of things on a single server and that made sense to adapt the number of httpds to the workload. Nowadays most people use one server per app : most of the time does not use all of its ressources, but performance is more predictable, and performance analysis and tuning is much easier.

Although there's a gotcha in a specific situation which is actually so common : Apache + mod_php (or whatever interpreter embedded in whatever httpd). Here mod_php totally changes the deal because it then becomes a question of app scaling and not httpd scaling, which is another (lenghty) subject, and requires a different approach (a 1000 preforked worker won't work).