My current httpd deployment setup involves generating new configuration files, relinking the config directory, then reloading the apache service. This process works fine for most of my hosts, however on of my proxy hosts, the httpd service will quit after issuing a reload.
Example:
/config/aaaa/*.conf
/config/bbbb/*.conf
/config/active -> /config/aaaa
And in my main httpd.conf file,
require /config/active/*.conf
The deployment process would remove the link, and relink a new config directory.
After issuing
systemctl reload httpd
The Error
I have the following error in my logs before the process just dies.
(28)No space left on device: AH02611: create: apr_shm_create(/etc/httpd/run/slotmem-shm-p975c3056_scanner_2.shm) failed
Debugging
I have plenty of disk space available:
[root@proxy3 log]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl_template-root 3.8G 3.4G 363M 91% /
tmpfs 989M 102M 888M 11% /run
//10.3.36.10/config3 3.8G 2.6G 1.3G 68% /config
The directory, when the server is running, is only 6M.
[root@proxy3 ~]# du -h /etc/httpd/run/
0 /etc/httpd/run/htcacheclean
6.0M /etc/httpd/run/
With ~1500 files
[root@proxy3 ~]# du -h /etc/httpd/run/* | wc -l
1521
Memories Available (should evict cache)
[root@proxy3 httpd]# free
total used free shared buff/cache available
Mem: 2025016 408128 113756 111216 1503132 1352584
My semaphores and ulimits seem fine
[root@proxy3 httpd]# sysctl -a | grep sem
kernel.sem = 32000 1024000000 500 32000
[root@proxy3 httpd]# ipcs -l
------ Messages Limits --------
max queues system wide = 32000
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 18014398509465599
max total shared memory (kbytes) = 18014398509481980
min seg size (bytes) = 1
------ Semaphore Limits --------
max number of arrays = 32000
max semaphores per array = 32000
max semaphores system wide = 1024000000
max ops per semop call = 500
semaphore max value = 32767
[root@proxy3 httpd]# ulimit
unlimited
Version Info
[root@proxy3 log]# httpd -v
Server version: Apache/2.4.37 (centos)
Server built: May 20 2021 04:33:06
[root@proxy3 log]# uname -r
4.18.0-305.19.1.el8_4.x86_64
What else can I do to debug / correct this failure mode? My current workaround is to simply restart the service after it quits, however this is only a band-aid.
You have only 350Mio available on your / filesystem. Are you sure it's enought ? In all cases, consuming more than 90% of disk space should be a critical fact ;)
Examine the free space and size of the directory from the error message:
Red Hat httpd package has a symlink /etc/httpd/run/ -> /run/httpd. Puts httpd DefaultRuntimeDir storage in a system tmpfs, without changing the convention of it living relative to ServerRoot. Note this means you should pay attention to /run.
What is log code 02611 anyway? Module mod_slotmem_shm which provides a shared memory API, Generic, so examine all modules loaded for if they use this to share state between workers. Approximate order of magnitude how large size can get, based on observations and capacity planning.
Regarding how to mitigate the problem, assuming a tmpfs is filling up, allocating a bit more memory to the host could give it more space. Or, you might tune your MPM to have fewer workers, or otherwise tweak the configuration to have less shared state. Finally, consider moving DefaultRuntimeDir to some temporary directory with sufficient space.
File backed shared memory maps are not System V shm you can see with ipcs. APR is portable and gives lots of options, but I think on this OS is file and mmap based.
On the topic of space, 363M available 91% used is not a lot free, even on a small host. Things like log files and package update transactions usually need more.