Forgive the cross post from Stack Overflow but I realized this was probably the better location to ask.
I recently rebuilt my server on AWS, upgrading from Amazon Linux to Amazon Linux 2, which like Linux 1 appears to be a fork of CentOS, now CentOS 7. My vhost config remains the same as the previous build while all the other conf files are the newer default files. (Just FYI, not sure it has any bearing).
Since the rebuild I can no longer do systemctl restart httpd. I get the error The Apache HTTP Server httpd pid (NNNN) already running.
I have done lots of googling and see that quite a few people have this error but it always turns out to be something different that they have done. I cannot think of anything "different" I did.
I have learned that the PID that I see as "already running", owned by root, is the "master process" and all the other PIDs, owned by apache, are the "worker processes". CentOS httpd running as both root and apache user
root 4461 1 0 07:18 ? 00:00:00 /usr/sbin/httpd -k restart
apache 4467 4461 0 07:18 ? 00:00:09 /usr/sbin/httpd -k restart
apache 4468 4461 0 07:18 ? 00:00:08 /usr/sbin/httpd -k restart
apache 4471 4461 0 07:18 ? 00:00:08 /usr/sbin/httpd -k restart
apache 4477 4461 0 07:18 ? 00:00:08 /usr/sbin/httpd -k restart
apache 4498 4461 0 07:18 ? 00:00:07 /usr/sbin/httpd -k restart
apache 5236 4461 0 07:19 ? 00:00:08 /usr/sbin/httpd -k restart
apache 5248 4461 0 07:19 ? 00:00:09 /usr/sbin/httpd -k restart
apache 5987 4461 0 07:20 ? 00:00:09 /usr/sbin/httpd -k restart
apache 5993 4461 0 07:20 ? 00:00:08 /usr/sbin/httpd -k restart
apache 5994 4461 0 07:20 ? 00:00:09 /usr/sbin/httpd -k restart
# systemctl restart httpd;systemctl status httpd;
Job for httpd.service failed. See "systemctl status httpd.service" and "journalctl -xe" for details.
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: failed (Result: protocol) since Thu 2021-12-23 19:56:05 UTC; 4ms ago
Docs: man:httpd.service(8)
Process: 31799 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=0/SUCCESS)
Main PID: 31799 (code=exited, status=0/SUCCESS)
Status: "Reading configuration..."
Dec 23 19:56:04 ip-10-5-11-55.ec2.internal systemd[1]: Starting The Apache HTTP Server...
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal httpd[31799]: httpd (pid 4461) already running
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: Failed to start The Apache HTTP Server.
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: Unit httpd.service entered failed state.
Dec 23 19:56:05 ip-10-5-11-55.ec2.internal systemd[1]: httpd.service failed.
Nothing unusual in the Apache error log:
[Thu Dec 23 07:18:58.777745 2021] [suexec:notice] [pid 4460] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Dec 23 07:18:58.802753 2021] [lbmethod_heartbeat:notice] [pid 4461] AH02282: No slotmem from mod_heartmonitor
[Thu Dec 23 07:18:58.802850 2021] [http2:warn] [pid 4461] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more
[Thu Dec 23 07:18:58.847386 2021] [mpm_prefork:notice] [pid 4461] AH00163: Apache/2.4.51 () PHP/7.4.21 configured -- resuming normal operations
[Thu Dec 23 07:18:58.847414 2021] [core:notice] [pid 4461] AH00094: Command line: '/usr/sbin/httpd'
If I kill this "master" PID, then I can do restarts no problem.
# kill 4461
# systemctl restart httpd;systemctl status httpd;
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2021-12-23 20:01:31 UTC; 6ms ago
Docs: man:httpd.service(8)
Main PID: 3091 (httpd)
Status: "Configuration loaded."
CGroup: /system.slice/httpd.service
└─3091 /usr/sbin/httpd -DFOREGROUND
Dec 23 20:01:31 ip-10-5-11-55.ec2.internal systemd[1]: Starting The Apache HTTP Server...
Dec 23 20:01:31 ip-10-5-11-55.ec2.internal systemd[1]: Started The Apache HTTP Server.
# ps -Af | grep httpd
root 3145 1 1 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 3147 3145 0 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 3153 3145 0 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 3171 3145 0 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 3177 3145 0 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
apache 3183 3145 0 20:01 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
Most people's advice is "just kill the process", Apache failes to start, already running but not maching pid file? but I need this to work unattended. What else could I be looking at?
Although I searched alot I did not find answers that were given for other flavors of Linux. I came to the answer when I noticed that httpd was originally started with "apachectl" and I was trying to restart using "systemctl" and thought this might be the issue. When I searched for this I found many answers. This appears to be the most concise: https://unix.stackexchange.com/questions/240528/apache-and-systemd
In others you see the same issue I experienced which is once you start with apachectl you must stick with it as systemd cannot know what you have done with apachectl as it uses a different config.
This can be solved by changing the SE Linux policy as a root user first enter the command
you may see these two in the bottom of the execution
then reboot the machine
then start httpd server