I have been running this instance of apache 2.4.17 for a long time. It is used as a proxy to a number of applications at the back. Recently I upgraded Piwik/Matomo, and it caused a bit of issue in the apache, but now it shoulbe fixed. However, after restart the physical server and apache instances numerous times, it will still crash apache pretty badly after a number of hours later, and need to apachectl restart to get it going again.
Checked apache's error.log, and there are quite a few entries like this:
[Sun Mar 21 16:15:21.126158 2021] [ssl:info] [pid 1274] [client x}:0] AH01998: Connection closed to child 28 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.136567 2021] [ssl:info] [pid 1271] [client 216.245.221.83:44352] AH01998: Connection closed to child 25 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.146466 2021] [ssl:info] [pid 1270] [client ?x}:0] AH01998: Connection closed to child 24 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.153012 2021] [ssl:info] [pid 1269] [client x}:0] AH01998: Connection closed to child 23 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.164055 2021] [ssl:info] [pid 1262] [client ?v}:0] AH01998: Connection closed to child 16 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.167462 2021] [ssl:info] [pid 1263] [client ?2):0] AH01998: Connection closed to child 17 with abortive shutdown (server <domain>:443)
[Sun Mar 21 16:15:21.169270 2021] [ssl:info] [pid 1264] [client ?x}:0] AH01998: Connection closed to child 18 with abortive shutdown (server <domain>:443)
where the client IP value is randomised... x}, ?x}, ?v} and etc.
There aren't anything strange in applications' log files.
How should I troubleshoot this problem?
The abortive shutdown messages just mean that the SSL sessions didn't terminate via the normal process - so a malformed client, or a robot, etc could just be abandoning connections. It's typically indicative of a client issue of some sort. Unless there are a ton of them, and users reporting issues, it's probably not your cause.
For further troubleshooting...
What does dmesg say around the time of the crashes?
Is there a pattern at all to the crashes?
Any cron jobs running?
What do system resources look like before you restart the failed Apache service?
Have you set the logs to full debug to see what it says?