Home / server / Questions / 243411
Accepted
hobodave
Asked: 2011-03-05 13:30:42 +0800 CST

How to troubleshoot Apache httpd that just went silent?

  • 772

I had one of our Apache httpd servers "die" in a weird way overnight. I have looked at all relevant logs and cannot find anything out of the ordinary.

This happened in our development environment thankfully, but I am rather concerned because our dev environment nearly identically mirrors our production environment.

Can anyone tell me what could have happened here?

Symptoms & Troubleshooting Steps

  • Apache httpd was completely unresponsive
  • Load average of 0.0 across the board
  • No log anomalies
  • ps aux | grep httpd appeared normal
  • iptables -L showed port 80 open
  • SELinux is enabled and properly configured: no AVCs logged
  • No configuration changes were made to any system components, no matter how insignificant
  • Apache httpd was reloaded at 17:11:25 server time to clear APC (php) cache
    • Successful requests were made after this reload through 17:20 after which I left for the day
    • Nothing remarkable happened after reload
  • The following morning the httpd processes were unresponsive
  • Telnet to port 80 connected successfully
    • Manual GET, POST, OPTIONS, and HEAD requests received no response whatsoever; nothing logged

Logs

access_log

172.22.44.41 - - [03/Mar/2011:17:20:18 -0600] "GET /admin.php?type=notifications HTTP/1.0" 200 44326 "http://umc.dev.example.com/admin.php" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"
172.22.44.41 - - [04/Mar/2011:14:45:16 -0600] "GET / HTTP/1.0" 200 62921 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13"

error_log

[Thu Mar 03 17:11:25 2011] [notice] SIGHUP received.  Attempting to restart
[Thu Mar 03 17:11:25 2011] [notice] Digest: generating secret for digest authentication ...
[Thu Mar 03 17:11:25 2011] [notice] Digest: done
[Thu Mar 03 17:11:25 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations
[Fri Mar 04 14:45:05 2011] [notice] caught SIGTERM, shutting down
[Fri Mar 04 14:45:05 2011] [notice] SELinux policy enabled; httpd running as context user_u:system_r:httpd_t:s0
[Fri Mar 04 14:45:05 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Fri Mar 04 14:45:05 2011] [notice] Digest: generating secret for digest authentication ...
[Fri Mar 04 14:45:05 2011] [notice] Digest: done
[Fri Mar 04 14:45:05 2011] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

ps aux | grep httpd

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      8477  0.0  1.9 408748  9876 ?        Ss   Feb24   0:03 /usr/sbin/httpd
apache   19933  0.4 14.6 436436 74692 ?        S    Mar03   0:01 /usr/sbin/httpd
apache   19936  0.4  8.4 419708 43148 ?        S    Mar03   0:01 /usr/sbin/httpd

/var/log/messages

Feb 27 04:02:01 webdev1 syslogd 1.4.1: restart.
Mar  4 14:43:35 webdev1 yum: Installed: strace-4.5.18-5.el5_5.5.x86_64

Versions

OS & Kernel

  • CentOS release 5.5 (Final)
  • Linux webdev1.int.example.com 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Apache httpd

Server version: Apache/2.2.3
Server built:   Aug 30 2010 12:28:40
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

PHP

  • php53u-5.3.5-3.ius.el5

Repositories used

CentOS Base & Updates for all packages with the following exception: php53u from IUS Community repository.

apache-2.2 httpd

1 Answers

  1. Best Answer

    Aside from disk full happening, there's not much to go there. If it doesn't happen again then it will fall on the weirdness category. Do you have the dmesg output to see if there was something wrong with the OS (oom killer, for example).

    If it happens again, then you will must use strace following forks to see what exactly is blocking the processes.

    • 4