Home / server / Questions / 727343
Accepted
D34DM347
Asked: 2015-10-08 05:43:54 +0800 CST

F5 LTM frequently kills processes with SIGKILL

  • 772

We have a BIP-IP 6400 LTM device that is killing processes with an alarming frequency. The CPU is consistently around 23% utilization, so that is not an issue.

Here is a sample from /var/log/ltm:

Oct  7 08:21:55 local/pri-4600 info bigd[3471]: reap_child: child process PID = 25338 exited with signal = 9
Oct  7 08:22:15 local/pri-4600 info bigd[3471]: reap_child: child process PID = 25587 exited with signal = 9
Oct  7 08:22:34 local/pri-4600 info bigd[3471]: reap_child: child process PID = 25793 exited with signal = 9
Oct  7 08:23:10 local/pri-4600 info bigd[3471]: reap_child: child process PID = 26260 exited with signal = 9
Oct  7 08:23:36 local/pri-4600 info bigd[3471]: reap_child: child process PID = 26584 exited with signal = 9
Oct  7 08:23:40 local/pri-4600 info bigd[3471]: reap_child: child process PID = 26647 exited with signal = 9
Oct  7 08:23:45 local/pri-4600 info bigd[3471]: reap_child: child process PID = 26699 exited with signal = 9
Oct  7 08:23:55 local/pri-4600 info bigd[3471]: reap_child: child process PID = 26805 exited with signal = 9
Oct  7 08:25:36 local/pri-4600 info bigd[3471]: reap_child: child process PID = 28079 exited with signal = 9
Oct  7 08:27:15 local/pri-4600 info bigd[3471]: reap_child: child process PID = 29286 exited with signal = 9
Oct  7 08:27:16 local/pri-4600 info bigd[3471]: reap_child: child process PID = 29307 exited with signal = 9
Oct  7 08:27:56 local/pri-4600 info bigd[3471]: reap_child: child process PID = 29793 exited with signal = 9
Oct  7 08:29:20 local/pri-4600 info bigd[3471]: reap_child: child process PID = 30851 exited with signal = 9
Oct  7 08:33:00 local/pri-4600 info bigd[3471]: reap_child: child process PID = 1122 exited with signal = 9
Oct  7 08:33:16 local/pri-4600 info bigd[3471]: reap_child: child process PID = 1299 exited with signal = 9
Oct  7 08:34:15 local/pri-4600 info bigd[3471]: reap_child: child process PID = 2054 exited with signal = 9
Oct  7 08:35:16 local/pri-4600 info bigd[3471]: reap_child: child process PID = 2784 exited with signal = 9
Oct  7 08:35:16 local/pri-4600 info bigd[3471]: reap_child: child process PID = 2807 exited with signal = 9
Oct  7 08:35:35 local/pri-4600 info bigd[3471]: reap_child: child process PID = 3015 exited with signal = 9
Oct  7 08:36:15 local/pri-4600 info bigd[3471]: reap_child: child process PID = 3601 exited with signal = 9

Is this normal? If not, what could be causing this to happen?

f5-big-ip

2 Answers

  1. bigd is the monitoring daemon on the BIG-IP and so this appears that a monitor that is in use is crashing. You should get a case open with support and upload your qkview to ihealth.f5.com. Here is a solution related to that error message:

    https://support.f5.com/kb/en-us/solutions/public/17000/000/sol17092.html

    • 1
  2. Best Answer

    This was a known bug in the 10.2.4 BIG-IP software we were running.

    From F5 support:

    ...you hit a known issue tracked internally as: bug ID539130 "bigd can deadlock while processing SIGCHLD causing bigd heartbeat failure and SIGABRT" -=Condition=- External monitors that run for a long time and are killed by the next iteration of the monitor, may cause bigd to crash and core, this causes a temporary lapse in health monitoring.

    The fix was to update the software with Hotfix-BIGIP-10.2.4-HF12-866.11-ENG.

    • 1