I've issue with apache. Everything begins about a week ago (so this may be related to latest system updates) - my monitoring tools SMS me about non-working website. I've checked it, and looks like request to trivial cgi (implemented in perl) took 30 seconds. After restarting apache problem was fixed for next few hours/days, and then happens again.
Initially I suspect php, because it was only update (5.3.15->5.3.18) which may affect apache. I've tried to disgrage it back to 5.3.15, but this won't fixed this issue.
Next, I did strace, and looks like delay happens while clone() syscall:
[pid 26659] 1356268557.675934 clone( <unfinished ...>
[pid 26659] 1356268587.684401 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xa1033768) = 32169 <30.008452>
cgi itself runs after that and works ok and fast:
[pid 32169] 1356268587.686614 execve("/var/www/mysite.com/mycgi.cgi", ["/var/www/mysite.com/mycgi.cgi"], [/* 26 vars */]) = 0 <0.000776>
Any ideas why this may happens? How and why clone(2) may block for 30 seconds?! And successfully return after that… It looks like some timeout inside clone() because it always exactly 30 seconds.
I'm using up-to-date Hardened Gentoo Linux, kernel 3.5.4-hardened-r1, apache 2.2.23 (prefork, mod_cgi).
Update: I probably should add: this server isn't under high load. Apache childs use only about 120/40 MB (VIRT/RSS). Server has 2GB RAM, and only 400MB used (without buffers/cache).
This happens because of CONFIG_GRKERNSEC_BRUTE=y kernel option.