I have a webapp running on some servers that have their Apache log files on an OCFS2 SAN volume. The internal webapp logging is done using log4perl, in pipe mode, with cronolog on the other end of the pipe to handle log file rotation. From time to time, right around midnight, when the log files are rotated, some of the servers will experience a broken pipe, which renders Apache unable to log, and generates all sorts of alerts and bad things.
I'm wondering if there is a way to determine the root cause of the pipe breaking. DTrace sounds like it could do it, with some clever programming, but I'm not sure.
Anyone have any ideas?
Thanks,
Kendall
My guess is that if chronolog is logging the broken pipe then it's possible you have logrotate or some other rotator competing with it.
logrotate
rotates apache's logs by having it close and reopen its log files (which makes it create new files after logrotate has moved the old ones), which could break the pipe. You might tryapache2ctl graceful
(this is what logrotate uses in debian, your distribution may vary) to see if this is what is triggering the condition.