I'm finding instances of syslog-ng writing out garbage followed by a blank kernel.emerg
line in one of our production environments. Example of one:
Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <Q▒b
+\c 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <;E0
Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <▒"▒l
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒▒▒e▒F
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒
Dec 21 00:14:58 someserver [kernel.emerg]
The kernel.emerg
line is of particular concern to me. According to man 3 syslog
:
LOG_KERN
kernel messages (these can’t be generated from user processes)
This seems to suggest that the kernel facility cannot be spoofed to. I can see where the system calls themselves might prevent such spoofing, but am I correct in thinking that there is nothing to stop a process from writing directly to /dev/log
and spoofing a facility of kernel
? I want to say that the only thing that could truly stop such spoofing would be the syslog daemon differentiating between whether it obtained a message from /proc/kmsg
vs. other sources.
- Distro is RHEL5.5. Kernel version is
2.6.18-194.8.1.el5
. These are not factors I have control over yet; consider me scolded. - The syslog daemon is a company-built
syslog-ng
3.1.4 package (32-bit and running on a 64-bit kernel, but I wouldn't expact that to be related). - There are some mailing list posts I found via Google suggesting that changes to
kmsg
in kernel 3.5 can create output errors like this, but that's definitely not the case here.
The messages are not coming in from the network. This is the only source defined:
source s_syslog {
# message generated by Syslog-NG
internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
unix-stream("/dev/log");
# messages from the kernel
file("/proc/kmsg" program_override("kernel: "));
};
Per the syslog-ng Administrators' Guide ( http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/en/syslog-ng-ose-v3.3-guide-admin-en/html-single/index.html#kernel-messages ), the kernel facility (at least as defined by default) reads directly from /proc/kmsg, which cannot be written to from userland.
(This is from memory, but I'm pretty sure RHEL 5.5 doesn't need klogd or even ksymoops for symbol-to-address output; check your docs on that, though..)
If you're worried (see my comment above) about some rogue process prepending the literal string "kernel: " to their message, you can always add a filter that would remove the string "kernel: " at the start of any message received to be sure. Alternately, define /proc/kmsg as a separate source with a separate destination.
EDIT: On looking more at that section:
"Note
If the message does not have a proper syslog header, syslog-ng treats messages received from files as sent by the kern facility. Use the default-facility and default-priority options in the source definition to assign a different facility if needed."