Home / server / Questions / 654407
Accepted
lsh
Asked: 2014-12-24 06:13:14 +0800 CST

syslog-ng prepending date to received logs, breaking patterns in patterndb

  • 772

I'm setting up central logging for our servers using syslog-ng + patterndb, however the logs the logging server is receiving from the client are prepended with the date, host and other data. This of course breaks all the patterns in patterndb so nothing matches.

Is there a way to do some preprocessing on the source log file before attempting to classify it or some other way to overcome this problem?

Cheers.

Relevant client conf:

source s_src {
   system();
   internal();
};

destination d_central_logging {
    syslog(192.168.1.1 transport("tcp") port("12345")); 
};

log {
    source(s_src);
    destination(d_central_logging);
};

Relevant server conf:

parser p_patterndb {
    db-parser(file("/var/lib/syslog-ng/patterndb.xml"));
};

source s_network {
    tcp(port(12345) flags(syslog-protocol));
};

filter f_class_unknown {
    match("unknown"
        value(".classifier.class")
        type("string")
    );
};

destination d_all {
    file("/tmp/all");
};

destination d_unknown {
    file("/tmp/unknown");
};

log {
    source(s_network);
    parser(p_patterndb);
    log {
        filter(f_class_unknown);
        destination(d_unknown);
    };
    log {
        destination(d_all);
    };
};

EDIT:

original log line: 10.0.2.2 - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

modified log line: Dec 23 13:59:08 192.168.33.44 264 <13>1 2014-12-23T13:42:50+00:00 devhost 10.0.2.2 - - [meta sequenceId="8"] - - [23/Dec/2014:13:42:49 +0000] "GET /assets/favicon.ico HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"

syslog

2 Answers

  1. Paul is right, the protocols used in the client-destination and the server-source are mismatched. As a result, the server does not recognize the incoming message as a properly-formatted syslog message, and prepends the host, date and other information.

    Regards,

    Robert

    • 0
  2. Best Answer

    ok, I've found the real reason why a match wasn't being made. I was using the update-patterndb command after confirming that the correct Apache patterns existed in the github.com/balabit/syslog-ng-patterndb repo. However, the update-patterndb command wasn't including .xml files, only .pdb files. I saw the compiled pattern file at /var/lib/syslog-ng/patterndb.xml changing but never checked it. As the man pages for update-patterndb say, it's only a thin wrapper around the pdbtool merge command. This works for me: 

    pdbtool merge -r --glob *.xml -D /etc/syslog-ng/patterndb.d/syslog-ng-patterndb/

    The directory /etc/syslog-ng/patterndb.d/syslog-ng-patterndb/ is where I have the repo cloned. I can confirm the Apache patterns are now extracting the content as expected by inspecting the compiled pattern file, running the pdbtool test command and seeing the structured results in those that receive the data downstream.

    • 0