Home / server / Questions / 151454
Accepted
dpb
Asked: 2010-06-16 10:49:36 +0800 CST

Unix/Linux simple log parser (since, until)

  • 772

Has anyone ever used/created a simple unix/linux log parser that can parse logs like the following:

timestamp log_message \n

Order the messages, parse the timestamp, and return:

  • All messages
  • Messages after a certain date (--since)
  • Messages before a certain date (--until)
  • Combination of --since, --until

I could write something like this, but wasn't sure if there was something canned. It would fit well in some automated reporting I'm planning on doing.

linux log-files unix

3 Answers

  1. Best Answer

    Take a look at this Python program I wrote to see if it comes close to what you're looking for or can be adapted to your needs.

    • 3

  2. Even if you find one I don't know if I would trust it. For example, since the timestamp isn't known ahead of time, the only what it could distinguish DD-MM-YYYY and MM-DD-YYYY would be to read ahead until either xx or zz with xx-zz-YYYY is greater than 12. I am sure there are other issues.

    Writing your own would be much easy and more reliable I think as you can use your language's standard string to datetime library and specify the date format specifiers explicitly.

    For example with Python:

        import re
    from datetime import datetime
    line_regex = re.compile(
        r'''
        \[
        (?P<day> \d{1,2} )
        /
        (?P<month> \w{3} )
        / 
        (?P<year> \d{4} )
        :
        (?P<hour> \d{2} )
        : 
        (?P<minute> \d{2} )
        : 
        (?P<second> \d{2} )
        \s
        (?P<timezone> -?\d{4} )
        \]
        ''', re.VERBOSE)


   new_entry['time'] = datetime.strptime(
        parsed_line['day'] +
        parsed_line['month'] +
        parsed_line['year'] +
        parsed_line['hour'] +
        parsed_line['minute'] +
        parsed_line['second']
        , "%d%b%Y%H%M%S"
   )

    If you do want a module / library that will try to figure out the format they do exist, one option for Perl is Date::Parse.

    • 2

  3. The only solution I've ever found for this which was even halfway decent was syslog-ng logging to a database (at which point it's reduced to simple SQL queries) -- Assuming you're logging everything centrally this doesn't add too much additional pain.

    (Insert obvious benefits & obvious caveats here)

    • 1