Home / server / Questions / 52124
Accepted
Edward Tanguay
Asked: 2009-08-10 23:11:42 +0800 CST

How can I use grep to find the most frequently recurring errors in my log file?

  • 772

I've got a PHP application that is generating large amounts of warnings and notices in the log file..

I'm fixing these one at a time.

I would like to be able to fix the most frequently occurring first in order to maximize the amount of effect my changes have on the size of the log file.

How can I use grep/sed/regex etc.find out which notices are most frequent?

I can't compare the whole line since the time stamp is at the beginning, but perhaps a listing/grouping/sorting of "on line ..."?

alt text

log-files analysis grep frequency sed

2 Answers

  1. Best Answer

    How about this?

    grep 'on line' /var/log/httpd/error_log | cut -d' ' -f6- | sort | uniq -c | sort -nr

    I.e.

    1. Get the right lines from the logfile
    2. Chop the date off the front
    3. Sort them
    4. Aggregate and count the duplicates
    5. Sort by number of duplicates
    • 8

  2. I think this is more concise, so you really could avoid 'cut'.

      grep -o 'on line [[:digit:]]*' /var/log/nginx/error.log | sort | uniq -c | sort -nr
    • 0