(I asked this question on stackoverflow but it might be better off here...)
I need to read through some gigantic log files on a Linux system. There's a lot of clutter in the logs. At the moment I'm doing something like this:
cat logfile.txt | grep -v "IgnoreThis\|IgnoreThat" | less
But it's cumbersome -- every time I want to add another filter, I need to quit less
and edit the command line. Some of the filters are relatively complicated and may be multi-line.
I'd like some way to apply filters as I am reading through the log, and a way to save these filters somewhere.
Is there a tool that can do this for me? I can't install new software so hopefully it's something that would already be installed -- e.g., less, vi, something in a Python or Perl lib, etc.
Changing the code that generates the log to generate less is not an option.
you should run secondary filters and counting scripts to analyze them
it is always a good idea to store the filtered output (rather than trying to look at it immediately).
It works better if your filters will reduce the actual file by a large amount
grep
','sed
', 'AWK
' usually suffice to process text log file very nicely.I have processed log files in the order of 10GB frequently with these things.
You can make your own tools with these things in '
bash scripts
'.vim
', it can handle large files too (but will take time, so give it filtered files)You certainly won't have this installed, but if this is going to be a regular thing you have to do it might be worth looking into Splunk. Splunk exists to index large datasets like this to help you find what you're looking for.