I am trying to analyze a series of application requests. There are web logs, application server logs and data server logs, all with slightly different timestamps on each line. I'd like to splice them together to see the various logs in the order they occurred.
Before I whip up a script (using awk and friends) to do this, is there a tool or existing project/recipe that does something similar?
Check out Splunk.
Disclaimer: i haven't done this before.
I think syslod is the deamon for logging in *nix systems, maybe there are better ones. But essentially works for similar goals. I think if your application can be configured to use syslogd instead of they having their own logging file then i think you can have a centralized place to see , compare all the logs. This requires you to know the configuration options of your application or tweaking its code if it does not have the options yet and knowledge about configuration options in syslogd or other similar tools. Happy hunting and learning :) .I would be interested to know how this is actually implemented. Please post the steps you took to achieve this is a blog or website or something so that can be found through google.
I ended up writing a python script.
Given a series of files, the script determines the type of file in each case (looking for keywords in the filename such as access_log ), and based on that type extracts a consistent timestamp value for each line. The script then splices the lines from each file into a sorted list of tuples (timestamp, file, log line), optionally filters for a given time period and prints the result.