Just downloaded a trial of Splunk, and am thinking of using it to monitor a Windows server base, with the associated apps, e.g.:
o Windows event logs / WMI queries (for Windows O/S, SQL Server, Exchange, etc)
o Apache/Jboss/Tomcat logs
o Oracle listener/db/etc logs
o Home-grown log files
Any short-but-sweet advice or gotchas?
Once you need it.. then you have to pay for it :P
Its what got me..
However its a really good application/tool..
Overall, you will probably be suprised how much you will log ( i was )...
I started off by loading a day of data for one of our applications and spending a few weeks just coming up with questions about the data I wanted answered: how many transactions per second of a particular customer, how busy are the busiest times for different transaction types, how can I search the logs for SLA violations, stuff like that.
It's a bit surprising how easy a lot of things are to search, and the more I searched the more ideas I had for new searches. Before long you'll get quite a catalog of saved searches.
The gotcha that got me in the beginning is making sure the time and hostname data is correct at index time. Some of our custom logs were not timestamped in a friendly format and took a few iterations to get it indexed correctly. Be sure to index a few small samples first to ensure everything looks correct before indexing a large collection of logs.
But, yeah, just imagine the questions you want answered about your data.
Splunk is great. Be SURE to know all the pricing before you get into it. It is really expensive.