Would you use Splunk?

One other thing to add. Our company recently looked into purchasing Splunk. We definitely had more than 500MB of logs to analyze and we found that their licensing model was outrageously expensive. Splunk has taken advantage of their increase in popularity and slowly increased their prices over the years. When we first looked at it 2 years ago, the limit on free was 1GB and the licensing fees were half of what they are now.

Splunk is a fantastic tool, but at it's current price, I would think hard about alternatives IMHO.

Install the logcheck package. It will scan the logs once an hour and email you anything it doesn't consider normal. Essentially, it emails anything that entered the logs in the last hour that it doesn't have a rule for ignoring. There are additional attack rules than include things which shouldn't be in the log. The email subject line varies depending on the reason things were picked up.

I generally build a local ignore file for it as I discover things which I consider normal, but don't have existing ignore rules.

The various syslog alternatives all support server consolidation, so you can forward the logs to a single server. However, I haven't been in the habit of doing it. The only system I forward logs off of is my OpenWRT firewall.

EDIT: I do use Splunk at work to search log files, although if I known the particular log I am looking for I am more likely to use less. It does have alert capabilities, but we don't use them. I expect they would alert on a match to a known record. This can lead to a lot of false negatives if you have new problems without an alert rule. I prefer to have false positives like I get from logcheck. Splunk may have better timeliness on alerts though.

I do get timely alerts from fail2ban on cases that cause it to trigger. It also maintains blacklist entries for the originating source.

Splunk isn't really in the same category of software as cPanel. From what I recall cPanel is a web based system management package. Splunk is a data analysis and alerting tool.

That said, according to our website:

"Download Splunk for free. You'll get all of the Enterprise features of Splunk for 60 days and you can index up to 500 megabytes of data per day. After 60 days, or anytime before then, you can convert to a perpetual Free license or purchase an Enterprise license to continue using the expanded functionality designed for multi-user Enterprise deployments. "

Sufficed to say, you can download and use Splunk, for free, on most average sized system log data sets.

As for diagnosing when your server is under attack, that's where Splunk would meet your needs. Check out my blog post from today where I show you how to setup iPhone alerts when someone attempts to login to your server with invalid credentials.

http://blogs.splunk.com/2010/08/16/how-to-use-notifo-to-receive-splunk-alerts-on-your-iphone/

Feel free to ping me back if you have any further questions or would like an extended demonstration license of our product.

I use and highly recommend http://papertrailapp.com for remote syslog. After you build your alerts and search filters its amazing and inexpensive!

We use Splunk for that kind of thing... Windows and Linux hosts all forward their logs to Splunk and there's some "saved searches" in splunk that generate alerts. Makes it possible to detect things like "more than X failed logins in the last 30 minutes" across all usernames and systems (the saved search is complex, though, in order to capture windows, linux system and various applications...). It's also great for searching through logs right now.

Splunk can be free for a small enough dataset.

You do need to scale the server appropriately, based on how much data you're sending to Splunk per day, how much you want to retain and how much searching you'll be doing. Otherwise it can bog down.

Before Splunk we used to use SEC on a syslog server that had all the logs forwarded to it. Much harder to write saved searches for, and doesn't really do anything to replace grep for searching for things after the fact. Still pretty decent, though.