We want to store logiles so that they are easily searchable and available for extended periods of time. The problem is that there are multiple servers that needs to be archived.
I've looked in to a few database based solutions. Database is pretty much the only way of storing this data so that it can be analyzed and searched. The problem is that the databases go one of 2 ways. Most solutions only focus on storing the data, which can be seen as a lack of indexes. The indexes are easy to create, but by creating indexes the database starts to grow out of control.
What I'm looking for is a solution that offers a balance between these extremes. I want to have a relatively good searching capabilities (searching for whole words or IP addresses) while the database stays relatively small.
For a good measure, I've seen a database with ~4GB of data grow in to 40GB just because of the indexes.
Splunk might just be what you're looking for. If you like FOSS better and don't need all the features of Splunk, take a look at graylog2.