I would like to setup a syslog server to forward all log file from all of my VMs and servers. I really don't much about what is out there. So I turn to the community, Something on Linux is fine, what I want more is alert ability like emails telling me something is not right. If there was something to sort the logs by source that would be cool.
Where would I want run the syslog server from? My admin WS or a server/VM?
Any input would be wonderful.
Thanks in advance.
You should start with the idea of a centralized log server (which should be a server, not your desktop machine). The simple way to do this is to set up the syslog that already exists on all your machines to ship logs to a central host. That central host then puts all the log messages into one file or a small collection of files.
Here's a central loghost mini-howto to get you started. Note that traditional syslog has a number of limitations such as the ability to only send logs over udp and no support for log rotation. Many people use the open source version of syslog-ng to address these issues, although that does come at the price of a more complex configuration.
Once you have all your logs going to a central location, you can use various tools to analyze them. I'm particularly interested in the new open source tool logstash as a way to do this. There are also non-free tools such as Splunk which previous posters have already commented on.
Since you're setting up a server to hold the syslog, it should be a server, not your workstation. You turn off your workstation sometimes - a server needs to be available. You need to install and configure your central syslog software, and then you need to configure agents on all of your other servers (and network devices is a great idea too) to send to the syslog server. Then, if your syslog software has the capability of sending alerts based on received events, you can configure that.
I haven't set one up in quite some time, and it was on Windows anyway (SolarWinds Orion), so I can't recommend or describe any specific packages, but that's the general idea. I have heard good things about Splunk, as a general add-on to make more use of syslogged data.
If you are a member of SAGE there is also "Building a logging infrastructure"