Home / server / Questions / 506787
Accepted
Eric
Asked: 2013-05-11 09:12:44 +0800 CST

Syslog-ng Load Balancing

  • 772

I am working on setting up a syslog-ng server to accept logs from various sources. The general setup is going to be:

Device1
Device2                               Syslog1
Device3   -->   Load balancer   -->   Syslog2   -->   Shared SAN storage
Device4                               Syslog3
Device5

I am using 514/udp and with this setup, I should easily have HA and be able to round robin and spread the load. My concern is how the storage is going to be shared and written to on the backend. Ideally I'd like each device to have a deviceX.log file for the day. What happens if syslog1 gets a large log file in from device1 and is still writing to device1.log when syslog2 gets a small log file in from device1 and starts trying to write to the same file at the same time?

I have read that in Linux if you append the file and don't open it in "write" mode, then you will never get the write protected error. I am not sure how syslog does this.

So my questions are:

  1. What shared file system would you recommend for the SAN to share across the syslog boxes?
  2. Is there a better way to do the multiple syslog servers to 1 file? If syslog just appends, then I should be fine.

Any recommendations would be greatly appreciated. Thanks in advance.

load-balancing

2 Answers

  1. Not sure what you are using for a load balancer, but I would recommend that on your load balancer you configure some sort of "stickiness" so that Device1 logs will always get directed to Syslog1, etc.

    This would address your concern about the log file, but it would also prevent what I would consider to be a problem that is just as bad.

    My concern with your example would be if there were a lot of log entries passed from Device1 to Syslog1 and then a smaller number passed from Device1 to Syslog2 is that Syslog2 might be able to insert the later syslog messages out of order into your log file.

    • 0
  2. Best Answer

    I found that any sort of cluster file system such as GlusterFS would work fine for multiple servers writing to one file. However due to the SAN infrastructure I have, that was not possible. Therefore I ended up writing a file per day per syslog cluster.

    Syslog1 --> 2012-07-29-sys01.log
Syslog2 --> 2012-07-29-sys02.log

    I then used the logmerge script and a cronjob to combine the files and sort them by the correct time. The cron job below runs the day after the files are done being written to.

    /local/bin/logmerge -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log | gzip -9 -c > /logs/date -d yesterday +\%Y-\%m-\%d.log.gz && rm -f /logs/date -d yesterday +\%Y-\%m-\%d-sys*.log

    • 0