Home / server / Questions / 670300
In Process
Prashant Lakhera
Asked: 2015-02-21 15:01:06 +0800 CST

Sending Subversion logs to Logstash

  • 772

My requirement is to send subversion logs(i.e username,revision number...) to logstash for parsing(then store it in elastic search and finally displayed it via kibana).Since subversion use its own file-based database(FSFS) and not plain text file,I have two options

  1. Run svn log via cron(on 1 min interval) and then send that file to logstash(really bad idea)
  2. Used subversion river plugin,I tried it and its not working as it not able to index the data at all.In between its development is stop almost 1 year ago.So no help at all

I also thought about post commit script that whenever any user checkin it will trigger it and then store the logs in text file,but its same as Point 1.

Any help/idea to do that is really appreciated

EDIT

We write this small post-commit as I mentioned in point 3 so that whenever user check-in we can save the metadata in some file and then via syslog we can transfer this log to logstash server.One of the biggest drawback of this approach is I am dealing with TB of data and 15+ checkin per min,this file goes really big(we can use logrotate) but at the same time facing locking condition issue(as multiple user try to check-in and writing to the same file) which will eventually lead to race condition and make the situation more worst.Pasting post-commit hook below so that it might be useful for other people

 #!/bin/sh

 REPOS="$1"
 REV="$2"

 LOG="/tmp/svn.log"

 var1=/usr/bin/svnlook info -r $REV $REPOS | tr '\n' '|'`
 var2=/usr/bin/svnlook changed -r $REV $REPOS | tr '\n' ' '`
 echo "r${REV}|${var1}|${var2}\n" | tee -a ${LOG} 2>&1
 echo " " | tee -a ${LOG} 2>&1
svn

2 Answers

  1. I see at least one convenient option: 1) Feed you SVN logs to syslog, most distributions use rsyslog now, so here is example for rsyslog (5.x) :

    $InputFileName /${path_to}/svn.log
$InputFileTag svn:
$InputFileStateFile /var/spool/rsyslog/svn_log

$InputFileSeverity notice
$InputFileFacility local7
$InputRunFileMonitor

:syslogtag, isequal, "svn:" @@${IP_of_logstash}:$PORT
&~

    Please note, that config will different for newer versions of rsyslog. version 8.x config:

    #reading SVN logs
input(type="imfile" File="/var/log/${path_to}/svn.log"
Tag="svn:"
StateFile="/var/spool/rsyslog/svn_log"
Severity="normal"
Facility="local7")

:syslogtag, isequal, "svn:" @@${IP_of_logstash}:$PORT
&~

    2) Configure logstash syslog listener and parser for logs

    In this case logs will not be stored additionally in syslog, but forwarded directly to logstash and syslog transport will take care of it.

    • 2

  2. One possibility could be Configuring Syslog to accept the subversion logs and send it to logstash.

    usefull links

    http://logstash.net/docs/1.1.9/outputs/syslog

    http://linux.die.net/man/5/syslog.conf

    http://www.commandlinefu.com/commands/view/11687/send-apache-log-to-syslog-ng

    • 0