Home / server / Questions / 285396
Accepted
Tad
Asked: 2011-06-30 08:48:08 +0800 CST

How to use Apache Piped Logs to regex replace out unwanted data in logs in real time?

  • 772

I have https_acces_log log files being generated by Apache that a certain web app is logging unwanted data into the file. I can't stop the app from logging the data, so I'd like to write a perl/python script which will take the data output by Apache and run a regex on it to replace the data in real time.

Assuming Perl is the right tool, here's where I'm at so far. Script produces no output right now. Help anyone?

$|=1;                               # Use unbuffered output
while (<STDIN>)                     # Loop through STDIN
{
    $Msg = $_;                      # Capture the line of input

    if ($Msg =~ m/&passwd=\w+GET/ )         #   Look for the string I don't want
    {
        $Msg =~ s/&passwd=\w+GET/&password=XXXXXXGET/g;
        print $Msg;                 # Print it
    }
    else
    {
        print $Msg\n;
    }
}
logging apache-2.2 perl regex

3 Answers

  1. Best Answer

    I may be mistaken here, but I believe you would need to have your script handle opening and writing to the log file you are expecting. Apache would not capture the stdout of your script and place it in the log file.

    I don't run my own custom scripts with piped logs, but I do use rotatelogs and you do have to give it the path to the log file to write. Such as:

    CustomLog "|/usr/sbin/rotatelogs /var/log/httpd/ssl_access_log 86400" common

    • 2

  2. You have an error on the last print statement - you can't use \n without quoting it. It's not needed though, as $Msg should still have a newline, so you can just use print $Msg. If you're using -w this will trigger a warning ("unopened filehandle"), so lines not matching your pattern will not be printed at all.

    Removing that your script seems to work - that is, it replaces &passwd=testGET with &password=XXXXXXGET. Is that what you need?

    Edit: I took it you were trying this in the command line prior to actually trying to use it on Apache. See @TCampbell's answer if you're testing it in Apache already.

    Edit 2: Do use perl -w or use warnings if you're not doing it already. Also consider use strict. You'll be a happier person overall.

    • 1

  3. Well if you need to do so, there is an example on how to create pipe in syslog:

    First create a named pipe using mkfifo: 

    
mkfifo -p /home/data/syslog.pipe

    Make syslog.conf to points to this file: 

    
*.info |/home/data/syslog.pipe
    Restart syslog: 
    
sudo pkill -HUP syslogd
    Create processing script that read the pipe 
    $ cat > foo
cat /home/data/syslog.pipe | while read input
do
echo "Do some stuff" ${input}
done

    But be careful, you may have no mangled messages if they arrive faster than our program can process them.

    • 0