Home / server / Questions / 144697
In Process
Sergei
Asked: 2010-05-25 03:19:58 +0800 CST

Log parser for ftp server

  • 772

Can anyone suggest good log reporting software for Proftpd?
I am looking for something at least as good as http://xferlogdb.sourceforge.net where log is fed into the database and dynamic web pages are built to retrieve historical data and statistics per user, time period and so on. Xferlogdb is very helpful , but unfortunately latest release is 2004

ftp logging reporting proftpd

3 Answers

  1. http://www.proftpd.org/docs/howto/Logging.html lists like 5 packages that can be used.

    Hey, a note though - be aware that most of these packages do NOT do ftp reporting properly, though it's probably not their fault.

    We do a lot of software downloads here and we bit this problem off before. Because so many people use download accelerators of one sort or another, when there's a download for a 5 MB file it is likely to show up in your logs (though I haven't used proftpd, it's true of wuftpd and ncftpd) as a bunch of different lines of some smaller increment - like 100 lines of 50k apiece. If you run this through most log analysis programs, they will all tell you "100 downloads of your file, sir!" Which is bad and wrong.

    Some FTP daemons - and it looks like proftpd does this - have a "completion-status" field at the end that is set to "i" for incomplete or "c" for completed download. So in theory you could look for the "c"'s and determine how many were successful or not. But IT'S A TRAP! That field is set by the client. If you inspect your logs carefully you'll see some naughty stretches of some that set every transfer - even for just a specific byterange chunk of the file - to "c". Even if it was set correctly, it would be hard to determine completion rates as a percentage of total without doing stuff like "Well, we'll assume all attempts to this one file from one IP over, say... an hour? Is the same attempt."

    There are only two fixes to this I know of.

    1. Have your FTP daemon clearly log when it sends the first byte and last byte of a file (still, maybe they missed chunks in the middle, but it's close). I know Akamai hacks their FTP server to do this, we suggested it to the ncftpd guy and he put it on his list but don't think it's done.

    2. Keep track of how big every one of your files is and do complicated math to determine if you think one IP downloaded about the right amount of file. Doesn't fix someone who makes ten attemptes and gets 1/10 of the file before the transfer fails each time, but again it's closer.

    You don't have to go to this trouble if you don't care, but I wanted to FYI you that you can NOT trust the numbers most log analysis programs spit out for FTP, they will be off by an order of magnitude. Verify real logs against them to understand what it is you're getting.

    • 2

  2. AWStats supports ftp.

    • 1

  3. The proftpd docs page has links to The Webalizer and FTPWebLog.

    • 1