Questions[goaccess](server)

I'm trying to use goaccess log analysis tool, to analyse vsftpd FTP server's logs. I'm aware that being a web server log analyser, goaccess is not the best tool for this. That being said, it's flexible enough with the log format and we're already using it to analyse the web server's log, so I decided to give it a try.

By default vsftpd has a very chatty log:

Mon Mar 23 06:00:00 2020 [pid 11111] CONNECT: Client "1.1.1.1"
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] OK LOGIN: Client "1.1.1.1", anon password "blablabla"
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] FAIL DOWNLOAD: Client "1.1.1.1", "/file1", 0.00Kbyte/sec
Mon Mar 23 06:00:00 2020 [pid 11111] [ftp] OK DOWNLOAD: Client "1.1.1.1", "/file2", 17500 bytes, 203.15Kbyte/sec

As modern servers ­—usually— have single a line of log for a particular request, my first step was to change the logging to xferlog format, which gives a single line per transfer.

# /etc/vsftpd.conf
xferlog_enable=YES
xferlog_std_format=YES
xferlog_file=/var/log/vsftpd.log

Now I have log lines like the following.

Sat Mar 28 06:00:00 2020 1 1.1.1.1 17500 /file1 b _ o a anonymous ftp 0 * c
Sat Mar 28 06:00:00 2020 1 1.1.1.1 0 /file2 b _ o a anonymous@host ftp 0 * i

As this is a download server only getting anonymous & passive FTP requests, I can assume the anonymous and anonymous@host are whatever the client provides for anonymous FTP.

How can I analyse this log with goaccess?

This is my log format:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

Any ideas why I'm getting?

Parsed 10 lines producing the following errors:

Token '123.123.123.83,' doesn't match specifier '%h'
Token '123.123.123.183,' doesn't match specifier '%h'

I'm new to goaccess and still learning, please bear with me...

Here's an a few lines from the access_log, the formatting might not look very readible though:

123.13.123.83, 123.123.196.82 - - [10/Jun/2018:08:31:12 +0200] "GET /index.php/faqs/14-faqs/101-liquor-authority-faq HTTP/1.1" 200 39940 "http
s://www.google.com/search?hl=en-GB&q=liquor+licence+application+fees&sa=X&ved=0ahUKEwiB_ZzvusjbAhVFC8AKHSS6BB8Q1QIIPCgF" "Mozilla/5.0 (BB10; T
ouch) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.1.2576 Mobile Safari/537.35+"
123.13.123.83, 123.123.196.82 - - [10/Jun/2018:08:31:13 +0200] "GET /cache/widgetkit/widgetkit-86051325.css HTTP/1.1" 304 - "http://www.abcdef
.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.1.2576 M
obile Safari/537.35+"
123.13.1123.83, 123.123.196.82 - - [10/Jun/2018:08:31:13 +0200] "GET /plugins/content/accordionfaq/css/css.php?id=accordion1&faq=blueaccordion
faq HTTP/1.1" 200 2369 "http://www.abcdef.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) AppleWebKit/537.35+
(KHTML, like Gecko) Version/10.3.1.2576 Mobile Safari/537.35+"
123.123.176.83, 123.123.196.82 - - [10/Jun/2018:08:31:13 +0200] "GET /cache/widgetkit/widgetkit-41cccc93.js HTTP/1.1" 304 - "http://www.abcdef
.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) AppleWebKit/537.35+ (KHTML, like Gecko) Version/10.3.1.2576 M
obile Safari/537.35+"
111.115.144.236, 123.123.196.82 - - [10/Jun/2018:08:31:13 +0200] "GET /index.php/2013-07-09-21-44-18/2013-07-11-05-51-45 HTTP/1.1" 200 32536 "
-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; ARM; Touch; WPDesktop)"
123.123.176.83, 123.123.196.82 - - [10/Jun/2018:08:31:14 +0200] "GET /media/widgetkit/widgets/mediaplayer/mediaelement/mediaelement-and-player
.js?_=1528612274054 HTTP/1.1" 200 63289 "http://www.abcdef.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) App
leWebKit/537.35+ (KHTML, like Gecko) Version/10.3.1.2576 Mobile Safari/537.35+"
123.123.176.83, 123.123.196.82 - - [10/Jun/2018:08:31:14 +0200] "GET /media/widgetkit/widgets/lightbox/js/lightbox.js?_=1528612274053 HTTP/1.1
" 200 17068 "http://www.abcdef.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) AppleWebKit/537.35+ (KHTML, lik
e Gecko) Version/10.3.1.2576 Mobile Safari/537.35+"
123.123.176.83, 123.123.196.82 - - [10/Jun/2018:08:31:14 +0200] "GET /media/widgetkit/widgets/spotlight/js/spotlight.js?_=1528612274055 HTTP/1
.1" 200 2462 "http://www.abcdef.com/index.php/faqs/14-faqs/101-liquor-authority-faq" "Mozilla/5.0 (BB10; Touch) AppleWebKit/537.35+ (KHTML, li
ke Gecko) Version/10.3.1.2576 Mobile Safari/537.35+"

The main task of the analyzer - referring sites, referrers URLs. Logs are collected in one place from 3 nginx server. I install awstats with a specific parameter LogFormat - all working fine, but the customer was not enough, we need statistics referring site and referrers URLs per day, awstats provides only for a month.

I found project goaccess, but again, in the demo, I did not see the possibility of statistics for a day (or a certain period).

logstash + elasticsearch + kibana - never work with this soft, i need solution as fast as can)

Please, advise me the solution, thank you)

When running GoAccess with my Nginx log file I'm getting this error:

Fatal error has occurred
Error occured at: src/goaccess.c - main - 1017
Nothing valid to process. Verify your date/time/log format.

Log lines example:

example.com 66.87.119.148 - - [27/May/2016:10:45:50 +0000] "GET /uploads/5458ac234488de92aa9cda3a/logo300.png HTTP/1.1" 200 11153 "http://example.com/page1" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13D15 Safari/601.1"
example.com 207.250.34.193 - - [27/May/2016:10:45:50 +0000] "GET /assets/5.bundle.js HTTP/1.1" 304 0 "http://example.com/page2" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"

~/.goaccessrc:

time-format %H:%M:%S
date-format %d/%b/%Y
log-format %^ %h - - [%d] "%r" %s %b "%R" "%u"

executing like this:

goaccess -f /example.com_access.log -p ~/.goaccessrc