Home / server / Questions / 625088
Accepted
Matt Beckman
Asked: 2014-08-30 14:20:39 +0800 CST

AWStats LogFormat for AWS Elastic Load Balancer

  • 772

Hoping someone can point out what is wrong with the LogFormat I'm attempting to use with the access logs generated by Amazon ELB.

Multi-line for ServerFault readability:

%time5 %elb %host %host_port %host_r %host_r_port %request_processing_time 
%backend_processing_time %response_processing_time %code 
%backend_status_code %received_bytes %bytesd %methodurl

Example log entry (after cleanup):

2014-08-28T17:59:14 awseb-e-2-AWSEBLoa-AAAAAAAA 123.123.123.123 44153 10.123.123.123 80 0.000046 0.536613 0.000045 200 200 0 13129 "GET /path/to/web/app HTTP/1.1"

Logs are cleaned up before getting sent to AWStats. I use the following replacements:

cat ${s3_logs_dir}/* \
| sed -e "s/\(\.[0-9]\{1,3\}\):\([0-9]\{2,5\}\)/\1 \2/g" \
| sed -e "s/\(:[0-9]\{2\}\)\(\.[0-9]\{6\}Z\)/\1/g" \
| sed -e "s/http:\/\/www\.example\.com:80//g" \
> ${combined_log} 2>>${log_file}

First I detach host from port, secondly I remove microseconds from the %time5 ISO date, and finally I tried removing the absolute URL from the request.

amazon-web-services

3 Answers

  1. Best Answer

    Turns out it appears to be a bug with %time5 LogFormat option in AWStats.

    I converted YYYY-MM-DDTHH:MM:SS to YYYY-MM-DD HH:MM:SS and used %time2, and the logs were parsed successfully.

    • 1

  2. Morning, now this is outdated, but with awstats Advanced Web Statistics 7.7 (build 20180105) and your parsing rules with this criteria it also match agent.

    In current elb logs is also logs type of request even not documented, so new format is

    LogFormat="%other %time5 %other %host %host_port %host_r %host_r_port %request_processing_time %backend_processing_time %response_processing_time %code %backend_status_code %received_bytes %bytesd %methodurl %uaquot"

    %time5 is working now.

    thx for the posting.

    Kind regards David

    • 1

    • there is no dot followed by 1 to 3 digit followed by 2 dot then 2 to 5 digit in sample but a space is already in place for the port separator

      sed -e "s/(.[0-9]{1,3}):([0-9]{2,5})/\1 \2/g"

    • there is no double dot followed by 2 digit fowwloed by a dot, 6 digit and 'Z'

      sed -e "s/(:[0-9]{2})(.[0-9]{6}Z)/\1/g"

    ...

    Also, you dont need to pipe the sed action, just separe action by a semi column s/pat11/pat12/opt1;s/pat21/pat22/opt2;other action

    • 0