Home / server / Questions / 24593
Accepted
Robert Munteanu
Asked: 2009-06-13 00:03:23 +0800 CST

Including hostname in apache logwatch reports

  • 772

When hosting multiple domains with apache it's useful to see the logwatch apache output with the virtual host name included, but I only get:

 --------------------- httpd Begin ------------------------

 Requests with error response codes
   400 Bad Request
      /: 1 Time(s)
      /robots.txt: 1 Time(s)

whereas I would like something like

 --------------------- httpd Begin ------------------------

 Requests with error response codes
   400 Bad Request
      example.com/: 1 Time(s)
      example.org/robots.txt: 1 Time(s)

How can I achieve this with logwatch?

apache-2.2 logwatch

6 Answers

  1. Best Answer

    Try this (works for me): Define LogFormat in your httpd.conf as

    LogFormat "%h %t [%V] \"%r\" %>s \"%{Referer}i\""

    With this particular case, you'll have remote_address, date/time, [The server name according to the UseCanonicalName setting], request, satus code and Referer (that's my desired format) and then put

    $LogFormat "%h %t %V \"%r\" %>s \"%{Referer}i\""

    in your services/http.conf LogWatch file. That will

    1. make apache put the hostname (Canonical or not, it depends if you use %v or %V)
    2. force LogWatch to understand your Apache Access log

    Here is an example of the a line in the log output with this particular set of directives:

    172.3.20.11 [01/Jun/2011:21:00:52 +0200] joomla.local "GET /images/tabs_back.png HTTP/1.1" 404 "http://joomla.local/templates/beez_20/css/personal.css"

    If we FOCUS ON ERROR CODES, and how are they treated in LogWatch, here are some changes you can made to /usr/share/logwatch/scripts/services/http: Add:

    my $my_host = ""; my $my_url = "";

    Then, about line 462, add this line to save our 4th column (HOST):

    $field{my_host} = $field{$log_fields[3]};

    And in line 560, after fmt_url is shorten ( if (length($field{url}) > 60) {...} ) add:

    $my_host = $field{$log_fields[3]};
$my_host = substr($my_host,1);
$my_url=$my_host . $fmt_url;

    Finally, change:

    $needs_exam{$field{http_rc}}{$fmt_url}++;

    by

    $needs_exam{$field{http_rc}}{$my_url}++;

    doing so, you'll have this in your Logwatch:

    Requests with error response codes
404 Not Found
joomla.local/images/tabs_back.png: 3 Time(s)

    I Hope it helps all you out

    • 3

  2. I had the same issue and solved it by changing the LogFormat in apache.conf (http://httpd.apache.org/docs/2.2/mod/mod_log_config.html)

    # LogFormat "%h %l %u %t \"%r\" %>s %O" common

# The default output has no info about the server name (%v).
# %m %U%q %H is strictly equivalent to %r.
LogFormat "%h %l %u %t \"%m %v%U%q %H\" %>s %O" common

    This generates the same output as the default, adding the canonical server name as a prefix. Eg:

    ... "GET www.example.com/apache_pb.gif HTTP/1.0" 200 2326 ...

    The pro is that you don't need any other customization (eg. on the logwatch side). The con is that you get a few extra characters for each logged line.

    • 3

  3. I don't think it is possible if you are logging all virtual domains into the same log file... The apache log will not differentiate between them.

    I would also suggest that you take a look at the open source OSSEC. We moved from logwatch to it, because it is in real time and allow centralized correlation (correlating things like ssh failed login with apache 400 errors).

    • 1

  4. I found a blog post detailing this problem and how someone resolved it. Don't know how this will affect log analysis, but noting it here as I found it useful.

    • 1

  5. You can get the hostname using the LogFormat directive in the apache configuration. You can use the following option

    %{Host}i

    I found this information in this link. Logwatch should be able to parse custom information.

    • 0

  6. Many thanks to @Syquus who put me on the right path to modifying the /usr/share/logwatch/scripts/services/http file.

    My file and solution was different but I thought I would share all the same.

    I use the standard vhost_combined LogFormat that Apache provides that looks like:

    LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined

    which outputs something like:

    example.org:80 1.1.1.1 - - [08/Oct/2013:16:55:01 +0000] "GET / HTTP/1.1" 200 6094 "-" "Opera/9.80 (X11; Linux x86_64; Edition Linux Mint) Presto/2.12.388 Version/12.16"

    I put this in the service configuration override at /etc/logwatch/conf/services/http.conf:

    $logformat = "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\""

    After finding the approximately correct places to make the changes for @Syquus's solution in /usr/share/logwatch/scripts/services/http, I thought simply changing the index from [3] to [0] would work - it didn't. I got incorrect segments of the path and even after traversing the whole hash/array, I didn't find the hostname. Debugging was frustrating because I'm new to Perl, but my solution was to add in matching for the %v which was being discarded and then modifying the url further down to include the domain name.

    Diff for my solution (I also removed the url truncation), YMMV:

    --- __http.2013-10-09   2013-10-09 13:11:48.000000000 +0000
+++ http        2013-10-09 14:36:59.000000000 +0000
@@ -132,6 +132,8 @@
 #  Build tables of the log format to parse it and determine whats what
 #

+my $my_url = "";
+
 my $detail = $ENV{'LOGWATCH_DETAIL_LEVEL'} || 0;
 my $ignoreURLs = $ENV{'http_ignore_urls'};
 my $ignoreIPs = $ENV{'http_ignore_ips'};
@@ -379,7 +381,10 @@
 $logformat =~ s/%[\d,!]*/%/g;
 while ($end_loop) {

-   if ($logformat =~ /\G%h/gc) {
+   if ($logformat =~ /\G%v/gc) {
+      $parse_string[$parse_index] .= "(\\S*?)";
+      $parse_field[$parse_index][$parse_subindex++] = "my_host";
+   } elsif ($logformat =~ /\G%h/gc) {
       $parse_string[$parse_index] .= "(\\S*?)";
       $parse_field[$parse_index][$parse_subindex++] = "client_ip";
    } elsif ($logformat =~ /\G%l/gc) {
@@ -437,7 +442,6 @@
 #
 #  Process log file on stdin
 #
-
 while (my $line = <STDIN>) {
    chomp($line);

@@ -580,11 +584,12 @@
          !((defined $ignoreURLs) && ($field{url} =~ /$ignoreURLs/)) &&
          !((defined $ignoreIPs) && ($field{client_ip} =~ /$ignoreIPs/)) ) {
       my $fmt_url = $field{url};
-      if (length($field{url}) > 60) {
-         $fmt_url = substr($field{url},0,42) . " ... " .
-                    substr($field{url},-15,15);
-      }
-      $needs_exam{$field{http_rc}}{$fmt_url}++;
+      #if (length($field{url}) > 60) {
+      #   $fmt_url = substr($field{url},0,42) . " ... " .
+      #              substr($field{url},-15,15);
+      #}
+      $my_url = $field{my_host} . $fmt_url;
+      $needs_exam{$field{http_rc}}{$my_url}++;
    }
    if (defined $field{userid} && $field{userid} ne "-" &&
          (eval $user_display) &&

    Should I decide to serve up secured content or content on a port other than :80 I might include it in the future. It should be obvious now how.

    Hope this helps!

    UPDATE

    Made some more changes, fixed a bug. Rather than keep editing this answer you can find my modifications here: https://bitbucket.org/ubiquitypress/logwatch

    • 0