Getting 408 errors on our logs with no request or user agent

Are you by any chance running your web servers in Amazon behind an Elastic Load Balancer?

It seems they generate a lot of 408 responses due to their health checks.

Some of the solutions from that forum thread:

• RequestReadTimeout header=0 body=0

  This disables the 408 responses if a request times out.

• Change the ELB health check to a different port.

• Disable logging for the ELB IP addresses with:

  SetEnvIf Remote_Addr "10\.0\.0\.5" exclude_from_log
  CustomLog logs/access_log common env=!exclude_from_log
  

And from this blog post:

• Adjust your request timeout to be 60 or above.

There are already quite a few good answers here, but I would like to hazard one additional note that hasn't been specifically addressed. As many of the previous commenters already mentioned, 408 indicates a timeout; and there are quite an array of circumstances in which timeouts occur on a web server.

With that said, 408 errors can be generated in a variety of cases where your server is being scanned for exploits. Clients in such cases rarely present a user agent and often end connections abruptly, resulting in an abortive shutdown for that connection that can generate a 408 error.

For example, let's say that I am a dastardly hacker who is scanning the internet for computers that are still vulnerable to the POODLE vulnerability. As such, I have written a script that opens connections to large chunks of IP addresses to find servers that will accept SSL version 3 - later on I will use that list to scan for the POODLE exploit specifically. All that this first script does is establish a connection using openssl to check for SSLv3, like this:

openssl s_client -connect [IP]:443 -ssl3

This command will, in many configurations of Apache, result in a 408 message exactly as you have described. Performing this command on two of my own servers resulted in this entry to the access log:

<remote IP address>  - - [04/Nov/2015:08:09:33 -0500] "-" 408 - "-" "-"

I wanted to make this clear as even in a situation where OP was not using any form of load balancing, 408 errors can occur in a variety of circumstances - some malicious, some indicating trouble with the client, and some indicating trouble with the server. (I noticed in the log provided by OP that a local IP was indicated as the remote IP, but OP did not specifically mention the use of a load balancer so I was not sure if OP had simply used a non-routable IP for the purposes of demonstration, as he did with the URL)

Anyway, even though my post if obviously way too late in the day to help the OP hopefully it might help others who arrive here looking for a solution to all of those damn timeout errors.

Something is connecting to the port and then never sending data. HTTP 408 is a "timeout" error. There's a good writeup here: http://www.checkupdown.com/status/E408.html

There are various reasons for a 408 Timeout. But lets start from a premise that everything's fine then at some point these 408's start appearing in your access log - i.e. 408 0 "-" "-".

Like many point out on the net, a 408 represents a connection is made but no request is sent in the appropriate time scale, therefore the server drops the connection with a 408. One arrogant individual actually responded to someone asking for help on this issue with - "What part of Timeout do you not understand".

I think that is very much a novice answer and demonstrates a total lack of understanding of how some security methods function with web-server software.

So back to the beginning, why am I seeing all these 408's. One thing you will have in common with the rest of us managing a server is the vast amount of attacks you receive on a daily basis. Now, what do you do about these? Well: you employ your chosen security methods to deal with them, this is what changes.

Let’s take a very easy example, Drop an IP address. Included in an iptabes file (rules.v4) you have "-A ufw-user-input -s 37.58.64.228 -j DROP". So along comes 37.58.64.228 the firewall recognises the IP and drops the connection. In a lot of configurations you wouldn't even know it's knocked on the door.

Now lets take a more advanced example, Drop the connection based on some criteria. Included in an iptabes file (rules.v4) you have "-A INPUT -p tcp -m tcp --dport 80 -m string --string "cgi" --algo bm --to 1000 -j DROP". This is different because in this iptable rule we are saying look at the first 1000 bytes of the request string and see if you can find a sub-string of "cgi" and if you do find that sub-string then don't go any further, just drop the connection.

Here the security method is good, but as far as your logs are concerned misleading. The 408 0 "-" "-" generated is the best apache can do under these circumstances. The connection was made and the request had to be accepted up to a point in order to apply the string comparison rule which ultimately results in a 408 because your rule met the criteria for the connection to be dropped. So, our little novice darlings couldn't be more wrong if they tried. A connection was made and a request was received (you just won't have visibility of it under these circumstances). Although a 408 is generated it is not a 'Timeout'; your server simply dropped the connection after the request was made in association with your firewall rule. There are many other rules, which would create the same 408 situation. Don't accept too literally the Server Error Code 408 explanation - 'Request Timed Out'.

Ideally there would be another Apache generated Error Code, for example - '499' which would mean 'Server Read Your Request And Decided It Just Couldn't Be Bothered Entertaining You - Sod Off HaHa'.

With the latest web-server software you can practically exclude DOS attacks and the new gene of browsers incorporating predictive capabilities don't cause this problem as some have suggested.

In short, the 408 is generated because the server did not respond to the request, so as far as the client is concerned the connection timed out, when in reality the server read the request but dropped the connection for reasons other than a timeout waiting for a request.

We had this very problem, and puzzled over it for quite a while. The best solution we came up with was suggested by AWS support's ELB team. It essentially hinges on ensuring that your httpd server's timeout settings are all greater than your ELB idle timeout setting (which defaults to 60 seconds).

• Ensure that your apache Timeout directive value is double the idle timeout setting of your ELB.
• Turn on the KeepAlive feature, ensure that MaxKeepAliveRequests is very large (either 0 for infinite or very high, like 2000), and that KeepAliveTimeout is greater than your ELB's idle timeout.

We found that the KeepAlive (and associated settings) setting specifically reduced the quantity of 408s to effectively 0 (we see a few, but very few).

I had this issue behind AWS Elastic Load Balancer. Health checks generated an awful amount of 408 responses in the log.

The only solution that worked for me was to have Load Balancer's Idle Timeout setting lower than its Health Check's Response Timeout.

A colleague recently remarked that while my last post gave a valid explanation how a 408 could have an association with a security measure, it offered no solution.

Piped Access log is my personal solution.

The following should work out-of-the-box on most Ubuntu configurations, and with minimal tinkering on other Apache configurations. I’ve chosen PHP because it’s the easiest to understand. There are two scripts: the first prevents a 408 being written to your access log. The second script sends all 408s to a separate log file. Either way the result is no more 408s in your access log. It's your choice which script to implement.

Use your favorite text editor, I use nano. Open the file where you have your 'LogFormat' and 'CustomLog' directives. Comment-out the originals with the usual # and add the following. You may find these directives in the file below.

sudo nano /etc/apache2/sites-available/default

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" AccessLogPipe

CustomLog "|/var/log/apache2/PipedAccessLog.php" AccessLogPipe env=!dontlog

NOTE: I don't log images in my access log. In my etc/apache2/httpd.conf file I include the line

SetEnvIfNoCase Request_URI ".(gif)|(jpg)|(png)|(css)|(js)|(ico)$" dontlog

If this is of no interest to you then remove the env=!dontlog from the CustomLog directive.

Now create one of the following PHP scripts (#!/usr/bin/php is a reference to the location of the interpreter, make sure that the location is correct for your system - you can do this by typing at the $ prompt; whereis php - this should return something like php: /usr/bin/php /usr/bin/X11/php /usr/share/man/man1/php.1.gz. As you can see #!/usr/bin/php is right for my setup).

sudo nano /var/log/apache2/PipedAccessLog.php

#!/usr/bin/php
<?php
  $file = '/var/log/apache2/access.log';
  $no408 = '"-" 408 0 "-" "-"';
  $stdin = fopen ('php://stdin', 'r');
  ob_implicit_flush (true);
  while ($line = fgets ($stdin)) {
    if($line != "") {
      if(stristr($line,$no408,true) == "") {
        file_put_contents($file, $line, FILE_APPEND | LOCK_EX);
      }
    }
  }
?>

sudo nano /var/log/apache2/PipedAccessLog.php

#!/usr/bin/php
<?php
  $file = '/var/log/apache2/access.log';
  $file408 = '/var/log/apache2/408.log';
  $no408 = '"-" 408 0 "-" "-"';
  $stdin = fopen ('php://stdin', 'r');
  ob_implicit_flush (true);
  while ($line = fgets ($stdin)) {
    if($line != "") {
      if(stristr($line,$no408,true) != "") {
        file_put_contents($file408, $line, FILE_APPEND | LOCK_EX);
      }
      else {
        file_put_contents($file, $line, FILE_APPEND | LOCK_EX);
      }
    }
  }
?>

Having saved the PipedAccessLog.php script; make sure that root has ownership by executing the following at the $ prompt.

sudo chown -R root:adm /var/log/apache2/PipedAccessLog.php

The PipedAccessLog.php script will need read/write and execute permissions so execute the following at the $ prompt.

sudo chmod 755 /var/log/apache2/PipedAccessLog.php

Finally to get everything working you need to restart the Apache service. Execute the following at the $ prompt.

sudo service apache2 restart

If your Apache logs are located elsewhere then change the paths to suit your configuration. Good Luck.

I have found that 408 errors are increasing in both number and frequency. The range of IP addresses they are originating from is also growing (these are logged to their own separate file). There are also evident log patterns displaying consecutive 408's from the same groups of IP that are not due to normal server timeouts in that the originator is attempting connections about 2 or 3 seconds apart in a cyclic pattern (there is no waiting for a timeout before another connection attempt) I see these as simple DDOS style connection attempts. In my opinion these are a type of confirmation message for the originator that a server is online... then they come back later with different tools.... If you increase your timeout interval, you are simply giving them a larger time_allocation to run their hack programs within.