Home / server / Questions / 20933
In Process
natacado
Asked: 2009-06-06 16:32:32 +0800 CST

ApacheBench: Request times going into the ether

  • 772

I have an Apache server that's proxying to Tomcat via mod_jk. I've set up a custom log in Apache to log the access times:

LogFormat "%h %l %u %t \"%r\" %>s %b %D" transfertimes
CustomLog /path/to/transfer-times.log transfertimes

I'm running ApacheBench on the web server itself. I've noticed that as I increase the concurrency, my 99th percentile request times appear to exponentially increase, although the 50th percentile remains relatively stable. As an example, ApacheBench will say that roughly 10 requests of 1000 take > 1 second to respond with a concurrency level of 100.

However, when I look at the transfer-times.log, no requests show up as greater than 1 second, based on the %D in the LogFormat. I'm trying to figure out what would cause the disparity between Apache's log and ApacheBench's reported access times. Normally I could attribute it to network latency, but I'm running this all on a single host. I'm thinking there must be something quirky with Linux TCP parameters or file descriptors I need to tune, but I'm not sure where to start.

performance linux apache-2.2 benchmark

2 Answers

  1. It's possible that it isn't actually apache that takes so long for the request to serve but that you run into some other limit already defined in the OS.

    Have a look at what netstat gives you about the number of accepted connections, it's very well possible that you will find large differences in the number of requests apache sees (and thus is serving within the time frame apache knows about) and the number of requests the OS already could take care about.

    netstat -tulpen

    may be of help here. Also have a look at what sysctl tells you about the limits, or maybe even iptables.

    I'm sorry I can't remember any details since it's been quite a while since I ran into such issues.

    • 1

  2. I think that your outlying request is sitting in the accept backlog due to insufficient workers either in apache, or in the mod_jk handoff.

    To properly proxy between apache and tomcat you need to set the tomcat connector to handle as many connections as apache will fork children. If you don't apache will have to block waiting for tomcat to free up a connection. This can lead to additional blocking as those workers waiting on tomcat will not be available to handle incoming connections.

    http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html

    ps. Don't take %D as gospel. It only measures the time between the apache child starting work (so doesn't count the accept/fork/handoff duration) and the time the last byte is sent via send() (so is highly variable due to OS buffering).

    pps. tuning AJP is a black art, IMHO the benefits of AJP (slightly easier to parse request, persistent connections) are no longer enough of a reason to eschew straight HTTP proxying.

    • 0