Ben Williams's questions

We have a MySQL-based application that is very response time sensitive. A MySQL stored procedure needs to respond on average in less than 10 milliseconds and the maximum acceptable response time is 50 milliseconds. We are hitting the average response time with flying colors (more like 3 milliseconds) but over a five hour load test we will get some (~50) responses greater than 50 milliseconds with a maximum response time sometimes as high as 265 milliseconds.

There is a lot of information out there about tuning MySQL and Linux for optimal performance, and I believe we've applied most of the best practices, but I can find very little about how to reduce the incidence of outliers and reduce maximum response time.

How can I diagnose the cause of these outlier response times and what Linux and MySQL tunings can I apply to reduce them?

I have Postfix configured to filter mail using the SpamAssassin spamd daemon. I noticed recently that some spam messages would be delivered to my inbox with no SpamAssassin headers added. I tracked this down to the fact that I had a daily cron job to update SpamAssassin with sa-update and restart spamd. When Postfix gets a mail while spamd happens to be restarting, it delivers it with no filtering.

My solution was to alter the cron job to stop Postfix before restarting spamd and restart it after. Is this the best solution? Can I somehow tell Postfix to pause all delivery? Can I change the call to spamd so that it blocks until spamd comes back?

Here's the SpamAssassin line from my /etc/postfix/master.cf:

spamassassin unix  -       n       n       -       -       pipe user=nobody argv=/usr/bin/spamc -u ${user} -e /usr/sbin/sendmail -oi -f ${sender} ${recipient}

I found references to using postsuper -h ALL to put delivery on hold but this only seems to hold messages currently in a Postfix queue. New messages that come in after I issue postsuper -h ALL are still delivered normally.

Tomcat seems to send an ETag header with each response by default. I'd like to disable these for reasons outlined here. I know I can strip them out in my Apache configuration, but is there any way to disable them on the Tomcat side?

How can I quickly select and log in to the multiple hosts that I need to access via ssh? I'm building a shell script like the one below that uses dialog to show a menu of hosts from the command line but I'm curious whether there's a better way.

#!/bin/dash
tempfile=`tempfile 2>/dev/null` || tempfile=/tmp/test$$
trap "rm -f $tempfile" 0 1 2 5 15

while [ 1 ]
do
    dialog --menu "Select Host" 0 0 0 \
    "hostname1" "Host description 1" \
    "hostname2" "Host description 2" \
    "hostname3" "Host description 3" 2> $tempfile

    HOSTNAME=`cat $tempfile`

    if [ "x" = "x$HOSTNAME" ]; then
        break
    fi

    ssh $HOSTNAME
done

I've set up Logrotate to rotate my JBoss Application Server 4.2.2.GA logs nightly. After the log files have been rotated and JBoss starts writing to them again, the new log files begin with as many NUL characters as there were characters in the previous log file, followed by the new log messages. For example, if the JBoss server.log file was 5000 bytes long then, after rotation, the new server.log file will begin with 5000 NUL characters. After several days, server.log begins with NUL characters equivalent to the characters in all the previous days log files combined. It seems as if JBoss is remembering its position in the log file and picking up where it left off in the truncated file. Here is my logrotate config for JBoss:

/apps/jboss-4.2.2.GA/server/default/log/*log {
    daily
    rotate 30
    compress
    notifempty
    copytruncate
    missingok
    nocreate
}

I can't restart JBoss nightly because that would be too much downtime. I also can't use the log4j DailyRollingFileAppender as it doesn't delete old log files. Has anyone gotten logrotate working properly with JBoss?

I'm trying to solve a common annoyance: when SSH'd into remote servers I often do things that produce a lot more output than I'm expecting. For example, grepping a large log file with a pattern that turns out to be too general. As garbage churns by I sit there hitting ^C and wishing I'd remembered to pipe to less. My best idea so far has been to try to configure my shell to always page output, I asked another question about that. Other than that, I only have less-than-optimal solutions:

• Wait patiently for the output to finish
• Train myself to always pipe to less
• Get a bigger pipe and a faster box so output goes faster

What do you do?

I'd like to configure bash to page the output of all commands. Essentially, I want bash to behave as if every command I enter ended with '| less'. Is this possible?

What kernel parameter or other settings control the maximum number of TCP sockets that can be open on a Linux server? What are the tradeoffs of allowing more connections?

I noticed while load testing an Apache server with ab that it's pretty easy to max out the open connections on the server. If you leave off ab's -k option, which allows connection reuse, and have it send more than about 10,000 requests then Apache serves the first 11,000 or so requests and then halts for 60 seconds. A look at netstat output shows 11,000 connections in the TIME_WAIT state. Apparently, this is normal. Connections are kept open a default of 60 seconds even after the client is done with them for TCP reliability reasons.

It seems like this would be an easy way to DoS a server and I'm wondering what the usual tunings and precautions for it are.

Here's my test output:

# ab -c 5 -n 50000 http://localhost/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
apr_poll: The timeout specified has expired (70007)
Total of 11655 requests completed

Here's the netstat command I run during the test:

 # netstat --inet -p | grep "localhost:www" | sed -e 's/ \+/ /g' | cut -d' ' -f 1-4,6-7 | sort | uniq -c 
  11651 tcp 0 0 localhost:www TIME_WAIT -
      1 tcp 0 1 localhost:44423 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44424 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44425 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44426 SYN_SENT 7831/ab
      1 tcp 0 1 localhost:44428 SYN_SENT 7831/ab