I run the systat -tcp command on a moderately loaded system and see something like this for a 5 second window:
/0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10
Load Average ||||||||||||
TCP Connections TCP Packets
0 connections initiated 21062 total packets sent
153 connections accepted 16291 - data
153 connections established 125 - data (retransmit by dupack)
11 connections dropped 1 - data (retransmit by sack)
0 - in embryonic state 4271 - ack-only
5 - on retransmit timeout 0 - window probes
0 - by keepalive 217 - window updates
0 - from listen queue 0 - urgent data only
148 - control
0 - resends by PMTU discovery
TCP Timers 21057 total packets received
9752 potential rtt updates 13471 - in sequence
12437 - successful 49 - completely duplicate
763 delayed acks sent 11 - with some duplicate data
140 retransmit timeouts 14 - out-of-order
0 persist timeouts 225 - duplicate acks
0 keepalive probes 12535 - acks
0 - timeouts 0 - window probes
80 - window updates
0 - bad checksum
Why is the sent ack-only number so much higher than the "delayed acks sent" number plus the dup data received numbers? What situations generate ack-only packets besides incoming data (which should go through delayed ack timer) and received dupes (immediate ack). Keepalive, I guess. But this box has no long-lived idle connections. Everything is short and furious. And there's a keepalive line item there too that says zero...
What am I missing about the tcp machine here?
You seem to be reasoning that ack-only packets should only be sent when the machine either gets a dupe, or when the timer expires, but I don't see why that would be true. There is another category, perhaps more common and important than either of them, which is that the ack window is almost full.
If the other machine is streaming data to you and sends enough packets to get close to the ack window, the local machine will send it an ack. In the common case that it has no other data to send back, it will send an ack-only packet.
So, why are there any delayed acks? Because we don't want to ack every single packet the client sends: that would generate excessive back traffic.
Suppose the recv window would allow for 10 incoming packets. The remote machine sends us one. There is no need to ack it right away, because they know they can send us up to 9 more. If they do in fact keep sending packets, then as we get close to filling the window, we'll acknowledge their transmissions.
On the other hand, if they send us just one packet, and then stop for a while, we do want to at least acknowledge we got that one packet, so that they know not to retransmit it and so they know the current state of the window.
The delayed ack timer distinguishes between these two cases: letting them carry on if they're sending lots of data, without letting data sit unacknowledged for too long.