Why does nf_conntrack_count keep increasing?

I'm using Ubuntu 11.10 & nginx. My server's currently doing about 350 rps (that's the load that's coming in). I use iptables to make sure connections on certain ports are restricted only to boxes I own.

I've noticed nf_conntrack_count keeps increasing. No matter what I push nf_conntrack_max to, nf_conntrack_count matches it within a day. Further, it doesn't match what netstat -tn tells me. Here are the numbers:

$ sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 649715
net.netfilter.nf_conntrack_max = 650000


$ netstat -tn | awk '{n[$6]++} END { for(k in n) { print k, n[k]; }}'
CLOSING 6
ESTABLISHED 2933
FIN_WAIT1 116
FIN_WAIT2 3447
LAST_ACK 35
SYN_RECV 79
TIME_WAIT 27141


$ sudo conntrack -L | awk '{n[$4]++}; END {for(k in n) { print k, n[k]; }}'
conntrack v1.0.0 (conntrack-tools): 648611 flow entries have been shown.
CLOSE 443
CLOSE_WAIT 2210
ESTABLISHED 645529
FIN_WAIT 45
LAST_ACK 50
SYN_RECV 74
TIME_WAIT 259

I don't want to keep increasing nf_conntrack_max until I know exactly what's happening. I definitely do not have 650,000 connections to my box (single IP, so I don't have that many ports).

Any idea what's going on or what I can do to explain it? If you need more numbers, I can probably get them.

Note that the majority of my connections are HTTP (the only exceptions being my ssh sessions), and keepalive timeout in nginx is set to 15 seconds. Also net.netfilter.nf_conntrack_tcp_timeout_time_wait = 1

Any help appreciated.