Questions[conntrack](server)

It first needs to store states. With some old BSD firewall that I used, I guess was named IPFW, I used to put a rule that sated "keep track of the state of the leaving packet", and this was placed on the outbound direction of interfaces. Then, another rule on the inbound direction that checked them against those states that were created by the rule on the outbound direction. So there used to be 2 rules: (1) to populate the states table, this was on the outbound direction, and (2) to lookup the states table, this was on the inbound direction.

But with connntrack, I see it applied on the INPUT chain, such as this rule:

iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

This makes me wonder, what is that statement actually doing?

• Is it saying that it will start tracking packets that match that rule by putting their information in the states table?
• Or is it saying that it already has the states information, and it is going to act against inbound messages based on it? (e.g. accept if they belonged to a previously accepted connection?). But, in this case, where did the states table get populated? Which rule does it? Or is it rule-less and implicit?

I'm using Ubuntu 11.10 & nginx. My server's currently doing about 350 rps (that's the load that's coming in). I use iptables to make sure connections on certain ports are restricted only to boxes I own.

I've noticed nf_conntrack_count keeps increasing. No matter what I push nf_conntrack_max to, nf_conntrack_count matches it within a day. Further, it doesn't match what netstat -tn tells me. Here are the numbers:

$ sudo sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
net.netfilter.nf_conntrack_count = 649715
net.netfilter.nf_conntrack_max = 650000


$ netstat -tn | awk '{n[$6]++} END { for(k in n) { print k, n[k]; }}'
CLOSING 6
ESTABLISHED 2933
FIN_WAIT1 116
FIN_WAIT2 3447
LAST_ACK 35
SYN_RECV 79
TIME_WAIT 27141


$ sudo conntrack -L | awk '{n[$4]++}; END {for(k in n) { print k, n[k]; }}'
conntrack v1.0.0 (conntrack-tools): 648611 flow entries have been shown.
CLOSE 443
CLOSE_WAIT 2210
ESTABLISHED 645529
FIN_WAIT 45
LAST_ACK 50
SYN_RECV 74
TIME_WAIT 259

I don't want to keep increasing nf_conntrack_max until I know exactly what's happening. I definitely do not have 650,000 connections to my box (single IP, so I don't have that many ports).

Any idea what's going on or what I can do to explain it? If you need more numbers, I can probably get them.

Note that the majority of my connections are HTTP (the only exceptions being my ssh sessions), and keepalive timeout in nginx is set to 15 seconds. Also net.netfilter.nf_conntrack_tcp_timeout_time_wait = 1

Any help appreciated.

I have a linux box which is setup as firewall/gateway for the network. Was just wondering why ss and other iproute2 tools show much less than iptables conntrack. Is it because the router function is happening in kernel only?

ss -na

Shows only two established connections where as a

conntrack -L -n

Shows 18 Established connections.

in a multi-ISP configuration, I'm routing and NATing specific traffic, e.g. VoIP, through specific interface - to a distinct provider. When one of the interfaces (or routes) becomes unavailable, all connections that were using it have to be dropped, and subsequent traffic has to be routed through the still working connection. Upon change in the status, I'm resetting and loading appropriate iptables and routing entries (it is "shorewall restart" - I'm using shorewall).

The problem is - the still present conntrack entries cause that the old (and now wrong) external address is still being used for NAT for those connections! After 'conntrack -D', the NAT works as expected again.

I'd like to delete only the conntrack entries belonging to the old external address or to solve the problem in a way that wouldn't affect connections through other interfaces.

E.g. - I'd like to delete all conntrack entries having reverse connection destination dst=old.ext.ip.adr, like

udp 17 164 src=192.168.158.3 dst=213.208.5.40 sport=5060 dport=5060 packets=178 bytes=104509 src=213.208.5.40 dst=old.ext.ip.adr sport=5060 dport=5060 packets=234 bytes=127268 [ASSURED] mark=256 secmark=0 use=2

What i've already tried:

# conntrack -D -r 212.108.43.143
^C (nothing happens, it just hangs)
# conntrack -D -r 213.208.5.40 -d 212.108.43.143
Operation failed: such conntrack doesn't exist

Thank you in advance! Best regards, Zrin

I have an Ubuntu 10.10 server with plenty of RAM, bandwidth and CPU. I'm seeing a strange, repeatable pattern in the distribution of latencies when serving static files from both Apache and nginx. Because the problem is common to both http servers, I'm wondering if I have misconfigured or poorly tuned Ubuntu's networking or cache parameters.

ab -n 1000 -c 4 http://apache-host/static-file.jpg:

Percentage of the requests served within a certain time (ms)
  50%      5
  66%   3007
  75%   3009
  80%   3011
  90%   9021
  95%   9032
  98%  21068
  99%  45105
 100%  45105 (longest request)

ab -n 1000 -c 4 http://nginx-host/static-file.jpg:

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     19
  75%   3011
  80%   3017
  90%   9021
  95%  12026
  98%  12028
  99%  18063
 100%  18063 (longest request)

The results consistently follow this kind of pattern - 50% or more of requests served as expected, then the remainder falling into discrete bands, with the slowest a few orders of magnitude slower.

Apache is 2.x and has mod_php installed. nginx is 1.0.x and has Passenger installed (but neither app server should be in the critical path for a static file). Load average was around 1 when each test was run (server has 12 physical cores). 5GB free ram, 7GB cached swap. Tests were run from localhost.

Here are the configuration changes I have made from Ubuntu server 10.10 defaults:

/etc/sysctl.conf:
    net.core.rmem_default = 65536
    net.core.wmem_default = 65536
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216
    net.ipv4.tcp_rmem = 4096 87380 16777216
    net.ipv4.tcp_wmem = 4096 65536 16777216
    net.ipv4.tcp_mem = 16777216 16777216 16777216
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.route.flush = 1
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.core.somaxconn = 8192 

/etc/security/limits.conf:
    * hard nofile 65535
    * soft nofile 65535
    root hard nofile 65535
    root soft nofile 65535

other config:
    ifconfig eth0 txqueuelen 1000

Please let me know if this kind of problem rings any bells, or if more information about the config would be helpful. Thanks for your time.

Update: Here's what I'm seeing after increasing net.netfilter.nf_conntrack_max as suggested below:

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      3
  99%      3
 100%      5 (longest request)