Home / server / Questions / 202157
In Process
Mike Conigliaro
Asked: 2010-11-16 12:05:01 +0800 CST

Mysterious HAProxy request errors

  • 772

I see tons of request errors on one of my frontend proxies (on the order of a few per second), but I can't figure out what's causing them. I've tried using the "show errors" command on the stats socket...

echo "show errors" | socat unix-connect:/var/run/haproxy.stat stdio

But this returns nothing. Debug logging doesn't give me any hints either. Is there some other place I should be looking?

Edit: Just to clarify, there is no "error message" per se (though it would sure help to have one). I'm just looking at the counter labeled "request errors" in the web interface and the socat output, and I'm trying to figure out what's incrementing it.

haproxy

2 Answers

  1. Have you looked at dmesg? A common problem with Proxy servers is to hit the max Linux connection tracking since each request is using to connections. If this is the case you will see ip_conntrack: table full, dropping packet. in dmesg. You can see the current count and raise it via sysctl or proc:

    [kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_max
131072
[kbrandt@lb01: ~] cat /proc/sys/net/netfilter/nf_conntrack_count
185

    You can also bypass connection tracking with the NOTRACK target, i.e.:

    sudo /sbin/iptables -t raw -A PREROUTING -p tcp --sport 80 -j NOTRACK

    Keep in mind that it is a security risk to disable tracking though, you don't want to do it unless you are already behind a stateful firewall.

    Can you post the errors you are seeing?

    • 1

  2. I suspect Michael is only seeing them on the stats web page. Most likely those errors are simply due to port scans. If your haproxy configuration has "option dontlognull", you won't get any log for them, but still they are correctly accounted for. You'd see them in the "show errors" if they were invalid HTTP request and this does not seem to be the case. Don't worry for a few per second, it's almost nothing once you've figured that they are just port scans. Some heavy sites are getting between hundreds and thousands a second !

    @Tim: recent versions have fixed an issue that older ones had which caused some of these errors not to be reported (they were incidently accounted in the backend where they're not retrieved). I'm sure it's just that.

    • 1