Home / server / Questions / 669596
Accepted
lexsys
Asked: 2015-02-19 06:16:50 +0800 CST

Configure HAProxy on Linux to accept 10K simultaneous connections

  • 772

I have HAProxy 1.5 running on Ubuntu 14.04 (modified). It accepts connections on http and https ports. Two backend applications process requests using persistent connection.

When I create around 2200 client connections haproxy stops accepting additional connections. But I want this system to accept at least 10K simultaneous connections.

Here is connection statistics:

# ss -s
TCP:   4119 (estab 4098, closed 6, orphaned 0, synrecv 0, timewait 6/0), ports 0

Transport Total     IP        IPv6
TCP       4113      4106      7
INET      4128      4117      11

I have already tuned maximum number of opened files for the process:

# cat /proc/1012/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            240017               240017               files

My haproxy.config file:

global
    log /dev/log syslog debug
    daemon
    user haproxy
    group haproxy
    maxconn 120000
    spread-checks 4

defaults
    log global
    timeout connect 30000ms
    timeout client 300000ms
    timeout server 300000ms

frontend http-in
    mode http
    bind :80
    option httplog
    option forwardfor
    reqadd X-Forwarded-Proto:\ http
    default_backend http-routers

frontend https-in
    mode http
    bind :443 ssl crt /opt/haproxy/cert.pem no-sslv3
    option httplog
    option forwardfor
    option http-server-close
    reqadd X-Forwarded-Proto:\ https
    default_backend http-routers

frontend ssl-in
    mode tcp
    bind :4443 ssl crt /opt/haproxy/cert.pem no-sslv3
    default_backend tcp-routers

backend http-routers
    mode http
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000

backend tcp-routers
    mode tcp
    balance roundrobin
        server node0 192.168.10.2:80 check inter 1000
        server node1 192.168.10.2:80 check inter 1000
linux

3 Answers

  1. Best Answer

    As far as I know, listen block maxconn is different then global maxconn. With global maxconn you limit the max number of connections you let the haproxy process handle.

    Listen / frontend section has its own maxconn, which limits the nubmer of connections per listener. So, try to set up maxconn in your frontend sections too, or at least set it up in default section.

    So either:

     defaults
     maxconn 10000

    or set it up per frontend.

    • 2

  2. What is the hardware configuration of your HAProxy machine? Cores, RAM, network interfaces, etc?

    We have run into many HAProxy performance issues. Here are some ideas:

    1. Split traffic into two HAProxy servers, with each acting as a backup for the other.

    2. Raise nbproc to reflect the number of cores on your particular hardware.

    3. If you are using layer 7 mode, see if you can drop down to layer 4 mode for more throughput.

    • 0

  3. Haproxy has changed since 1.5 version, but as the management guide points on section "5. File-descriptor limitations", these are the limits that you can encounter:

    1- Max number of file descriptors. Automatically set at startup by Haproxy.

    Rule maxconn value x 2 ~= max file descriptors

    On conf file:

    maxconn 500000

    Form web stats report:

    maxconn = 500.000
maxsock = 1.000.070

    And from system:

    # pidof haproxy
# cat /proc/630/limits | grep 'open files'
Limit                     Soft Limit             Hard Limit             Units 
Max open files            1.000.070              1.000.070              files

    Identificable error -> strace accept() or socket() return "-1 EMFILE"

    2- System-wide file descriptors. Set at boot based on the amount of memory

    # cat /proc/sys/fs/file-max
3.074.560

    Identificable error -> strace accept() or socket() return "-1 ENFILE"

    3- Per-process hard limit on the number of file descriptors.

    # cat /proc/sys/fs/nr_open 
1.048.576

    Identificable error -> strace accept() or socket() return "-1 ENFILE"

    • 0