Home / server / Questions / 226845
Accepted
Chris
Asked: 2011-01-26 12:58:25 +0800 CST

Haproxy Falling over under high volume

  • 772

I am having some issues with HAProxy configuration. I have been playing around with it to try and make it more resilient to high server loads and Denial of Service. However, I felt it was working fine until suddenly I was victim of a (D)DoS attack - Haproxy was reporting the backend as down even though I could still access it fine via the direct port.

Could somebody please check my HAProxy config and see if there is somewhere I am messing up or why I would be experiencing this.. I just can't seem to understand why this is happening.

Thanks in advance (and after of course) .

global

    # Global Max Connections
        maxconn                                 20000
        # Various Other Settings
        pidfile                                 /var/run/haproxy.pid
        stats socket                            /var/run/haproxy.stat mode 600 level admin
        stats timeout                           5m
        chroot                                  /usr/share/haproxy
        daemon
        # User Settings
        user                                    haproxy
        group                                   haproxy

defaults
        # Default configuration settings for Haproxy
        retries                                 2
        maxconn                                 19500
        timeout server                          10s
        timeout client                          10s
        timeout queue                           10s
        timeout connect                         10s
        timeout http-request                    10s
        # Error files
        errorfile 503 /etc/phpconf/haErrors/503.http

frontend Connection_Handler
        default_backend Primary
        bind :80
        mode                                    http
        option                                  forwardfor
        option                                  http-server-close
        maxconn                                 20000
        # Check if cookie exists
        #acl cookie_set hdr_sub(cookie) authorized=1
        # If cookie doesn't exist try and set it
        #redirect prefix * set-cookie authorized=1 if !cookie_set

        # If the cookie is still not set, send it to blocked backend
        #use_backend Cookie_Block if !cookie_set

        ## (D)DoS Mitigation ##
        # Setup stick table
        stick-table type ip size 1m expire 10m store gpc0
        # Configure the DoS src
        acl src_DoS src_get_gpc0(Connection_Handler) gt 0
        # Use DoS tarpit if src_DoS
        use_backend DoS_Tarpit if src_DoS
        # If not blocked, track the connection
        tcp-request connection track-sc1 src if ! src_DoS

listen Statistics_Engine
        mode http
        bind                                    XX.XXX.XX.XX:9012
        stats                                   enable
        stats uri                               /admin?stats=true

        stats auth                              admin:Password
        stats hide-version
        stats refresh 2s
        #stats scope # Add this option to provide stats for a singular backend

backend Primary
        # Option Configs
        option                                  httpclose
        option                                  redispatch
        option                                  abortonclose

        ## (D)DoS Mitigation ##
        # The following table is recording the IP, connection rate and bytes out rate
        stick-table type ip size 200k expire 10s store conn_rate(5s)

        # Track request and enforce rules
        tcp-request content track-sc2 src
        # Mark as abuse if exceeding connection rate
        acl conn_rate_abuse sc2_conn_rate gt 80
        # Mark as abuse if over X bytes
        acl data_rate_abuse sc2_bytes_out_rate gt 200000

        # Set ACL rule to enforce on frontend
        acl mark_as_DoS sc1_inc_gpc0 gt 0
        # Block connections marked as DoS
        tcp-request content reject if conn_rate_abuse mark_as_DoS
        #tcp-request content reject if data_rate_abuse mark_as_DoS

        # Configure Server
        mode http
        option forwardfor
        server Primary_HTTP 0.0.0.0:1080 check addr 127.0.0.1 port 80 inter 3000 rise 2 fall 3 maxconn 20000
        #fullconn 1024

backend Conn_Tarpit
        # Tarpit for connections
        mode http
        timeout tarpit 20s
        reqitarpit .
        errorfile 503 /etc/phpconf/haErrors/tarpit_503.txt

backend Cookie_Block
        # Block connections that will not take on a cookie
        mode http
        reqdeny .
        errorfile 503 /etc/phpconf/haErrors/503_cookie.txt

backend DoS_Tarpit

        # Tarpit for suspected attacks
        log 127.0.0.1 local1 info
        timeout tarpit 10s # Tarpit for 10 seconds
        errorfile 500 /etc/phpconf/haErrors/500_DoS.txt
        mode http
        reqitarpit .
haproxy

1 Answers

  1. Best Answer

    I see nothing obviously wrong in your config, you seem to have already taken care of correctly tuning your settings (especially maxconn). Is conntrack loaded on this machine ? The connection table might be full, preventing checks and connections from establishing to the server.

    Also, have you checked how many concurrent connections were sent to the server ? It's possible that the server is alternatively going up and down due to the load.

    Check the kernel log messages for any unexpected error.

    • 4