Home / server / Questions / 964532
Accepted
basin
Asked: 2019-04-26 00:53:13 +0800 CST

haproxy not performing health check after DNS restored

  • 772

My desired haproxy behavior is the following:

  1. don't fail to start when failed to resolve the backend address
  2. use libc to resolve addresses at start time; no DNS servers should be in config
  3. use libc to resolve addresses at run time
  4. always retry name resolution, if previous attempt failed
  5. Don't spam my DNS server every 10 seconds. Only resolve name when needed

_

  • for (1) I've added none to init-addr
  • for (2) I've added libc to init-addr

Not sure what to do for (3). The resolvers section seems to support DNS only, what the hell? If I add a nameserver there and resolvers to my server string, it starts to work, but it sends a DNS request every 10 seconds. I don't want that. When I'm making a client connection I'm willing to wait for haproxy to resolve the backend using libc, synchronously.

I disconnect from network, make sure that the backend is unresolvable and restart haproxy. Then I connect to network, make sure that the backend is now resolvable and try to use a client. Haproxy still refuses to retry connecting to the backend until I restart or reload it:

Apr 25 11:34:28 il systemd[1]: Starting HAProxy Load Balancer...
Apr 25 11:34:28 il haproxy[3411]: [WARNING] 114/113428 (3411) : parsing [/etc/haproxy/haproxy.cfg:49] : 'server app1' : could not resolve address 'basinasrv', disabling server.
Apr 25 11:34:28 il haproxy[3411]: Proxy main started.
Apr 25 11:34:28 il haproxy[3411]: Proxy app started.
Apr 25 11:34:28 il haproxy[3411]: [NOTICE] 114/113428 (3411) : New worker #1 (3412) forked
Apr 25 11:34:28 il systemd[1]: Started HAProxy Load Balancer.
...
Apr 25 11:46:21 il haproxy[3412]: 127.0.0.1:45302 [25/Apr/2019:11:46:21.946] main app/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 1/1/0/0/0 0/0 "CONNECT serverfault.com:443 HTTP/1.1"

_

defaults
    # never fail on address resolution
    default-server init-addr libc,none
    #default-server init-addr last,libc,none
    option log-health-checks
    option httplog

global
    ca-base /etc/ssl/certs
    maxconn     20000
    log         /dev-log local0 debug
    user        haproxy
    chroot      /usr/share/haproxy
    pidfile     /run/haproxy.pid
    daemon

frontend  main
    bind :3129
    mode                 http
    log                  global
    maxconn              8000
    timeout              client  30s
    default_backend             app

backend app
    log                  global
    mode        http
    reqadd      Proxy-Authorization:\ Basic\ xxxxxx
    timeout     connect 5s
    timeout     server  30s
    timeout     queue   30s
    server  app1 basinsrv:443 check ssl verify required ca-file ca-certificates.crt
haproxy

2 Answers

  1. Best Answer

    use libc to resolve addresses at run time; no DNS servers should be in config
    always retry name resolution

    As far as I know that is simply not possible.

    To use hostnames rather than ip-addresses in your HAproxy configuration you either allow HAproxy to do the resolving at startup (with all the issues that you already imply) OR you need to define one or more resolvers where you instruct HAproxy how to resolve those hostnames while running.

    See my answer here for an example, with most of the tuning options for DNS resolving.

    • 1

  2. OPTION 1: You can modify the hosts file for each URL in HAProxy. I have 2 servers balanced in HAProxy server1 and server2.

    Edit hosts file:

    # vim /etc/hosts

10.0.0.2    www.server1.com
10.0.0.3    www.server2.com

    you must add as servers as you have then just restart the server and it will start HAproxy like a charm, but, you will need to change your IPs manually if they change.

    OPTION 2: Use crontab to start Haproxy after the system reboot.

    #crontab -e

@reboot sleep 60 && systemctl start haproxy

    OPCION 3: If you are using cockpit Centos/RHEL, install it if you don't.

    Check if it is installed:

    rpm -qa | grep cockpit

    Enable cockpit:

    systemctl enable --now cockpit.socket

    Go to your server using your browser:

     https://<IP>:9090

    Then, go to Services menu > Timers > Create Timer

    And in the menu set the follow:

    Name     : Start HAPROXY
Command  : systemctl start haproxy 
Timer set: On Reboot
           60 seconds
    • 0