I have an HAProxy server that I'm using as a L7 load balancer for my k8s nodes. My cluster is istio enabled and I have an istio-ingressgateway service exposed via NodePort
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway NodePort 10.11.140.167 <none> 15021:30301/TCP,80:31916/TCP,443:31517/TCP,15012:30768/TCP,15443:32020/TCP 11d
From the HAProxy server I'm trying to perform health-checks on /healthz/ready
endpoint. I'm using HAProxy 1.8 and my haproxy.cfg
is as follows:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
pidfile /var/run/rh-haproxy18-haproxy.pid
user haproxy
group haproxy
daemon
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
spread-checks 21
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
# https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 10000
balance roundrobin
frontend http-80
bind *:80
mode http
option httplog
default_backend www-80
backend www-80
balance roundrobin
mode http
option httpchk /healthz/ready HTTP/1.1
http-check expect status 200
server backendnode1 155.13.200.29:31916 check port 30301 fall 3 rise 2 inter 1597
server backendnode2 155.13.200.28:31916 check port 30301 fall 3 rise 2 inter 1597
server backendnode3 155.13.200.27:31916 check port 30301 fall 3 rise 2 inter 1597
frontend health-80
bind *:8080
acl backend_dead nbsrv(www-80) lt 1
monitor-uri /haproxy_status
monitor fail if backend_dead
listen stats # Define a listen section called "stats"
bind :9000 # Listen on localhost:9000
mode http
stats enable # Enable stats page
stats hide-version # Hide HAProxy version
stats realm Haproxy\ Statistics # Title text for popup window
stats uri /haproxy_stats # Stats URI
stats auth haproxy:passwd
I am using HTTP/1.1
for backend health-check because istio-ingressgateway
is not accepting HTTP/1.0
requests, it results in error code 426
.
Hitting the backend from HAProxy server results in success:
curl -I http://155.13.200.29:31916/healthz/ready
HTTP/1.1 200 OK
date: Fri, 11 Jun 2021 07:21:09 GMT
x-envoy-upstream-service-time: 0
server: envoy
transfer-encoding: chunked
However, the HAProxy health-check still won't pass. I'm getting the following errors:
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[11795]: [WARNING] 161/071821 (11795) : Former worker 11798 exited with code 0
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
I understand that status code 400
occurs for bad requests. Is there something miss configured in my haproxy.cfg
? I feel like it's the way I am trying to send HTTP/1.1
request for health-check. But, I'm not sure what else to add or what to modify in the configuration.
Check the manual, you can't specify version without specifying the method:
in your case, I'd try
Also: learn about tcpdump, very interesting to watch the communication between systems and find out what's wrong.