bakadevops's questions

I have a requirement to load balance to endpoints https://api1.app.com and https://api2.app.cpm that are currently outside GCP.

GCP provides Internet NEGs to plugin endpoints external to GCP to the HTTP(S) External load balancer. However, it has limitations.

Mainly:

1. Load balancing is currently not supported on internet endpoints – the requests are only proxied to the endpoint. Google Edge infrastructure terminates your user connections and then directs the connections to your internet endpoint.

2. No health checking is performed for the internet NEG. If your internet endpoint becomes unreachable or if it is specified as an FQDN but cannot be resolved, Cloud CDN sends a 502 (Bad Gateway) message in response to user requests.

3. You can only use internet NEGs on the Premium network service tier, which is the default tier.

Reference

I cannot really use Internet NEGs to load balance my endpoints. Is there any other option available in GCP to achieve load balancing for endpoint external to GCP using GCP Load Balancers?

Can I plugin HAProxy or any other third party load balancing solutions in between the GCP Load Balancers and my backend endpoints?

If so, can we configure domain names such as https://api1.app.com and https://api2.app.cpm as backend in haproxy.cfg?

I have an HAProxy server that I'm using as a L7 load balancer for my k8s nodes. My cluster is istio enabled and I have an istio-ingressgateway service exposed via NodePort

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                      AGE
istio-ingressgateway   NodePort    10.11.140.167   <none>        15021:30301/TCP,80:31916/TCP,443:31517/TCP,15012:30768/TCP,15443:32020/TCP   11d

From the HAProxy server I'm trying to perform health-checks on /healthz/ready endpoint. I'm using HAProxy 1.8 and my haproxy.cfg is as follows:

global
    log         /dev/log    local0
    log         /dev/log    local1 notice
    chroot      /var/lib/haproxy
    pidfile     /var/run/rh-haproxy18-haproxy.pid

    user        haproxy
    group       haproxy
    daemon
    stats       socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners

    spread-checks  21

    # Default SSL material locations
    ca-base     /etc/ssl/certs
    crt-base    /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # An alternative list with additional directives can be obtained from
    #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3

    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 10000
    balance                 roundrobin


frontend http-80
        bind *:80
        mode http
        option httplog
        default_backend www-80

backend www-80
        balance roundrobin
        mode http
        option httpchk /healthz/ready HTTP/1.1
        http-check expect status 200
        server backendnode1 155.13.200.29:31916 check port 30301 fall 3 rise 2 inter 1597
        server backendnode2 155.13.200.28:31916 check port 30301 fall 3 rise 2 inter 1597
        server backendnode3 155.13.200.27:31916 check port 30301 fall 3 rise 2 inter 1597


frontend health-80
    bind *:8080
    acl backend_dead nbsrv(www-80) lt 1
    monitor-uri /haproxy_status
    monitor fail if backend_dead

listen stats # Define a listen section called "stats"
    bind :9000 # Listen on localhost:9000
    mode http
    stats enable  # Enable stats page
    stats hide-version  # Hide HAProxy version
    stats realm Haproxy\ Statistics  # Title text for popup window
    stats uri /haproxy_stats  # Stats URI
    stats auth haproxy:passwd  

I am using HTTP/1.1 for backend health-check because istio-ingressgateway is not accepting HTTP/1.0 requests, it results in error code 426.

Hitting the backend from HAProxy server results in success:

curl -I  http://155.13.200.29:31916/healthz/ready
HTTP/1.1 200 OK
date: Fri, 11 Jun 2021 07:21:09 GMT
x-envoy-upstream-service-time: 0
server: envoy
transfer-encoding: chunked

However, the HAProxy health-check still won't pass. I'm getting the following errors:

Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode2 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 2ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[11795]: [WARNING] 161/071821 (11795) : Former worker 11798 exited with code 0
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 11 07:18:22 hap-server01 haproxy[12348]: Server www-80/backendnode3 is DOWN, reason: Layer7 wrong status, code: 400, info: "HTTP status check returned code <3C>400<3E>", check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

I understand that status code 400 occurs for bad requests. Is there something miss configured in my haproxy.cfg? I feel like it's the way I am trying to send HTTP/1.1 request for health-check. But, I'm not sure what else to add or what to modify in the configuration.

I initially created an EnvoyFilter to apply idle_timeout of 5s to outbound requests originating from workloads with label app: mecha-dev.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: tcp-idle-timeout
spec:
  workloadSelector:
    labels:
      app: mecha-dev
  filters:
    - listenerMatch:
        listenerType: SIDECAR_OUTBOUND
        listenerProtocol: TCP
      filterName: envoy.tcp_proxy
      filterType: NETWORK
      filterConfig:
        idle_timeout: 5s

However, the filter did not work for me and I found out it was because my cluster is using istio v1.2.5 which is a very old version and does not support EnvoyFilters properly and the envoy API for the same have been deprecated.

Therefore, I deleted the EnvoyFilter using kubectl.

But, even after deletion of the filter the istio-proxy sidecar containers keep on showing me the following error again and again:

[2021-05-11 11:59:14.109][22][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.155.0.113_443: Proto constraint validation failed (TcpProxyValidationError.StatPrefix: ["value length must be at least " '\x01' " bytes"]): idle_timeout {
  seconds: 5
}

I deleted the pods and re-created new ones but this error keeps on showing up. Is there anything else I need to clean up or delete to get rid of this error?

I have created an EnvoyFilter to apply TCP idle timeout to outbound requests. Here's my filter configuration:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: tcp-idle-timeout
spec:
  workloadSelector:
    labels:
      app: mecha-dev
  filters:
    - listenerMatch:
        listenerType: SIDECAR_OUTBOUND
        listenerProtocol: TCP
      filterName: envoy.tcp_proxy
      filterType: NETWORK
      filterConfig:
        idle_timeout: 5s

The istio version is 1.2.5, the envoyproxy version it uses is 1.11.0-dev.

I was successfully able to create the filter but it does not seem to have any effect. I checked istio config dump but I couldn't find my filter there, so I think my filter configuration is wrong.

I followed this document to create the filter.

The document does not give a lot of information. Since Istio version 1.2.5 is an old version, it wouldn't support my earlier filter configuration:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: listener-timeout-tcp
  namespace: istio-system
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          sni: some.app.com
          filter:
            name: envoy.filters.network.tcp_proxy
    patch:
      operation: MERGE
      value:
        name: envoy.filters.network.tcp_proxy
        typed_config:
          '@type': type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy
          idle_timeout: 5s

My main requirement is to create a filter that only applies to outbound requests going to some.app.com. I tried converting this filter to make it version 1.2.5 compatible (the first configuration) but I think it's not working.

What's wrong with my EnvoyFilter?