Home / server / Questions / 1044816
Accepted
minaev
Asked: 2020-12-04 02:42:45 +0800 CST

HAProxy backend server returns "SSL handshake error"

  • 772

I know it's a frequently asked question which often means there's a problem with certificate validation. It doesn't seem to be the case, because I do not verify the certificate.

This is how my server specification looked in the beginning:

server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl verify none check resolvers mydns

Later it evolved to

server 1.base.maps.ls.hereapi.com [email protected]:443 ssl verify none force-tlsv12 check resolvers mydns resolve-prefer ipv4

But it always returns the same error:

Server freehere_maps_redirect/1.base.maps.ls.hereapi.com is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure"

URL is the real server name. There are 3 more servers in this backend (2.base.maps, 3.base.maps, 4.base.maps...), and all of them return the same error. These servers are not under my control.

I also tried to enable certificate verification, but, of course, it didn't fix the problem. No need to say that curl and other attempts to connect to the backend server work flawlessly.

HA-Proxy version 1.8.4-1deb90d 2018/02/08 running on CentOS7.

Your ideas will be greatly appreciated :)

Update: Here is the output of a successful request made with curl:

$ curl -v --tls-max 1.2 'https://1.base.maps.ls.hereapi.com/maptile/2.1/maptile/newest/normal.day/11/525/761/256/png8?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

*   Trying 13.33.243.105:443...
*   Trying 2600:9000:2118:be00:2:b190:a500:93a1:443...
* Connected to 1.base.maps.ls.hereapi.com (13.33.243.105) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=NL; ST=Noord-Brabant; L=Eindhoven; O=HERE Global BV; CN=*.base.maps.ls.hereapi.com
*  start date: May 10 17:54:34 2020 GMT
*  expire date: May 11 17:54:34 2021 GMT
*  subjectAltName: host "1.base.maps.ls.hereapi.com" matched cert's "*.base.maps.ls.hereapi.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55b593a60020)
> GET /maptile/2.1/maptile/newest/normal.day/11/525/761/256/png8?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/2
> Host: 1.base.maps.ls.hereapi.com
> user-agent: curl/7.72.0
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< content-type: image/png
< content-length: 30653
< access-control-allow-origin: *
< cache-control: public,max-age=86400
< date: Thu, 03 Dec 2020 12:51:37 GMT
< etag: 4742ccdf6d
< expires: Fri, 04 Dec 2020 12:51:37 GMT
< last-modified: Thu, 19 Nov 2020 20:58:37 GMT
< server: nginx
< x-nlp-irt: D=68945
< x-ols-tid: hjBM-ZZ2dSrOCS0R3IovQk_icd0LanCeb81VKszvSQzzTLygjksosg==
< x-served-by: i-094a2ddaf9791ccf9.eu-west-1b
< x-cache: Hit from cloudfront
< via: 1.1 228e9f9ffd3a938a52da99b2c67d587f.cloudfront.net (CloudFront)
< x-amz-cf-pop: HEL50-C1
< x-amz-cf-id: gALOyDPx1QY7dsP0NlaEBivgFHjgP8kLWPw5l2U2DfS1ZCEXOg7FDQ==
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination
* stopped the pause stream!
* Connection #0 to host 1.base.maps.ls.hereapi.com left intact

Update2: I tried to replace HTTP checks with TCP checks by modifying the backend to:

backend freehere_maps_redirect
        http-send-name-header Host
        http-request set-uri http://%[req.hdr(Host)]%[path]?apiKey=xxxxxxxxxxx&%[query]
        option tcp-check
        tcp-check connect port 443
        server 1.base.maps.ls.hereapi.com [email protected]:443 check

tcp-check passes, but I still can't get any results from the uplink server. On the other hand, it may point to a deeper problem, because I get HTTP 400 (Bad request) from CloudFront:

<H1>400 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Bad request.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.

Update 3: Some more details. I tried to use ssldump to view the SSL handshake process. Here's how it looks with haproxy:

1 1  0.1478 (0.1478)  C>S  Handshake
      ClientHello
        Version 3.3 
        cipher suites
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        TLS_DH_DSS_WITH_AES_256_GCM_SHA384
        TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
        TLS_DH_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
        TLS_DH_RSA_WITH_AES_256_CBC_SHA256
        TLS_DH_DSS_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DH_RSA_WITH_AES_256_CBC_SHA
        TLS_DH_DSS_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA
        TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_DH_DSS_WITH_CAMELLIA_256_CBC_SHA
        TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
        TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_GCM_SHA384
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        TLS_DH_DSS_WITH_AES_128_GCM_SHA256
        TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
        TLS_DH_RSA_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
        TLS_DH_RSA_WITH_AES_128_CBC_SHA256
        TLS_DH_DSS_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DH_RSA_WITH_AES_128_CBC_SHA
        TLS_DH_DSS_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_SEED_CBC_SHA
        TLS_DHE_DSS_WITH_SEED_CBC_SHA
        TLS_DH_RSA_WITH_SEED_CBC_SHA
        TLS_DH_DSS_WITH_SEED_CBC_SHA
        TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA
        TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_DH_DSS_WITH_CAMELLIA_128_CBC_SHA
        TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
        TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_GCM_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_SEED_CBC_SHA
        TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_IDEA_CBC_SHA
        TLS_ECDHE_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
        TLS_ECDH_RSA_WITH_RC4_128_SHA
        TLS_ECDH_ECDSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
1 2  0.3019 (0.1541)  S>C  Alert
    level           fatal
    value           handshake_failure
1 3  0.3019 (0.0000)  S>C  Alert
    level           warning
    value           close_notify
1    0.3024 (0.0004)  S>C  TCP FIN
1    0.3112 (0.0087)  C>S  TCP FIN

And here's the successful exchange using curl:

New TCP connection #1: localhost.localdomain(60630) <-> server-13-33-243-73.hel50.r.cloudfront.net(443)
1 1  0.2425 (0.2425)  C>S  Handshake
      ClientHello
        Version 3.3 
        cipher suites
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xcca9
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xcca8
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
        Unknown value 0xccaa
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_AES_256_GCM_SHA384
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_AES_128_GCM_SHA256
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        compression methods
                  NULL
1 2  0.3793 (0.1367)  S>C  Handshake
      ServerHello
        Version 3.3 
        session_id[32]=
          f8 f2 8b 5b 48 eb bb 7f d8 5c 70 e0 9c 86 30 0d 
          f7 3d 6c 52 2f 66 b7 33 84 09 1f bb 25 14 d9 f6 
        cipherSuite         TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        compressionMethod                   NULL
1 3  0.3994 (0.0201)  S>C  Handshake
      Certificate
1 4  0.3994 (0.0000)  S>C  Handshake
      ServerKeyExchange
1 5    0.3994   (0.0000)    S>C    Handshake
        ServerHelloDone
etc.

Does it mean it's the ciphers mismatch problem? But the TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 selected by the remote server during curl exchange is also present in the ciphers list offered by haproxy.

Tried also to add sni str(d2t4vhngii5504.cloudfront.net) or sni str(1.base.maps.ls.hereapi.com), but it didn't help either.

ssl haproxy

1 Answers

  1. Best Answer
    server ... [email protected]:443 ssl ... check ...

    The server in question requires SNI. Thus check-sni name instead of check need to be used. From the documentation:

    check-sni
    This option allows you to specify the SNI to be used when doing health checks over SSL. It is only possible to use a string to set . If you want to set a SNI for proxied traffic, see "sni".

    Therefore it should be:

    server ... [email protected]:443 ssl ... check check-sni 1.base.maps.ls.hereapi.com
    • 2