ingress returns 502 / TLS backend communication issue

i am operating a k8s cluster with cert-managerto obtain letsencrypt certs. I just set up nifi (https://github.com/cetic/helm-nifi) within this cluster, where the workload exposes a TLS secured port which is abstracted by a respective service.

i set up an ingress which routes the traffic to the respective nifi service. The certs is properly obtained as expected, but i keep getting a 502 bad gatewaymessage when trying ot access the service via ingress.

The ingress controller reveals the reason:

2020/11/07 15:45:20 [error] 10687#10687: *154495249 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 134.3.215.120, server: https://[my-domain], request: "GET /favicon.ico HTTP/2.0", upstream: "https://10.42.1.137:8443//favicon.ico", host: "https://[my-domain]", referrer: "https://[my-domain]" 

this is my ingress definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
 
    ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/secure-backends: "true"
  generation: 1
  labels:
    app: nifi
    chart: nifi-0.5.5
    heritage: Helm
    release: nifi
  name: nifi-ingress
  namespace: nifi
  resourceVersion: "138180326"
  uid: a6b561a8-1fc0-11eb-b2e1-6a9097f8076d
spec:
  rules:
  - host: [my-domain]
    http:
      paths:
      - backend:
          serviceName: nifi
          servicePort: 5422
        path: /
  tls:
  - hosts:
    - [my-domain]
    secretName: nifi-tls

it appears that ingress wants to establish a TLS connection which is rejected by the pod (quite obviously), but i have no idea how to fix this. It seems apparent that installing e.g. a self-signed cert on the nifi pod would do the job, but I have no idea how to do that and I rather believe i am doing something wrong here since this should work out of the box.

Thankful for any hint or idea.

UPDATE / clarification:

I set it up with a standard configuration now to make things a bit easier to understand. The core issue is from the ingress controller log:

2020/11/18 23:01:25 [error] 15529#15529: *166867031 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: xxx, server: [nifi-ingress-hostname], request: "GET /favicon.ico HTTP/2.0", upstream: "https://[nifi-pod]:8443/favicon.ico", host: "[nifi-hostname]", referrer: "https://[nifi-ingress-hostname]" 

[nifi-ingress-hostname] is the ingress which routes traffic to the nifi service on port 8443. The service forwards the traffic to [nifi-pod]:8443. the service on [nifi-pod] listens on 8443 for encrypted communication.

However the issue seems to be that the last step in the communication ingress to pod does not work since the ssl handshake fails. I added those annotations to the ingress:

    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"

to pass the encrypted traffic to the pod directly, but this had no effect. Somehow it seems that either i have to change the helm chart to change the service on the pod to http rather than https or ensure an encrypted connection with self-signed certs or am i missing anything?