Home / server / Questions / 1159948
Accepted
Lasse Michael Mølgaard
Asked: 2024-05-26 11:27:05 +0800 CST

How to make ingress-nginx use provided certificate to user?

  • 772

I am installing Home Assistant in my K3S Kubernetes cluster through Ansible and I want to use a Let's Encrypt certificate for my ingress that has been generated by Cert-Manager running on my cluster.

I am using the following playbook when installing Home Assistant:

---
- name: Create HomeAssistant namespace
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Namespace
      metadata:
        name: home-system

- name: Generate certificate for HomeAssistant
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: cert-manager.io/v1
      kind: Certificate
      metadata:
        name: homeassistant-tls
        namespace: home-system
      spec:
        secretName: homeassistant-tls-secret
        commonName: "homeassistant.example.com" 
        dnsNames:
        - "homeassistant.example.com"
        issuerRef:
          name: letsencrypt-staging
          kind: ClusterIssuer

- name: Get TLS certificate secret
  kubernetes.core.k8s_info:
    api_version: v1
    name: homeassistant-tls-secret
    kind: Secret
    namespace: home-system
  register: tls_secret
  
- name: Create ConfigMap for HomeAssistant
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: home-config
        namespace: home-system

      data:
        fullchain.pem: "{{ tls_secret.resources[0].data['tls.crt'] | b64decode }}"

        privkey.pem: "{{ tls_secret.resources[0].data['tls.key'] | b64decode }}"

        groups.yaml: |-

        scripts.yaml: |-

        scenes.yaml: |-

        known_devices.yaml: |- 
    
        automations.yaml: |-

        configuration.yaml: |-

          # Loads Default configuration - DO NOT REMOVE
          default_config:

          # Load frontend themes from the themes folder
          frontend:
            themes: !include_dir_merge_named themes
          
          # Text-to-speech
          tts:
            - platform: google_translate
          
          group: !include groups.yaml
          automation: !include automations.yaml
          script: !include scripts.yaml
          scene: !include scenes.yaml

          http:
            use_x_forwarded_for: true

            trusted_proxies:
            - 127.0.0.1
            - 10.42.0.0/16
            - 10.43.0.0/16
            - 192.168.0.0/24

            ssl_certificate: /ssl/fullchain.pem
            ssl_key: /ssl/privkey.pem

- name: Define storage space for HomeAssistant
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: homeassist-pvc
        namespace: home-system
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: small-longhorn # Store data on only 2 nodes instead of 3
        resources:
          requests:
            storage: 5Gi

- name: Add HomeAssistant StatefulSet
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: apps/v1
      kind: StatefulSet
      metadata:
        name: homeassistant
        namespace: home-system
        labels:
          app: homeassistant
      spec:
        serviceName: home-svc
        replicas: 1
        selector:
          matchLabels:
            app: homeassistant
        template:
          metadata:
            labels:
              app: homeassistant
          spec:
            volumes:
            - name: home-config-vol
              persistentVolumeClaim:
                claimName: homeassist-pvc

            - name: home-config-file
              configMap:
                name: home-config

            containers:
            - name: homeassistant
              image: lscr.io/linuxserver/homeassistant:latest
              ports:
                - containerPort: 8123
              volumeMounts:
              - name: home-config-vol
                mountPath: /config

              - name: home-config-file
                mountPath: /config/configuration.yaml
                subPath: configuration.yaml
                
              - name: home-config-file
                mountPath: /config/automations.yaml
                subPath: automations.yaml

              - name: home-config-file
                mountPath: /config/scripts.yaml
                subPath: scripts.yaml

              - name: home-config-file
                mountPath: /config/scenes.yaml
                subPath: scenes.yaml

              - name: home-config-file
                mountPath: /config/groups.yaml
                subPath: groups.yaml

              - name: home-config-file
                mountPath: /ssl/fullchain.pem
                subPath: fullchain.pem

              - name: home-config-file
                mountPath: /ssl/privkey.pem
                subPath: privkey.pem

              env:
                - name: PUID
                  value: "1000"
                - name: PGID
                  value: "1000"
                - name: TZ
                  value: "Etc/UTC"

            securityContext:
              capabilities:
                add:
                - NET_ADMIN
                - NET_RAW
                - SYS_ADMIN

              priviledged: true

- name: Define HomeAssistant service ports
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: v1
      kind: Service
      metadata:
        name: home-svc
        namespace: home-system
        annotations: 
          metallb.universe.tf/address-pool: default-pool
      spec:
        type: LoadBalancer
        externalTrafficPolicy: Local
        selector:
          app: homeassistant
        ports:
          - name: "8123"
            port: 8123
            targetPort: 8123

- name: Add HomeAssistant ingress
  kubernetes.core.k8s:
    state: present
    definition:
      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: homeassistant
        namespace: home-system

        annotations:
          kubernetes.io/ingress.class: nginx
          cert-manager.io/cluster-issuer: letsencrypt-staging
          nginx.ingress.kubernetes.io/backend-protocol: HTTPS
          nginx.ingress.kubernetes.io/ssl-redirect: "true"
          nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

      spec:
        ingressClassName: nginx
        tls:
        - hosts:
          - homeassistant.example.com
          secretName: homeassistant-tls

        rules:
        - host: 'homeassistant.example.com'
          http:
            paths:
            -  path: /
               pathType: Prefix
               backend:
                service:
                  name: home-svc
                  port:
                    number: 8123

I have replaced the actual FQDN with a name belonging to example.com. :-)

A few key points:

Home Assistant supports TLS encryption.

My first job is therefore to create a Let's Encrypt certificate, which I am naming homeassistant-tls. It has the associated secret homeassistant-tls-secret.

Next up is to extract certificate and key from the secret homeassistant-tls-secret and use it to create the files fullchain.pem and privkey.pem in the ConfigMap home-config.

The files contains the base64 decoded certificate and key.

The certificate and key files are then stored in the /ssl folder inside the container, which leads back to the last important file defined in the ConfigMap: configuration.yaml.

In order to enable TLS encryption we need to add the following lines to the http section of configuration.yaml:

  ssl_certificate: /ssl/fullchain.pem
  ssl_key: /ssl/privkey.pem

I tried calling the files tls.crt and tls.key, but apparently Home Assistant did not like it. :-)

All I know for sure is the files has to be PEM encoded.

After running the playbook I get the following output:

root@central:~# kubectl get svc -n home-system
NAME       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)          AGE
home-svc   LoadBalancer   10.43.23.186   192.168.0.203   8123:30630/TCP   81m

root@central:~# kubectl get ingress -n home-system
NAME            CLASS   HOSTS                       ADDRESS         PORTS     AGE
homeassistant   nginx   homeassistant.example.com   192.168.0.200   80, 443   85m

The website is working when I enter the address: https://192.168.0.203:8123/ or https://homeassistant.example.com in the browser.

However when I inspect the associated certificates I get the following:

  • https://192.168.0.203:803 (LoadBalancer) points at the Let's Encrypt certificate, which as far as I can tell is because I defined the certificates in the /ssl folder.
  • https://homeassistant.example.com (Ingress) points at a generic Kubernetes cluster certificate.

So how do I also make Ingress use the correct certificate? :-)

Changing the secretName to homeassist-tls-secret does not work for certain. :-)

kubernetes

2 Answers

  1. Best Answer

    The problem you're facing is that the ingress controller isn't always picking up the name of the game containing your Let's Encrypt Certificates for homeassistant.example.com .

    To fix your problem:

    Instead of specifying the secretName for your ingress appear,utilize annotations to reference the TLS secret and cluster issuer;

    annotations:


              certmanager.K8s.Io/provider: letsencrypt-staging
              nginx.Ingress.Kubernetes.Io/ssl-secret:homeassistant-tls-secret

    After updating the ingress appear with annotations its advocated to restart the nginx-ingress controller pod to choose up the changes

    By the usage of annotations you're providing greater context to the ingress controller. It can leverage the cluster issuer to validate the certificates and use the referenced secret homeassistant-tls-secret for serving secure visitors on homeassistant.example.Com

    The standards and configuration steps associated with the usage of ingress-nginx with Lets Encrypt certificates through annotations are normally the same for k8s and k3s.

    For more information see this Medium blog by Ivan Khramov

    • 1

  2. A quick reformatting and explanation to my comment to Sai Chandini Routhu's answer:

    Ivan Khramov's Medium Blog was almost on point to what was happening, although his implementation was not compatible with my cluster, but considering his blog post is 6 years old, I will attribute that to Kubernetes has made som renaming meanwhile.

    As stated in my comment there are no annotation named nginx.ingress.kubernetes.io/ssl-secret.

    The closest match is nginx.ingress.kubernetes.io/proxy-ssl-secret: secretName, but even that did not work when experimenting.

    What I did find was working is that if I renamed secretName in my ingress configuration from homeassistant-tls to homeassistant-ingress-tls then I will get the result I want.

    For reference see the configuration below that is working as intended:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: homeassistant
  namespace: home-system

  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    cert-manager.io/cluster-issuer: letsencrypt-staging
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"

spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - homeassistant.example.com
    secretName: homeassistant-ingress-tls
  rules:
  - host: 'homeassistant.example.com'
    http:
      paths:
      -  path: /
          pathType: Prefix
          backend:
          service:
            name: home-svc
            port:
              number: 8123

    The reason why it is working is because the ingress is calling cert-manager to create a new certificate that is essentially a copy of the certificate installed in the container.

    (I haven't checked if the hash code on both certificates are the same).

    Afterwards I am left with two certificates as seen here:

    root@central:~# kubectl get certs -n home-system
NAME                        READY   SECRET                      AGE
homeassistant-tls           True    homeassistant-tls-secret    10h
homeassistant-ingress-tls   True    homeassistant-ingress-tls   42m

    (I guess I should do some renaming in my definition of homeassistant-tls). :-)

    This makes my browser always see the Let's Encrypt certificate, no matter if I call the website via the ip address and thereby hitting my Load Balancer or via DNS and hitting the Ingress.

    I am happy now. :-)

    • 0