Home / server / Questions / 1020310
Accepted
CodeChimp
Asked: 2020-06-07 09:22:41 +0800 CST

How do I add certificates to Kubernetes to allow images to be pulled from a custom Harbor repository?

  • 772

I am finding all sorts of walkthroughs on how to add certificates to be used in the pods themselves, but I can't seem to find info on how to setup Kubernetes to allow a self-signed cert for pulling images from a Harbor instance running inside the cluster. I have the ca cert imported to the system's trusted certs (system is running Ubuntu 18.04), but I am guessing Kubernetes uses it's own trusted certs store somewhere, similar to how Java ignores the system's trusted certs and relies on keystore files?

Edit To be more specific, what I am trying to do is deploy a custom Docker image stored in my Harbor instance that is running in my Kubernetes cluster. I have the certs for Harbor imported into my host systems OS, and I can run:

docker login <url_to_harbor>
docker pull <url_to_harbor>/library/custom/image:latest

and it works fine from CLI, but if I try to create a deployment yaml like so:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: custom-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom
  template:
    metadata:
      labels:
        app: custom
    spec:
      containers:
      - name: custom
        image: <url_to_harbor>/library/custom/image:latest
...

and run kubectl apply -f custom-deploy.yaml I get the following error when I get pods:

custom-deployment-6ff68947f6-8jj2p            0/1     ImagePullBackOff   0          13s

And if I get a description on the failed pod I see:

  Warning  Failed          18s                kubelet, node3     Failed to pull image "<url_to_harbor>/library/custom/image:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://<url_to_harbor>/v2/: x509: certificate signed by unknown authority
  Warning  Failed          18s                kubelet, node3     Error: ErrImagePull

I have restarted the host machine since I imported the CA certs. I added the certs by moving the root CA cert file into /usr/local/share/ca-certificates and then running sudo update-ca-certificates.

certificate-authority kubernetes ubuntu-18.04 self-signed-certificate

2 Answers

  1. Best Answer

    The method of importing CA cert is correct, but you have to do that on every cluster node: all masters and workers. Kubernetes simply relies on CAs installed on underlying operating system.

    • 3

  2. If you're in a situation where you may not want to install the CA cert on every node, you can also run a DaemonSet which configures the ca-certificate. Note, this will only work if Kubernetes is configured with the containerd runtime.

    apiVersion: v1
kind: ConfigMap
metadata:
  name: trusted-ca
  namespace: kube-system
data:
  ca.crt: |+
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: setup-script
  namespace: kube-system
data:
  setup.sh: |
    echo "$TRUSTED_CERT" > /usr/local/share/ca-certificates/ca.crt && update-ca-certificates && systemctl restart containerd
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: node-custom-setup
  labels:
    k8s-app: node-custom-setup
spec:
  selector:
    matchLabels:
      k8s-app: node-custom-setup
  template:
    metadata:
      labels:
        k8s-app: node-custom-setup
    spec:
      hostPID: true
      hostNetwork: true
      initContainers:
      - name: init-node
        command: ["nsenter"]
        args: ["--mount=/proc/1/ns/mnt", "--", "sh", "-c", "$(SETUP_SCRIPT)"]
        image: debian
        env:
        - name: TRUSTED_CERT
          valueFrom:
            configMapKeyRef:
              name: trusted-ca
              key: ca.crt
        - name: SETUP_SCRIPT
          valueFrom:
            configMapKeyRef:
              name: setup-script
              key: setup.sh
        securityContext:
          privileged: true
      containers:
      - name: wait
        image: k8s.gcr.io/pause:3.1

    Source: http://hypernephelist.com/2021/03/23/kubernetes-containerd-certificate.html

    These instructions were written prior to the implementation of a host registry directory in cri in containerd 1.5, so if running a new version, it may be possible to configure without restarts. See Registry Configuration - Introduction.

    Finally, if you're running on a cloud managed cluster, they may provide a custom configuration for managing CA Certificates. E.g., AKS - Custom CA

    • 0