Home / server / Questions / 1065444
Accepted
Antoine
Asked: 2021-06-03 06:02:35 +0800 CST

How can I find which kubernetes certificate has expired?

  • 772

I have a kubeadm installed kubernetes cluster. Recently it stopped working. kubelet is running but seems stuck in initialization phases. I think the root cause is this recurring log in kube-apiserver:

1 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2021-06-02T13:18:50Z is after 2021-05-29T15:48:22Z

So there is a certificate issue, also kubectl is failing with unauthorized. The thing is, kubeadm certs check-expiration seems happy, and I even manually checked a few yaml config files (base64 decoded certificates, and run them through openssl to check the date). Nevertheless, I asked kubeadm to renew all certificates and rebooted everything, to no effect.

Any idea how I can identify which certificate exactly has expired ?

ssl-certificate kubernetes kubeadm

3 Answers

  1. Best Answer

    [acknowledgment and reference] I was helped by a kubernetes' dev here

    The expired certificate was /var/lib/kubelet/pki/kubelet/pki/kubelet-client-2020-*.pem. The certificates in /var/lib/kublet/pki/ are not handled by kubeadm cert but by kubelet itself, so it's supposed to be renewed automatically, but for some reason this didn't happen as planned for us. The kubelet-client-current.pem had been renewed, but something was still using an old (and expired) certificate.

    Here is how I fixed the issue:

    • /etc/kubernetes/kubelet.conf was obsolete, in particular using default-user instead of system:node:node_name. I deleted the file, created a kubeadm conf file and ran kubeadm init phase kubeconfig kubelet to recreate a clean kubelet.conf
    • /var/lib/kublet/pki/kubelet-client-current.pem is supposed to be a symlink, which was not the case for me. So I removed it.
    • restart kubelet and apiserver (kill the pod using containerd, docker, etc. since kubectl is unavailable) and wait for a new kubelet-client-current.pem to be created ; it should be a symlink.
    • run kubeadm init phase kubelet-finalize all
    • restart kubelet again
    • run kubeadm certs renew all
    • reboot (or restart kubelet and all control plane pods)
    • update your kubectl conf from /etc/kubernetes/admin.conf
    • 3

  2. Similar issues seems to be linked to NTP desynchronization.
    Try forcing time synchronization (run as root):

    # service ntp stop
# ntpd -gq
# service ntp start
    • 0

  3. Renew Kubernetes certificates (RUN on all master node)

    #kubeadm alpha certs check-expiration
#kubeadm init phase kubelet-finalize all
#kubeadm alpha certs renew all
#cd /etc/kubernetes
#kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > kubelet.conf
#mkdir -p $HOME/.kube
#cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#chown $(id -u):$(id -g) $HOME/.kube/config
#restart server
#kubectl get nodes
    • 0