Home / server / Questions / 1157039
Accepted
benjimin
Asked: 2024-03-28 19:52:15 +0800 CST

What does kubelet do if you delete the node?

  • 772

In a Kubernetes cluster, if you delete a node from the API server, how does the corresponding machine's kubelet respond?

Does it send a request to the API server to recreate the node resource, or does it shutdown and restart the machine?

kubernetes

1 Answers

  1. Best Answer

    From the source code, when kubelet runs it spawns a goroutine to continuously update the node object (via the kubernetes API server) to match the kubelet's own internal data. (The kubelet is able to create a node object when it first starts up, but once the node object exists, the current kubelet will not reattempt to create the node object if it vanishes.) The node sync will start failing after the node object gets deleted (flooding logs that would be visible on the host using journalctl -u kubelet) but has no direct feedback on the functioning of the kubelet.

    Meanwhile, the control plane will respond to the deleted node by garbage collecting the orphaned pod objects and (via deployment controllers etc) rescheduling them to other nodes. Since the kubelet also continuously monitors relevant pod objects, this will trigger it to terminate corresponding containers that were running on the original host (including daemonsets).

    In a dynamically scaled cluster, the cluster-autoscaler uses cloud provider APIs to shut down and relinquish machines that have been drained of pods (and also launches new machines as needed). Cluster-autoscaler may not be prepared to handle when a machine that it has been managing becomes unrepresented by any node object. Note the responsibility for deleting nodes is supposed to belong to the cloud controller, but the cloud node controller can't recreate the node object for machines that are still running (as it can't tell whether the machine is still running kubelet, indeed sysadmins could have repurposed that box for some role unrelated to the kubernetes cluster). Consequently, the zombie host may be left idling.

    Note, ideally kubelet itself might restore and maintain the node object (so that manually deleting a node object, like manually deleting a pod belonging to a deployment, would have no enduring effect), and the node object would only get removed when either the kubelet gracefully exits or the control plane loses contact with it. But that's not the current behaviour.

    • 1