Home / server / Questions / 1011997
Accepted
Meghana B Srinath
Asked: 2020-04-13 08:08:48 +0800 CST

How to suspend GKE node instances without deleting and re-creating the nodes

  • 772

We have a GKE cluster with autoscaling enabled. We are having google schedulers to shut down instances daily at a specific time. This also shuts down the GKE nodes. Since these clusters has autoscaling enabled and has a minimum node count, the nodes are getting recreated to match the minimal level, even after they are shut down by the script. Hence we are temporarily making the node count to 0 in the cluster and manually changing it back to the required number, when the nodes need to be started. When the node count is made 0, the original nodes are deleted from the cluster and new nodes are created when nodes are started. Is there any way in which nodes can just be shut down through the scripts(like normal GCP instances) instead of deleting and re-creating? There is a functionality in AWS autoscaling group where the suspend settings can be changed. So is there anything similar in GCP as well?

virtual-machines google-kubernetes-engine autoscaling

1 Answers

  1. Best Answer

    When you are using GKE Cluster Autoscaler you will change nodes number specified in your CA configuration based on some metrics.

    Cluster autoscaler works on a per-node pool basis. When you configure a node pool with cluster autoscaler, you specify a minimum and maximum size for the node pool. Cluster autoscaler increases or decreases the size of the node pool automatically, based on the resource requests (rather than actual resource utilization) of Pods running on that node pool's nodes. It periodically checks the status of Pods and nodes, and takes action

    As was mentioned in GKE docs - Decreasing the size of your cluster.

    Do not use both Cluster Autoscaler and manual resize commands simultaneously on a node pool, as this can cause interactions that result in unstable and/or incorrect node pool size.

    In this situation when you manually change node pool to lower than it was specified in configuration, Cluster autoscaler mechanism worked as designed and change number of nodes to minimal.

    As workaround for this situation you could consider to turn off Cluster Autoscaler before manually resizing the node pool. When you would need autoscaling of the nodes you would need to run CA again.

    Just for additional information, when you are using Cluster Autoscaling you can also use Horizontal Pod Autoscaler.

    • 3