We're trying to figure out a way to periodically have system security patches for our K8s cluster, to keep our system safe and meet the security requirements.
our K8s clusters are running in different clouds, AWS, Azure, Bare metal, etc.
for clouds, we can change our IAM image to update to the latest, replace the old image, launch new nodes, and drain the old nodes. For bare metal one, we need to drain the old nodes, and then patch, and add them back.
Not sure if there is any other way to do that automatically. we don't want to do this work each month in each clouds. maybe there is a better solution?
Packer by HashiCorp is a free tool for automating the creation of machine images.
Packer can build machine images for different clouds including AWS, Azure, and self hosted virtualisation platforms like VMware.
Using Packer, you can consume published reference machines (Amazon published AMI's for example), automate the install of patches, updates, and any custom configuration, before publishing the patched image back to your cloud platform, ready to be consumed.
Packer can also be used to seed standard configuration management tools into cloud images. For example perform ansible or puppet runs, and/or bake in the necessary configuration for these to run when a instance starts up.
Terraform (available for free), also by HashiCorp, lets you automate the configuration of compute platforms, such as AWS/Azure/VMware, allowing you to update the configuration automatically.
If you wished to deploy or update the AMI used by nodes in a running Kubernetes cluster, you could likely do this by creating a new launch configuration pointing at the new Packer built image, using Terraform.
One way you could potentially reduce the amount of work, is to modify the boot script (cloud init/user data) of the instance group, to run something like a 'yum update' on boot. This way you could use the pre-made images, and simply need to update the AMI ID to the latest version, each time these are released. This could likely be done using Terraform.