We have a rather extensive set of yaml configuration files that we use to define the deployments, stateful sets, namespaces, services, etc. resources that should be created in the kubernetes API on a cluster.
We have experimented with a few tools like terraform and ansible for deploying the underlying compute and k8s cluster, and they work fine for applying configuration management at that level.
What I haven't found is a good way to intelligently automate the deployment and updating of these resources. We use source control to manage changes to these resource definitions and feed those changes into the test and production clusters with kubectl apply -f
.
Often the change is something simple like updating the image tag of for a pod in a deployment. In this case a simple patch of the proper image
property on the deployment is all that is needed.
For some of the cannonical resources I've played a bit with kubernetes terraform provider. It's rather cool as it is property aware and can do things like decide between an entity teardown/rebuild and a simple patch.
It falls down, however, in it's speed of development. It is hard to do anything beyond vanilla release k8s. This makes the tool useless for custom resources like used with operators. There are similar providers that will apply your k8s yaml definitions by shelling out to kubectl
, but they are not property-aware.
Any pointers on solutions would be appreciated before I start applying some bash-fu.