Managing k8s deployment configs - Check changes before running`kubectl apply`

Robin Winslow Source

We currently run a number of our websites in a single Kubernetes cluster.

The config for the whole cluster and each of the services within it is contained in a set of configuration files that we keep in a "deployment-configs" repository, so that changes to that config can be reviewed using a standard Git workflow, before being applied to the cluster.

So at the moment, we have Jenkins jobs that kubectl apply -fs effectively all the files in that repository, which we run whenever something changes.

This feels rather brittle - if somehow a critical breaking change made its way into that repository then running that Jenkins job could bring down all of our production sites in one fell swoop.

The alternative is to selectively apply bits of the config when they change - e.g. one per service. The downside here is that we increase the chance that the actual shape of the cluster would be more likely to diverge from that described by the repository.

Is there a way to ask Kubernetes exactly what changes are going to be applied before actually applying them? This would help, 'cos we could do a checking step before applying the config to the whole cluster.

Or does anyone know of any other strategies for managing the config for a Kubernetes cluster?



answered 4 months ago VonC #1

kubectl diff (feature 491) should now (for Kubernetes 1.11 actually, Q2 2018) work correctly.

Issue 61145 has just been closed by PR 61227

Local and Live functions where doing and returning the same thing, giving empty results by default.
Fix the local function by copying the objects before fetching the live version.

That would help when it comes to ask Kubernetes exactly what changes are going to be applied before actually applying them.

comments powered by Disqus