Valuable lessons in over-engineering the Core of Kubernetes Kops, Kris Nova

This is a creation in Article, where the information may have evolved or changed.

Kris Nova, of Microsoft, shares lessons on over-engineering from she experience contributing to Kubernetes Kops. In hers spare time, she runs a Kubernetes special Interest Group bringing Kubernetes to AWS, all and working on Microsoft Acs. She ' s writing a book called Cloud Native Infrastructure. She has contributed to the Go core codebase and helps moderate community discussions.

"Who's here have used Kubernetes in production?" Almost everyone in the packed hostel raises their hand.

Kubernetes Kops

Kubernetes Kops is short for "Kubernetes Operations". It ' s a cluster management tool. Think of it like and for kubectl clusters.

Kops models a cluster, then realizes it in a cloud

It's a statically linked CLI tool written in Go that imports the Kubernetes libraries ( import "k8s.io/kubernetes" ).

For the sake of this talk, think of Kubernetes as this wonderful, beautiful API, black-boxed. Kops calls into as API to translate your config files into a working cluster.

Kops takes a cluster definition to a working Kubernetes deployment in the cloud.

Here's what's some of the configuration YAML looks like:

apiVersion: “kops/v1alpha2”kind: “Cluster”kubernetesVersion: “1.7.0”networkCIDR: “172.20.0.0/16”masterPublicName: “api.nivenly.com”

How does does Kops work?

Roughly, here's what it does:

1. State Store:the declarative model of what the cluster looks like
2. Unmarshal structs:unmarshals configuration into Go structs
3. Map:maps these structs to tasks
4. Tasks:generate a list of tasks that should is running in production
5. Compare new tasks with the current tasks running
6. Reconcile:make necessary changes to production to reflect configuration

Kops 1.4

To understand the pitfalls of over-engineering, we need to go back on time to Kops 1.4. In Kops 1.4, the tasks is defined in YAML files, templated with the standard Lib's package text/template . They wanted the task definitions to is declarative, so YAML seemed like a natural choice.

Here's a snippet of what a task model file looked like:

From here, the logic would is translated into actual tasks via a rough pipeline of:

1. Embed the templated YAML in a. Go file
2. Importtext/template
3. Parsed at runtime to translate into tasks

Here's what happens in Kops 1.4 if you want to the change and re-deploy your configuration:

Note This iteration requires changing production state, so it's a relatively high latency cycle time. And so it really sucks if a deployment fails after a few iterations through this cycle.

To avoid such frustrating issues caused by runtime errors, they had to get really good at dealing with YAML. At times they spent more time looking at YAML than actual Go code. At this point, they had-over-$ YAML files.

Kops 1.5 (Next release)

For the release of Kops 1.5, there were a lot of new feature requests (from GITHUB issues, but only 3 maintainers). But their development process was brittle. Too many fragile YAML files. Kubernetes moves fast and Kops is desperately trying to keep up but falling behind.

They couldn ' t easily test all the logic embedded in their YAML files. Issues would often pop up at runtime.

So, in a nutshell:

Kris describes being an open-source maintainer as kind of the movie, The Sandlot. A Bunch of friends getting together to play ball, write code. The Beast (the big scary Dog in the movie, the steals the Kids ' baseball), the "the" the issue of dealing with the IR Overly complex YAML. Kris was "Smalls" and the character who confronts the Beast.

Kris convinced the team to scrap all their templated YAML files. In one commit, they removed the entire Kops models directory where the YAML is.

How does they do it? They just rm -rf d the entire directory and then observed what broke. And then they fixed the issues one bug at a time.

What did they remove exactly?

In a nutshell, they got rid of a custom programming language they had inadvertently created via YAML and text/template . Their YAML models were effectively written in a Turing-complete language composed of Yaml and text/template directives. They had been heavily abusing the FuncMap type in text/template .

Clearly, this is suboptimal. Given text/template that can is used as a programming language, let's compare it with Go.

Errors

• go:enforced Error Handling
• Text/template:panic

Compiling

• Compile Error on Invalid syntax
• Compiles just fine, no errors or warnings

Debugging

• Stack traces with line numbers
• "Something broke"

Developing

• Good IDE support
• No IDE support

Testing

• go test
• N/A

So, looking at the scoreboard, it ' s go:5, text/template:0.

So, they moved all their task definitions from YAML to Go.

They turned lines of YAML into ten Go structs. They wrote real tests that exercised real code. They deleted entire packages.

They removed a lot of complexity, which is getting in the the to not just technical progress, but also community developm Ent. People started saying, "I get Kops now." I want to help. How does I contribute? "Kops development became exciting again.

In a nutshell, "we fixed our shit." Ultimately, the lesson was learned was:

Simple Go Wins.