Go for Control-Plane Development

The subset of Go that matters when you are writing controllers, admission webhooks, and platform APIs — context propagation, the error model, concurrency failure modes, and production diagnostics.

Overview

Go for control-plane software

This topic is not a Go tutorial. It is the slice of Go you get asked about when the job is writing control-plane software: operators, admission webhooks, platform APIs and CLI tooling that talks to the Kubernetes API. It assumes you can write a loop and a struct, and explains the model behind the idioms in controller-runtime, client-go and Terraform providers.

A control-plane process is mostly a loop that observes remote state, compares it with desired state and issues remote writes.

The three primitives

Everything hard about that loop is either cancellation or concurrency. Go aims a primitive squarely at each, plus a third that keeps failure explicit.

A remote call must not outlive its deadline or process shutdown — context.Context — Hung workers, uncancellable writes during a rollout
Many loops run at once and share caches — Goroutines and channels — Serial bottlenecks, blocked reconcilers
Failure must stay visible — The error value — Stack unwinding that hides control flow

Versions worth knowing

Go 1.26 is current as of February 2026; 1.25 still supported under the two-release policy — Shows you track the toolchain
Go 1.22 made loop variables per-iteration — Deleted the most common goroutine bug, but only for modules whose go.mod declares go 1.22 or newer
Modules, not GOPATH, are the dependency model — Describing GOPATH workflows signals you have not written Go since 2019

> [!TIP] Your operator, your webhook and the tooling that patches CRs in CI are all Go binaries. Interviewers here are almost never asking about syntax.

What an interviewer probes

The real questions are behavioral: what happens when the API server closes a watch mid-reconcile, whether worker goroutines exit on SIGTERM, and whether you can find a leak in a process you cannot restart.

> [!IMPORTANT] These five questions separate people who have shipped a controller from people who have read about one.

Key concepts

context.Context: An immutable value carrying a cancellation signal, an optional deadline, and request-scoped values. Done() returns a channel closed on cancel or deadline; Err() returns context.Canceled or context.DeadlineExceeded. Derived with context.WithCancel, WithTimeout, WithDeadline, WithValue. Passed as the first argument named ctx; never stored in a struct field. Every Kubernetes client method takes one.
Goroutine: A function scheduled onto an OS thread by the Go runtime, starting with a few kilobytes of stack that grows on demand. Cheap enough that tens of thousands are routine. There is no goroutine ID and no way to kill one from outside — a goroutine exits only by returning, so every long-lived goroutine must select on a context Done channel or it leaks.
select: Blocks until one of several channel operations is ready, choosing pseudo-randomly among ready cases. A default case makes the whole select non-blocking. The canonical control-plane form is select on ctx.Done() versus a result channel versus a time.After timer — that one statement expresses cancellation, success, and timeout together.
Error wrapping: fmt.Errorf("get pod %s: %w", name, err) embeds err in a chain reachable by errors.Is (compare against a sentinel value) and errors.As (assert into a concrete type). Using %v instead of %w formats the text but breaks the chain deliberately, hiding an implementation detail from callers. Wrapping makes the wrapped error part of your public API.
Accept interfaces, return structs: Take the narrowest interface a function needs as a parameter (io.Reader, client.Reader) and return a concrete type. Interfaces are satisfied implicitly, so the interface is declared in the consuming package, not the producing one. This is what makes a reconciler testable against a fake client without the reconciler importing a test package.
Struct tags: Backtick-quoted metadata on struct fields read at runtime by reflection: json, yaml, protobuf. In a CRD API type the json tag names the field in the serialized object and omitempty controls whether a zero value is emitted. controller-gen reads these tags plus marker comments to generate the CRD OpenAPI schema, so a wrong tag becomes a wrong API.
Race detector: go test -race and go build -race enable a ThreadSanitizer-based runtime that flags unsynchronized concurrent access where at least one access is a write. Roughly 5-10x CPU and memory cost. Reports only races actually executed, so its value is proportional to how concurrent your tests are.
pprof: Sampling profiler built into the runtime. Profiles: cpu, heap (alloc and inuse), goroutine, block, mutex, threadcreate. Exposed over HTTP by importing net/http/pprof, or written from tests with -cpuprofile / -memprofile. go tool pprof renders text, graphs, and flame graphs. The goroutine profile at debug=2 is the fastest leak diagnosis available.

When to use it

Writing a Kubernetes operator, admission webhook, or scheduler extender — controller-runtime and client-go are Go-only
Building a platform API or internal CLI that must ship as a single static binary to machines you do not control
Any long-lived service where you need deterministic shutdown, in-process profiling, and low memory overhead per concurrent request
Extending existing infrastructure — Terraform providers, Prometheus exporters, CSI drivers, and CNI plugins all have Go plugin interfaces
Replacing a pile of Bash or Python glue whose failure mode is "it worked on the build box"

Interview questions

Explain context.Context and how it flows through a controller. What actually breaks if you ignore it?
How does Go error handling work in a control-plane service, and when is panic acceptable?
Walk me through the concurrency bugs that actually bite in controllers and how you find them.
A controller pod has been running for three days and its RSS has grown from 80 MB to 1.4 GB. Walk me through diagnosing it in place.
Implement graceful shutdown for a Go service that serves HTTP, runs background workers, and holds a leader-election lease.
How would you structure and test a Go control-plane package so it is actually testable?