Writing Kubernetes Operators in Go

Authoring an operator with kubebuilder and controller-runtime: the Manager/Cache/Client architecture, idempotent level-triggered reconciliation, finalizers, status conditions, RBAC markers, and envtest.

Overview

Writing an operator that survives production

The concepts topic covers what a CRD is, what the Operator Framework capability levels mean and when an operator is the right answer. This topic assumes all of that and deals with the part that takes the time: writing one that survives production.

Tooling

kubebuilder is the scaffolding tool and controller-runtime is the library underneath. Operator SDK's Go path wraps the same controller-runtime, so what you learn here transfers directly.

kubebuilder init — A module, a Makefile, a Dockerfile and a config/ Kustomize tree
kubebuilder create api --group apps --version v1alpha1 --kind Cluster — api/v1alpha1/cluster_types.go (your Go API type) and internal/controller/cluster_controller.go (your reconciler), both registered
make manifests — controller-gen turns marker comments into CRD YAML and RBAC ClusterRoles

Idea 1: level-triggered reconciliation

Reconcile is handed a namespace and name, nothing else. It is not told whether the object was created or updated, or whether a child was deleted. It must fetch current state, compare it against the spec and close the gap.

> [!IMPORTANT] Run Reconcile twice in a row and the second run must be a no-op. Idempotency lets the framework requeue freely, resync periodically and recover from a restart that dropped every in-flight event.

Code that assumes it sees every event is edge-triggered and will be wrong within a week of shipping.

Idea 2: the API server uses optimistic concurrency

Every object carries a resourceVersion, and an Update fails with 409 Conflict if yours is stale. In a controller this is normal: your status write races a user edit, another controller's annotation or the garbage collector. A controller that treats conflicts as failures alerts constantly and fixes nothing.

Idea 3: deletion is cooperative

The garbage collector cannot clean up a cloud load balancer, DNS record or database. A finalizer converts deletion into an observable state; you clean up, then remove it.

> [!WARNING] Get finalizers wrong in one direction and you leak external resources; in the other, objects wedge in Terminating and take their namespace with them.

What an interviewer probes

Why does Reconcile take only a key? — Level-triggered, deduplicated queue
Get immediately after Create through the cached client? — The cache may not have the object yet
A 409 on status update? — Re-fetch and retry, patch, or server-side apply
Where do ownerReferences work? — Kubernetes children only; external resources need a finalizer
A finalizer never cleared? — The namespace wedges in Terminating
Is your reconciler idempotent? — Genuinely, not just usually

Key concepts

Manager: The controller-runtime top-level object. Owns the shared Cache, Client, Scheme, metrics and health servers, webhook server, and leader election. mgr.Start(ctx) runs every registered Runnable and blocks until ctx is cancelled. All controllers in one binary share its Cache, so one process opens one set of watches.
Cache and split Client: The Cache is a set of shared informers backing an in-memory store per watched Kind. mgr.GetClient() returns a split client — reads from the Cache, writes to the API server — with no read-after-write guarantee. mgr.GetAPIReader() reads uncached, directly from the API server, for the cases where you need consistency.
Level-triggered reconciliation: Reconcile receives only a NamespacedName and must derive all action from the object as it is now. Events are coalesced by key in the workqueue, so many rapid changes produce one reconcile. The consequence is a hard requirement: reconcile must be idempotent and must never assume it observed every intermediate state.
ctrl.Result and requeue: Returning an error requeues the key with exponential backoff (default 5ms base, 1000s cap) and increments the error metric. Returning ctrl.Result{RequeueAfter: d} requeues after a fixed delay with no error recorded — the correct form for polling an external system. Returning an empty Result and nil error means done until the next event.
ownerReference: A field in metadata linking a child object to its owner. controllerutil.SetControllerReference sets it with controller: true. It drives both Kubernetes garbage collection (delete the owner, children cascade) and the Owns() event mapping. It does not work across namespaces, and a namespaced object cannot own a cluster-scoped one.
Status subresource and conditions: Enabled by // +kubebuilder:subresource:status. Splits writes so /status updates do not modify spec and do not bump metadata.generation. Conditions are the standard status shape: type, status (True/False/Unknown), reason, message, lastTransitionTime, observedGeneration — managed with meta.SetStatusCondition from apimachinery.
Finalizer: A string in metadata.finalizers that blocks actual deletion. A delete request sets metadata.deletionTimestamp; the object remains until every finalizer is removed. Controllers add one with controllerutil.AddFinalizer, perform external cleanup when deletionTimestamp is non-zero, then RemoveFinalizer. A finalizer nobody removes wedges the object, and then its namespace, in Terminating.
envtest: A test harness that starts a real etcd and kube-apiserver from binaries fetched by setup-envtest, loads your CRDs from config/crd/bases, and hands your tests a rest.Config. It has no kube-controller-manager and no kubelet, so there is no garbage collection, no Pod scheduling, and namespaces never finish terminating.

When to use it

Automating operational knowledge for a stateful system — the reconcile loop encodes the runbook for backup, failover, version upgrade, and scaling
Exposing a platform capability as a Kubernetes API so consumers declare intent in YAML and get it reconciled, rather than filing tickets
Managing resources outside the cluster whose lifecycle must track a Kubernetes object — cloud databases, DNS records, IAM roles, TLS certificates
Coordinating multi-object rollouts where ordering, health gating, and rollback cannot be expressed by a Deployment alone
Replacing a CI job that runs kubectl apply on a cron — a controller reconciles continuously and corrects drift, which a cron job cannot

Interview questions

Why does Reconcile take only a NamespacedName, and what does level-triggered actually mean for the code you write?
Walk me through kubebuilder scaffolding. What does each generated piece do, and what does make manifests actually generate?
Your status update fails with "the object has been modified; please apply your changes to the latest version." Explain the mechanism and how you fix it properly.
Explain finalizers, and walk through what happens when one is never removed.
How do you test an operator? Compare unit tests, envtest, and end-to-end, and be specific about what envtest cannot do.
What do you configure before an operator is production-ready — leader election, caching, RBAC, and CRD validation?