Monitoring and Troubleshooting

The Flux observability stack — Prometheus metrics, Kubernetes events, and controller logs — plus a systematic playbook for diagnosing a Kustomization or HelmRelease that will not go Ready.

Overview

Failures that keep running

Flux is a set of controllers running a continuous loop, so the interesting failures are not crashes. They are loops that keep running and keep failing. A Kustomization Ready=False for six hours is still reconciling every interval, still emitting events, still burning API calls; nothing is down, the deployment simply is not happening.

> [!TIP] Interviewers probe whether you reach for these layers in the right order. Reaching for logs first is the tell of someone who has not done this at scale.

Three signal layers

Prometheus metrics — Reconcile duration histograms per kind and object, controller-runtime error counters, work queue latency, controller memory
Kubernetes events — The reconciliation narrative: the Reason and Message the controller recorded, annotated with the source revision
Controller logs — The raw error from git, kustomize, the Kubernetes API or Helm

Where readiness metrics come from

In current Flux 2.x the controllers do not export the Ready condition of your custom resources. They export gotk_reconcile_duration_seconds and cache metrics.

> [!IMPORTANT] Without that kube-state-metrics config you cannot alert on Flux at all.

Two alerting paths

Most teams need both.

The notification-controller path lands reconciliation failures in the same on-call pipeline as everything else, rather than a separate chat channel nobody reads at 3am.

The troubleshooting playbook

flux check — Are controllers and CRDs healthy?
flux get all -A --status-selector ready=false — What is the blast radius?
flux events --for Kind/name — What did the controller decide?
flux logs --level=error --all-namespaces — What is the raw error?
flux trace — Which Git revision produced this live object?
flux diff kustomization — What would this change do before merging?

Failure taxonomy

Every class below presents as the same top-level symptom, Ready=False, and each has a completely different fix.

> [!TIP] Naming the condition Reason for each and saying what you would change is the difference between having read the docs and having run Flux in production.

Key concepts

gotk_reconcile_duration_seconds: Histogram exported by every GitOps Toolkit controller on port 8080 at /metrics, labelled by kind, name, and namespace. The bucket, sum, and count series let you compute p50/p99 reconcile latency per object. Rising p99 for one object usually means a large kustomize build or a health check that is waiting out its timeout.
gotk_resource_info: Info metric produced by kube-state-metrics using CustomResourceStateMetrics, not by the Flux controllers. Labels include customresource_kind, exported_namespace, name, ready, and suspended, plus kind-specific labels like revision, source_name, chart_name, and url. This is the metric all Flux alerting rules are written against.
controller_runtime_reconcile_total: Standard controller-runtime counter labelled by controller and result. A rising result="error" rate is the fastest aggregate signal that a controller is in a retry loop, and it fires before any per-object alert because it does not need the 15 minute hold window.
flux events --for: Reads the Kubernetes events a controller emitted for one object: flux events --for Kustomization/my-app. Each event carries Type (Normal or Warning), a machine-readable Reason such as ReconciliationSucceeded or GitOperationFailed, a human message, and the source revision as an annotation. This is the fastest path from symptom to root cause.
flux trace: Walks a live Kubernetes object back up the delivery pipeline: flux trace -n apps deployment my-app prints the owning Kustomization or HelmRelease, its source, and the exact revision that produced it. The answer to "who deployed this and from which commit" without reading any YAML.
flux diff kustomization: Builds a Kustomization from a local path, performs a server-side dry-run, and prints the diff. Exit status 0 means no differences, 1 means differences found, greater than 1 means error — which makes it usable as a pull request gate. Flags include --path, --local-sources for offline repo substitution, and --recursive.
Suspend as break-glass: Setting spec.suspend: true stops reconciliation for one object without touching the controllers. It is the correct incident action when you need to hand-patch a resource, and the most common source of silent GitOps drift when nobody unsuspends it. Alert on gotk_resource_info{suspended="true"}.
Field manager conflict: kustomize-controller applies with server-side apply under its own field manager. If another controller (an HPA writing replicas, a mutating webhook, a human kubectl edit) owns a field Flux also declares, the apply conflicts. Resolved per-resource with the kustomize.toolkit.fluxcd.io/ssa annotation set to Merge, IfNotPresent, or Ignore instead of the default Override.

When to use it

Standing up a new Flux installation — deploy kube-prometheus-stack with the CustomResourceState config before you have your first incident, not after
Any cluster where more than one team commits to the GitOps repo, so a broken path in one tenant does not silently stop delivery for another
Debugging why a merged pull request has not reached the cluster, when the pipeline is green but the workload has not changed
Diagnosing a HelmRelease stuck in a remediation loop, where every interval reinstalls or rolls back and nobody notices
Capacity work: deciding whether slow reconciles are a controller sizing problem, a repository size problem, or a single pathological object

Interview questions

Walk me through diagnosing a Kustomization that has been Ready=False for an hour.
A GitRepository source will not become Ready. What are the failure modes and how do you tell them apart?
Explain the difference between a dry-run validation failure, a health check timeout, and a dependsOn deadlock.
Flux applied a change and deleted a resource nobody expected to lose. How does prune work and how do you prevent that?
A HelmRelease reports "upgrade retries exhausted". What happened and what do you do?
You get "field is immutable" and separately a field manager conflict under server-side apply. Explain both and fix them.