GitOps Principles

Weaveworks coined the term in 2017; OpenGitOps (CNCF, 2021) formalized it. Four principles: declarative, versioned + immutable, pulled automatically, continuously reconciled. The operational model that makes Kubernetes deployments auditable, recoverable, and rollback-trivial.

Overview

What GitOps is

GitOps is the operating model where git is the source of truth for declarative infrastructure and applications, and software agents continuously reconcile actual system state to match git. Alexis Richardson coined the term at Weaveworks in 2017; the OpenGitOps project (CNCF Sandbox since 2021) formalized the principles in v1.0 of the spec.

Why GitOps wins for Kubernetes

Auditability — Every production change is a commit with author, timestamp, message and diff; git log is the audit log and PR review is change control. SOC 2 / ISO 27001 / FedRAMP audits get significantly easier
Rollback simplicity — git revert creates a reversing commit and the agent reconciles to it: the same mechanism forward and back
Disaster recovery — The cluster is replaceable: spin up a new one, install Argo CD, point it at the repo, and state restores from git
Multi-environment consistency — dev, staging, prod are subdirectories or branches; promotion is a PR; drift between environments is a git diff
Security — Deployment credentials never leave the cluster; CI no longer needs kubectl access to prod, so CI compromise doesn't compromise prod

Where it doesn't fit cleanly

Non-Kubernetes infrastructure — Argo CD primarily targets K8s resources — Atlantis or Spacelift for Terraform-managed AWS
Imperative operations — Migrations, secret rotations, one-time data fixes; GitOps has weak primitives — Jobs or hooks, though the model strains
Per-deploy variables — CI-generated image SHAs must land in manifests — Write back to git via image-update controllers, adding operational complexity

Three load-bearing concepts

The four principles — Declarative, versioned + immutable, pulled, continuously reconciled; all four must hold
Pull, not push — The cluster agent pulls from git; CI never pushes to the cluster. This inverts the deployment direction: the fundamental security and operational improvement
Continuous reconciliation — Drift is detected and remediated as normal operation, not as an exception

Key concepts

Declarative state: Systems describe what the desired state is, not how to achieve it. Imperative (not GitOps): Declarative (GitOps): Why declarative matters: - Idempotent: re-applying the same manifest produces the same outcome. - Observable: you can compare two manifests and see what's different. - Reversible: revert the file → agent reconciles back. No reverse command list to maintain. This is why GitOps needs Kubernetes (or another declarative API). You can't easily GitOps "ssh to box, apt install nginx" — the imperative shape doesn't fit.
Pull-based reconciliation: Architectural inversion: instead of CI pushing to the cluster, an agent in the cluster pulls from git. Traditional CI/CD push sends the CI runner straight to the Cluster API via kubectl apply — the CI runner needs cluster credentials, so a CI compromise is a cluster compromise. GitOps pull flips the arrow: the cluster agent reaches out to the Git repo via git fetch — the cluster needs git credentials, and CI never touches the cluster at all. See the Visual Explanation diagram below for the side-by-side flow. The agent runs as a Kubernetes controller (Argo CD's argocd-application-controller, Flux's source-controller + kustomize-controller). Watches Application/Kustomization CRDs that reference git URLs. Polling vs webhook: - Default: poll git every 3 minutes. - Optimization: configure git webhooks to notify the agent immediately. Reduces lag from ~3min to seconds. Authentication: - Read-only Deploy Key per repo (SSH). - HTTP token with read-only scope. - GitHub App with restricted permissions. The pull model is GitOps's biggest security improvement over push-based CI/CD.
Continuous reconciliation + drift remediation: Reconciliation loop runs every ~3 minutes regardless of whether git changed. Drift sources: - Manual cluster changes (kubectl edit, kubectl scale). - Other controllers mutating resources (HPA scaling deployment.replicas). - Cloud provider mutations (AWS adding tags). - Time-based changes (certificates expiring). Three response modes per Application: Mode 1: self-heal: true. Agent automatically reverts manual changes. For stable production where any non-git change is wrong. Mode 2: self-heal: false. Agent detects drift, marks Application as OutOfSync, sends notification. User clicks Sync to remediate. Mode 3: prune: false. Agent doesn't delete resources that exist in cluster but not in git. ignoreDifferences for HPA-managed replicas: Without this, agent fights HPA constantly. Always set for HPA-managed deployments.
GitOps tools comparison: Two dominant CNCF Graduated tools: Argo CD (graduated Dec 2022). Originated at Intuit. UI-rich; Application CRD model; Helm/Kustomize/Jsonnet/plain YAML; App-of-Apps + ApplicationSet for multi-app; strong RBAC + SSO. Flux v2 (graduated Nov 2022). Originated at Weaveworks (the GitOps coiners). GOTK (GitOps Toolkit) — multiple controllers. More composable; thinner UI; multi-tenancy via K8s native primitives. Use Argo CD when: polished UI matters; centralized control plane managing many clusters; rich app catalog UX. Use Flux when: K8s-native composability matters; multi-tenancy via namespaces; smaller surface. Many large orgs run both: Argo CD for application teams (UI), Flux for platform teams (composability). Other GitOps-adjacent tools: - Anthos Config Sync (Google) — GKE GitOps. - Rancher Fleet — multi-cluster for SUSE Rancher. - Atlantis — PR-driven Terraform (not strictly GitOps but adjacent). - Spacelift / env0 — hosted Terraform with GitOps mode.
Recipe: secrets in GitOps: Plain-text secrets in git is a non-starter. Three working patterns: External Secrets Operator (ESO) — dominant 2026 pattern: ESO fetches from AWS Secrets Manager / Vault / GCP Secret Manager and creates a K8s Secret. Rotation in cloud → propagates automatically. Sealed Secrets (Bitnami) — simpler: Encrypted SealedSecret CRD committable to git. Controller decrypts using cluster's private key. Simple but tied to one cluster. SOPS + KMS — encrypted YAML in git, decrypted at apply time using cloud KMS. Argo CD plugin handles decryption. Strong audit trail in CloudTrail. Pick ESO + AWS Secrets Manager for production at scale. Sealed Secrets for OSS / single-cluster.

When to use it

Kubernetes deployments at any non-trivial scale (>5 services or >2 environments)
Multi-cluster K8s where consistent state across clusters matters
Compliance-driven environments needing audit trail of every prod change
Disaster-recovery scenarios where cluster rebuild from git is essential
Replacing fragile CI-driven kubectl apply pipelines

Interview questions

What are the four GitOps principles, and what does each rule out?
Walk through what happens when you push a commit to a GitOps cluster.
When does GitOps NOT fit, and what do you use instead?
How do you handle secrets in a GitOps repo?
Quick-fire interview answers — GitOps essentials.