Security, Secrets and Supply Chain

How to put secrets in Git without putting secrets in Git — SOPS, Sealed Secrets, External Secrets — plus signed-commit and cosign artifact verification, controller RBAC, and workload identity.

Overview

The week-two wall

Every GitOps adoption hits the same wall in week two. The repository is the source of truth for cluster state, and then someone needs a database password. Git is the worst possible place for one: history is immutable, every developer has a full clone, mirrors and backups multiply the copies, and there is no revocation primitive.

Five approaches to secrets

Plain Kubernetes Secrets — Cluster only — Bootstrap credentials only; anything more is explicitly discouraged
Decryption operators, e.g. Sealed Secrets — Ciphertext in Git, private key in-cluster — Supported
SOPS — Ciphertext in Git; kustomize-controller decrypts at reconcile time, no extra controller or CRD — The deeply integrated path
External secret operators — A dedicated store, synced into the cluster — Common alternative
CSI drivers — Mounted as files; no Secret objects at all — Supported

SOPS, the one to know cold

Backends: age, OpenPGP, AWS KMS, GCP KMS, Azure Key Vault, and HashiCorp Vault or OpenBao.

.sops.yaml creation rules — Developers never have to remember key material
encrypted_regex limited to data and stringData — Diffs stay reviewable and Kustomize can still resolve the resource

Why External Secrets Operator became common

SOPS and Sealed Secrets both put ciphertext in Git, so rotation is a commit and revocation is impossible for anything already cloned. ESO leaves the material in a system built for secrets and syncs it in; the repository holds only a reference.

Rotation — A commit — In the store, no Git change
Revocation — Impossible for cloned ciphertext — Real
Repo describes full state — Yes — No
Runtime dependency — None — The external store

Integrity: spec.verify

Supply chain integrity asks a different question: did this manifest come from someone I trust?

> [!IMPORTANT] Verification failure gives you a stopped rollout rather than a compromised one.

Controller posture

Flux installs with cluster-admin: defensible on a single-tenant cluster, indefensible on a shared one.

spec.serviceAccountName impersonation — Applies run as tenant identities
Restricted PodSecurity on flux-system — Controllers stay locked down
NetworkPolicy — Only authorized components reach the source-controller artifact server
Workload identity — No cloud credential stored as a Secret

Flux's controllers already run as non-root with read-only root filesystems and dropped capabilities. Your job is not to undo that.

> [!WARNING] Avoid the flags the docs explicitly warn against: --insecure-kubeconfig-exec and --insecure-kubeconfig-tls.

Key concepts

SOPS: Secrets OPerationS. Encrypts values inside a structured file (YAML, JSON, ENV) while leaving keys and structure in plaintext. Backends: age, OpenPGP, AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault or OpenBao. kustomize-controller decrypts on demand at reconcile time, so no extra controller and no extra CRD are required.
.sops.yaml creation rules: A repository-root file mapping path_regex to a recipient key and an encrypted_regex. It means a developer runs sops secret.yaml and the right key is selected automatically. Restricting encrypted_regex to ^(data|stringData)$ keeps apiVersion, kind, and metadata readable so diffs are reviewable and Kustomize can resolve the resource.
spec.decryption: The Kustomization field enabling SOPS: provider is sops, secretRef names a Secret holding the age key, PGP key, or cloud credentials. Alternatively serviceAccountName names a ServiceAccount bound to a cloud identity, so KMS access uses workload identity and no static credential exists in the cluster.
age: A modern file encryption tool with small keys and no configuration surface, generated by age-keygen. The common SOPS backend when there is no cloud KMS. The private key is stored as a Secret in flux-system and referenced by spec.decryption.secretRef. Losing it means every encrypted value in the repository is unrecoverable, so it needs an offline backup.
Sealed Secrets: A Bitnami controller that generates a 4096-bit RSA key pair at startup and persists it as a Secret. kubeseal encrypts with the public certificate, producing a SealedSecret CR that only that controller can decrypt. Encryption needs no cluster access once the cert is fetched. The controller key pair must be backed up or every SealedSecret in Git becomes undecryptable after a cluster rebuild.
External Secrets Operator: Syncs from a remote store (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, 1Password) into Kubernetes Secrets via ExternalSecret and SecretStore resources. Git holds only a reference, never ciphertext. Rotation happens in the store and propagates without a commit. The tradeoff is a runtime dependency and cluster state no longer fully described by the repository.
spec.verify on GitRepository: Enables commit and tag signature verification. secretRef names a Secret of trusted public keys — PGP armored keys under .asc-suffixed keys, SSH public keys in authorized_keys format under .sshpub-suffixed keys, both types allowed in one Secret. mode is HEAD, Tag, or TagAndHEAD. Success sets a SourceVerified condition to True.
Keyless cosign verification: spec.verify.provider: cosign on an OCIRepository with matchOIDCIdentity instead of secretRef. Each entry carries issuer and subject regular expressions matched against the signing certificate; matchers are OR-ed. Asserts that an artifact was signed by a specific CI workflow identity rather than by whoever holds a long-lived key, removing key storage and rotation entirely.

When to use it

Any repository that will ever contain a credential — decide the secrets pattern before the first one is committed, because a leaked secret in Git history is a rotation event, not a cleanup task
Regulated environments requiring signed provenance — spec.verify with PGP or SSH on GitRepository, cosign or notation on OCIRepository, so an unsigned change cannot reach the cluster
Shared clusters where Flux controllers must not apply with cluster-admin — impersonation plus --default-service-account plus per-tenant RBAC
Cloud-hosted clusters where storing an IAM key as a Kubernetes Secret is itself the finding — annotate controller ServiceAccounts for IRSA, GKE Workload Identity, or Azure Workload Identity instead
Organisations already running Vault, AWS Secrets Manager, or Azure Key Vault — External Secrets Operator keeps rotation in the system built for it and keeps Git free of ciphertext entirely
Air-gapped or high-assurance clusters where remote Kustomize bases and cross-namespace references must be structurally impossible, not just discouraged

Interview questions

Walk through setting up SOPS with age for a Flux repository, end to end.
Compare SOPS, Sealed Secrets, and External Secrets Operator. Which do you pick and why?
How do you make Flux refuse to deploy anything that is not signed?
Flux installs with cluster-admin. How do you reduce that, and what remains irreducible?
How do you give Flux access to a private registry and a cloud KMS without storing any credentials?
A secret was committed unencrypted to the Flux repository three weeks ago. Walk me through the response.