Secret Management

Where secrets should live, the GitOps challenge, and modern secret-rotation patterns.

Overview

Why secret management matters

Secrets are credentials that grant access: API keys, database passwords, TLS private keys, OAuth tokens, encryption keys. Improper secret management is one of the most common causes of security incidents.

> [!IMPORTANT] Assume every secret will eventually leak. Rotate so a leaked credential stops working quickly.

Where secrets live

Source code — Cloud-managed secret store
Config files committed to the repo — HashiCorp Vault
Env vars baked into container images — Kubernetes secrets with Sealed Secrets or External Secrets
Slack, email or chat history — Parameter store with KMS encryption
CI/CD logs

Secret store options

1. Cloud-managed store — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault — IAM-controlled, encrypted at rest, audit-logged; built-in rotation for some types such as RDS passwords — Cloud-specific
2. HashiCorp Vault — Self-hosted or HCP Vault — Most flexible; dynamic, just-in-time DB credentials — Steeper operational overhead
3. Kubernetes secrets plus helpers — Native Secrets, Sealed Secrets, External Secrets Operator — Native Secrets are only base64-encoded, and encryption at rest must be enabled explicitly; Sealed Secrets are safe to commit; ESO syncs from a cloud store — More moving parts
4. Parameter store with KMS — AWS SSM Parameter Store — Cheap; tight AWS integration — Fewer features than Secrets Manager

The GitOps challenge

GitOps wants everything in Git, and secrets in Git would be a security disaster.

Sealed Secrets — Encrypt with a public key; only the in-cluster controller's private key decrypts; the encrypted blob goes in Git
External Secrets Operator — Git holds references (SecretStore and ExternalSecret); ESO pulls values from Vault or the cloud at sync time
SOPS with age or PGP — Encrypt YAML files; decryption keys live in cloud KMS or local keychains; Mozilla SOPS is the dominant tool

Each trades complexity for security; pick based on your team's tooling.

Secret rotation

Vault's DB plugin is the dynamic example: short-lived, 1-hour DB credentials issued per service. It requires Vault or similar, though cloud-managed alternatives are progressing.

Detecting leaked secrets

Commits — GitHub secret scanning detects 200+ secret types and alerts the owner
Pre-commit and CI — gitleaks, trufflehog
Cloud behavior — AWS GuardDuty flags compromised access keys
Traps — Honey credentials: fake secrets that alert when used

> [!WARNING] First response to a leaked secret: rotate immediately, then assess scope: where it was used and what an attacker could reach.

The 12-factor pattern, updated

Heroku's 12-factor manifesto (2011) made "config in environment variables" canonical. The intent, decoupling code from config, was good, but env vars baked into images are not secrets-safe.

The modern reading: inject env vars at runtime from the secret store. The container starts, reads secrets via SDK or sidecar, and the image never holds them.

Common mistakes

Long-lived service-account credentials in deployment env vars — Rotation requires a redeploy
One credential across environments — A dev leak compromises prod
Secrets in logs — Hard to purge from log retention
Manual rotation nobody runs — Nothing rotated in 18 months

The 2026 best practice

Source of truth — A cloud-managed secret store
Delivery — Runtime injection, no secrets in images
Rotation — Automated per secret type
Detection — Scanning at commit, CI and runtime
Highest stakes — Dynamic secrets via Vault

Key concepts

Cloud secret stores: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault. Default for cloud-native apps.
HashiCorp Vault: Self-hosted or HCP. Most flexible; supports dynamic secrets. Higher operational overhead.
Sealed Secrets: Encrypt secrets with public key; commit to Git; in-cluster controller decrypts. GitOps-friendly.
External Secrets Operator: Git references; ESO pulls real values from cloud / Vault at sync time. Modern GitOps pattern.
Dynamic secrets: Just-in-time credentials per request. Short-lived; eliminate leak risk. Vault DB plugin canonical example.
Secret scanning: GitHub native + gitleaks / trufflehog at commit / CI / runtime. Multi-layer detection.

When to use it

Greenfield service — set up secret management from day 1
Audit existing systems — find secrets in code or config; migrate to secret store
Postmortem with credential leak — what was the secret-management gap?
Compliance / audit prep — secret rotation and audit logs are common asks

Interview questions

How do you handle secrets in a Kubernetes + GitOps deployment?
What is "dynamic secrets" and why is it powerful?
A secret was committed to a public repo. Walk me through the response.