Identity, Secrets and Config Errors — Workload Identity, Stale ConfigMaps and Secret Stores

Pods that run fine but cannot reach a cloud API, read yesterday's config, or never start because the secret store said no: workload identity on EKS, AKS and GKE, ConfigMap delivery, Secrets Store CSI and External Secrets.

Overview

This layer covers how a workload proves who it is and how it gets its settings. The failures are unusual because the pod usually looks healthy. It is Running and Ready, yet every call to a cloud API returns access denied, or it keeps using a value you changed an hour ago. Nothing in kubectl get pods looks wrong, so these are the errors most often misread as application bugs.

You work outside-in along the chain. For identity the chain is: ServiceAccount, the admission webhook that injects a token, the projected token itself, the cloud trust rule that accepts it, and finally the permission on the resource. For configuration the chain is: the object in the API, the delivery method (environment variable, volume, subPath), the kubelet sync, and whether the application re-reads what it was given. Find the first broken link and you have the cause.

The one idea to keep: the object in the API is not what the container sees. Always compare the two directly, with kubectl exec or a test pod that uses the same ServiceAccount, before you change anything.

Symptom to first command

AccessDenied ... AssumeRoleWithWebIdentity — Trust policy sub does not match the pod's namespace and ServiceAccount — aws iam get-role --role-name orders-role --query Role.AssumeRolePolicyDocument
InvalidIdentityToken: No OpenIDConnect provider found — Cluster OIDC issuer not registered in IAM — aws iam list-open-id-connect-providers
SDK uses the node role, or NoCredentialProviders — Webhook did not inject; pod predates the annotation — kubectl get pod -n prod -o yaml and look for AWS_ROLE_ARN
AADSTS70021: No matching federated identity record found — Federated credential issuer or subject mismatch — az identity federated-credential list --identity-name orders-id -g rg-prod
ConfigMap changed, pod still has old value — Value delivered as an env var or subPath — kubectl get pod -n prod -o yaml and look for envFrom / subPath
MountVolume.SetUp failed for volume "secrets-store" — Identity cannot read the secret, or wrong name — kubectl events -n prod --for pod/
ExternalSecret SecretSyncedError — SecretStore invalid, or key missing in the vault — kubectl get externalsecret,secretstore -n prod
ExpiredToken or AADSTS700024 after hours of uptime — App read the token file once and cached it — kubectl exec -n prod -- env then check how the app builds its client

What you will be able to do

Find the broken link in an IRSA, EKS Pod Identity, AKS Workload ID or GKE Workload Identity chain in five commands.
Explain why a ConfigMap edit did nothing, and pick between restart, checksum annotation, Reloader and live reload.
Read a Secrets Store CSI FailedMount or an ExternalSecret SecretSyncedError and act on it directly.
Say how the same failure looks and is fixed on EKS, AKS and GKE.

Key concepts

Workload identity: Exchanging a Kubernetes ServiceAccount token for short-lived cloud credentials, so no static key lives in the pod. IRSA, EKS Pod Identity, Entra Workload ID and Workload Identity Federation for GKE are the cloud versions.
Projected ServiceAccount token: A signed, audience-bound, expiring JWT the kubelet writes into the pod and rewrites at 80% of its lifetime or after 24 hours. Default lifetime is one hour, minimum ten minutes.
Trust policy subject: The rule on the cloud side that says which token may assume the role, normally system:serviceaccount:: plus an audience. A mismatch is the most common identity failure.
EKS Pod Identity: The newer AWS option: an association links namespace and ServiceAccount to a role, a node agent serves credentials, and no IAM OIDC provider is needed.
Delivery method: How a ConfigMap or Secret reaches a container: env vars are fixed at start, volumes update after the kubelet sync and cache delay, subPath mounts never update.
Checksum annotation: A hash of the config placed in the pod template, so a config change is a template change and triggers a normal rollout.
Immutable ConfigMap: A ConfigMap or Secret with immutable: true. Updates are rejected, the kubelet stops watching it, and the setting cannot be reverted without deleting the object.
SecretProviderClass: The namespaced object that tells the Secrets Store CSI driver which provider and which vault objects to mount, and optionally which Kubernetes Secret to sync.
refreshInterval: How often External Secrets Operator re-reads the vault for an ExternalSecret. Defaults to one hour; zero means sync once.

When to use it

A pod is Running and Ready but every call to S3, Key Vault, GCS or another cloud API returns access denied.
You changed a ConfigMap or Secret and the application still behaves as before.
Pods sit in ContainerCreating with a FailedMount event for a secrets-store volume.
An ExternalSecret shows SecretSyncedError, or a rotated password never reached the pods.
A workload fails with ExpiredToken or AADSTS700024 after running for hours.
You are preparing for an SRE or platform interview that covers IRSA, Pod Identity, Entra Workload ID or GKE Workload Identity.

Interview questions

After moving the orders service from the default namespace to prod, every S3 call fails with AccessDenied on AssumeRoleWithWebIdentity. Nothing else changed. Walk me through it.
The orders ServiceAccount has the right role annotation, yet aws sts get-caller-identity inside the pod shows the node's instance role. Why?
We changed LOG_LEVEL in the orders ConfigMap ten minutes ago and nothing happened. How do you fix it now, and how would you design config rollout so this does not recur?
Security rotated the database password in AWS Secrets Manager. Some orders pods now fail to authenticate and others work. Walk me through it.
How does workload identity differ between EKS, AKS and GKE, and where does each one usually break?
After adding a new node pool, new orders pods on those nodes stay in ContainerCreating with FailedMount for secrets-store, while pods on old nodes are fine. What happened?
A nightly batch job that runs about 26 hours fails near the end with ExpiredToken. Shorter jobs never fail. What do you look at?