Kubernetes — Stuck Rollouts, Version Upgrades and Mistakes

Working a stuck rollout and a CrashLoopBackOff to an undo-or-fix decision, planning a two-minor-version upgrade on kubeadm, EKS, AKS and GKE, and recovering from and preventing mistakes.

Overview

Most production incidents start with a change. A new image rolls out and half the pods never become ready. A cluster upgrade removes an API, a label or a default that something depended on. Someone runs a command against the wrong context. This page covers the three change-driven failures interviewers ask about most: the stuck rollout, the cluster version upgrade, and the mistake.

The common thread is that Kubernetes reports, it does not decide. A stalled Deployment gets a condition, not an automatic rollback. A version upgrade enforces skew rules, but it does not check your add-ons, admission webhooks or manifests for you. A kubectl delete namespace is accepted, asynchronous and cannot be taken back. Your process, your guardrails and your rollback plan are what turn each of these into a short incident instead of a long one.

This page links to the other pages in this guide instead of repeating them. Pod states and exit codes are in Kubernetes Error Triage, the CrashLoopBackOff mechanism in Pod Startup & Scheduling Errors, exit codes and probes in Container Errors, stuck namespaces and finalizers in Control Plane Errors, drain and PDB failures in Node & Scaling Errors, and ReadWriteOnce rollout deadlocks in Storage Errors.

Symptom to first command

kubectl rollout status hangs, some pods new, some old — kubectl describe deployment orders -n prod (Conditions, NewReplicaSet) — Reading a stuck rollout
error: deployment "orders" exceeded its progress deadline — kubectl get rs -n prod -l app=orders — Reading a stuck rollout
New pods Pending, ImagePullBackOff or 0/1 Ready — kubectl get pods -n prod -l app=orders -o wide — Working the stuck rollout
Pod in CrashLoopBackOff — kubectl logs POD -n prod --previous — CrashLoopBackOff end to end
Asked to upgrade the cluster two minor versions — kubectl version and kubectl get nodes — Upgrading two minor versions
Workloads or controllers break right after an upgrade — kubectl get --raw /metrics (deprecated API metric) — Removed APIs and add-ons
Node drain hangs during an upgrade — kubectl get pdb -A — Drains, PDBs and managed node upgrades
Something was deleted or applied in the wrong place — kubectl config current-context — Mistakes

What you will be able to do

Read a Deployment's conditions and ReplicaSets to say exactly why a rollout stopped, and do the maxSurge and maxUnavailable arithmetic out loud.
Debug a CrashLoopBackOff from logs, exit code, termination message, probes and kubectl debug, without guessing.
Plan an upgrade across two minor versions: one control-plane step at a time, within the documented skew, with removed APIs, add-ons, webhooks and PDBs checked first.
State what rollback really means on kubeadm, EKS, AKS and GKE, which is different on each.
Recover from a wrong-context delete, a bad apply or GitOps drift, and name the guardrails (RBAC, admission policy, server dry-run, diff, PDBs) that prevent each one.

Key concepts

progressDeadlineSeconds: Seconds the Deployment controller waits before reporting Progressing=False with ProgressDeadlineExceeded; default 600. Kubernetes only reports it and keeps retrying.
pod-template-hash: Label the Deployment controller adds to each ReplicaSet and its pods; the suffix of the ReplicaSet name, used to tell new pods from old.
maxSurge and maxUnavailable: Rolling update limits, default 25% each; maxSurge rounds up, maxUnavailable rounds down, and they cannot both be 0.
rollout undo: Restores a previous ReplicaSet's pod template only; needs revision history (revisionHistoryLimit default 10) and does not work on a paused Deployment.
Version skew policy: Kubelet and kube-proxy may be up to three minors older than kube-apiserver and never newer; kubectl within one minor; HA API servers within one minor; no skipped minors.
Deprecated API metric: apiserver_requested_deprecated_apis, set to 1 for each deprecated API requested since the API server started, with a removed_release label.
Eviction API: The API drains use; it honors PodDisruptionBudgets and answers 200 (allowed), 429 (budget does not allow it now) or 500 (misconfiguration).
unhealthyPodEvictionPolicy: PDB field: IfHealthyBudget (default) protects unhealthy pods of a disrupted app; AlwaysAllow lets drains evict them.
Two-step control plane upgrade (GKE): New binary while emulating the previous minor for a 6 hour to 7 day soak, during which rollback is possible; then the emulated version is upgraded and rollback ends.
ValidatingAdmissionPolicy: In-process CEL admission rules, stable since 1.30, bound with Deny, Warn or Audit actions; applies to every client, unlike client-side checks.

When to use it

Use this page when a Deployment rollout hangs, reports ProgressDeadlineExceeded, or leaves old and new pods running side by side.
Use it when a pod is in CrashLoopBackOff and you need the full debugging sequence, not just the mechanism.
Use it before planning any Kubernetes minor-version upgrade, especially one that spans more than one minor version or a managed service.
Use it after a destructive or unintended command (delete, apply, edit, scale) to recover in the right order.
Use it to design guardrails: RBAC, admission policies, pipeline checks, PDBs and GitOps settings.

Interview questions

A prod deployment rollout appears stuck. kubectl rollout status has been hanging for 20 minutes, half the pods are up, the other half aren't. What's your process?
You're asked to upgrade a prod cluster's Kubernetes version by two minor versions. What's your actual risk assessment and rollout plan?
How do we debug a pod in CrashLoopBackOff state?
Does Kubernetes roll back a Deployment automatically when it hits ProgressDeadlineExceeded?
Walk me through maxSurge and maxUnavailable for 10 replicas, and why a 1-replica Deployment can behave differently.
Someone ran kubectl delete namespace in production. What do you do now, and how do you stop it happening again?
How do you find out whether anything still uses an API that the target version removes?
A node drain hangs during a cluster upgrade. Why, and what do you do?
Can you roll back a control plane upgrade on EKS, GKE, AKS or kubeadm?
An engineer fixed production with kubectl edit, and an hour later the fix disappeared. What happened?