Pod Startup and Scheduling Errors — ImagePullBackOff, CrashLoopBackOff, Pending, Evicted

The failures between an accepted manifest and a running process: image pulls, crash loops, missing config, unschedulable pods, init containers, pod IP exhaustion and node-pressure eviction, with the commands and the cloud-specific causes.

Overview

These errors happen after the API server has accepted your manifest but before your application does useful work. The Pod object exists, so kubectl get pods lists it, but it is stuck somewhere on the way to Running, or it got there and was thrown off the node. Each status name is a place on that path, and the Events at the bottom of kubectl describe pod say exactly what went wrong there.

Work through it in order, starting with one question. Has the pod been given a node? If not, the scheduler is the only component involved, and its one-line summary (0/N nodes are available: ...) counts every node and the reason each was rejected. If it has a node, the kubelet on that node owns everything that follows: admission, the network sandbox, the image pull, init containers, building the container config, and starting the process. Reading the node assignment first cuts the problem in half.

The idea to keep: a status like CrashLoopBackOff or ImagePullBackOff is a retry timer, not a diagnosis. The cause is in the Events, the previous container's logs, or the exit code. Strong engineers go straight to those three places and never restart pods hoping the problem goes away.

Symptom to first command

ErrImagePull / ImagePullBackOff — Wrong tag, missing pull credentials, no route to the registry — kubectl describe pod -n prod (read the Failed to pull line)
CrashLoopBackOff — The process exits at startup, or a liveness probe kills it — kubectl logs -n prod --previous
CreateContainerConfigError — A referenced ConfigMap, Secret or key does not exist — kubectl describe pod -n prod
InvalidImageName — The image string does not parse (uppercase repository, empty tag, unrendered template) — kubectl get pod -n prod -o jsonpath='{..image}'
Pending with Insufficient cpu — Requests do not fit any node's free allocatable — kubectl events -n prod --for pod/
Pending with taint, affinity or spread messages — Placement rules exclude every node — kubectl describe pod -n prod
Init:CrashLoopBackOff / Init:0/2 — An init container fails or never exits — kubectl logs -n prod -c
ContainerCreating with FailedCreatePodSandBox — The CNI plugin cannot give the pod an IP — kubectl describe pod -n prod then check subnet free IPs
Evicted (phase Failed) — The kubelet reclaimed memory or disk under node pressure — kubectl describe node (Conditions)
SchedulingGated — A scheduling gate was never removed — kubectl get pod -n prod -o jsonpath='{.spec.schedulingGates}'
OutOfcpu, OutOfpods, UnexpectedAdmissionError — The kubelet refused the pod after it was bound — kubectl get pod -n prod -o jsonpath='{.status.message}'
Pods vanish, event Preempted — A higher-priority pod took their place — kubectl get events -n prod --field-selector reason=Preempted

What you will be able to do

Read the scheduler's 0/N nodes are available line and name the rule or resource that blocks each group of nodes.
Explain the back-off behind CrashLoopBackOff and ImagePullBackOff, including the 1.32 to 1.35 changes to it.
Debug a container that dies too fast to exec into, with --previous logs, exit codes and kubectl debug --copy-to.
Tell node-pressure eviction, preemption and kubelet admission rejection apart, and say which pods go first.
Map each failure to the EKS, AKS or GKE component that really causes it: ECR and node roles, ACR and kubelet identity, VPC CNI prefix delegation, Azure CNI Overlay, GKE pod ranges.

Key concepts

Back-off (CrashLoopBackOff, ImagePullBackOff): The kubelet waits between retries, starting at 10 seconds and doubling to a 300 second cap; a container that runs 10 minutes resets it. Since 1.35 a node can lower the crash cap with crashLoopBackOff.maxContainerRestartPeriod (beta).
Requests versus usage: The scheduler places pods by the sum of requests against node allocatable, never by measured usage, so over-requested pods block scheduling on idle nodes.
Allocatable: Node capacity minus kube-reserved, system-reserved and the hard eviction threshold; it is the budget the scheduler and kubelet admission use.
Taints, tolerations and affinity: Taints on nodes repel pods that do not tolerate them; nodeSelector and node affinity make pods require node labels. A toleration permits a node but does not attract the pod to it.
Topology spread constraints: Rules that bound how unevenly an app's pods sit across zones or nodes (maxSkew). With DoNotSchedule they can leave pods Pending after a zone is lost.
Native sidecar container: An init container with restartPolicy: Always, GA in 1.33. It starts in init order, does not block later steps once started, and runs for the life of the pod.
Pod sandbox: The pod-level environment, network namespace and IP, created by the runtime and the CNI plugin before any container starts; failures show as FailedCreatePodSandBox.
Node-pressure eviction: The kubelet failing pods when memory, disk, inodes or PIDs cross thresholds. It ranks by usage over request, then priority, then overage; QoS class is not an input.
Preemption: The scheduler removing lower-priority pods so a higher-priority pending pod can fit; victims get DisruptionTarget with reason PreemptionByScheduler.
Scheduling gates: Names in spec.schedulingGates that keep a pod out of scheduling until a controller removes them; the pod shows SchedulingGated. GA since 1.30.

When to use it

A pod is listed but never reaches Running: Pending, ContainerCreating, Init or a back-off status.
A deploy stalls with ImagePullBackOff, CrashLoopBackOff or CreateContainerConfigError on new replicas.
Pods stay Pending even though dashboards show idle CPU across the cluster.
Scale-up events leave pods in ContainerCreating on EKS, or new nodes fail to join on GKE.
Pods are disappearing as Evicted, Preempted or OutOfcpu and you need to know which one and why.

Interview questions

After a release, the new ReplicaSet shows ImagePullBackOff on about half the pods while the rest run fine. Walk me through it.
A pod is in CrashLoopBackOff and kubectl logs --previous prints nothing. How do you find the cause?
Pods are Pending with 0/12 nodes are available: 12 Insufficient cpu, and the cluster autoscaler is not adding nodes. What do you check?
One Availability Zone had an outage. Afterwards, some StatefulSet pods stay Pending even though the other zones have plenty of room. Why?
During a large scale-up on EKS, new pods sit in ContainerCreating with FailedCreatePodSandBox. How would the same problem look on AKS and GKE?
Several nodes report DiskPressure and pods are being evicted. The team says BestEffort pods always go first, so they are not worried about their Guaranteed database. Are they right?
A pod is stuck in Init:0/2. The first init container waits for Postgres and the app also needs the Cloud SQL Auth Proxy. What is wrong and how do you redesign it?