Kubernetes Error Triage — Symptoms, Exit Codes and Cloud Equivalents

The hub of the Kubernetes production errors guide: the outside-in method, the first five minutes of commands, every symptom mapped to its layer and page, pod states and exit codes decoded, and the EKS, AKS and GKE equivalents.

Overview

A Kubernetes failure almost never shows up where it starts. A user sees a 503, but the cause is a readiness probe. A pod sits in Pending, but the cause is a quota, a taint or a cloud account with no spare IP addresses. This page is the entry point to the whole category: it teaches the method, the commands you run in the first five minutes, and which page covers each error in depth.

The method is outside-in. Start where the user is and walk inward along the request path (DNS, cloud load balancer, Ingress or Gateway, Service, EndpointSlice, Pod, container, dependencies). For anything that never started, walk the lifecycle path instead (apply, admission, scheduler, kubelet, container runtime, probes). Every hop either passes the request on or leaves a specific signal behind.

The one idea to keep: a status code, a pod state or an exit code is a location, not a diagnosis. A 504 says the request reached a backend that did not answer in time. ImagePullBackOff says the pod was scheduled and the kubelet could not fetch the image. Read the signal as 'how far did it get', and you know which layer to open before you touch a log.

Symptom to first command

Edge and Ingress — 502, 503, 504, 404 from the proxy, TLS errors, EXTERNAL-IP pending — kubectl get endpointslices -n prod -l kubernetes.io/service-name=orders — Ingress & Load Balancer Errors
Control plane — kubectl timeouts, Forbidden, webhook denials, quota, stuck namespaces — kubectl get --raw='/readyz?verbose' — Control Plane Errors
Scheduling and startup — Pending, ImagePullBackOff, CrashLoopBackOff, Init:*, sandbox errors, Evicted — kubectl events -n prod --for pod/ — Pod Startup & Scheduling Errors
Container — OOMKilled, exit 1, 126, 127, Running but 0/1, old code still running — kubectl logs -n prod --previous — Container Errors
Service networking — i/o timeout, DNS failures, port mismatch, errors on every rollout, mesh sidecars — kubectl get networkpolicy -n prod — Service Networking Errors
Storage — FailedMount, PVC Pending, Multi-Attach, disk full, resize not visible — kubectl describe pvc data-orders -n prod — Storage Errors
Nodes and scaling — NotReady, no scale-up, drain blocked, stuck Terminating, HPA without metrics — kubectl describe node — Node & Scaling Errors
Identity, secrets, config — AccessDenied from the cloud, stale config, secret mount failures — kubectl get sa orders-sa -n prod -o yaml — Identity/Secrets/Config Errors
HTTP semantics — 400, 401, 403, 405, 408, 409, 410, 413, 414, 422, 426, 429, 431, 499, 500 — curl -sv https://api.corp.com/api/orders -o /dev/null — HTTP 4xx/5xx Reference

What you will be able to do

Read any status code, pod state or exit code as a position on the request or lifecycle path, and name the next place to look.
Run the first five minutes of any incident from memory: events, previous logs, EndpointSlices, ephemeral debug containers and API server health.
Map every one of the 58 symptoms in this guide to its layer, its first command and its page.
Translate a Kubernetes object into the EKS, AKS or GKE service behind it, and find where that cloud reports the real error.
Run an incident out loud in an interview: mitigate, diagnose, prevent, then write a blameless postmortem.

Key concepts

Outside-in triage: Walk the request path from the client inward, or the lifecycle path from apply onward, and stop at the first hop that fails.
EndpointSlice: The object that lists the ready pod IPs and ports behind a Service. It replaced Endpoints, which is deprecated as of Kubernetes 1.33.
Pod phase versus STATUS: The phase is one of Pending, Running, Succeeded, Failed or Unknown. The STATUS column in kubectl shows a more specific reason, such as CrashLoopBackOff or Terminating.
CrashLoopBackOff: The kubelet waiting before restarting a container that keeps exiting, with a delay that doubles from 10 seconds up to 5 minutes by default.
128 + signal rule: An exit code above 128 means the process was killed by a signal; subtract 128 to get the signal number.
Ephemeral container: A temporary debug container added to a running pod with kubectl debug, sharing its namespaces, for images that have no shell.
Grace period: terminationGracePeriodSeconds, 30 by default: the time between the start of shutdown and SIGKILL. The preStop hook counts against it.
Gateway API: The Kubernetes successor to Ingress, with role-oriented resources (GatewayClass, Gateway, HTTPRoute). Supported on EKS, AKS and GKE.
Mitigate, diagnose, prevent: The incident order: stop user impact with a reversible action, find the root cause with saved evidence, then add a guardrail.

When to use it

An alert fired and you do not yet know which layer is failing.
You see a status code, pod state or exit code and need to know what it proves and which page covers it.
You are preparing for an SRE, platform or DevOps interview with a live troubleshooting round.
You need the EKS, AKS or GKE service behind a Kubernetes object, or where that cloud logs the real error.
You are writing a runbook or postmortem and want a standard first-five-minutes checklist.

Interview questions

The site is down. You have kubectl access and nothing else. Go.
After every deploy, users see 502s for about twenty seconds. Probes are correct. Walk me through it.
A container restarts with exit code 137, but the reason is Error, not OOMKilled. What is going on?
New pods are stuck in Pending after a node group change on EKS. How would this differ on AKS and GKE?
kubectl itself times out. Is the cluster down?
You rolled back the deploy, but errors continue. What now?
How do you run the incident and the postmortem, not just the debugging?