HTTP 4xx and 5xx in Kubernetes — Which Hop Sent It and How to Fix It

How to tell which hop wrote a 4xx or 5xx (CDN, cloud load balancer, Ingress, mesh sidecar, app or API server) and how to fix 400, 401, 403, 405, 408, 499, 409, 410, 414, 431, 422, 426, 429 and 500.

Overview

A status code on its own tells you very little. Between a user and the orders pods in prod there can be a CDN or web application firewall (WAF), a cloud load balancer, an Ingress controller or Gateway, a service mesh sidecar and finally the application. Any of those hops can answer on its own, and each one uses the same numbers to mean slightly different things. Separately, kubectl, controllers and GitOps tools talk to the Kubernetes API server, which returns HTTP codes too, and there 409, 410 and 422 have precise meanings of their own.

So work outside-in and ask one question first: which hop wrote this response? The response headers, the shape of the body and the access log of each proxy answer that in a minute. Only then read the code. A 403 with an nginx HTML body and nothing in the app log is an allow-list; the same 403 as JSON from the app is a business rule; RBAC: access denied is the mesh.

The one idea to keep: find the hop, then the code, then the cause. Most wasted hours on 4xx and 5xx come from debugging the application for a response it never produced.

The examples use ingress-nginx because it is still everywhere, but the Kubernetes project retired it in March 2026: no more fixes or security patches. Gateway API is the successor, and many of its implementations are built on Envoy, so learn to read Envoy's responses and response flags as well as nginx's.

Which hop answered

CDN or WAF in front — server: cloudflare plus cf-ray; via naming CloudFront plus x-amz-cf-id; x-azure-ref from Front Door — The CDN or WAF log, not the cluster
Cloud load balancer — server: awselb/2.0 on ALB-generated errors; Microsoft-Azure-Application-Gateway/v2; via: 1.1 google on GKE — ALB access log elb_status_code; App Gateway access log; LB statusDetails
ingress-nginx — server: nginx with no version and a small HTML page with nginx — Controller access log: upstream_status is - when nginx answered itself
Envoy (Istio, Envoy Gateway) — server: istio-envoy or envoy; plain-text bodies such as no healthy upstream; x-envoy-upstream-service-time only when the upstream answered — Access log %RESPONSE_CODE% and %RESPONSE_FLAGS% (UH, UF, UT, NR, DC, RL)
Your application — Its own JSON error body and headers — App log line with the same X-Request-ID
Kubernetes API server — A JSON Status object with reason and code; kubectl prints Error from server (Reason) — Audit log; a kubectl error without from server never left your machine

Every code in one table

400 — Ingress, LB, app, API server — Malformed request, HTTP sent to an HTTPS port, a header line too large for nginx, or an unknown field under strict validation — curl -sv and the controller log
401 — Ingress auth, mesh, app, API server — No valid credentials; from kubectl an expired or unmapped identity — kubectl auth whoami
403 — WAF, Ingress allow-list, mesh, API server — Known caller, refused by a rule — Did the app log it? Then the client IP the proxy saw
404 — Ingress, Envoy NR, app — No route matched, or the app has no such path — Ingress and Gateway page for rules
405 — App, sometimes a proxy — Method not allowed on that path; often a CORS preflight — curl -i -X OPTIONS and the Allow header
408 — Ingress, LB — Client did not finish sending the request in time — Client network and upload size
409 — API server, app — Stale resourceVersion, server-side apply field conflict, or AlreadyExists — --show-managed-fields
410 — API server — Watch or list resourceVersion older than the retained history — Controller relist behavior
413 — Ingress, LB, WAF — Body larger than the proxy allows — Ingress and Load Balancer Errors page
414 — Ingress, LB — Request line (URL) too long — large-client-header-buffers
422 — API server, app — Object parsed but failed validation, or a ValidatingAdmissionPolicy denied it — The field path in the message
426 — Envoy, WebSocket server — HTTP/1.0 sent to Envoy, or a WebSocket path called without a valid upgrade — The caller's HTTP version
429 — Ingress, mesh, API server, registry — Rate limited; but ingress-nginx sends 503 and most cloud WAFs 403 by default — Which of the four sources
431 — Envoy, app servers — Request headers too large — Token and cookie size
444, 494-497 — ingress-nginx internal — No response (444), header too large (494), client certificate errors (495, 496), HTTP to HTTPS port (497); the client sees 400 for 494-497 — Controller log
499 — ingress-nginx, App Gateway v2 — Client closed the connection before the response; Envoy logs 0 with DC, ALB logs 460 — Backend latency
500 — App, nginx auth subrequest, API server — Unhandled error; a failing auth-url service; an unreachable admission webhook — App log by request ID
502, 503, 504 — Ingress, LB, mesh — Bad upstream reply, no healthy upstream, upstream too slow — Ingress and Load Balancer Errors page

Symptom to first command

400 with an nginx page — HTTP to the HTTPS port, or a header line over 8 KB — curl -sv https://api.corp.com/api/orders
401 on every request — Auth service rejecting everyone, or no sign-in redirect — kubectl describe ingress orders-ingress -n prod
403 and nothing in the app log — Allow-list, WAF or mesh policy — kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=100
405 or a CORS error in the browser — Preflight not handled, or a rewrite hits the wrong handler — curl -i -X OPTIONS https://api.corp.com/v1/orders
408 or 499 in the proxy log — Slow client upload, or a backend slower than the client's timeout — kubectl logs -n ingress-nginx deploy/ingress-nginx-controller \ — grep ' 499 '
409 from kubectl or a controller — Two writers on one field — kubectl get deploy orders -n prod --show-managed-fields -o yaml
410 or too old resource version — Watch fell behind the history window — kubectl logs -n deploy/
414, 431 or Request Header Or Cookie Too Large — Long URL, large JWT or many cookies — curl -s -o /dev/null -w '%{size_request}\n' ...
422 is invalid — Field failed validation or is immutable — kubectl apply -f deploy.yaml --dry-run=server
426 Upgrade Required — HTTP/1.0 into Envoy, or a broken WebSocket upgrade — kubectl logs -n prod -c istio-proxy
429 — Edge rate limit, mesh, API Priority and Fairness, or registry — Read the body and the server header
500 — App exception, failing auth service, or webhook — kubectl logs deploy/orders -n prod --since=10m

What you will be able to do

Name the hop that produced any 4xx or 5xx from its headers, body and logs before touching the application.
Explain the mechanism behind each code, including the proxy-specific ones (444, 494-497, 499, ALB 460 and 464, Envoy 0 DC).
Resolve 409, 410 and 422 from kubectl, controllers and GitOps pipelines, including server-side apply field conflicts.
Say how each code shows up on EKS, AKS and GKE, where the cloud load balancer or WAF often answers first.
Set the guardrail that stops each one recurring: timeout budgets, header buffers, field ownership, rate-limit status codes, request IDs.

Key concepts

Hop attribution: Working out which component wrote a response, from its server header, body format and whose access log has the request, before interpreting the code.
upstream_status: The ingress-nginx access log field with the backend's status. A dash means nginx answered the request itself.
Envoy response flags: Short codes in Envoy's access log that explain a response: UH no healthy upstream, UF connection failure, UT timeout, NR no route, RL rate limited, DC client disconnected.
499 and its equivalents: nginx's log-only code for a client that closed the connection before a response. Envoy logs 0 with DC, an ALB logs 460, Application Gateway v2 logs 499.
resourceVersion: An opaque version on every Kubernetes object. Updates with a stale one get 409; watches from one older than the retained history get 410.
Field manager: The owner server-side apply records for each field in managedFields. Applying a field another manager owns returns 409 unless you use --force-conflicts.
Watch bookmark: A watch event that carries a fresh resourceVersion without an object change, so a reconnect can resume inside the history window instead of getting 410.
Strict field validation: kubectl's default --validate=strict. The API server rejects unknown or duplicate fields with 400 BadRequest instead of dropping them.
API Priority and Fairness: The API server's flow control. It classifies requests into flow schemas and priority levels and sheds excess with 429 and Retry-After.
large_client_header_buffers: nginx's buffers for long request lines and headers, 4 of 8 KB by default. A longer URL gets 414 and a longer header line gets 400.

When to use it

You have a status code from a user, a dashboard or kubectl and need to know which hop produced it.
A 4xx is rising at the edge and the application log shows nothing.
A pipeline, GitOps tool or controller fails with 409, 410 or 422 from the API server.
Large tokens, WebSockets, gRPC or long uploads fail only in the cluster, not locally.
You are choosing timeouts, header buffers or rate limits across a CDN, load balancer, Ingress and mesh.

Interview questions

A user sends a screenshot showing only a status code. How do you work out which hop produced it?
Office users get 403 on api.corp.com while everyone else works. The app logs nothing. The cluster runs ingress-nginx behind a cloud network load balancer. Walk me through it.
Your CI pipeline started failing with an apply conflict on .spec.replicas right after the team added an HPA. What happened and what do you change?
The ingress-nginx dashboard shows a jump in 499s but almost no 5xx. The product team says it is just mobile users on bad networks. Do you agree?
After the identity team added group claims to tokens, some users on your AKS cluster get 400 Request Header Or Cookie Too Large, and a few get 431. Traffic goes Application Gateway, ingress-nginx, Istio sidecar. How do you fix it?
Your operator's logs fill with "too old resource version" and it keeps crash-looping. How do you reason about it?
Your team moved API rate limiting from ingress-nginx to AWS WAF in front of the ALB on EKS. Client SDKs that handled 429 now fail hard. Why, and how would this differ on GKE?