Control Plane Errors — API Server, RBAC, Webhooks, Quotas and Finalizers

The failures that stop you before any workload runs: unreachable API servers, Forbidden, broken webhooks, stuck namespaces, quota, exec and logs dial errors, expired certificates, a full etcd and 429 throttling.

Overview

Control-plane errors stop you before any workload is involved. Every kubectl command, CI deploy and controller loop is an HTTPS call to the API server, and that call passes through a fixed chain: connect and TLS, authentication, API Priority and Fairness, authorization, admission, then storage in etcd. A few requests (exec, logs, port-forward) then leave the control plane again and dial the kubelet on the node. Each error on this page is one link in that chain breaking.

The method is outside-in: first prove you are talking to the right cluster and can reach it at all. Then read the rejection message word by word, because the API server nearly always names the rule that stopped you: the user, the verb, the webhook, the quota, the finalizer. Guessing from the HTTP code alone is how people end up editing RBAC to fix a webhook. The code-level theory (401 versus 403, 409, 410, 422, 429 and the 5xx family) lives in the companion topic on API server status codes; this page is about the operational failures and what to type.

The one idea to keep: find the link, then fix only that link. A connection timeout, an expired certificate, a Forbidden, a webhook outage and a full etcd all look like 'kubectl is broken', and their fixes have nothing in common.

Symptom to first command

connection refused, i/o timeout, TLS handshake timeout — Wrong context, blocked network path, or unhealthy API server — kubectl config current-context
Error from server (Forbidden): ... cannot resource — No Role or ClusterRole grants that verb — kubectl auth can-i -n prod --as=
failed calling webhook or admission webhook ... denied the request — Webhook down (Fail policy) or a real policy denial — kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
Namespace stays Terminating — Finalizer with no controller, or an unavailable APIService — kubectl get namespace staging -o jsonpath='{.status.conditions}'
Deploy applied but no new pods; exceeded quota in events — ResourceQuota full, or a pod without requests or limits — kubectl events -n prod --types=Warning
error dialing backend on exec or logs only — API server cannot reach the kubelet on port 10250 — kubectl get --raw /api/v1/nodes//proxy/healthz
x509: certificate has expired or sudden Unauthorized everywhere — Control-plane or client certificates expired — sudo kubeadm certs check-expiration
etcdserver: mvcc: database space exceeded on every write — etcd hit its space quota and raised the NOSPACE alarm — etcdctl alarm list
A controller logs too many requests (429) — API Priority and Fairness is shedding that client's flow — kubectl get --raw /metrics \ — grep apiserver_flowcontrol_rejected_requests_total

What you will be able to do

Tell a reachability problem from a credential problem from a dead control plane in under a minute, with the command that proves which.
Read a Forbidden, webhook or quota message and turn its facts straight into the exact fix.
Recover a cluster locked by a failing webhook, a full etcd or expired certificates, in the right order and without making it worse.
Explain how each failure looks on EKS, AKS and GKE, where the control plane is managed and some of these fixes belong to the provider.
Name the guardrail (policy, alert, spec field or pipeline check) that stops each one from coming back.

Key concepts

Request chain: Every API call passes connect and TLS, authentication, API Priority and Fairness, authorization, admission, then etcd; each error on this page is one link failing.
RBAC is additive: There are no deny rules, so an RBAC Forbidden always means no Role or ClusterRole grants that verb on that resource in that scope.
failurePolicy: What the API server does when it cannot get an answer from a webhook. The default, Fail, rejects the request; Ignore lets it through.
ValidatingAdmissionPolicy: CEL rules evaluated inside the API server (GA in 1.30). No webhook pod, no network hop, so no webhook outage.
Finalizer: A marker on an object that a controller must remove before the object is deleted. With the controller gone, the object waits forever.
ResourceQuota: A namespace cap on summed requests, limits and object counts, enforced at pod admission. Only non-terminal pods count against compute quota.
API server to kubelet path: logs, exec, attach, port-forward and cp make the API server dial the kubelet on port 10250; on AKS and some GKE clusters this runs through a Konnectivity tunnel.
kubeadm certificate lifetime: Leaf certificates last one year and the CA ten years; kubeadm upgrade renews the leaves, and kubeadm certs renew does it by hand.
etcd space quota and NOSPACE: etcd enforces a backend size quota (2 GiB default, 8 GiB suggested maximum). Past it, a NOSPACE alarm makes the cluster read and delete only.
API Priority and Fairness: Classifies requests into FlowSchemas and priority levels with separate concurrency and queues, and answers 429 with Retry-After when a level is full.

When to use it

kubectl or a CI job cannot reach the cluster, or fails with x509, Unauthorized or Forbidden.
Creates or updates fail cluster-wide with a webhook error, or writes fail while reads still work.
A deploy reports success but the new pods never appear.
A namespace has been Terminating for longer than it should.
kubectl exec, logs or port-forward fail while every other command works.
A controller falls behind and logs 429 too many requests.

Interview questions

On Monday morning kubectl against the self-managed prod cluster fails, and it worked on Friday. Walk me through it.
You installed a policy engine with a validating webhook on Pods. An hour later a node died and now no pods can be created anywhere, including the policy engine's own. How do you recover?
After upgrading the cluster to 1.35, the CI smoke test that runs kubectl exec as a read-only ServiceAccount starts failing with Forbidden. Nothing in RBAC changed. Why?
A pipeline applies a new Deployment version, reports success, but the new pods never appear and the old ones keep serving. Walk me through it.
kubectl logs and exec work for some pods and time out for others. How would you debug it, and how does it differ on EKS, AKS and GKE?
A namespace has been Terminating for a day after someone uninstalled an operator. What do you do, and what do you refuse to do?
Writes to a self-managed cluster fail with etcdserver: mvcc: database space exceeded. Recover it, then tell me what would be different on EKS.