Node and Scaling Errors — NotReady, Scale-Up, Drain, Terminating and HPA

How to diagnose the failures above any one pod: nodes going NotReady, autoscalers that will not add or remove nodes, drains blocked by PodDisruptionBudgets, pods stuck Terminating, and HPAs that cannot read metrics, on EKS, AKS and GKE.

Overview

These failures sit above any one pod. A node stops reporting, the cluster will not grow, a drain never finishes, a pod will not die, or an autoscaler sits still. They rarely look like one broken service. They look like several unrelated workloads degrading at once, and that pattern is itself the clue: when unrelated services fail together, check the node and capacity layer before you open any application log.

Work from the control loop outward. Every item on this page is a controller that has stopped acting: the node lifecycle controller, the scheduler, the Cluster Autoscaler or Karpenter, the Eviction API guarded by a PodDisruptionBudget, the kubelet finishing a pod, or the HPA controller reading metrics. Each one writes down why it did nothing, in a node condition, an event, a status ConfigMap or an HPA condition. Your job is to find that written reason, not to guess.

The one idea to keep: capacity problems are almost always accounting problems. The scheduler, the autoscalers and the HPA all reason about resource *requests*, not real usage, and the eviction machinery reasons about *budgets*, not health. Wrong numbers in a spec produce most of the incidents on this page.

Symptom to first command

Node shows NotReady or Unknown — Kubelet or runtime down, CNI not initialized, node lost network — kubectl describe node \ — grep -A12 Conditions
Pods stay Pending, no node arrives — Node group at max, pod fits no instance type, cloud capacity or quota — kubectl events -n prod --for pod/
kubectl drain retries forever — PDB allows zero disruptions — kubectl get pdb -A
Pod stuck Terminating — Node unreachable, finalizer, volume will not unmount — kubectl get pod -n prod -o jsonpath='{.metadata.finalizers}'
HPA target shows — metrics-server down or missing CPU requests — kubectl describe hpa orders -n prod
Node has DiskPressure or PIDPressure, pods evicted — Logs, images or a fork bomb exhausting the node — kubectl describe node \ — grep -E 'Pressure\ — Taints'
Nodes stay mostly empty and never go away — Scale-down blocked by PDBs, local storage or annotations — kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

What you will be able to do

Read a node's conditions and taints and say whether the kubelet is reporting a problem or has stopped reporting at all.
Explain why the autoscaler did not add a node from its events and status, and fix the node group, the pod spec or the quota that caused it.
Unblock a drain without deleting the PDB, and write budgets that cannot deadlock.
Decide when force-deleting a Terminating pod is safe, and use the out-of-service taint instead when a node is truly gone.
Diagnose an HPA stuck at from the metrics APIService down to missing requests, and tune scaling behavior on purpose.

Key concepts

Node Lease: A Lease object in kube-node-lease that the kubelet renews every 10 seconds. If the node lifecycle controller sees no renewal for node-monitor-grace-period (50 seconds since 1.32), the node becomes Unknown.
Taint-based eviction: The not-ready and unreachable NoExecute taints evict pods after their toleration expires. DefaultTolerationSeconds gives every pod 300 seconds for both.
Node-pressure eviction: The kubelet evicting pods itself when a signal such as nodefs.available or pid.available crosses its threshold. It ignores PodDisruptionBudgets.
Eviction API: The API that drain, autoscalers and upgrades use to remove pods. It checks PodDisruptionBudgets and answers 429 when an eviction would violate one.
PodDisruptionBudget: A floor on healthy pods during voluntary disruption, written as minAvailable or maxUnavailable. unhealthyPodEvictionPolicy: AlwaysAllow lets not-Ready pods be evicted.
NotTriggerScaleUp: The event the Cluster Autoscaler writes on a pending pod when no node group it manages could fit that pod.
Karpenter NodePool and NodeClaim: A NodePool sets which instances Karpenter may launch and its limits. Each NodeClaim is one requested node, and its status explains launch failures.
Finalizer: A key in metadata.finalizers that stops an object from being deleted until the controller that added it removes it.
Out-of-service taint: node.kubernetes.io/out-of-service, applied by hand to a node confirmed shut down. It force deletes its pods and detaches volumes (stable since 1.28).
Resource metrics API: metrics.k8s.io, served by metrics-server through the aggregation layer. The HPA divides this usage by requests to get utilization.

When to use it

Several unrelated services degrade at the same time and no single deploy explains it.
A node shows NotReady or Unknown, or pods on one node are being evicted while the node still reports Ready.
Pods stay Pending after a traffic spike and no new node arrives.
A cluster or node pool upgrade stalls on draining, or a pod will not leave Terminating.
An HPA shows targets, sits at maxReplicas, or scales slower than expected.
The node count stays high after load drops and the compute bill keeps growing.

Interview questions

Three nodes in one availability zone go NotReady at once. Pods on them keep serving errors for about five minutes before moving. Walk me through it.
A traffic spike arrives, the HPA raises replicas, but new pods stay Pending for 20 minutes. What happened and how do you fix it?
A managed node group upgrade has been stuck for an hour on draining. How do you find the blocker, and how does this differ on EKS, AKS and GKE?
A node lost power. The StatefulSet pod orders-db-0 on it has been Terminating for 40 minutes and the database is down. What do you do?
After the team enabled a service mesh, the orders HPA stopped scaling and shows /70%. Explain.
Pods on one node keep getting Evicted, yet the node shows Ready. Unrelated workloads are affected. How do you work it?
Traffic dropped by half overnight but the node count stayed the same for a week. Finance is asking why. How do you find out?