Kubernetes Pod Scheduling

Kubernetes pod scheduling determines how the control plane assigns pods to nodes using a multi-phase pipeline of filtering, scoring, and binding decisions. Mastering scheduling lets you control placement for availability, performance, cost, and fault isolation across your cluster.

Overview

Overview

Pod scheduling is how the control plane decides which node runs each new pod. The default kube-scheduler watches for unscheduled pods and runs each through a pipeline of filter and score plugins before committing a node. Knowing that pipeline is the key to clusters that stay available across zones, avoid resource contention and scale well.

The scheduling framework

The framework, stabilized in Kubernetes 1.19, replaced the older predicates-and-priorities model with named extension points. Each accepts plugins that read pod and node state, change decisions or call external systems, so teams can ship custom placement logic as compiled Go plugins without forking the scheduler.

  • Queueing — PreEnqueue
  • Filtering — PreFilter, Filter, PostFilter
  • Scoring — PreScore, Score
  • Commit — Reserve, Permit, PreBind, Bind, PostBind

Placement controls

  • Node affinity — Node labels — "Only nodes with an SSD", "prefer us-east-1a"
  • Pod affinity / anti-affinity — Other pods' labels, through a topology key — Co-locate chatty pods, or separate replicas across failure boundaries
  • Topology spread constraints — Pod counts per topology domain — Balance replicas with a numeric maxSkew tolerance
  • Taints and tolerations — Nodes repel pods — Dedicated nodes, defective hardware, NoExecute eviction

> [!TIP] requiredDuringSchedulingIgnoredDuringExecution is a hard filter; preferredDuringSchedulingIgnoredDuringExecution is a soft scoring hint.

Topology spread constraints (introduced in 1.16, GA in 1.19) are simpler to operate than anti-affinity: declare maxSkew and topologyKey, and the scheduler enforces the balance. They also work with the cluster autoscaler, which can request nodes in underrepresented zones.

Taints are the inverse of affinity: nodes repel pods unless a pod opts in. Combined with PriorityClass and preemption, they control where workloads land and which are evicted first when resources are scarce.

QoS classes and eviction

  • BestEffort — None — First
  • Burstable — Requests below limits, or partial — Second, those over their requests
  • Guaranteed — Requests equal limits on every container — Last resort

> [!IMPORTANT] The QoS class decides who survives memory pressure. Design workloads with it in mind to avoid unexpected terminations.

Key concepts

Scheduler Extension Points and the Scheduling Framework
The Kubernetes scheduling framework defines a sequence of well-known extension points that plugins implement. The Filter extension point determines whether a node is eligible for a pod. The Score extension point assigns a numeric priority to each eligible node. Reserve claims the node in the scheduler's internal cache to prevent double-booking between the scheduling cycle and the async binding cycle. Permit can block the binding cycle until an external condition is met, which batch schedulers use to ensure all pods in a group are schedulable before any is bound. Bind writes the PodBinding object to the API server. A minimal custom filter plugin in Go looks like this: The plugin is registered in the scheduler configuration and compiled into the scheduler binary or loaded as a separately deployed scheduler with a custom KubeSchedulerProfile. Multiple profiles can coexist, and pods opt in to a profile by setting schedulerName in the pod spec.
Node Affinity: Required vs Preferred
Node affinity uses label selectors against node labels. The requiredDuringSchedulingIgnoredDuringExecution field acts as a hard filter: if no node matches, the pod stays pending indefinitely. The preferredDuringSchedulingIgnoredDuringExecution field adds a weighted score bonus without blocking scheduling. matchExpressions supports operators In, NotIn, Exists, DoesNotExist, Gt, and Lt, giving you expressive multi-label rules that are not possible with the older nodeSelector map. The ignored-during-execution suffix means an already-running pod is not evicted if a node's labels change after the pod is placed. A future requiredDuringSchedulingRequiredDuringExecution field is planned but not yet available in stable releases.
Pod Affinity and Anti-Affinity with topologyKey
Pod affinity and anti-affinity use label selectors against other pods and a topologyKey to define the scope of the constraint. When topologyKey is topology.kubernetes.io/zone, the rule applies at zone granularity: all nodes in a zone share the same value for that label, so a pod anti-affinity at zone scope prevents two matching pods from being placed in the same zone. When topologyKey is kubernetes.io/hostname, the constraint is per-node, ensuring no two matching pods share a single machine. This example places api-server pods on distinct nodes (hard) and distinct zones (soft), while preferring to co-locate with a redis-cache pod on the same node to reduce cache latency. Anti-affinity rules scale poorly in large clusters because the scheduler must compare the candidate pod against every running pod that matches the selector, making scheduling O(pods) per node candidate.
Topology Spread Constraints
Topology spread constraints express the desired distribution of pods across topology domains without the O(pods) cost of anti-affinity. The maxSkew field is the maximum allowed difference between the domain with the most matching pods and the domain with the fewest. The topologyKey identifies the node label that defines domains. The whenUnsatisfiable field is either DoNotSchedule (hard) or ScheduleAnyway (soft). The minDomains field, added in Kubernetes 1.25, specifies the minimum number of eligible domains; if fewer domains exist, the global minimum is treated as zero. With three zones (a, b, c) and this constraint, the scheduler will not place a pod in zone a if zone a already has two more web pods than zone c. The minDomains: 3 declaration tells the scheduler to assume the global minimum is zero when fewer than three zones have eligible nodes, preventing the constraint from relaxing when the cluster has only one or two zones available. The second constraint softly encourages even per-node spread but does not block scheduling if nodes are unbalanced.
Taints and Tolerations
A taint is a key-value pair with an effect applied to a node. The NoSchedule effect prevents new pods from being scheduled on the node unless they have a matching toleration. The PreferNoSchedule effect softly discourages scheduling. The NoExecute effect evicts existing pods that lack a toleration and blocks new ones; pods with a matching toleration can optionally include tolerationSeconds to specify how long they survive before eviction. The system automatically adds taints for node conditions such as not-ready, unreachable, memory-pressure, disk-pressure, pid-pressure, and unschedulable. DaemonSet pods receive automatic tolerations for these system taints so they remain on degraded nodes to perform cleanup work. The tolerationSeconds: 300 pattern is a common production pattern for giving long-running pods five minutes to finish in-flight requests before the kubelet forces eviction.
QoS Classes and Eviction Order
Kubernetes assigns one of three Quality of Service classes to every pod based on how resource requests and limits are configured. Guaranteed pods have every container with CPU and memory requests equal to their limits. Burstable pods have at least one container with a request below its limit, or limits set without requests. BestEffort pods have no resource requests or limits on any container. During memory pressure, the kubelet evicts pods in this order: BestEffort first because they have no memory floor, then Burstable pods that are consuming more memory than their request, then Burstable pods consuming at or below their request, and finally Guaranteed pods. The OOM killer uses a similar ordering via oom_score_adj, which is set to 1000 for BestEffort containers, proportionally lower for Burstable containers based on how much they exceed their request, and -997 for Guaranteed containers so the kernel kills them last. Setting accurate resource requests is therefore both a scheduling concern and a runtime reliability concern.
PriorityClass and Preemption
PriorityClass assigns a numeric priority to pods. When a high-priority pod cannot be scheduled because all nodes are full, the scheduler's PostFilter extension point runs the preemption plugin, which identifies victim pods on candidate nodes whose removal would make room. The scheduler evicts the minimum set of lowest-priority pods needed to free enough resources, then places the pending pod. Setting preemptionPolicy: Never on the batch-low class means those pods will not preempt other pods even though they have a priority number, which is useful for workloads that should only run on spare capacity. System-critical components like kube-dns and metrics-server use the built-in system-cluster-critical and system-node-critical classes with priority values of 2000000000 and 2000001000 respectively, which are above the maximum user-configurable value of 1000000000 to ensure they cannot be accidentally outranked.

When to use it

  • Distributing stateless application replicas across multiple availability zones to survive a zone outage without downtime
  • Co-locating a sidecar or data cache pod on the same node as its primary workload to minimize inter-pod network latency
  • Dedicating GPU nodes to machine learning training jobs using taints so general workloads do not consume expensive GPU capacity
  • Preventing two replicas of a stateful service from landing on the same node, ensuring a single-node failure does not cause quorum loss
  • Controlling eviction priority during a node memory pressure event by assigning critical system components a high PriorityClass
  • Enabling the cluster autoscaler to provision nodes in the correct zone by expressing topology spread constraints that reveal where pods are underrepresented
  • Restricting noisy-neighbor tenants to specific node pools using node affinity combined with PodDisruptionBudgets to bound disruption during node maintenance
  • Scheduling batch jobs with low PriorityClass so long-running analytical workloads are preempted gracefully when real-time serving pods arrive

Interview questions

  • How would you configure a deployment to spread 10 replicas evenly across 3 availability zones, ensuring no zone has more than 1 pod extra compared to any other zone?
  • What is the difference between pod anti-affinity and topology spread constraints for distributing pods, and when should you prefer one over the other?
  • Walk through the order in which the kubelet evicts pods during a node memory pressure event. What determines where each pod falls in the eviction order?
  • Explain the mechanics of pod preemption. What happens step by step when a high-priority pod cannot be scheduled on any node?
  • Your cluster autoscaler is not scaling up even though several pods have been pending for 10 minutes due to insufficient CPU. What are the likely root causes and how do you diagnose each one?

Sign up free to read the full topic

More DevOps topics

Start free on Cariara · Coding problems · System design studio