What Happens on kubectl apply

From one YAML file to running Pods

Follow one kubectl apply -f app.yaml end to end: the client merge, authentication, RBAC, admission and etcd in the API server, then the controllers, the scheduler and the kubelet that turn a stored Deployment into Pods behind a Service.

Overview

kubectl apply -f app.yaml is the command most engineers type every day, and "what happens when you run it?" is a favorite interview question because it touches every part of Kubernetes. The honest answer has two halves.

The first half is a single HTTPS request. kubectl reads the file, works out which API each object belongs to, computes what to change, and sends it to the API server. The API server checks who you are, whether you are allowed to do it, lets admission plugins change or reject the object, validates it, and stores it in etcd. It answers 201 Created or 200 OK, and kubectl prints deployment.apps/web created. At that moment nothing is running yet.

The second half is reconciliation. Controllers, the scheduler and the kubelet each watch the API server for objects they care about. The Deployment controller sees a new Deployment and creates a ReplicaSet; the ReplicaSet controller creates three Pods; the scheduler picks a node for each; the kubelet on that node starts the containers and reports status back. When the Pods pass their readiness probes, they join the Service's EndpointSlice and traffic reaches them. No component calls another: they all read and write objects through the API server, and each one keeps comparing the desired state with the actual state until they match.

The example used throughout: app.yaml holds a Deployment web (3 replicas, image ghcr.io/acme/web:1.4.2) and a Service web of type ClusterIP, applied to namespace shop.

The interviewer listens for four things. Do you separate the synchronous request from the asynchronous reconciliation? Do you know the order of the API server's checks (authentication, authorization, mutating admission, validation, validating admission, storage)? Can you name who creates what (Deployment controller → ReplicaSet, ReplicaSet controller → Pods, scheduler → binding, kubelet → containers)? And can you say where a broken deploy shows up, since a successful kubectl apply says nothing about whether the Pods will start?

Key concepts

Declarative configuration
Three-way merge
Server-side apply and field managers
Authentication and RBAC
Admission control
etcd and resourceVersion
Controllers and watches
Owner references
Scheduling framework
Kubelet and CRI/CNI/CSI
EndpointSlices
Reconciliation loop

Functional requirements

Apply a file of several objects in one command, creating the ones that do not exist and updating the ones that do
Change only the fields the file manages, leaving fields set by other tools (an autoscaler, a controller) alone
Reject a request from an unknown identity, or one without permission, before anything is stored
Let cluster policy add defaults and refuse objects that break rules, before they are stored
Store the object durably and return its new resourceVersion to the client
Turn a stored Deployment into the requested number of running, ready Pods
Roll out a new image gradually, keeping the service available
Send Service traffic only to Pods that are ready

Non-functional requirements

Idempotent: running the same apply again changes nothing and reports unchanged
Safe under concurrency: two writers cannot silently overwrite each other (optimistic concurrency on resourceVersion)
Self-healing: a deleted Pod or a failed node is replaced without any new command
Auditable: every request is attributed to an identity and can be written to the audit log
Loosely coupled: each component only talks to the API server, so one can restart without the others noticing
Responsive: the API request itself finishes in well under a second on a healthy cluster; Pods follow in seconds

Interview questions

What happens when you run kubectl apply -f app.yaml?
In what order does the API server check a request?
What is the difference between client-side and server-side apply?
kubectl apply succeeded but no Pods are running. Where do you look?
How does the scheduler choose a node?
How does a rolling update work, and how does it fail?
Why do Kubernetes components not call each other directly?
How is kubectl apply different from terraform apply?