Kubernetes the Hard Way

Overview

Overview

Kubernetes the Hard Way means bootstrapping a cluster entirely by hand: generating every certificate, writing every config file and starting every component yourself instead of using an installer like kubeadm. Kelsey Hightower popularized it as a learning tool, and it is still the most effective way to build real intuition for how Kubernetes works. Engineers who have done it know why a cluster fails to start, how to recover from certificate expiry, and what each component owns.

Component responsibilities

• etcd — Strongly consistent key-value store (Raft); holds all cluster state: nodes, pods, services, Secrets, ConfigMaps
• kube-apiserver — The only component that talks to etcd; every request passes authentication, authorization and admission
• kube-scheduler — Watches unscheduled pods; filters out nodes that cannot host the pod, then scores the rest
• kube-controller-manager — Control loops that converge actual state toward desired state
• kubelet — Turns a PodSpec into CRI, CNI and CSI calls and reports pod status back
• kube-proxy — Programs the kernel so Service ClusterIPs reach pod IPs

> [!IMPORTANT] Every control plane component is stateless and reads the world from etcd, so etcd's health is the cluster's health. Sizing for quorum, backup and restore, compaction and defragmentation are core skills.

One gate in front of etcd

Because all state goes through this pipeline, a well-configured API server with RBAC and admission controllers is a complete security boundary around the cluster's persistent state.

Watches, not polling

The scheduler and controller manager hold long-lived HTTP connections to the API server and receive streaming notifications when their resources change. The scheduler reacts to new unscheduled pods within milliseconds, and controllers converge with minimal latency.

The kubelet's three interfaces

• CRI (Container Runtime Interface) — Any conformant container runtime
• CNI (Container Network Interface) — Any network plugin
• CSI (Container Storage Interface) — Any storage driver

The kubelet also keeps monitoring running containers and updates pod status, so controllers and users see what is actually running.

kube-proxy modes

• iptables — Chains of DNAT rules — Fine for smaller clusters; rule counts become a bottleneck at scale
• IPVS — The kernel's IP Virtual Server module — More efficient load balancing in large clusters

Key concepts

etcd Raft Consensus and Quorum

etcd uses the Raft distributed consensus algorithm to ensure that all members of the cluster agree on the same state. Raft elects a single leader node that accepts all writes and replicates them to followers. A write is committed only after a majority quorum of nodes acknowledges it. For a cluster of N members, quorum requires floor(N/2)+1 nodes to be healthy. A 3-member cluster tolerates 1 failure, and a 5-member cluster tolerates 2 failures. When bootstrapping etcd for Kubernetes the Hard Way, you generate a unique set of TLS certificates for each member and for clients such as the API server. The initial cluster string tells each member how to find the others. Compaction removes old revisions so that disk usage does not grow unboundedly. Defragmentation reclaims the freed space on disk. Both operations should be automated in production.

kube-apiserver Authentication and Authorization Chain

Every request to the kube-apiserver passes through a pipeline before touching etcd. The first stage is authentication, where the server tries each enabled authenticator in order. The most common authenticators are x509 client certificate authentication, bearer token authentication, and webhook token authentication. When an authenticator succeeds it attaches a user identity consisting of a username, UID, and groups to the request context. The second stage is authorization. Kubernetes supports multiple authorization modes including RBAC, ABAC, Node, and Webhook. RBAC is the standard choice. The server checks whether the authenticated identity has a RoleBinding or ClusterRoleBinding that grants the requested verb on the requested resource. If no authorizer approves the request, it is rejected with 403. After authorization, admission controllers run. Mutating controllers run first and may modify the object. Validating controllers run second and may only approve or reject. Built-in controllers like LimitRanger add default resource requests, ServiceAccount injects a service account token, and NodeRestriction prevents kubelets from modifying objects outside their own node scope. Custom webhooks registered as MutatingWebhookConfiguration or ValidatingWebhookConfiguration plug into the same pipeline.

Scheduler Filter Predicates and Score Priorities

The kube-scheduler selects a node for every unscheduled pod through a two-phase algorithm. The filter phase eliminates nodes that cannot satisfy the pod's requirements. The score phase ranks the surviving nodes and selects the winner. This design keeps the algorithm extensible because both phases are composed of independently registered plugins. Common filter plugins include NodeResourcesFit, which checks that the node has sufficient CPU and memory for the pod's requests; NodeAffinity, which enforces required node selector terms; TaintToleration, which rejects nodes whose taints the pod does not tolerate; and PodTopologySpread, which enforces topology spread constraints. Common score plugins include NodeResourcesBalancedAllocation, which prefers nodes where resources are used more evenly; InterPodAffinity, which scores nodes higher if they already host pods that this pod wants affinity with; and ImageLocality, which gives a bonus to nodes that already have the pod's container images pulled. Custom scheduler plugins implement the Plugin interface and register with the framework by returning themselves from a factory function registered with the out-of-tree scheduler. This allows operators to add domain-specific scheduling logic without forking the scheduler binary.

TLS Bootstrapping and kubeconfig Anatomy

Every component in a Kubernetes cluster authenticates to the API server using TLS certificates signed by the cluster's certificate authority. In a from-scratch deployment you generate a CA, then use it to sign certificates for each component. The common name in the certificate determines the username the API server sees, and the organization field maps to Kubernetes groups. A kubeconfig file encodes three pieces of information: how to reach the cluster, which identity to present, and which context links them together. Each component has its own kubeconfig that points to the API server's load-balanced address and includes its signed certificate and key. Kubelet certificates use the Node authorization mode, where the username must be system:node:NODENAME and the group must be system:nodes. The NodeRestriction admission plugin then limits each kubelet to only reading and writing objects related to its own node and the pods assigned to it.

kubelet Sync Loop and CRI/CNI/CSI Integration

The kubelet's primary job is to reconcile the desired pod state delivered by the API server with the actual state of containers running on the node. It runs a continuous sync loop that compares the list of pods that should be running with the list of containers that are actually running and takes action to close any gap. When the kubelet decides a pod needs to start, it calls the container runtime through the CRI gRPC interface. The CRI defines two services: RuntimeService for pod and container lifecycle operations and ImageService for image management. The kubelet first calls RunPodSandbox to create the pod's network namespace and pause container, then calls CreateContainer and StartContainer for each container in the spec. After the sandbox is created, the kubelet invokes the CNI plugin to configure networking for the pod's network namespace. The CNI plugin assigns an IP address, sets up routing, and potentially connects the namespace to an overlay network. The kubelet passes the pod's namespace path to the plugin and receives back the assigned IP address, which it then reports in the pod status. For storage, the kubelet calls the CSI driver to attach volumes to the node and mount them into the pod's filesystem. The kubelet coordinates with the attachdetach controller in kube-controller-manager, which is responsible for attaching volumes to nodes, while the kubelet itself is responsible for mounting attached volumes into pod containers. The kubelet also runs liveness, readiness, and startup probes. Liveness probe failures cause the kubelet to restart the container. Readiness probe failures cause the kubelet to remove the pod's IP from endpoint slices. Startup probe failures prevent liveness and readiness probes from running until the container has had time to initialize.

kube-proxy iptables vs IPVS Mode

kube-proxy is responsible for implementing Kubernetes Services by programming the node's kernel networking layer. When a Service is created, the API server allocates a ClusterIP from the service CIDR. kube-proxy watches Service and EndpointSlice objects and translates them into kernel networking rules so that packets destined for the ClusterIP are forwarded to one of the healthy pod IPs. In iptables mode, kube-proxy creates chains in the NAT table. For each Service there is a KUBE-SVC chain. For each endpoint there is a KUBE-SEP chain that performs DNAT to the pod IP. The KUBE-SVC chain uses statistic match rules to distribute traffic randomly across the KUBE-SEP chains. In IPVS mode, kube-proxy creates virtual servers in the kernel's IPVS subsystem. Each Service ClusterIP becomes an IPVS virtual server and each endpoint becomes a real server entry. IPVS maintains connection state in a hash table, which scales much better than iptables as the number of Services grows because iptables rules are evaluated linearly while IPVS lookups are O(1). kube-proxy also writes iptables rules in IPVS mode to handle hairpin traffic and masquerading, so both subsystems are active simultaneously in IPVS mode. IPVS mode also supports additional load balancing algorithms beyond round-robin, including least connection and shortest expected delay. For clusters with fewer than a few hundred Services, the performance difference between the two modes is negligible. Beyond that scale, IPVS mode is strongly preferred because iptables rule processing time grows linearly with the number of rules while IPVS lookup time remains constant.

etcd Backup and Restore Procedure

etcd backup is one of the most critical operational procedures in a Kubernetes cluster. Because etcd holds all cluster state, a successful restore from backup is the recovery path after a catastrophic control plane failure. The standard tool for backups is etcdctl snapshot save, which takes a consistent point-in-time snapshot of the data directory. To restore, stop all API servers first because they cache state and writing new state while a restore is in progress causes inconsistency. Then restore each etcd member from the same snapshot using a unique data directory and unique member name per node. The --skip-hash-check flag is useful when restoring from a snapshot that was taken on a different cluster. Backups should be automated to run at least every 30 minutes in production and stored in object storage outside the cluster. The backup retention policy should account for recovery time objective requirements.

When to use it

• When preparing for a senior platform engineer or SRE interview where interviewers ask about internal Kubernetes mechanics rather than surface-level kubectl usage
• When debugging a cluster that will not start after a certificate rotation or etcd failure and you need to understand which component depends on which certificate
• When designing a high-availability control plane and you need to reason about etcd quorum requirements and API server load balancing
• When evaluating whether iptables or IPVS mode is appropriate for a cluster based on the number of Services and performance requirements
• When onboarding to a bare-metal Kubernetes deployment where no managed control plane exists and every component must be configured explicitly
• When writing a custom scheduler plugin or admission webhook and you need to understand how the scheduler framework and admission chain are structured
• When performing a security audit and you need to trace exactly which certificates authenticate which components and what RBAC permissions each control plane identity holds

Interview questions

• How do you recover a Kubernetes cluster after losing quorum in the etcd cluster?
• Walk through the complete path from kubectl apply to a running pod, naming every component and every API call involved.
• What is the difference between scheduler filter predicates and score priorities, and how do you write a custom plugin?
• Describe the kubelet reconciliation loop in detail. What happens when the desired state diverges from actual state?
• Compare kube-proxy iptables mode and IPVS mode. When does the performance difference matter and how do you switch between them?

More DevOps topics

• Kubernetes RBAC
• Kubernetes Runtime Classes
• Kubernetes Secrets Management
• Kubernetes Security: Pod Security and Runtime Hardening
• Kubernetes Services
• Kubernetes Storage
• Kubescape and Kubernetes Runtime Security
• Kustomization and Reconciliation
• Lease and Reservation Control Planes
• libstdc++, libc++, glibc and Symbol Versioning
• Lifecycle Hooks and Tests
• Linking Deep Dive