Platform Provisioning on Bare Metal: From Silicon to Kubernetes

Racks of servers to healthy Kubernetes nodes, at fleet scale

Design the system that takes thousands of CPU and GPU servers across your own sites from powered-off silicon to healthy Kubernetes nodes, keeps their firmware and OS current in safe waves, and replaces failed hardware automatically.

Overview

You run your own data centers. Trucks deliver racks of servers: most are general CPU nodes, some are GPU nodes with eight accelerators each. Your company also runs Kubernetes on AWS, Azure and Google Cloud, and the platform team wants every cluster, in the cloud or on your floor, to be created and upgraded the same way. Your job is the part the hyperscalers normally hide: the system that takes a machine from powered-off silicon to a healthy Kubernetes node without a person at a keyboard, keeps its firmware and operating system current, and replaces it when it breaks.

This is a fleet design question, not a how-to. One machine can be installed by hand from a USB stick. The interviewer wants to hear how thousands are handled: where the truth about each machine lives, how a machine moves through its lifecycle, how you know the hardware is good and has not been tampered with, how you change firmware on 12,000 servers without an outage, and what happens to the provisioning system when it fails.

Two other topics cover what this one does not repeat. The Bare-Metal Provisioning topic explains the boot chain itself (DHCP options, iPXE chainloading, UEFI HTTP boot) and the tools' state machines one by one. Deploying Kubernetes on Bare Metal: End to End builds a single cluster by hand. Here those are building blocks.

Questions to ask first

How big is the fleet, and where? — 12,000 servers in 6 sites, about 2,000 per site, growing 25% a year
What kinds of hardware? — 80% CPU nodes, 20% GPU nodes with 8 GPUs each; two to three vendors and a few models per generation
What runs on them? — Kubernetes only, many clusters per site, created with Cluster API like the cloud clusters
Who racks and cables? — A data center team or the vendor; they scan serials and record rack positions, then leave
Do BMCs speak Redfish? — Current generations yes; some older machines only IPMI
Security needs? — Hardware must prove its firmware and boot chain before it gets cluster credentials
How fast must a failed node be replaced? — Capacity back within 15 minutes from spares (a target we chose); the broken part fixed within days

The same building blocks on each platform

The point of the hybrid platform is that clusters look alike everywhere. The table shows who does each job on each substrate; on bare metal every row is yours.

Rack, test and repair hardware — AWS (instance retirement on failed hardware) — Azure (maintenance, live migration) — Google — You: intake, burn-in, RMA
Power and boot control — EC2 API — Azure Resource Manager — Compute Engine API — BMC through Redfish or IPMI
Inventory — EC2 describe calls — Azure Resource Graph — Cloud Asset Inventory — NetBox plus inspection data
Get a machine — RunInstances, Auto Scaling — Virtual Machine Scale Sets — Managed instance groups — Ironic (via Metal3), Tinkerbell or MAAS
OS image — AMI — Compute Gallery image — Compute Engine image — Talos or Flatcar image, pinned by digest
Cluster lifecycle — Cluster API AWS provider, or EKS — Cluster API Azure provider, or AKS — Cluster API GCP provider, or GKE — Cluster API with the Metal3 or Tinkerbell provider
Boot integrity — NitroTPM measured boot — Trusted Launch — Shielded VM — UEFI secure boot, TPM 2.0, Keylime
Node repair — Replace the instance — AKS node auto-repair — GKE node auto-repair — MachineHealthCheck plus Metal3 remediation

The interviewer listens for four things. Do you separate intended state (what you bought and where it should be) from discovered state (what inspection found)? Can you draw the lifecycle of one machine as a state machine with an owner for every transition? Do you treat hardware as untrusted until it passes burn-in and attestation? And do you roll every change, firmware above all, in waves with a gate that can stop it?

What you will be able to do

Take a delivered rack to Ready Kubernetes nodes with no hands on keyboards, and name the component that owns each step.
Keep a source of truth and a discovered inventory, and use the difference between them to catch wrong cabling, wrong parts and swapped servers.
Gate a node's entry into a cluster on burn-in results and TPM attestation.
Plan a firmware campaign for 12,000 servers in waves, with limits per cluster and rack, and compute how long it takes.
Detect, drain and replace failed hardware automatically, without the remediation itself causing an outage, and size the spare pool.

Key concepts

Out-of-band management (BMC, Redfish)
Source of truth vs discovered inventory
Machine lifecycle state machine
Zero-touch provisioning
Burn-in and acceptance
Immutable OS images
Cluster API on bare metal
Measured boot and attestation
Firmware waves
Remediation ladder and RMA

Functional requirements

Accept a delivered rack from its purchase record and rack plan, and enroll each server by its BMC with no manual typing of MAC addresses or disk names
Inspect every server and compare what was found with what was ordered and where it should be
Bring firmware and BIOS settings to the approved baseline for the hardware model, then run burn-in tests
Install a pinned, signed OS image and join the node to the Kubernetes cluster that claimed it
Refuse cluster credentials to any node whose measured boot does not match an approved state
Roll firmware, BIOS, NIC and OS changes across the fleet in waves, with automatic halt on failure
Detect unhealthy nodes, drain them, try the cheapest repair first and send failed parts to RMA
Securely erase disks before a server changes owner, goes to RMA or is retired

Non-functional requirements

Zero-touch: a correctly racked and cabled server needs no further human action to become a node (a goal we chose)
Reproducible: the same machine reprovisioned months later gets the same image bytes and firmware baseline
Site independence: a site keeps provisioning and remediating when the central services or the WAN are down
Safe change: no change takes more than 5% of a cluster, or 2 servers of one rack, out at once (limits we chose)
Remediation never makes things worse: a mass failure stops automatic repair and pages a person
Auditable: every lifecycle transition records who or what caused it and why
Replacement capacity from spares within 15 minutes of a node being declared failed (a target we chose)

Interview questions

Walk me through a new rack becoming Kubernetes nodes.
IPMI or Redfish, and why does it matter at fleet scale?
How do you update BIOS and NIC firmware on 12,000 servers safely?
A node goes NotReady at 3 a.m. What happens, step by step?
How do you know a server has not been tampered with?
Which OS do you run on the nodes, and why not a normal distribution?
What happens when the provisioning system itself is down?
How is this different from just using cloud instances?
How do you size spares and plan capacity?