Bare-Metal Provisioning

How a machine with no operating system becomes a cluster node: the network boot chain, out-of-band management, discovery and inspection, imaging, and the tool landscape from Ironic and MAAS to Tinkerbell and Metal3.

Overview

Why bare metal is your problem

Every cloud abstraction rests on somebody provisioning bare metal. If you run GPUs, storage, telco edge, or anything where the hypervisor tax or the neighbor noise is unacceptable, that somebody is you.

The problem statement is narrow and unforgiving: turn a machine with no operating system, no agent, and no credentials into a trusted, inventoried, reimageable cluster node, remotely, at rack scale, repeatably.

Two channels into a machine that has not booted

You have no in-band access to a machine that has not booted. Everything begins with two channels, and provisioning is the choreography of both.

Network boot path — Lets an unprovisioned machine pull code over the wire
Baseboard management controller — Controls power and boot device while the machine is off

> [!WARNING] Get the sequencing wrong and you get the classic symptoms: a node that reinstalls itself in a loop, or a node that boots from an empty disk and sits at a firmware prompt forever.

The tooling landscape

The landscape is smaller than it looks because everything descends from the same model.

OpenStack Ironic — The mature engine — ironic-api; ironic-conductor processes owning nodes via consistent hashing; hardware types composing power, management, boot, deploy, inspect and RAID interfaces; Ironic Python Agent ramdisk for in-band work
Canonical MAAS — Batteries-included appliance — Region and rack controllers running DHCP, TFTP, HTTP and DNS; machines move New, Commissioning, Ready, Deployed; curtin installs, cloud-init configures first boot
Tinkerbell — Kubernetes-native decomposition — Smee for DHCP and iPXE, Tootles for metadata, Rufio driving BMCs over IPMI and Redfish, Tink server and agent running Workflows from Templates against Hardware records, all in HookOS
Metal3 — The bridge — BareMetalHost CRD backed by standalone Ironic, plus the Cluster API provider so scaling a MachineDeployment provisions physical servers like EC2 instances

Firmware is configuration

BIOS and UEFI settings are configuration: they drift, and they are load-bearing. Two machines identical on the purchase order and different in one BIOS toggle will behave differently under load, and nothing in your Kubernetes stack will tell you which toggle.

SR-IOV — Must be enabled for virtual functions
Above 4G Decoding — Must be on or large-BAR cards will not enumerate
IOMMU (VT-d on Intel, AMD-Vi on AMD) — Governs whether passthrough works at all

> [!TIP] Declaring settings through Redfish attribute registries and vendor tooling, and enforcing them on every provision, is the only way this stays sane past a few dozen machines.

Failures that do not exist in cloud

In cloud these are somebody else's SLO. On bare metal they are your pager.

Node will not POST — Nothing happens after power-on
Marginal DIMM — Passes memtest for six hours, throws correctable ECC errors under production load
Renamed NIC — Different predictable name after a firmware update silently breaks network config
Fan curve — GPU thermally throttles at 90% utilization, benchmark 15% slow with no profiler explanation
Moved disk — Serial moved from sda to sdc after a controller reset, image lands on the wrong device

Where interviewers push

Boot sequence — Exact DHCP and boot steps, including why iPXE re-DHCPs
Channels — In-band versus out-of-band, and what you can still do when one is gone
Inspection — Discovered inventory versus a CMDB
Reprovisioning — Idempotent, including secure erase between tenants
Firmware — How it becomes version-controlled configuration
Capacity — "Can you scale out?" starts with a procurement lead time

Key concepts

PXE, TFTP, and DHCP options 66 and 67: The legacy network boot chain. The NIC firmware DHCPs with a client architecture identifier in option 93; the server returns option 66 (boot server) and option 67 (bootfile name); the client fetches that file over TFTP. TFTP is UDP, unauthenticated, and small-block, so it does not scale to a rack booting at once — it exists only to deliver the next stage.
iPXE chainloading: A second-stage bootloader delivered over TFTP that replaces the NIC firmware network stack and adds HTTP, HTTPS, iSCSI, and scripting. iPXE re-issues DHCP with user-class "iPXE" so the server can branch and return an HTTP script URL instead of the binary again. Missing that conditional produces the classic infinite chainload loop.
UEFI HTTP Boot: The modern path. The client advertises DHCP vendor class "HTTPClient" in option 60 and receives a full URL in option 67, downloading the boot image over HTTP or HTTPS directly from firmware. No TFTP server and no chainload stage. Requires current UEFI firmware, which is why mixed-vintage fleets still carry the PXE path alongside it.
Out-of-band management (IPMI and Redfish): A separate processor on the board, reachable on its own network, that works when the host is powered off. IPMI over LAN uses RMCP+ on UDP 623 for power, boot device, and Serial-over-LAN. Redfish is the HTTPS REST successor: /redfish/v1/Systems for power and boot override, /redfish/v1/Managers for the BMC, /redfish/v1/Chassis for sensors, plus VirtualMedia for booting an attached ISO.
Discovery and inspection ramdisk: A minimal in-memory OS network-booted specifically to enumerate the machine from the inside — the Ironic Python Agent under Ironic and Metal3, HookOS under Tinkerbell. It reports CPU, memory, block devices with serials, NICs with MACs and LLDP neighbours, PCI and GPU inventory, and firmware versions. This is discovered truth and it supersedes any spreadsheet.
Full-disk image versus scripted install: Image deploy writes a prepared raw or qcow2 image to the target disk and grows the filesystem — fast, identical everywhere, no repository dependency at install time. Kickstart, preseed, and autoinstall run the distribution installer against an answer file — slower and less reproducible but able to vary per machine. Most fleets use an image plus cloud-init for identity.
Provisioning state machine: The shared abstraction across tools. Ironic: enroll, manageable, available, active, error with verifying, inspecting, cleaning, deploying, deleting, servicing between them. Metal3 BareMetalHost: registering, inspecting, preparing, available, provisioning, provisioned, deprovisioning, deleting. MAAS: New, Commissioning, Ready, Allocated, Deploying, Deployed, Releasing, Broken.
Cleaning and secure erase: The step between one tenant and the next. Automated cleaning runs on the way back to available and can be configured with metadata erase (fast, wipes partition tables and metadata) or a full overwrite or ATA/NVMe secure erase. Skipping it leaves the previous tenant data readable by the next occupant, which is the bare-metal equivalent of not deleting an EBS volume.

When to use it

GPU and accelerator fleets, where passthrough, NVLink topology, and thermal behaviour make the hypervisor layer either impossible or expensive
Latency-sensitive or throughput-bound workloads — trading, packet processing, NVMe storage — where virtualization overhead and noisy neighbours are the dominant cost
Regulated or sovereign environments that require physically isolated, auditable hardware rather than a shared tenancy attestation
On-premise and colocation Kubernetes, where the cluster autoscaler has no cloud API to call and nodes must be provisioned from racked inventory
Telco and edge sites with thousands of small, remote, physically inaccessible machines that must be provisioned and reimaged entirely over the BMC

Interview questions

Trace the full boot chain from power-on to a running installer, naming every protocol and why each step exists.
Compare IPMI and Redfish, and explain what you can and cannot do when only one channel is available.
What does discovery and inspection actually collect, and why do you insist on it rather than a hardware inventory database?
How do you treat firmware and BIOS settings as version-controlled configuration, and which settings matter for GPU nodes?
What failure modes exist on bare metal that simply do not exist in cloud, and how do you design for them?
How does bare metal change capacity planning, and how do you present that to people used to cloud elasticity?