Bare-metal Provisioning & Fleet Firmware

Overview

Why this topic

A GPU cluster of a few hundred nodes is too large to install by hand and too expensive to install wrong. Every server must boot from the network, report what it is, receive the same image as its neighbors, and later take BIOS, BMC, NIC and GPU firmware updates without a technician at the rack.

The interview questions here are about control at a distance. How do you power cycle a hung node in another building, why does a node land in the wrong state, and how do you push a firmware update to 512 nodes without breaking all of them at once.

Interview questions

• A GPU node in another data center is hung and not answering SSH. Walk me through getting it back without a technician.
• Design how a new rack of 32 GPU servers goes from power-on to a ready Kubernetes node with no manual steps.
• How do you keep firmware consistent across a 1,000-node GPU fleet, and how do you know when it is not?
• Plan a GPU VBIOS and NIC firmware rollout to 512 production nodes. How do you stage it, and when do you stop?
• Compare MAAS, OpenStack Ironic and Tinkerbell. Which would you choose for an on-prem GPU cluster, and why?
• A node finishes deploy but PXE boots and reinstalls itself on every reboot. Another never gets past the iPXE prompt. Debug both.

More AI Infrastructure topics

• AI Accelerator Software Stacks: TPU, Trainium, MTIA & Bring-up
• Batch GPU Scheduling on Kubernetes: Kueue, Volcano & Gang Scheduling
• CUDA Kernel Profiling, Occupancy & the Roofline
• CUDA Memory Hierarchy & Access Patterns
• Disaggregated Prefill-Decode Architecture
• Distributed Training: NCCL, Collectives & Parallelism
• DPUs in GPU Clusters: BlueField & DOCA
• EDA Licenses & Compute Farms
• Emulation & FPGA Prototyping Pipelines
• Firmware CI with Hardware in the Loop
• GPU Architecture & AI Hardware
• GPU Cluster Fabrics: InfiniBand & RoCE