GPU Node Bring-up & Burn-in
Taking an 8-GPU HGX node from the loading dock to the scheduler: firmware settings, PCIe and NVLink checks, DCGM diagnostics, nccl-tests burn-in, admission gates and the RMA path.
Overview
Why this topic
A new 8-GPU HGX node is the most expensive server in the data center and the one most likely to arrive with a fault. A loose riser, a GPU trained at x8 instead of x16, a dead NVLink, a Fabric Manager that does not match the driver: each one passes a power-on test and then slows or hangs every training job that lands on it.
Bring-up is the process that proves a node is healthy before the scheduler can see it. Burn-in is the part of that process that loads the hardware hard enough, for long enough, that marginal parts fail on your bench instead of in week three of a training run.
> [!IMPORTANT] A single slow GPU makes a synchronous training job run at that GPU's speed. One bad node in a 512-GPU job costs the whole job, so admission gates exist to protect the fleet, not the node.
Interview questions
- Walk me through bringing up a new 8-GPU HGX node, from power-on to the scheduler.
- One GPU on a new node shows Width x8. How do you diagnose it, and what do you check before blaming the GPU?
- CUDA fails on a new HGX node with error 802, but nvidia-smi lists all 8 GPUs. What is wrong?
- How do you use nccl-tests to admit a node, and what numbers do you gate on?
- Design the burn-in and admission policy for a fleet of hundreds of GPU nodes.
- A GPU fails burn-in. Walk me through triage and the RMA process.
Sign up free to read the full topic
More AI Infrastructure topics
- Emulation & FPGA Prototyping Pipelines
- Firmware CI with Hardware in the Loop
- GPU Architecture & AI Hardware
- GPU Cluster Fabrics: InfiniBand & RoCE
- GPU Driver & Software Stack Release Validation
- GPU Fleet Health: XID Errors, ECC & DCGM
- GPU Quotas, Utilization & Chargeback On-prem
- GPU Virtualization: Passthrough, vGPU, SR-IOV & MIG in VMs
- GPUs on Kubernetes: GPU Operator, Device Plugin, MIG & Time-slicing
- LLM Inference: KV Cache, Batching & Scheduling
- Manufacturing Test Data & Yield Pipelines
- Mixture of Experts (MoE) Inference
Start free on Cariara · Coding problems · System design studio