Power & Cooling for GPU Racks
Power and cooling for on-prem GPU racks: per-node draw and rack density, 3-phase PDUs, air versus liquid cooling, nvidia-smi power caps and throttle reasons, PUE, and the failures that take racks down.
Overview
Why power is the real capacity limit
On a GPU cluster, the number of servers you can run is usually set by power and cooling, not floor space. A rack that held 5 to 15 kW of CPU servers now has to hold one or a few 8-GPU nodes that each draw about 10 kW (DGX H100) to 14 kW (DGX B200) at full load. Every watt delivered to a GPU leaves as heat that the room or a liquid loop has to remove.
- B200 (HGX B200) — Up to 1,000 W per GPU — DGX B200 is air cooled; GB200 racks are liquid cooled
- H100 SXM, H200 SXM — Up to 700 W per GPU — Configurable lower with a power limit
- H100 PCIe — Up to 350 W — Air-cooled card form factor
- A100 SXM — 400 W — The previous generation
- 8-GPU H100 node (DGX H100) — About 10.2 kW maximum — GPUs, CPUs, NVSwitch, NICs, fans, drives
- 8-GPU B200 node (DGX B200) — About 14.3 kW maximum — Six 3.3 kW power supplies
- GB200 NVL72 rack — About 120 to 132 kW — Vendor rack specs; liquid cooled, 72 GPUs in one rack
- Classic enterprise rack — 5 to 15 kW — Air cooled
- Dense GPU rack — 40 kW and up — Usually needs liquid assistance
> [!IMPORTANT] Budget with the vendor's maximum system power, not a typical draw. A training job runs every GPU near its limit for weeks.
Interview questions
- Plan the power for one rack of 8-GPU H100 servers on 415 V three-phase. How many servers fit?
- Training throughput dropped 15% on some nodes. nvidia-smi shows clocks below normal. How do you find out why?
- Compare air cooling, rear-door heat exchangers and direct liquid cooling for a new GPU hall.
- A coolant leak alarm fires in a liquid-cooled GPU row at 3 a.m. What should happen, automatically and by hand?
- How would you add power awareness to scheduling on a cluster whose racks are near their power limit?
- A rack PDU breaker tripped on feed A. Nodes stayed up. What do you do next?
Sign up free to read the full topic
More AI Infrastructure topics
- GPUs on Kubernetes: GPU Operator, Device Plugin, MIG & Time-slicing
- LLM Inference: KV Cache, Batching & Scheduling
- Manufacturing Test Data & Yield Pipelines
- Mixture of Experts (MoE) Inference
- Multi-tenant GPU Clusters & Confidential Computing
- Post-silicon Validation & Bring-up Labs
- PyTorch torch.compile, FSDP2 & Training Optimization
- Quantization & Model Compression
- Rack-scale NVLink: Operating GB200/GB300 NVL72
- ROCm & the AMD GPU Stack
- Serving LLMs On-prem: Triton, NIM, vLLM & the Model Store
- Silicon Root of Trust & Key Provisioning in Manufacturing
Start free on Cariara · Coding problems · System design studio