GPU Cluster Fabrics: InfiniBand & RoCE
How on-prem GPU clusters move gradients between nodes: rail-optimized fat trees, InfiniBand and lossless RoCEv2, GPUDirect RDMA, NCCL settings, and how to find the one bad link behind a slow all-reduce.
Overview
Why the fabric decides training speed
Inside one 8-GPU server, NVLink moves data between GPUs at hundreds of GB/s. The moment a job spans two servers, every gradient all-reduce also crosses the scale-out fabric: one NIC per GPU, cables, leaf and spine switches. That path carries about a ninth of NVLink's bandwidth (a 400 Gb/s NIC moves 50 GB/s each way; NVLink 4 on an H100 or H200 moves 450 GB/s each way, 900 GB/s in total, and NVLink 5 on Blackwell doubles that), so it sets the pace of every step.
Two technologies carry this traffic on-prem.
- InfiniBand (IB) — Native IB verbs, credit-based link flow control — Design: a sender never transmits without receiver credits — A subnet manager (OpenSM, or UFM which runs one)
- RoCEv2 — RDMA verbs inside UDP/IP (UDP port 4791) over Ethernet — Configuration: PFC + ECN + DCQCN on every NIC and switch — Your normal Ethernet tooling, plus QoS settings
Both expose the same RDMA verbs API, so NCCL uses the same NET/IB transport for each. The difference is who keeps the network from dropping packets.
Interview questions
- Design the fabric for a 32-node H100 training cluster. What topology, speeds and switch counts do you pick, and why?
- A multi-node training job's all-reduce is three times slower than last week. Walk me through finding the cause.
- How do you make RoCEv2 lossless, and what goes wrong when you get it half right?
- Explain GPUDirect RDMA. How do you prove it is working, and what breaks it?
- What does SHARP do, and when would you turn it on?
- Your training job hangs at NCCL init on a new cluster. What do you check?
Sign up free to read the full topic
More AI Infrastructure topics
- Distributed Training: NCCL, Collectives & Parallelism
- DPUs in GPU Clusters: BlueField & DOCA
- EDA Licenses & Compute Farms
- Emulation & FPGA Prototyping Pipelines
- Firmware CI with Hardware in the Loop
- GPU Architecture & AI Hardware
- GPU Driver & Software Stack Release Validation
- GPU Fleet Health: XID Errors, ECC & DCGM
- GPU Node Bring-up & Burn-in
- GPU Quotas, Utilization & Chargeback On-prem
- GPU Virtualization: Passthrough, vGPU, SR-IOV & MIG in VMs
- GPUs on Kubernetes: GPU Operator, Device Plugin, MIG & Time-slicing
Start free on Cariara · Coding problems · System design studio