Rack-scale NVLink: Operating GB200/GB300 NVL72

Overview

Why this topic

An NVL72 rack is not 18 servers that happen to share a cabinet. Its 72 GPUs form one NVLink domain, so a job can treat GPU memory on another tray almost like memory next door. That changes operations: the network that matters most is now managed from switch trays inside the rack, a job needs a memory-sharing service (IMEX) set up for exactly its nodes, and one bad tray, cable cartridge or cooling loop affects the whole domain.

Interviewers for GPU platform roles ask how you bring such a rack up, how you schedule onto it, and what you do when part of the fabric fails mid-job.

Interview questions

• Walk through taking a new NVL72 rack from power-on to job-ready. What do you check at each step?
• How do you schedule Slurm jobs so they stay inside one NVLink domain?
• Run multi-node NVLink workloads on Kubernetes. How does the GPU stack create the IMEX domain?
• An NVLink switch tray fails during a long training job. What happens, and what do you do?
• Two teams must share one NVL72 rack without seeing each other's GPU memory. How do you split it?
• What are the failure domains of an NVL72 rack, and how do they change how you size jobs and spares?

More AI Infrastructure topics

• Mixture of Experts (MoE) Inference
• Multi-tenant GPU Clusters & Confidential Computing
• Post-silicon Validation & Bring-up Labs
• Power & Cooling for GPU Racks
• PyTorch torch.compile, FSDP2 & Training Optimization
• Quantization & Model Compression
• ROCm & the AMD GPU Stack
• Serving LLMs On-prem: Triton, NIM, vLLM & the Model Store
• Silicon Root of Trust & Key Provisioning in Manufacturing
• Slurm for GPU Clusters
• Speculative Decoding & Parallel Token Generation
• Storage for GPU Training: Parallel Filesystems & GPUDirect Storage