Distributed Training: NCCL, Collectives & Parallelism

AllReduce, AllGather, ReduceScatter, and Broadcast collective operations. NCCL topology awareness, ring vs tree algorithms. Overlapping computation with communication. Data, tensor, pipeline, and sequence parallelism strategies.

Overview

Why communication decides scaling

Training large models across hundreds or thousands of GPUs means coordinating gradient exchange, weight distribution and activation movement. The volume and latency of collective operations decide how well training scales. NCCL (NVIDIA Collective Communications Library) is the standard collective library for GPU clusters.

Collective operations

A collective involves all GPUs (or a subset) in a communicator. Four matter most for LLM training.

AllReduce — A tensor — The sum of all tensors — Aggregating data-parallel gradients
ReduceScatter — A tensor of size P — A different 1/N shard of the sum — FSDP gradient reduction in the backward pass
AllGather — A shard of size P/N — The full tensor of size P — FSDP weight gathering before forward; tensor-parallel activations
Broadcast — One GPU sends its tensor — A copy of that tensor — Initializing weights from a checkpoint

> [!NOTE] AllReduce = ReduceScatter followed by AllGather. For P parameters, a ring AllReduce sends 2*(N-1)/N * P per GPU, and the total volume is 2P * N_gpu floats.

Topology awareness and algorithms

NCCL detects the interconnect topology from NVLink, PCIe and InfiniBand bandwidth measurements, then picks an algorithm.

Ring — Uniform bandwidth — Each GPU sends to one neighbor, forwarding chunks until all data is distributed
Tree — Large communicator groups where ring bandwidth is limited by link saturation — Tree-shaped reduction
Flat all-reduce — Inside a DGX node with NVSwitch all-to-all connectivity — Reaches near-peak NVLink bandwidth

Overlapping computation with communication

The key optimization in data-parallel training is hiding gradient communication behind backward compute.

DDP — Local — AllReduce for a layer starts as soon as its backward finishes, while earlier layers keep computing
FSDP2 (PyTorch 2.1+) — AllGather weight shards before each layer, prefetching the next layer's AllGather during current compute — ReduceScatter gradients before moving to the previous layer

> [!TIP] Nsight Systems shows the NCCL stream alongside the compute stream. NCCL idle while compute runs means under-used bandwidth, and the reverse means compute is waiting on communication.

Parallelism strategies

Data (DP) — The batch; full model replicated — AllReduce of gradients — Only models that fit on one GPU
Tensor (TP) — Individual weight matrices by column or row (Megatron-LM); for Y = XW, W split column-wise — AllReduce of partial results — Bandwidth-intensive; practical within a node over NVLink, not across nodes
Pipeline (PP) — Layers into stages on different GPUs — Point-to-point activations between adjacent stages — Pipeline bubbles; mitigated by the 1F1B interleaved schedule
Sequence (SP) — The sequence dimension for attention and normalization — Combined with TP — For very long sequences
Expert (EP) — MoE experts across GPUs — All-to-all token routing — For Mixture-of-Experts models

SHARP and in-network reduction

NVIDIA SHARP offloads AllReduce computation to the InfiniBand switch fabric, doing partial reductions at switch aggregation points.

Interview questions

Compare FSDP and DDP for distributed training. When would you choose each?
How do you overlap communication with computation in distributed training?