PyTorch torch.compile, FSDP2 & Training Optimization

torch.compile internals (TorchDynamo, TorchInductor), FSDP2 sharding strategies, activation checkpointing, mixed-precision training, and the PyTorch profiler workflow for training throughput optimization.

Overview

PyTorch is the dominant framework for large-model training in 2025-2026. High throughput depends on three layers: the compiler stack (torch.compile), memory management (FSDP2, activation checkpointing) and numerical precision (mixed precision, loss scaling). Together they decide whether a job reaches 60% or 40% of theoretical hardware peak.

torch.compile

torch.compile (PyTorch 2.0+) is a JIT compiler that turns Python-level PyTorch code into optimized native code through three stages.

TorchDynamo — Captures the graph of tensor ops at the Python bytecode level — Untraceable Python (data-dependent if statements, print calls, external libraries) causes a graph break and a fallback to Python; minimizing graph breaks is critical
AOT Autograd — Turns the forward graph into a joint forward-backward graph — Lets the backend fuse across forward and backward, impossible in eager mode
TorchInductor — Default backend code generator — Loop tiling, pointwise fusion, persistent reductions; autotunes kernel parameters with the Triton autotuner

> [!NOTE] On LLM training, torch.compile gives a 1.2-1.5x speedup over eager mode by fusing RMSNorm, RoPE and residual additions, cutting memory bandwidth and removing Python interpreter overhead.

Activation checkpointing

Backprop keeps all forward activations to compute gradients, so activation memory grows as O(L * B * S * H). For LLaMA-3 70B with L=80, B=4, S=4096, H=8192 that is ~21 GB before attention maps.

No checkpointing — O(L * B * S * H), all activations kept — Baseline
Full checkpointing — ~10-15x reduction; O(sqrt(L)) checkpoints for O(sqrt(L)) memory — 30-35% extra compute
Selective checkpointing — Most of the memory benefit (checkpoints only the heaviest ops, such as attention) — Less overhead than full

Mixed precision training

BF16 mixed precision (standard) — BF16 on tensor cores — BF16 activations and gradients — FP32 master weights, momentum and variance; update applied in FP32, then cast to BF16 — Standard for LLM training
FP8 via Transformer Engine — FP8 in forward and backward — BF16 residuals and normalizations — Same as BF16 — ~25% less memory than BF16, 2x tensor core throughput for GEMMs; accuracy held by per-tensor history-based scaling (amax over a sliding window)

FSDP2 with async prefetch

FSDP2's main gain over FSDP1 is async layer prefetching on a separate NCCL stream, so communication hides behind compute.

FULL_SHARD (ZeRO-3) — Parameters, gradients, optimizer state — Minimum — Maximum — Largest memory savings
SHARD_GRAD_OP (ZeRO-2) — Gradients, optimizer state — Higher — Less — Lower communication
HYBRID_SHARD — ZeRO-3 within a node, full replica across nodes — More intra-node memory — Less inter-node — Multi-node jobs bottlenecked on InfiniBand

PyTorch Profiler workflow

The profiler captures CPU and GPU activity at kernel granularity, and the Chrome trace shows CUDA kernels, their durations and CPU-GPU synchronization.

GPU idle gaps — Launch overhead or CPU bottleneck — Find the CPU-side stall
Very short kernels — Poor batching — Consider fusing
Long runs of non-fused kernels — Missed fusion — Compile opportunity

> [!TIP] Use the profiler schedule wait/warmup/active: skip N steps (wait), warm up for M steps, then capture K active steps, for a clean steady-state trace without first-step JIT noise.

Interview questions

How would you use torch.compile to speed up LLM training and what are the failure modes to watch for?
What is the complete optimization checklist for maximum GPU MFU on LLaMA-3 70B training?