CUDA Kernel Profiling, Occupancy & the Roofline

Nsight Systems and Nsight Compute workflow, occupancy tuning, warp stall analysis, ILP, warp divergence, and CUDA Graphs. How to go from a slow kernel to a fast one systematically.

Overview

Measure before you optimize

Profiling is the empirical method of AI performance engineering; without measurement, optimization is guesswork. NVIDIA provides two complementary profilers, and knowing how to read and act on their output is an essential skill for any AI systems engineer.

Nsight Systems versus Nsight Compute

Scope — System-level timeline — Per-kernel microarchitecture
How it works — CPU and GPU activity side by side — Re-executes the target kernel with hardware performance counters on
Shows — Running kernels, CPU-GPU sync points, PCIe/NVLink transfers, NCCL collectives, NVTX ranges — Memory Workload, Compute Workload, Occupancy, Warp Stall analysis
Question it answers — "Why is the GPU idle?" — "What is this kernel waiting on?"

> [!NOTE] Gaps between GPU kernels in Nsight Systems point to CPU-side bottlenecks, inefficient launch patterns or blocking synchronizations.

Occupancy

Occupancy is active warps on an SM divided by the maximum an SM supports (64 on Hopper). It is not the goal; it is how the GPU hides memory latency. At 100% the scheduler always has a ready warp while others stall, and at 25% memory stalls leave execution units idle.

Warp stall analysis

Nsight Compute's Warp Stall Reasons shows what the scheduler was waiting on when it could not issue.

Long Scoreboard — Waiting on a global memory load — Tile into SMEM, prefetch, reduce global traffic
Short Scoreboard — Waiting on a shared memory load — Restructure access to avoid bank conflicts
Barrier — Threads waiting at __syncthreads — Fewer barriers, or overlap compute with synchronization
No Instruction — Warp has nothing to execute — Increase grid or block size

Instruction-level parallelism

ILP means several independent instructions in flight within one thread. It is the lever to pull when register pressure stops you adding more threads per SM.

Technique: unroll a loop body 2-4x so the compiler can interleave independent loads and FMAs.

CUDA Graphs

CUDA Graphs record kernel launches and memory operations once, then replay them with a single driver call.

CPU cost per kernel — ~5-20 microseconds — Amortized across the graph
Total for many small inference kernels — 1-5 ms per token — One driver call
Constraint — None — Static kernel parameters; variable sequence lengths need conditional graph nodes (Hopper+) or one graph per length bucket

> [!TIP] vLLM, TensorRT-LLM and SGLang capture CUDA Graphs for decode, where batch shapes are more predictable, and run prefill eagerly. This hybrid keeps ~80-90% of the replay benefit while staying flexible for prefill.

Interview questions

How would you systematically profile and optimize a slow GEMM kernel using Nsight tooling?
What is CUDA Graph pre-warming and why does it matter for LLM inference latency?