GPU Architecture & AI Hardware

NVIDIA GPU architecture from streaming multiprocessors through tensor cores, HBM memory, NVLink, and Blackwell-class superchips. Hardware-software codesign and "mechanical sympathy" for AI workloads.

Overview

Throughput, not latency

Understanding GPU architecture at the hardware level is the prerequisite for every optimization decision in AI systems. CPUs are built for low-latency serial execution. GPUs are throughput processors: they give up single-thread performance to run thousands of threads at once and hide memory latency through massive parallelism.

Streaming multiprocessors and the execution hierarchy

A GPU is organized into Streaming Multiprocessors (SMs). The execution hierarchy maps directly onto that hardware.

CUDA cores — FP32 math
Tensor cores — Mixed-precision matrix math
Load/store units — Memory access
Shared memory / L1 block — On-chip data shared by a thread block

Grid to block: a CUDA grid contains thread blocks, and each thread block executes on exactly one SM.
Block to warp: threads in a block execute in groups of 32 called warps, the hardware unit of execution.
SIMT: all 32 threads in a warp execute the same instruction in lockstep.

> [!WARNING] Warp divergence, when threads in the same warp take different branches, serializes those paths and reduces effective throughput.

Tensor cores

Tensor cores are the most important hardware unit for AI workloads. They perform fused multiply-add (FMA) on matrix tiles, D = A * B + C; on Hopper H100 a tensor core performs a 16x16 matrix FMA in BF16 in a single clock cycle. This is the engine behind GEMM and attention.

The programmer or the compiler must tile the problem to match the hardware tiles. Operations that cannot be expressed as tensor-core GEMM fall back to CUDA cores and run dramatically slower.

> [!WARNING] Operations that miss tensor cores run at CUDA core throughput, roughly 30x slower than BF16 tensor core peak on H100. Verify tensor core utilization in Nsight Compute before declaring a GEMM optimized.

Memory hierarchy

GPU memory has several levels, each trading capacity for bandwidth and latency.

Registers — Per thread — Typically 255 per thread, shared across the SM — 1 cycle
Shared memory (L1) — Per thread block — 32-256 KB per SM — ~10-20 cycles; the key resource for tiling
L2 cache — All SMs — 50 MB — 100-200 cycles
HBM — Whole GPU — 80 GB (H100 SXM5), 3.35 TB/s peak — 500-800 cycles (~500 ns)

HBM bandwidth is the bottleneck for memory-bound kernels like attention decode.

> [!TIP] The latency hierarchy spans 4 orders of magnitude. Every kernel optimization is fundamentally about moving data closer to registers before operating on it.

NVIDIA Blackwell

Blackwell (2024-2025) introduces several advances critical for large model inference.

System — 36 Grace CPUs and 72 Blackwell GPUs over NVLink 5.0
GPU-to-GPU bandwidth — 1.8 TB/s
CPU-to-GPU bandwidth — 900 GB/s
Memory per GPU — 192 GB HBM3e at 8 TB/s
Tensor cores — 4th generation, with FP4 support
Transformer Engine — Second generation

> [!NOTE] The key inference innovation: KV cache can live on different chips and stream over NVLink at full bandwidth, enabling KV cache disaggregation across a 72-GPU island without going through the CPU.

NVLink and multi-GPU topology

NVLink is NVIDIA's proprietary GPU-to-GPU interconnect. NVSwitch builds an all-to-all NVLink fabric within a node, removing the bandwidth reduction of tree topologies.

NVLink 4.0 (Hopper), between paired GPUs — 900 GB/s bidirectional
PCIe 4.0 — ~64 GB/s

The gap between intra-node NVLink and inter-node InfiniBand is why tensor parallelism is preferred within a node and pipeline or data parallelism across nodes.

Goodput versus throughput

Goodput is the fraction of theoretical hardware throughput doing useful work. An H100 might offer 3.35 TB/s HBM bandwidth and 1979 TFLOPS BF16 tensor performance; goodput measures how much of those peaks your workload actually reaches. The roofline model diagnoses whether a kernel is compute-bound or memory-bound, and maximizing goodput is the central objective of all kernel optimization work.

MIG and MPS

How it shares — Partitions a GPU into up to 7 isolated instances — Multiple CUDA processes share the GPU without spatial partitioning
Dedicated resources — SM slices, L2 cache slices, HBM slices — None
Isolation — Hardware isolation — Trusted workloads
Best for — Multi-tenant serving with strict SLA and QoS guarantees — Higher occupancy and utilization when batches are small

Interview questions

Explain the GPU execution hierarchy from grid to thread and how it maps to hardware.
What is the roofline model and how do you use it to diagnose a slow kernel?
Compare NVLink, InfiniBand, and PCIe for multi-GPU AI systems.