Quantization & Model Compression

Overview

Quantization lowers the numerical precision of model weights (and optionally activations) from 32-bit or 16-bit floating point to fewer bits. It saves memory, bandwidth and compute; the risk is quantization error that compounds through the layers and costs accuracy.

Data types

• FP32 — 8, 23 — Standard full precision — Optimizer states and loss scaling; not used for inference
• BF16 — 8, 7 — FP32's dynamic range, lower precision than FP16; robust to overflow; H100 tensor cores 1979 TFLOPS — Standard LLM training and inference precision
• FP16 — 5, 10 — More precision than BF16, narrower range (max ~65504); needs loss scaling in training — Older models
• FP8 e4m3 — 4, 3 — Hopper computes FP8 at 2x BF16 throughput — Weights and activations in the forward pass; production standard for H100 inference in 2025-2026
• FP8 e5m2 — 5, 2 — Wider range — Gradients
• INT8 — 8-bit signed integer — Needs a per-tensor or per-channel scale — W8A16 or W8A8; SmoothQuant is the standard W8A8 technique
• INT4 / FP4 — 4-bit — Blackwell runs FP4 at 2x INT8 throughput — Weight-only W4A16 with FP16 activations; GPTQ and AWQ are the standard PTQ algorithms

Post-training quantization (PTQ)

PTQ quantizes a finished model using a small calibration set (100-512 samples) to set quantization parameters, with no retraining.

• GPTQ (Frantar et al., 2022) — INT4 weights — Layer by layer, minimizes the L2 reconstruction error (W * X - W_q * X)^2 using second-order Hessian information estimated from calibration data; quantizes column by column and updates unquantized weights to compensate. Perplexity within 0.5-1.0 of FP16 for LLaMA-3 70B, in about hours of CPU time
• AWQ (Lin et al., 2023) — INT4 weights — Weight channels fed by high-activation inputs matter most. Searches a per-channel scale that scales these salient channels up before quantization and the preceding activation down by the same factor, so the math is unchanged
• SmoothQuant (Xiao et al., 2022) — W8A8 — Activations have large outliers that are hard to quantize. Scales activations by 1/s and weights by s per channel so both become easy to quantize to INT8

NVIDIA Transformer Engine

Transformer Engine (TE) is NVIDIA's library for FP8 training and inference on Hopper and Blackwell.

• Per-layer precision: picks FP8 or BF16 for each layer based on the operation type.
• Calibration: per-tensor, history-based amax tracking.
• Inference split: FP8 for GEMMs (doubling tensor core throughput), BF16 for residuals and activations where precision matters more.

> [!NOTE] Production TE + FP8 deployments reach 1.7-1.9x the throughput of BF16 on H100.

Structured sparsity (2:4)

A100 and H100 support 2:4 structured sparsity: in every 4 consecutive weights, at most 2 are non-zero.

• 2:4 sparse vs dense GEMM — 2x
• 2:4 sparse + FP8 vs FP16 dense — 4x

> [!WARNING] 2:4 sparsity needs sparse-aware fine-tuning, or SparseGPT for post-training application.

Interview questions

• Compare GPTQ and AWQ for deploying a LLaMA-3 70B model in INT4. Which would you choose and why?

More AI Infrastructure topics

• Disaggregated Prefill-Decode Architecture
• Distributed Training: NCCL, Collectives & Parallelism
• GPU Architecture & AI Hardware
• LLM Inference: KV Cache, Batching & Scheduling
• Mixture of Experts (MoE) Inference
• PyTorch torch.compile, FSDP2 & Training Optimization
• ROCm & the AMD GPU Stack
• Speculative Decoding & Parallel Token Generation