Disaggregated Prefill-Decode Architecture

Separating compute-bound prefill and memory-bound decode onto independent GPU pools for independent scaling. KV cache transfer mechanisms, routing, SLO-aware scheduling, and disaggregated KV cache pools.

Overview

Disaggregated prefill-decode (PD disaggregation) resolves the core tension in LLM inference: prefill and decode have opposite compute profiles, yet a co-located deployment makes them share GPUs and interfere. PD disaggregation gives each phase its own GPU pool, so each pool can be sized in hardware and tuned in software on its own.

Why prefill and decode are different

Prefill runs the whole prompt in parallel; decode produces one token per step. That single difference sets the hardware each phase wants.

Work per step — Whole prompt in parallel (e.g. 2048 tokens) — One token per request
Linear layers — Large matrix-matrix products — Matrix-vector products at batch 1 or small B
Arithmetic intensity — High, hundreds of FLOPs/byte — About B * 2 / weight_bytes_per_output; below 1 FLOPs/byte at B=1
Bound by — GPU compute (tensor cores) — HBM bandwidth
Buy hardware for — TFLOPS per dollar — HBM bandwidth per dollar

> [!TIP] The best prefill GPU and the best decode GPU can be different hardware generations; PD disaggregation lets you mix them.

The co-located interference problem

On shared GPUs, a long prefill competes with running decode requests, and their inter-token latency rises while the GPU is busy with prefill compute. This is head-of-line blocking at the GPU level.

Chunked prefill only moves the tradeoff around:

Smaller chunks — Longer — Lower
Larger chunks — Shorter — Bigger latency spikes

PD disaggregation architecture

A prefill cluster and a decode cluster run independently, joined by a KV transfer fabric.

KV transfer is on the TTFT critical path

A 2048-token KV cache for a 70B GQA model is about 655 MB. How long it takes to move depends on the link.

InfiniBand NDR (cross-rack) — 50 GB/s — ~13 ms
NVLink 4.0 (same rack) — 900 GB/s — ~0.7 ms

> [!NOTE] NVIDIA NIXL (NVLink-based Transfer Library) is built specifically for low-latency KV cache migration.

Disaggregated KV cache pools

A further step separates KV cache storage from the decode GPU. A pool on CPU DRAM, SSD or a dedicated KV cache server holds active requests' caches, and the decode GPU streams blocks in as needed.

This lets KV caches exceed decode GPU HBM, which matters for long-context (128K+ token) workloads.

Production implementations

Mooncake — Kimi/Moonshot, 2024 — Separate prefill and decode pools; a KV cache scheduler sends requests to decode workers where their KV cache is already resident, cutting transfer cost
DistServe — Berkeley, 2024 — Research formalization: the best prefill:decode GPU ratio depends on the prompt/generation length ratio and the SLO targets

Interview questions

Design a PD disaggregated serving system for a high-traffic LLM endpoint with TTFT < 500ms and TPOT < 50ms SLOs.