LLM Serving

Overview

Why LLM inference is its own discipline

Autoregressive generation creates constraints no other model has: KV cache memory pressure, two-phase computation and variable sequence lengths. Naive serving approaches fail badly at scale because of them.

> [!IMPORTANT] GPU memory, not compute, is the throughput ceiling for LLM serving.

Prefill and decode

Inference runs in two phases with different bottlenecks.

• Prefill — Processes the whole prompt in parallel — Compute — Fast
• Decode — Generates one token at a time, reading the full KV cache every step — Memory bandwidth — Dominates for long outputs

The KV cache

The KV cache stores key and value attention activations for every token. Its size scales as 2 × layers × heads × head_dim × sequence length × batch size. A 70B model serving a 4K-context request uses multiple GB of GPU memory per request for the KV cache alone.

> [!NOTE] Past 32K context the KV cache grows linearly while attention compute grows quadratically. Mitigations include paged KV cache, prefix sharing, KV cache quantization, sparse attention and sliding windows.

Continuous batching

Static batching makes short requests wait for the longest one in the batch. Continuous (inflight) batching drops finished requests and adds new ones at every decode step, keeping GPU utilization near 100%.

The serving landscape

• vLLM (UC Berkeley, 2023) — PagedAttention: KV cache in non-contiguous pages, near-100% memory use; continuous batching, prefix caching, speculative decoding, tensor and pipeline parallelism — Default for self-hosted open-weight models
• TensorRT-LLM (NVIDIA) — Compiles the model graph per GPU architecture with kernel fusion — Highest throughput per GPU-dollar on NVIDIA; needs a build per model-GPU pair
• SGLang (UC Berkeley, late 2023) — RadixAttention: KV cache in a prefix tree shared across requests — Chatbots and agents with shared system prompts or RAG context
• HuggingFace TGI — Production Rust + Python stack, tight HF Hub integration — Still deployed; lost ground to vLLM after 2023
• llama.cpp — C++ engine for CPU and Apple Silicon; powers Ollama and LM Studio — Edge and developer tooling, not production servers

Quantization

Running 70B models in FP16 needs multiple H100s. Quantization shrinks memory at a small quality cost.

• FP8 — 8 — Native on H100/B100; halves FP16 memory with minimal accuracy impact
• INT8 — 8 — Widely supported across frameworks
• GPTQ — 4 — Post-training quantization with calibration data
• AWQ — 4 — Protects activation-sensitive weights; better quality than GPTQ at the same bit width
• GGUF — 2-8 — llama.cpp container format; powers Ollama and LM Studio

Throughput versus latency

A bigger batch raises throughput (tokens per second per GPU) but also p99 latency; a smaller batch lowers latency but wastes GPU.

• Continuous batching: keeps the GPU busy without waiting on slow requests.
• Speculative decoding: a draft model proposes tokens and the target model verifies them in parallel, cutting latency without losing throughput.
• Prefix caching: skips prefill for shared prefixes, cutting time to first token (TTFT).

Interview questions

• Explain PagedAttention and why it was a breakthrough for LLM serving throughput.
• Compare vLLM, TensorRT-LLM, and SGLang. What workload does each optimize for?
• Explain the throughput-latency tradeoff in LLM serving and what techniques improve both simultaneously.
• How would you design a self-hosted LLM serving infrastructure for a B2B SaaS product serving 1000 concurrent users?
• What quantization formats exist for LLMs in 2026 and when would you use each?

More MLOps & LLMOps topics

• Agent Tool Security
• Azure Machine Learning Platform
• Data Versioning and Dataset Management
• DevOps to MLOps Transition
• Feature Stores
• Hyperparameter Tuning and AutoML
• LLMOps
• ML Experiment Tracking
• ML Governance and Compliance
• ML Monitoring and Alerting
• ML Pipeline Orchestration
• ML Security and Adversarial Robustness