Large-Scale AI Inference Serving

Multi-Tenant Model Serving

Design a platform that serves large language models and many smaller models to many teams on shared GPUs, streaming tokens with low latency while keeping expensive GPUs busy.

Overview

An inference platform takes models that teams have trained and runs them for real traffic. Teams hand over a model; the platform gives back an endpoint, and takes care of GPUs, batching, scaling, rollouts and cost reports. The hard case, and the one this design centers on, is a large language model (LLM): billions of parameters, answers streamed token by token.

The design follows from two facts. First, an LLM answers in two phases: prefill reads the whole prompt at once, then decode produces one token per step, and each step has to read all of the model's weights from GPU memory. Second, every token a request has seen leaves behind a KV cache entry (the attention keys and values) that stays in GPU memory until the request ends. So memory, not arithmetic, usually limits how many requests one GPU can serve at once.

The platform's job is to keep GPUs full without letting any single user wait too long: batch many requests into each decode step, manage KV memory tightly, route requests to replicas that have room, and scale replicas that take minutes to start.

Questions to ask first

Which models? — One 70B-parameter chat model (Llama 3 70B shape) plus many small models (embeddings, classifiers)
How much traffic? — 10 million chat requests a day, peaks at 3 times the average
How long are requests? — About 1,000 prompt tokens and 300 output tokens
What latency matters? — Time to first token under 1 s and about 50 ms between tokens at p99 (targets we chose)
Interactive only? — No: batch jobs too, which can wait hours and fill idle GPUs
One team or many? — Many teams; each needs isolation and a usage report

What you will be able to do

Explain prefill and decode, and why decode is limited by memory bandwidth.
Size the KV cache from a model's shape and work out how many requests fit on a replica.
Explain continuous batching and paged KV memory, and show why they raise throughput.
Size a GPU fleet from traffic, and plan autoscaling around multi-minute cold starts.
Roll out a new model version safely and share GPUs across many teams.

Key concepts

Prefill and decode
Time to first token
KV cache
Paged attention
Continuous batching
Tensor parallelism
Multi-model packing
Warm pools
Canary rollout
Speculative decoding

Functional requirements

Deploy a model version to an endpoint with a hardware profile, minimum and maximum replicas
Serve streaming generation (tokens as they are produced) and one-shot calls for small models
Accept batch jobs that run when capacity is free and report results later
Shift traffic between model versions by weight, with shadow traffic that is not returned to users
Scale replicas up and down from load signals, including to zero for idle small models
Record usage (tokens, GPU-seconds) per team and endpoint

Non-functional requirements

Time to first token under 1 s and time per output token under 50 ms at p99 for chat (targets we chose)
High GPU use: batch as many requests per step as latency targets allow
Isolation: one team's spike or bad request cannot take down another team's endpoint
Rollouts and rollbacks without dropped requests
Available at 99.9% for interactive endpoints (a target we chose); batch may wait

Interview questions

Why is LLM decoding limited by memory bandwidth, and what does that mean for design?
How big is the KV cache, and how many requests fit on a replica?
What is continuous batching and why does it help?
What does PagedAttention change?
How do you size the GPU fleet?
How do you autoscale when a replica takes minutes to start?
How do you roll out a new model version safely?
How do many teams share the fleet fairly?