ChatGPT / LLM System

Large Language Model Serving Platform

Design a chat service that runs a large language model on GPUs: stream each answer token by token, batch many users on one GPU, manage the KV cache, and enforce per-customer rate limits.

Overview

A user types a message and the answer appears word by word. Behind that, a model with billions of parameters runs on a GPU and produces the answer one token (a word piece of a few characters) at a time. Each new token needs a full pass through the model, so a 300-token answer is 300 passes.

Three things make this unlike a normal web service. A request lasts seconds, not milliseconds. It holds GPU memory the whole time: the KV cache, the attention keys and values of every token so far, which the model reuses instead of recomputing. And the hardware is expensive, so the design lives or dies by how many requests one GPU serves at once.

This problem is about serving a trained model, not training it. It works for any model; the numbers below are interview assumptions and a 13-billion-parameter model is used for the memory arithmetic only because its KV cache size is published.

Questions to ask first

How many users and requests? — 10 million daily users, 10 requests each a day
How long are prompts and answers? — About 1,000 tokens in (with history), 300 tokens out
Streaming or whole answers? — Stream tokens as they are made; the API also offers whole answers
One model or several sizes? — Several; each runs on its own pool of GPU servers
Who are the customers? — Chat users and API customers, with limits per organization
Must we store conversations? — Yes, so users can resume them

What you will be able to do

Explain prefill and decode, and why decode is limited by memory bandwidth.
Show how continuous batching beats static batching, with a small simulation.
Size the KV cache and explain how paging it lets more requests share a GPU.
Stream tokens over server-sent events, and handle a client that goes away.
Enforce requests-per-minute and tokens-per-minute limits and shed load under overload.

Key concepts

Prefill and decode
Time to first token
Continuous batching
KV cache
PagedAttention
Prefix caching
Server-sent events
Tokens per minute
Speculative decoding
Cache-aware routing

Functional requirements

Send a list of messages and a model name; get the assistant's reply
Stream the reply token by token, or return it whole
Store conversations so a user can resume them
Offer several models and generation settings (temperature, maximum tokens)
Check inputs and outputs against a safety policy
Limit and meter usage per organization and API key

Non-functional requirements

First token within 1 second at p95 for short prompts (a target we chose)
At least 20 tokens a second per stream after that, faster than people read (a target we chose)
Keep GPUs busy: batch many requests per step instead of one request per GPU
Under overload, answer 429 or queue briefly; never let every request slow down together
The API stays available 99.9% of the time; a failed GPU server loses only its running requests

Interview questions

Why is LLM serving limited by memory rather than compute?
Walk me through continuous batching.
What does PagedAttention change?
How do you stream the answer, and what if the client disconnects?
How would you rate limit an LLM API?
How do you choose which GPU server gets a request?
How do you scale up for a traffic spike?
What is speculative decoding and when does it help?
Where does conversation history live, and how is the prompt built?