Latency vs Throughput

How long one request takes and how many a system finishes each second: percentiles and tail latency, why waits grow as servers get busy, batching and pipelining, honest measurement, and latency SLOs.

Overview

Latency is how long one request takes. Throughput is how many requests a system finishes each second. Fundamentals introduced both, along with percentiles and Little's Law. This chapter goes deeper, into the questions that separate a memorized answer from real experience: why the slowest 1% of requests matters more than the average, why a server that is 90% busy is so much slower than one that is 50% busy, and how a load test can report a p99 that is 200 times too good.

The two numbers pull against each other. Most ways to get more throughput, such as batching, running servers hotter or queueing work, make some requests wait longer. Most ways to cut latency, such as spare capacity or sending a slow request twice, cost throughput or money. A strong answer names the trade and puts numbers on it.

The examples come from Google's paper The Tail at Scale, the Google SRE books, and the documentation of Kafka, PostgreSQL, Redis, Prometheus and gRPC. The queueing results are standard formulas, and the code on this page computes every number it quotes.

What you will be able to do

  • Read a latency distribution and explain why p99 and p99.9 matter more than the mean, especially when one request fans out to many servers.
  • Explain how hedged and tied requests cut tail latency, and when they make things worse.
  • Predict how latency grows with utilization, and choose a utilization target with a reason.
  • Use batching and pipelining to raise throughput, and say what each costs in latency.
  • Measure latency without averaging percentiles or falling into coordinated omission, and write a latency SLO with an alert on its error budget.

Key concepts

  • Percentiles: p50, p99, p99.9
  • Tail latency and fan-out
  • Hedged and tied requests
  • Utilization and queueing
  • Batching
  • Pipelining and requests in flight
  • Bandwidth-delay product
  • Latency histograms
  • Coordinated omission
  • SLOs and error budgets

Interview questions

  • Why is the average latency a poor metric, and how would you report latency for a service on 50 servers?
  • A page fans out to 100 backend calls, each with a p99 of one second. What do users see, and what would you do?
  • What is a hedged request, and when would you not use one?
  • Why does latency rise so sharply as a server gets busy, and what utilization would you plan for?
  • How does batching trade latency for throughput? Give an example with numbers.
  • A client calls an API in another region, 80 ms away, one request at a time. How fast can it go, and how would you speed it up?
  • Your load test reports a p99 of 5 ms, but users see stalls of several seconds. What could be wrong?
  • How would you write a latency SLO for an API, and alert on it?
  • What does Amdahl's law say about adding servers?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio