Backpressure and Load Shedding

What a system should do when work arrives faster than it can finish it: bound every queue, let slow consumers set the pace, shed load early at the edge, and make clients back off instead of retrying harder.

Overview

Every service has a maximum rate at which it can finish work. When requests arrive faster than that, the difference has to go somewhere. It can wait in a buffer, it can be thrown away, or the sender can be told to slow down. Backpressure is the third option: a signal that travels from the slow part of a system back toward the fast part, so the producer sends only as much as the consumer can take.

Systems without it fail in a slow, confusing way. A queue with no limit accepts everything, so nothing looks wrong at first. But the backlog grows every second, each new request waits longer, callers time out and retry, which adds more work, and eventually the process runs out of memory. By then the queue is full of requests whose callers have already given up.

Load shedding is the companion technique for the edge of the system, where you cannot slow the sender down directly: reject some requests quickly and cheaply, so the ones you accept still finish in time.

What you will be able to do

  • Work out how fast a backlog grows and how long a new request will wait, from two rates.
  • Choose between buffering, dropping and slowing the producer for each hop of a design.
  • Explain pull-based flow control in TCP, HTTP/2, Reactive Streams and Kafka consumers.
  • Design load shedding at an API: in-flight limits, priorities, deadlines and the right status codes.
  • Make clients back off with exponential delays, jitter and retry budgets, so retries do not multiply the overload.

Key concepts

  • Rate mismatch
  • Bounded queues
  • Buffer, drop or slow down
  • Pull-based flow control
  • TCP and HTTP/2 windows
  • Reactive Streams demand
  • Load shedding
  • Deadlines
  • Backoff with jitter
  • Retry budgets

Interview questions

  • What is backpressure, and why are unbounded queues dangerous?
  • Design backpressure for a pipeline: API, queue, workers, database.
  • How do TCP and Reactive Streams implement backpressure?
  • Your service is overloaded. How do you shed load without making it worse?
  • How do you stop retries from causing an outage?
  • Kafka consumer lag keeps growing. What do you do?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio