Load Shedding

How and when to deliberately reject requests so the rest succeed. Priority-based shedding, queue management, the AWS framing.

Overview

Reject some requests so the rest succeed

Load shedding is the discipline of rejecting some requests so that others succeed. It feels wrong, because engineers want to serve every request.

The right reframe: serving 70% of requests successfully is much better than failing 100% because the system collapsed.

SRE Book Chapter 21, Handling Overload, is the canonical treatment: "Avoid serving requests that you can't serve gracefully. The point of load shedding is to reduce the load on the server such that it can return to a healthy state and serve the rest of its load."

The three patterns

1. Reject incoming connections — LB or API gateway with a concurrent-request cap returns 503 immediately; existing requests continue — Fast; shed requests consume nothing
2. Drop in-flight requests — A bounded request queue rejects new requests past a threshold, or drops the oldest, CoDel-style — Stops the queue growing without bound
3. Priority-based shedding — Requests tagged by priority: premium, paid feature, internal vs external; drop low priority first — Standard for free, pro and enterprise tiers

When to start shedding

Too early — Rejecting requests you could have served: wasted capacity, unhappy customers
Too late — Degradation has begun, a cascade is starting, recovery is hard
Sweet spot — Just before the system would otherwise degrade

Queue depth — Over threshold
p99 latency — Over target
CPU — Past the saturation point
Memory — Beyond headroom
Connection pool — Over 90%

> [!TIP] Shed before the SLI breaches. With a "p99 < 500ms" SLO, shed when p99 climbs toward it, such as 300ms and rising, not at 600ms.

The AWS four-level scheme

1 — All requests served — Normal
2 — Background work deprioritized — Cron jobs and async tasks paused; user-facing reads and writes prioritized
3 — Read-only — Writes rejected, reads served
4 — Critical-only — Only health checks and admin commands; all user traffic rejected

Each level has alerting, so engineers know which mode the service is in.

Coordinated shedding via priority headers

Internal services propagate X-Priority: critical | high | normal | low, and each service sheds on it.

API call from a logged-in user — Critical
API call from an anonymous user — Normal
Background job — Low
LB health check — Critical, so the LB does not remove the instance

When overloaded, drop low first, then normal, then high, never critical.

Anti-patterns

Unbounded queues — Requests pile up until OOM or a timeout cascade
Synchronous full thread pool — Every thread waits, so new requests cannot even be acknowledged; reject before threads
No shed-rate metric — You cannot tell whether shedding happens or how much
Shedding on error rate only — Too late; shed on load indicators such as queue depth and latency

Useful libraries

Netflix concurrency-limits — Adaptive concurrency limiting that computes the optimal in-flight count
Hystrix, resilience4j — Bulkheads enforce concurrency limits per dependency
Envoy, Istio — Rate-limit filters and concurrency caps
AWS API Gateway, Cloudflare — Rate limits at the edge

Key concepts

Load shedding: "Reduce the load on the server such that it can return to a healthy state" — SRE Book Ch 21. Reject some so others succeed.
Priority-based shedding: Requests tagged by priority; drop low first. Standard for tiered services.
Bounded queue: Hard cap on request queue depth; reject when full. Prevents OOM cascade.
Quarantine mode: Health-check-failed instance rejects low-priority traffic but stays in pool for critical. Better than removing it.
AWS shedding levels: Normal → background-deprioritized → read-only → critical-only. Explicit modes; engineers know the state.
Adaptive concurrency limit: Calculate optimal in-flight count from latency / throughput data (Netflix concurrency-limits). Auto-tunes.

When to use it

Pre-launch — bound the request queue; configure shedding before you need it
Reviewing capacity plans — what happens at 1.5× peak? Shed gracefully or collapse?
Postmortem with cascade involved — was shedding configured? Did it kick in?
Critical events (Black Friday, viral moments) — pre-tune shedding aggressiveness

Interview questions

Why is "serve all requests" worse than "drop some requests"?
How would you implement priority-based shedding?
What's the difference between rate limiting and load shedding?