Performance Optimization

Where the latency goes, where to put cache, how deadlines propagate, and the SRE Book's rules for not making it worse.

Overview

Faster without breaking it

Performance optimization is making something faster without breaking it. Many teams instead make changes that are faster most of the time and create subtle correctness or reliability issues at the tail.

The SRE Book's framing (Ch 21 and 22): performance is not a single number. The tail, p99 and p99.9, is what users feel. A change that improves the median by 20% but adds 5× to the p99 is usually a regression.

Latency budgets

Every request has a budget, say 500ms, apportioned across components. Exceed any budget and you fail fast: a degraded result beats the right result too late.

DB lookup — 100ms
Cache check — 5ms
RPC fan-out — 50ms each, in parallel
Render — 30ms

> [!TIP] Latency budgets keep a system that is good at p95 from becoming one that is bad at p99.

Deadline propagation

SRE Book Ch 22: RPCs inherit and decrease deadlines as they descend the stack, so a server doesn't waste cycles on a request whose caller already gave up.

Caching tiers

1. CPU cache / page cache — Free — Automatic
2. In-process memory — Microseconds — Eviction logic in your hands
3. Distributed cache (Redis, Memcached) — Sub-millisecond — Network hop; consistency is best-effort
4. CDN edge — Milliseconds — Geographic distribution; needs cache-control headers

> [!WARNING] SRE Book caution: caches can become load-bearing "capacity caches." A cold cache after restart can stop the service handling normal load.

Two failure shapes to recognize

Bimodal latency — When 5% of requests never complete, the mean looks fine but p99 explodes and threads exhaust. Monitor distributions, not means
GC death spiral (Java, SRE Book Ch 22) — Insufficient CPU, more GC, more CPU consumed, more in-flight requests, more memory pressure, more GC

> [!NOTE] The fix for a GC death spiral is usually capacity; the skill is recognizing the spiral.

Key concepts

Latency budget: Per-request deadline apportioned across components. Sum-of-budgets ≤ total deadline; each component fails fast when over budget.
Deadline propagation: RPCs inherit and decrease their parent's deadline. SRE Book Ch 22 verbatim: "RPCs inherit and decrease deadlines as they descend the stack so a server doesn't waste cycles on a request whose caller already gave up."
Bimodal latency: 5% of requests never complete → mean looks fine, p99 explodes. Monitor distributions; mean latency hides bimodal failure.
Capacity cache: A cache that becomes load-bearing — service can't handle normal load without it. Cold-start after restart is an outage. SRE Book Ch 22 warning.
Tail tolerance: The fundamental SRE position: optimize for p99 and p99.9. Mean-improving changes that hurt the tail are regressions.
GC death spiral: Java case: insufficient CPU → more GC → more CPU → more in-flight → more memory pressure → more GC. Diagnostic: sustained GC % > 30% under load.

When to use it

Pre-launch performance reviews — does the request budget actually add up?
Latency-regression investigations — what changed at the tail, not the mean?
Cache-tier design — picking which level matches the actual access pattern
Production incidents involving timeouts — almost always a deadline-propagation problem

Interview questions

What's deadline propagation and why does it matter?
How do you decide which caching tier to use?
What's bimodal latency and why does it matter?