Alerting Philosophy

The SRE Book's alerting principles, multi-window multi-burn-rate, and avoiding alert fatigue.

Overview

Where observability meets human attention

Bad alerting is the leading cause of SRE burnout. Good alerting is rare and load-bearing.

The SRE Book's core principle (Ch 6, paraphrased): alert on symptoms, not causes. "Did something break?" is a better question than "Is the database CPU above 90%?"

Most causes don't matter. A DB at 90% CPU might be serving traffic perfectly; paging on it wakes an SRE for nothing. Paging only when users are affected (latency spike, error-rate spike) means every page is a real problem.

Five qualities of a good alert (SRE Book Ch 6 + Workbook)

1. Actionable — Someone can do something. "FYI, X happened" belongs on a dashboard
2. Precise — Says what's broken and where: "p99 latency on checkout-service > 500ms in us-east-1" beats "something is slow"
3. Timely — Fires fast enough to mitigate before users notice or it gets worse
4. Not noisy — Low false-positive rate; nobody snoozes it reflexively
5. Linked to a runbook — The page comes with documented investigation steps

Speed vs precision: multi-window, multi-burn-rate

Fast alerts have more false positives; precise alerts come too late. The canonical solution (SRE Workbook Ch 5) AND-gates a long window (high precision, slow) with a short window (confirms it's still happening).

At 14.4× the normal burn rate, you exhaust 2% of a 30-day budget in 1 hour. That's actionable now; pair it with a 5-minute window confirming "still burning."

Page-rate-fast — 1h — 5m — 14.4× — PAGE: fast, urgent — Explosive incidents
Page-rate-medium — 6h — 30m — 6× — PAGE: slower, sustained — Sustained incidents over a few hours
Ticket-rate-slow — 3d — 6h — 1× — TICKET: no page — Slow creep over days

> [!IMPORTANT] All three shapes need a response; only the first deserves an urgent page.

Alert fatigue

SREs reflexively dismiss pages — Every alert must be actionable
New SREs don't know which pages matter — Every page has a runbook
"Always firing" alerts never get fixed — Track noisy alerts; kill any that fire more than 3× without action

> [!TIP] Review every alert that fired, weekly. Was it actionable? Did the runbook help? Should it fire faster, slower, or be deleted?

Key concepts

Alert on symptoms: "Did something break for the user?" beats "is some internal metric over a threshold?". Symptoms = SLI deviation; causes = internal saturation.
Multi-window multi-burn-rate: Long window for trend (precision) + short window for currency (speed) + burn-rate threshold based on error budget. SRE Workbook Ch 5 canonical formula.
Page vs ticket: Page = wake someone now. Ticket = work item for next business day. Slow burns get tickets; fast burns get pages.
Actionable: A first quality of any alert — someone can do something. If no action, it's a dashboard, not a page.
Runbook: Investigation + mitigation steps for an alert. Every alert links to one. SRE's test: a new on-call could resolve from the runbook alone.
Alert fatigue: SREs ignore pages because too many fire without merit. Cure: kill noisy alerts, every alert actionable.

When to use it

Designing alerts for a new service — start from SLOs, derive burn-rate alerts
Reviewing on-call burden — count pages/week, identify noisy alerts to kill
Postmortems — was the alert that fired the right one? Did it page early enough?
On-call handbook — every page should link to a runbook

Interview questions

Walk me through multi-window multi-burn-rate alerting.
Why is "DB CPU > 90%" usually a bad page?
How do you fix alert fatigue on an on-call team?