Alert Fatigue and On-Call Burnout

Identify, measure, and systematically reduce alert noise to restore effective on-call rotations.

Overview

Overview

Alert fatigue is the state where noisy, low-signal alerts have taught on-call engineers to treat pages as background noise. It is one of the most dangerous failure modes in reliability engineering because it silently degrades incident response, often without anyone deciding to stop responding.

Symptoms you can measure

  • MTTA trends upward over weeks while alert volume stays constant — Engineers no longer treat pages as urgent
  • Alerts acknowledged without opening runbooks — No investigation of root cause
  • Many alerts fire for one issue and nobody responds — The alert storm drowned the signal
  • On-call is the most dreaded assignment — The rotation is burning people out

Root causes

  • Conservative static thresholds — CPU alert at 70% fires every afternoon peak — Engineers learn to ignore CPU alerts
  • No priority tiers — A connection pool at 95% pages like a single 500 response — Everything urgent means nothing is urgent
  • No runbooks — Same diagnostic steps worked out from scratch each time — Higher cognitive load, less investigation
  • Redundant alerts — One root cause watched from many angles — A single failure generates 20 simultaneous pages

Alert on symptoms, not causes

The Google SRE approach, documented in *Site Reliability Engineering*, fixes this at design time. Alert on what users feel, not on causes that may or may not affect them.

  • SLO burn rate, error rate, latency p99 — CPU usage, memory usage, disk I/O

> [!TIP] A CPU spike that does not affect response latency should log a metric, not page anyone.

The weekly alert audit

For every alert that fired in the past week, ask three questions. Any alert that answers no to all three is deleted, downgraded to a ticket, or automated.

  • Did it require human action? — It should not page
  • Was the action documented? — Add a runbook or downgrade
  • Could the action be automated? — Automate it and drop the page

> [!IMPORTANT] Target fewer than five pages per on-call shift, each one a genuine user-facing issue.

Burn-rate alerting, the structural fix

  • 1 hour — 14.4x — 2% of the 30-day budget gone in an hour; exhausted in about 2 days — Page now
  • 6 hours — 6x — 5% of the budget gone; exhausted in about 5 days — Urgent, calmer response

> [!NOTE] These alerts are specific, actionable and tied directly to user impact.

Key concepts

Alert Fatigue
The desensitization of on-call engineers to alerts due to excessive volume or noise, resulting in slow or absent responses to genuine incidents.
MTTA
Mean Time To Acknowledge — the average time from an alert firing to an engineer acknowledging it; a leading indicator of alert fatigue when trending upward.
SLO Burn Rate
The rate at which error budget is being consumed relative to the target rate — the foundation of Google's multi-window, multi-burn-rate alerting approach.
Symptom-Based Alerting
Alerting on user-visible impact (high error rate, high latency) rather than internal causes (CPU, memory) — the core SRE principle for reducing noise.
Alert Hierarchy
A tiered structure classifying alerts by urgency and required response: page (immediate action), ticket (next business day), log (no action needed).

When to use it

  • Diagnosing why on-call engineers are slow to respond despite receiving pages
  • Redesigning an alert hierarchy to separate critical pages from informational notifications
  • Explaining the SRE approach to symptom-based vs cause-based alerting
  • Auditing an existing alert set to identify and eliminate noise
  • Setting up SLO burn rate alerting to replace threshold-based pages

Interview questions

  • What is the Google SRE recommendation for what to alert on, and why does it reduce noise?
  • How do you run an alert audit to systematically reduce noise in an existing alerting system?
  • How do suppression windows and alert grouping help reduce on-call burden without losing signal?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio