Debugging a Broken Alerting Pipeline
Systematic approach to diagnosing why alerts stopped firing during a production incident.
Overview
Overview
A broken alerting pipeline is a silent failure: the system is on fire and nobody gets paged. Diagnose it by segmenting it into stages and checking each stage independently.
Stages and what breaks
- Silences and inhibitions — An active silence, maintenance window or stale inhibit rule suppresses the alert (most common cause)
- Metric scrape and storage — Targets DOWN, metrics missing
- Rule evaluation — Rule errors; the alert never reaches firing state
- Alertmanager routing — Routing tree sends the alert to the wrong receiver
- Notification delivery — The provider never receives or delivers the event
> [!TIP] Check stages in order of most-common failure first, not in pipeline order.
Monitoring the monitor
> [!IMPORTANT] A dead man's switch is the only continuous end-to-end test of the pipeline: the signal is the absence of a ping.
Interview questions
- Monitoring alerts stopped firing during a production incident. How do you debug a broken alerting pipeline?
- What is a dead man's switch in the context of alerting and why is it the only reliable end-to-end pipeline test?
Sign up free to read the full topic
More Observability topics
Start free on Cariara · Coding problems · System design studio