Alerts Not Firing

Alert silence, inhibition, scrape failures, rule misconfiguration, dead man's switch, and alertmanager routing.

Overview

Silent alerts are an existential threat to reliability: if alerts fail to fire, you learn about outages from customers. Finding out why an expected alert did not fire means checking the whole pipeline, from metric collection through rule evaluation to notification routing.

Common failure points

Target scrape failure — Prometheus has no data for that target, so rules evaluating to absent data may not fire — Status > Targets; the absent() function alerts on missing metrics
Rule misconfiguration — Typo in the PromQL, a label that changed, or a threshold that no longer matches the metric's scale — promtool check rules, or run the query in the Prometheus UI
Alertmanager silence — A maintenance-window silence is still active — Alertmanager UI, Silences; old silences never removed are a frequent cause
Inhibition — A "cluster down" alert inhibits all component alerts; the cluster is degraded but not fully down — Alertmanager inhibition rules

Dead man's switch

The Watchdog alert always fires to confirm the whole alerting pipeline works. If the Watchdog stops arriving, the pipeline itself is broken.

> [!TIP] Route the Watchdog to a dedicated receiver that alerts when it goes quiet.

Key concepts

Dead man's switch (Watchdog): An always-firing alert (expr: vector(1)) that continuously sends notifications. If it stops, the alerting pipeline is broken. Route it to a separate receiver that triggers if quiet for >5 minutes.
Alertmanager inhibition: Suppresses alerts matching a target matcher when a source alert is firing. Used to reduce noise (e.g., suppress service alerts when the entire cluster is down). Can cause missing alerts if misconfigured.
Alertmanager silence: Temporary suppression of alerts matching specific matchers. Created during planned maintenance. Forgotten silences are a common cause of alerts never reaching on-call after maintenance ends.
absent() function: Prometheus function that returns 1 if no time series match the selector. Use it to alert when a metric disappears (target down, label changed, service stopped exporting).
for clause: Alert fires only after condition is true for this duration (e.g., for: 5m). If the condition resolves before 5 minutes, the alert never fires. Prevents flapping but can delay detection.

When to use it

Investigating why an expected alert did not fire during a production incident
Auditing alerting configuration after a postmortem
Setting up a dead man's switch to detect alerting pipeline failures
Explaining why an Alertmanager silence from last month is still suppressing alerts

Interview questions

During a postmortem, you discover that an alert should have fired 30 minutes before the outage was detected. How do you diagnose why it did not fire?

Alerts Not Firing

Overview

Overview

Common failure points

Dead man's switch

Key concepts

When to use it

Interview questions

More Troubleshooting topics