Alert Correlation & Grouping

When one incident produces a thousand alerts via cascading dependencies, on-call engineers cannot triage. Correlation strategies (temporal, topological, causal), platforms (PagerDuty AIOps, BigPanda, Moogsoft, Datadog Incidents, ServiceNow ITOM), and what realistic noise reduction (70-90% achievable) looks like in production.

Overview

The alert storm

Alert correlation and grouping addresses the most immediate operational pain in AIOps. When one root cause (database connection pool exhaustion, a network partition, a shared cache failure) cascades through the dependency graph, every downstream service fires its own alerts: 50 to 200 pages within two minutes. Finding the one root cause among 150 notifications is too much cognitive load, and mean time to acknowledge climbs.

Every alert is truthful

Alert storms are not a sign of bad alerting; they reflect how distributed failures cascade. Each alert (auth service, gateway latency, frontend timeouts, error rates on every caller) is correct.

> [!NOTE] The problem is that 150 truths about one incident are worse than one summary.

Correlation strategies

Temporal — Alerts within a configurable window, typically 1 to 15 minutes — Sharp-onset incidents where everything breaks at once — Slow-burn incidents that accumulate over hours; unrelated incidents that happen at the same time
Topological — Alerts from services that share a dependency edge in the service map — Links auth service, API gateway and downstream services; excludes the unrelated batch job that failed at the same time — Needs topology from APM auto-discovery (Datadog, Dynatrace), service mesh (Istio, Linkerd) or CMDB
Causal — Traces upstream to the root entity; Dynatrace Davis walks the Smartscape graph to the node with no healthy upstream — Produces a tree (one root, many downstream effects) instead of a flat group — Depends on a complete topology graph

Realistic noise reduction

Vendor case studies — 95 percent and above
Real production, steady state after two to four months of tuning, clean metadata — 70 to 90 percent
Closing the gap from 90 to 99 percent — Requires very high-quality topology data; not achievable in all environments

Prerequisite: alert hygiene

Every alert needs four fields. Correlation tools fed "Error in service" with no service field produce meaningless groups.

> [!IMPORTANT] Input data quality determines correlation quality more reliably than the choice of correlation algorithm.

Interview questions

How would you design an alert correlation system for a 300-service microservices platform?
What distinguishes PagerDuty AIOps from BigPanda and how would you choose between them?
How do you measure whether alert correlation is actually working?
Explain how change correlation improves alert triage and what data pipelines you need to support it.
What does a well-tuned alert grouping configuration look like in PagerDuty AIOps?