AIOps and SRE

AIOps augments Site Reliability Engineering by automating SLO burn rate alerting, reducing toil below 30%, and accelerating postmortems through LLM-generated incident timelines. The result is faster incident response and engineers spending more time on reliability improvements rather than repetitive operational tasks.

Overview

Where AIOps plugs into SRE practice

AIOps touches five SRE practices: SLO alerting, toil, postmortems, runbooks and on-call routing. In each, it automates the mechanical part and leaves judgment with the engineer.

SLO and SLI predictive alerting

Traditional SLO alerting is reactive: you get paged after the SLO is already breached. AIOps uses trend detection on error rate or latency to project a breach time, for example "at the current error rate you will exhaust the 30-day error budget in 4 hours," so engineers act before customers are hit.

Google's Site Reliability Workbook formalizes this as burn rate alerting, with two thresholds for two failure modes.

Threshold — 2% of the 30-day budget in a 1-hour window — 5% in a 6-hour window
Meaning — Linear extrapolation exhausts the budget in 50 hours — Will exhaust the budget if unaddressed, but not an emergency
Response — Page the on-call engineer immediately — Ticket, not a page

AIOps platforms automate the burn rate calculation and route the alert, so no one builds and maintains custom dashboards per service.

Toil reduction

Toil is manual, repetitive, automatable work that scales linearly with service growth: double the services, double the toil, unless it is automated.

Google's ceiling — No more than 50%
AIOps target — Below 30%

Measurement baseline: count engineer-hours per week on specific tasks, such as pod restarts on OOM kill, cache flushes on stale-data alerts, queue consumer restarts on deadlock patterns, log rotation and backup verification. This baseline anchors every automation ROI calculation.

AIOps-assisted postmortems

Postmortems take time to write well. An LLM reads the incident timeline (alert fired at T+0, acknowledged at T+3 minutes, restored at T+45 minutes) and generates the scaffolding.

Narrative timeline — Definitive root cause conclusion
Contributing services — Action items with named owners and due dates
Customer impact summary — Prevention mechanisms against recurrence

Tools in this space: PagerDuty postmortem generation, Incident.io AI incident summary, Datadog Bits AI incident review and Rootly.

> [!WARNING] LLMs are not reliable for causal reasoning about novel system behavior, so the root cause stays with the SRE.

The runbook automation trust ladder

1. Traditional runbook — Engineers read and execute each step manually
2. Machine-readable runbook — Steps run via Rundeck, Ansible or AWS Systems Manager Run Command
3. LLM-assisted lookup — "How do we restart the payments service?" returns the runbook through semantic RAG search
4. LLM-recommended, human-approved — LLM recommends the next step, SRE approves, automation executes, SRE verifies
5. Semi-autonomous — A human starts the runbook; automation runs the rest, pausing only at high-risk actions

> [!IMPORTANT] Climb the ladder one rung at a time over months, not all at once.

Alert routing and on-call

AIOps routes alerts by combining service ownership tags with historical resolution patterns. If the Identity team has resolved auth service alerts 94% of the time, new auth alerts go straight there. PagerDuty and OpsGenie both support ML routing rules learned from historical incidents.

Interview questions

Explain Google's error budget burn rate alerting model and how AIOps automates it.
How should an engineering organization measure and reduce toil using AIOps?
Walk through the runbook automation trust ladder and explain why it must be climbed incrementally.
How does AIOps-assisted postmortem generation change the SRE postmortem process, and what are its limits?