Runbooks

The runbook structure that works under stress, when to update them, and the path to runbook automation.

Overview

What a runbook is for

A runbook tells an on-call engineer "if X happens, do Y." Good runbooks are the difference between a 5-minute mitigation and a 60-minute investigation; bad ones are stale wiki pages nobody reads.

> [!IMPORTANT] A runbook must be scannable in 30 seconds by a stressed engineer at 3am. Walls of text go unread.

The structure that works under stress

1. One-line summary — What the runbook is for; searchable, in the alert text, quoted in pages
2. Pre-conditions — Which alerts fire and what the symptom looks like
3. Initial diagnosis — Which dashboards and queries to check first, in priority order
4. Decision tree — "If you see X, do A. If you see Y, do B. If neither, escalate."
5. Mitigation steps — Concrete, copy-pasteable commands, numbered when multi-step
6. Verification — "After mitigation, check that [specific metric] is back to normal."
7. Escalation — "If the above doesn't work, page [X]; provide [Y] info."
8. Post-incident — "File a [postmortem ticket / followup task]."

The brevity discipline

Lead with the action — Background context, unless essential; link to it instead
Bullet points — Paragraphs
Diagrams where they help — Walls of text

Runbook formats

Wiki page: Confluence, Notion, internal docs — Most common; risk of staleness
Markdown in the repo — Better: reviewed via PR, lives with the code
Embedded in the alert — The alert message carries the runbook URL or an excerpt
Slack workflow or chatops — Interactive: "type !runbook checkout-down"

The 2026 trend: runbook automation

Executable runbooks automate the steps, so routine mitigations are auto-remediated and the engineer is informed without needing to act. That cuts page volume and time to mitigate.

Rundeck — Orchestrates scripted procedures with an audit trail
StackStorm — Event-driven automation
AWS Systems Manager — Managed automated runbook execution
Custom Lambdas plus alert webhooks — Alert fires, Lambda runs the mitigation, engineer is notified

The pattern is not well understood — It may apply the wrong fix
Side effects are not bounded — It can cause new damage
It masks the underlying bug — Recurring problems stay hidden
It is misconfigured — It makes things worse

> [!TIP] Every auto-remediation should alert even when it succeeds, so the underlying pattern stays visible.

Runbook hygiene

Update after every incident — If the runbook missed what happened, fix it
Test in game days — Simulate the alert and verify the runbook works
Review quarterly — Stale links, deprecated commands, removed dependencies
Link from every alert — Alert to runbook URL gives instant context

The runbook orphan problem

A detailed runbook exists, but nobody knows it does. The on-call engineer cannot find it, finds an outdated copy, or finds it after the incident.

The minimum-viable runbook

If you cannot write a full runbook for a new alert, write the minimum and iterate.

Meaning — "This alert means X."
Where to look — "Check dashboard at [link]."
Escape hatch — "If unsure, escalate to [name/team]."

Key concepts

Runbook structure: Summary + pre-conditions + diagnosis + decision tree + mitigation + verification + escalation. Scannable.
Brevity discipline: Scannable in 30 seconds. Bullets, not paragraphs. Stressed engineer can't read walls of text.
Embedded in alert: Alert message links the runbook URL. Click → context. Removes "where was that runbook?" friction.
Runbook automation: Rundeck / StackStorm / Lambdas execute the steps. Reduces MTT-Mitigate; eliminates routine pages.
Hygiene cadence: Update after every incident. Test in game days. Review quarterly. Without hygiene, runbooks decay.
Minimum-viable runbook: 3 lines: meaning, dashboard, escalation. Better than nothing. Iterate.

When to use it

Every new alert — write a minimum runbook before the alert fires for the first time
Postmortem follow-up — update runbook with what we learned
Quarterly hygiene — review existing runbooks for staleness
On-call training — new engineers read N runbooks during onboarding

Interview questions

What does a good runbook look like?
How do you keep runbooks from going stale?
When should you automate a runbook vs keep it manual?