Incident Command

The Google IRT structure (modeled on FEMA ICS): three roles, why "IC does not type," and when to escalate.

Overview

Separate command from execution

Google's incident response framework (SRE Book Ch 14) is modeled on FEMA's Incident Command System (ICS), the structure US wildland firefighters and emergency responders use. The core idea is to separate command from execution.

The roles

Incident Commander (IC) — Decides priorities ("we mitigate first, investigate later"), delegates, runs the bridge, decides escalation, ends the incident when mitigated
Operations Lead (Incident Response Lead) — Reads dashboards, runs commands, makes changes; pulls in SMEs; reports findings to the IC
Communications Lead — Posts on the status page if customer-impacting; updates the internal Slack channel every 15-30 minutes; briefs executives; drafts customer comms
Planning Lead (incidents past 4 hours) — Manages shift handoffs, owns the timeline document, keeps context when the IC changes

> [!IMPORTANT] "The IC does not type" is load-bearing. An IC who is also fixing the issue can't see the bigger picture, coordinate others or communicate. In one well-known war story, an IC debugging while running the call missed a parallel issue that compounded the outage.

Handoff rules (SRE Book Ch 14)

Declare early — Declare an incident when more than one engineer is involved, or when users are impacted
First responder is IC — By default, until handed off
Hand off explicitly — "John, you are now IC. I am no longer IC." Acknowledged. Otherwise two people think they're IC and decisions don't get made

One channel per purpose

> [!TIP] Don't mix them. Anyone asking "what's the latest?" should catch up from chat or the doc without joining the bridge.

Escalation toolkit

Another IC — Peer support; rare
A senior engineer — Deep technical knowledge needed
Leadership — Financial-impact incidents
Vendor support — Cloud incidents: AWS, GCP, Datadog

End of incident

The IC declares "incident resolved" when the symptom is mitigated. That is not the same as "root cause identified."

> [!IMPORTANT] Mitigation first; root cause later, in the postmortem.

Key concepts

Incident Commander (IC): Coordinates the response; does not execute mitigations themselves. The "does not type" role.
Ops Lead: Executes the mitigation. Reads dashboards, runs commands, pulls in SMEs.
Comms Lead: Updates status page, Slack, executives, customers. Owns the external narrative during the incident.
Explicit handoff: "John, you are now IC. I am no longer IC." Acknowledged. Otherwise two ICs and unclear decisions.
Mitigation before root cause: IC declares "resolved" when symptom is mitigated, not when root cause is found. Root cause goes into the postmortem.
FEMA ICS lineage: Google's IRT framework is modeled on the US wildland firefighting Incident Command System — same structure, same separation of roles.

When to use it

Active incident — assigning IC + Ops Lead + Comms Lead is the first 60 seconds
On-call training — every new SRE should have shadowed an IC and run an exercise
Postmortem — was the incident command structure correct? Were roles confused?
Designing the on-call rotation — who can be IC? It is a learned skill, not all SREs are ready.

Interview questions

Walk me through the incident roles and why each exists.
When should you escalate vs handle it yourself?
You're paged at 3am as IC. Walk me through the first 5 minutes.
When does IC declare an incident resolved?