Reliability Math

Every incident and reliability number you may be asked to compute on the spot — timeline metrics, SLI ratios, error budgets, RTO/RPO checks, downtime cost. Formulas are cloud-agnostic; only the metric names change per provider.

Overview

The arithmetic sheet

Interviews and postmortems ask you to *compute*, not just define. Every formula here should be doable in your head or on a whiteboard.

The organizing principle: the math is universal, the metric names are not. An availability SLI is "good events over valid events" whether you read it from Prometheus, CloudWatch, Cloud Monitoring or Azure Monitor. Learn the formula once and keep a translation table for the provider in front of you.

Timeline metrics

All of them are means: sum the per-incident durations and divide by the incident count.

MTTD: Mean Time To Detect — issue start → monitoring fires — total detection delay / incidents
MTTA: Mean Time To Acknowledge — alert fires → human acks the page — total ack delay / pages
MTTI: Mean Time To Identify — issue start → team knows and owns it — total identification delay / incidents
MTTR: Mean Time To Repair / Restore — work begins → user impact gone — total repair duration / incidents
MTBF: Mean Time Between Failures — steady-state reliability frequency — total operational time / failures

The convention trap

Two conventions are in wide use, and they are not compatible. Read this before you add anything up.

Nested — Issue start to resolved; MTTD + MTT-Mitigate + MTT-Resolve. Detection is *inside* MTTR — MTTR — Adding MTTI to MTTR double-counts detection
Additive — From the end of detection (MTTI, or MTTD+MTTA) — MTTI + MTTR — Quoting MTTR alone understates user-visible downtime

> [!IMPORTANT] State your boundaries before you compute: "MTTR measured from detection" or "MTTR measured from issue start." Interviewers test whether you know the ambiguity exists; naming it scores better than picking a side.

Worked example (additive convention)

60 min cumulative detection delay across 5 incidents — MTTI = 12 min
150 min cumulative repair across the same 5 — MTTR = 30 min
Total downtime per incident — 12 + 30 = 42 min
720 h (30 days) of operation, 3 outages — MTBF = 240 h

Cross-check against the availability identity:

Availability = MTBF / (MTBF + MTTR_total) = 240 / (240 + 0.7) = 99.71%

> [!WARNING] That is *below* a 99.9% SLO: 42 minutes × 3 outages is 126 minutes against a 43.2-minute monthly budget. Timeline metrics and the error budget must tell the same story; if they don't, one is measured wrong.

SLI formulas: one shape, four dialects

Every request-driven SLI is the same ratio:

SLI = good_events / valid_events × 100

"Valid" excludes what isn't your responsibility (404s for routes that don't exist, requests rejected upstream). "Good" follows the SLI definition: 2xx for availability, 2xx and under the threshold for latency.

Load balancers give you totals and errors, not successes, so the practical form is a subtraction:

Metric-name translation table

Total requests — http_requests_total (counter) — ALB RequestCount; API GW Count — loadbalancing.googleapis.com/https/request_count — App Gateway TotalRequests; Front Door RequestCount
Server errors — http_requests_total{status=~"5.."} — ALB HTTPCode_Target_5XX_Count / HTTPCode_ELB_5XX_Count; API GW 5XXError — same metric filtered response_code_class = 500 — App Gateway FailedRequests; App Service Http5xx
Latency distribution — http_request_duration_seconds_bucket (histogram) — ALB TargetResponseTime (p99 statistic) — https/total_latencies (distribution) — BackendLastByteResponseTime
Ratio mechanism — PromQL division — Metric Math expression — MQL ratio / Managed Prometheus — Metric math or KQL over logs

The same availability SLI in each dialect

Prometheus (generic):

Prometheus (Kubernetes NGINX ingress):

AWS CloudWatch Metric Math, binding b = RequestCount (Sum) and a = HTTPCode_Target_5XX_Count (Sum):

GCP — Managed Service for Prometheus takes the PromQL above verbatim; native MQL divides request_count filtered to non-5xx by unfiltered request_count
Azure — Log Analytics KQL over diagnostic logs, below

Latency SLI = (requests served under X ms) / valid requests × 100

In Prometheus the histogram bucket gives the numerator directly: the le label is cumulative, so the bucket *is* the count under the threshold.

> [!TIP] This is why a threshold SLI beats a percentile SLI: it aggregates across instances correctly, and percentiles do not.

Error budget and burn rate

Budget — 0.1%
As time — 43.2 min/month
As volume — 10,000 allowed failed requests

> [!TIP] Quote the denomination your audience thinks in: engineers usually want requests, executives usually want minutes.

Burn-rate thresholds are derived, not memorized:

burn = (budget fraction × window hours) / alert window hours

14.4× — 0.02 × 30 × 24 / 1 — 2% of budget in 1 h — page on-call
6× — 0.05 × 30 × 24 / 6 — 5% of budget in 6 h — page on-call
1× — 0.10 × 30 × 24 / 72 — 10% of budget in 3 days — ticket, low priority

Pair every long window with a short confirmation window (1 h + 5 min, 6 h + 30 min) so a burst that already stopped doesn't page anyone.

RTO and RPO: targets vs measured reality

> [!IMPORTANT] An RTO is a cap, so compare it against p95 or worst-case downtime, never the mean. Comparing a cap to an average passes roughly half the time by construction. MTTR of 30 min against an RTO of 45 min proves nothing; the incidents that breach RTO are exactly the ones a mean hides.

When the check fails, the remedies are the same everywhere; only the product names change:

Health-checked failover — Route 53 / Application Recovery Controller — Cloud DNS + Global LB — Traffic Manager / Front Door
Automated recovery runbook — Systems Manager Automation — Cloud Workflows — Azure Automation
RPO = 0 storage — Multi-AZ synchronous replication — Spanner, regional PD sync — Cosmos DB strong, zone-redundant

Cost of downtime

Using the numbers from section 1:

$100M ARR / 8,760 h — $11.4K/h revenue exposure
200 engineers × $100/h loaded × 50% blocked — $10K/h productivity
$21.4K/h × 0.7 h (42 min) — ~$15K per incident
× 5 incidents — ~$75K

> [!TIP] That $75K is the budget ceiling for the fix. Reliability spend is justified only while marginal cost is below marginal downtime avoided: the "100% is the wrong target" argument in dollars instead of nines.

Key concepts

MTTI / MTTA: Mean Time To Identify / Acknowledge. The detection-and-ownership phase. MTTA specifically measures alert-fire → human ack and is the cleanest on-call health signal. Under the additive convention these sit BEFORE MTTR; under the Google SRE convention detection is nested inside MTTR as MTTD.
Total downtime per incident: MTTI + MTTR under the additive (ITIL/PagerDuty) convention. Under the nested (Google SRE) convention MTTR already spans the whole incident and nothing is added. Always state which you mean before summing.
Good / valid ratio: The universal SLI shape: good_events / valid_events. Load balancers expose totals and errors rather than successes, so in practice you compute ((total − errors) / total) × 100 — algebraically identical.
Metric-name portability: The formula never changes across clouds; only the metric identifiers do. Requests: RequestCount (AWS ALB) / request_count (GCP) / TotalRequests (Azure) / http_requests_total (Prometheus). Knowing the shape lets you write an SLI in a console you have never opened.
Burn rate derivation: burn = (budget fraction × window hours) / alert window hours. 14.4x is 0.02 × 30 × 24 / 1 — 2% of a 30-day budget in one hour. Deriving it beats memorising it, because the constant changes when the window does.
Cap vs. mean: RTO and RPO are caps. Validating them against MTTR (a mean) is a category error — use p95 or worst-case observed recovery time.

When to use it

Any interview that hands you incident timestamps and asks for a number — compute, then state your MTTR boundary convention
Writing an SLI in a provider you have not used before — start from good/valid, then look up the two metric names
Postmortem review — cross-check that timeline metrics and error-budget consumption agree
DR drill sign-off — compare p95 recovery time to RTO, not the mean
Building the business case for reliability headcount or infrastructure spend

Interview questions

We had 5 incidents. Cumulative detection delay 60 min, cumulative repair 150 min. What is our average downtime, and what does it imply?
Write me an availability SLI. I will not tell you which cloud we are on.
Our RTO is 45 minutes and our MTTR is 30 minutes. Are we compliant?
Justify spending on reliability. Give me a number.
Where does the 14.4x burn rate come from, and would it change?