Testing for Reliability

The SRE testing hierarchy — unit through production — and how each layer targets a different class of reliability failure.

Overview

Testing to build confidence in production

SRE Book Chapter 17 frames testing not as quality assurance but as a confidence-building exercise for production. Traditional QA asks "does the feature work?"; SRE testing asks "does the system remain reliable under real production conditions?"

> [!IMPORTANT] "If you haven't tested your recovery process, you don't have a recovery process." DR tests, chaos engineering and game days all validate this layer.

The six-layer hierarchy

Unit — Algorithmic bugs in individual components — Run in milliseconds; should number in the thousands
Integration — Contract violations between components, such as service A expecting a specific schema from B — Verifies the schema
System — Emergent failures that appear only when all components interact — Whole stack in a production-like environment
Stress — The capacity cliff and the failure mode — Find the breaking point in a controlled test, not during a Black Friday spike
Canary — Failures that appear only with real user data — New binary on a fraction of live traffic
Disaster recovery — Recovery procedures that no longer work — Deliberately destroy components; an untested DR procedure is not a DR procedure

Key concepts

Unit test: Validates a single function or class in isolation. Catches algorithmic bugs. Should be the most numerous, fastest, and cheapest tests in the suite.
Integration test: Validates that two or more components interact correctly — typically service-to-service API contracts, database schema adherence, or event format correctness.
System test: Runs the full service stack in a production-like environment. Catches emergent failures that only appear when all components interact simultaneously.
Stress test: Applies load beyond the service's designed capacity to identify the breaking point, the failure mode, and the recovery behavior. Answers "what happens when we hit our traffic spike?"
Canary test: Routes a small fraction of live production traffic to the new binary. Real users, real data — catches failures that staged tests miss because production traffic is unpredictable.
DR test: Deliberately destroys or disables a component and validates that documented recovery procedures produce recovery within RTO. An untested DR procedure is not a DR procedure.

When to use it

"How do you ensure a service stays reliable after changes?" — walk through the testing hierarchy
Justifying investment in integration or system tests when unit coverage is already high
"What testing does your team do before releasing to production?"
Designing a testing strategy for a new service with a reliability SLO

Interview questions

Why can't you rely on unit tests alone to validate service reliability?
How do you design a stress test for a service whose peak traffic you've never seen?