Configuration Management

SRE Workbook approach to configuration as code: validation, staged rollout, blast radius containment, and the pitfalls that make config changes the leading cause of production incidents.

Overview

Configuration is a first-class artifact

SRE Workbook Chapters 14 and 15 treat configuration as an engineering artifact: not a collection of YAML files someone edits at 2am, but a versioned, validated, staged deployment pipeline with automated rollback.

The central insight: "A change in configuration is a change in behavior." Configuration changes cause the majority of major production incidents; not code, not hardware, but human-written configuration that was not validated before reaching production.

> [!IMPORTANT] If rolling back a bad configuration takes longer than restarting the service, the configuration system is not safe for production.

Four principles from the Workbook

1. A schema — Every field typed, range-bounded and validated before acceptance; a system that accepts arbitrary strings will eventually accept a malformed value that crashes the service
2. Staged changes — No change reaches 100% of instances at once: 1% canary, automated validation, 10%, 50%, 100%
3. Rollback faster than restart — Otherwise the configuration system is unsafe
4. Versioned with code — A binary at version N expects a configuration schema versioned and compatible with version N

Key concepts

Configuration as code: Configuration is stored in version control, reviewed like code, validated by automated tooling, and deployed through the same staged pipeline as binary changes.
Schema validation: Every configuration field is typed and range-bounded. Invalid configuration is rejected before deployment, not after it reaches production instances.
Configuration canary: The new configuration is applied to 1% of instances first. Metrics are compared against the 99% running the previous config. Rollout proceeds only if no regressions are detected.
Configuration drift: When production configuration diverges from version-controlled configuration through ad-hoc manual edits. Drift is the primary reason why "it works in staging" incidents occur.
Feature flags: Configuration-driven toggles that enable or disable code paths without a binary deployment. Separate code deployment from behavior change, enabling gradual exposure and instant rollback.

When to use it

"What percentage of incidents are caused by configuration changes?" — most industry data says 30-70%
Designing a configuration system for a service that operates across multiple environments
"How do you safely roll out a configuration change?" — the staged rollout pattern
Justifying investment in configuration validation tooling vs. manual review

Interview questions

Why do configuration changes cause more production incidents than code changes, and how do you prevent it?