Data Integrity

SRE Book Chapter 26 — reading what you wrote: checksums, soft deletes, backup validation, and why data loss is harder to recover from than downtime.

Overview

What you read is what you wrote

SRE Book Chapter 26 frames the whole topic: "The goal of data integrity is to ensure that data is what you expect it to be — that what you read is what you wrote."

That is distinct from availability (the data is accessible) and durability (the data is persisted). A system can be available and durable while silently corrupting what it stores.

> [!IMPORTANT] "Data loss is almost always harder to recover from than an availability failure." A 4-hour outage ends at hour 4; lost customer records may never come back.

Three categories of loss and corruption

Physical corruption — Bit rot, hardware failure, cosmic-ray bit flips — Checksums at the storage layer, RAID and replication at the hardware layer
Logical corruption — Application bugs writing bad data, migrations transforming data incorrectly, bugs in backup and restore — Application-layer checksums, migration testing, backup validation through test restores
Deletion — User deletes, admin mistakes, software bugs deleting the wrong data — Soft deletes with retention, delayed hard deletes, point-in-time recovery

The asymmetry

Service unavailable for 4 hours — Back to full operation at hour 4
10% of customer records lost — May never fully recover; the data is gone

> [!TIP] This asymmetry justifies investing more in data integrity than availability: an extra 9 of availability is worth less than a comprehensive backup validation program.

Key concepts

Data integrity: The guarantee that data read from a system is identical to data written to it. Distinct from availability (data is accessible) and durability (data is persisted).
Checksum: A hash of the data content stored alongside the data. On read, the stored checksum is recomputed and compared — a mismatch indicates corruption between write and read.
Soft delete: Marking a record as deleted (deleted_at timestamp) without removing it from storage. Enables recovery from accidental deletion. Hard delete occurs after a retention period (e.g., 30 days).
Backup validation: Regularly performing test restores from backup to verify that the backup contains correct, uncorrupted data and that the restore procedure works. An untested backup is not a backup.
Point-in-time recovery (PITR): The ability to restore a database to its state at any specific timestamp within a retention window. Requires continuous transaction log shipping, not just periodic snapshots.

When to use it

"How do you ensure data written to your service is the same data read back?"
"What is your backup strategy and how do you know it works?"
Designing a storage system that must handle both hardware failures and application bugs
"What is the difference between durability and integrity?"

Interview questions

How do you design a data integrity strategy for a service that cannot tolerate data loss?