Backups That Were Never Actually Completing

A 14-month-old backup job was logging success but writing zero bytes — discovered only when a recovery was attempted.

Overview

A daily backup job reported success every morning for 14 months while writing nothing, and a disk failure cost 6 days of data.

What happened

Every morning Slack said "Backup completed successfully," and no alert ever fired. Then the database host had a disk failure, and the team found the backup S3 bucket empty.

Why it broke

S3 answered the PUT with a 404. The backup script treated any exit without an exception as success, and posted the success message regardless of the HTTP response code.

Nothing else would have caught it:

No restore drill had ever been run.
No monitoring watched the bucket's object count or size.

Impact

Time the job silently failed — 14 months
Only surviving backup — Weekly RDS automated snapshot
Age of that snapshot at the failure — 6 days
Data lost — 6 days

The fix

A backup counts only once the object is verified and a restore has been proven.

Key concepts

Backup verification: Periodically performing an actual restore from backup to verify the backup is complete, uncorrupted, and restorable within the RTO.
Dead man's switch: An alert that fires if an expected event does NOT happen within a time window — for backups, alerting if no new backup object appears in S3 within 25 hours.
Restore drill: A scheduled exercise where the team actually restores from backup to a test environment and verifies data integrity, typically monthly or quarterly.

When to use it

Explaining why backup verification is as important as backup creation
Discussing SRE toil and the danger of "green" dashboards that are not actually checked
Designing backup and restore testing procedures

Interview questions

How do you design a backup monitoring system that actually validates backups are working?