Checksums and Data Integrity

How systems notice that data changed when it should not have: CRCs and cryptographic hashes, end-to-end checks, Merkle trees for comparing replicas, content addressing, ETags, HMACs against tampering, and scrubbing.

Overview

A checksum is a short value computed from a block of data, such as 4 bytes for a CRC-32 or 32 bytes for SHA-256. Store or send it with the data, compute it again later, and compare: if the two values differ, the data changed. Checksums are how storage systems, networks and replication protocols notice corruption that nothing else reports.

That corruption is real and quiet. A study of 1.53 million disk drives in production storage systems over 41 months found more than 400,000 checksum mismatches: data that came back wrong with no error from the disk. Network paths corrupt packets in ways that link-level checks miss. Memory, firmware and software bugs damage data in between.

The same idea, with stronger functions, does more than catch accidents. Cryptographic hashes name data by its content (Git, content-addressed stores), Merkle trees let two replicas find their differences quickly, ETags let HTTP caches skip downloads, and HMACs detect deliberate tampering.

What you will be able to do

  • Choose between a CRC, a fast hash, a cryptographic hash and an HMAC for a given job.
  • Design end-to-end integrity checks and say which gaps per-hop checks leave.
  • Explain how Merkle trees let replicas find differences in a few steps.
  • Use content addressing and ETags in a design, and know their limits.
  • Plan scrubbing and repair so corruption is found before the last good copy is gone.

Key concepts

  • CRC32C
  • Cryptographic hashes
  • Silent data corruption
  • End-to-end integrity
  • Merkle trees
  • Content-addressable storage
  • ETags and conditional requests
  • HMAC
  • Scrubbing and repair

Interview questions

  • CRC, MD5, SHA-256 or HMAC: how do you choose?
  • What is silent data corruption, and how would you protect a storage system against it?
  • How do Merkle trees help replicas find differences?
  • How do ETags work, and how do they prevent lost updates?
  • Design end-to-end integrity for a file upload service.
  • Why use HMAC instead of hashing the secret together with the message?
  • What is content-addressable storage, and what are its trade-offs?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio