Caching

How caches make reads fast and take load off databases: where they sit, how to read and write through them, what to evict, how to keep copies fresh, and what goes wrong at scale.

Overview

A cache is a small, fast store that keeps copies of data that is slow or expensive to get, so the next request for it can be answered quickly. The slow source is usually a database, but it can also be another service, a heavy computation or a server far away. When the cache has the data, that is a hit. When it does not, and the request has to go to the source, that is a miss.

A cache does two jobs. It cuts latency, because reading nearby memory is much faster than running a query or crossing a network. And it cuts load, because every hit is a read the database never sees. The second job is often the bigger one: if the cache answers 99% of reads, the same database can carry about a hundred times as many reads as it could alone.

The price is that a cache holds copies, and copies go out of date. Most hard questions about caching are versions of one question: when the source changes, when does the copy change, and what does a user see in between? This chapter covers where caches sit, the numbers behind hit ratios, the patterns for reading and writing, eviction, invalidation and the races it causes, and the failures that show up at scale.

What you will be able to do

  • Place caches at the right layers of a design and say what each one holds.
  • Work out the mean read time and the database load from a hit ratio, and the hit ratio a latency target needs.
  • Choose between cache-aside, read-through, write-through, write-behind and refresh-ahead.
  • Explain eviction and invalidation, and the race that leaves an old value in the cache.
  • Handle stampedes, hot keys and a cache outage, and choose between Redis and Memcached.

Key concepts

  • Cache layers
  • Hit ratio
  • Cache-aside
  • Write-through and write-behind
  • LRU, LFU and TTL
  • Invalidation
  • Stale sets and leases
  • Stampedes
  • Hot keys
  • Redis and Memcached

Interview questions

  • Where would you add caches to a read-heavy web app, and what would each one hold?
  • Walk through cache-aside. What can go wrong when the data changes?
  • The hit ratio falls from 99% to 95%. What happens to latency and to the database?
  • How do you choose a TTL?
  • LRU or LFU? Which eviction policy would you set on a Redis cache?
  • How do you keep a cache consistent with the database?
  • What is a cache stampede, and how do you prevent it?
  • One product key gets 500,000 reads a second. What do you do?
  • Redis or Memcached?
  • What happens when the cache cluster goes down?

Sign up free to read the full topic

More System Design topics

Start free on Cariara · Coding problems · System design studio