Partitioning & Sharding
How to split a large dataset into pieces: range, hash and directory partitioning, choosing a shard key, fixing hotspots, living with cross-shard queries, and moving data without downtime.
Overview
Partitioning splits one large table into smaller pieces, called partitions, by the value of a column: the partition key. When the pieces live on different machines, it is usually called sharding, and each piece is a shard. The goal is the same either way: no single machine has to store all the data or handle all the writes.
A single database can grow a long way, first on a bigger machine and then with read replicas. Replicas copy all the data, so they add read capacity but not write capacity or storage. Sharding is the step that adds both, and it is the hardest step to undo, which is why interviewers probe it.
Almost every sharding problem comes back to one decision: the key. It decides which queries touch one shard and which touch all of them, whether load spreads evenly or piles onto one machine, and how painful it is to add machines later. This chapter covers the ways to split data, how to pick the key, and what breaks when you get it wrong.
What you will be able to do
- Explain the difference between partitioning inside one database and sharding across many.
- Compare range, hash and directory partitioning, and say where each fits.
- Choose a shard key for a given product and defend it against hotspots and cross-shard queries.
- Explain why hash mod N reshuffles data and how consistent hashing and logical shards avoid it.
- Describe how to move to more shards without downtime.
Key concepts
- Partitioning vs sharding
- Range partitioning
- Hash partitioning
- Consistent hashing
- Directory and logical shards
- Shard key
- Hotspots and salting
- Scatter-gather queries
- Online resharding
Interview questions
- How do you choose a shard key?
- Why is hash mod N a problem, and how does consistent hashing help?
- How do you handle a hotspot in a sharded database?
- How would you add shards without downtime?
- What makes cross-shard queries expensive, and how do you avoid them?
- What is the difference between partitioning and sharding, and when is partitioning in one database enough?
- How do you keep a value unique across shards, such as an email address?
Sign up free to read the full topic
More Databases & SQL topics
Start free on Cariara · Coding problems · System design studio