Data Partitioning
How to split one dataset across many machines: choosing the partition key, hash and range placement, hot keys, queries that cross partitions, and moving data when the cluster grows.
Overview
Data partitioning means splitting one large dataset into smaller pieces, called partitions or shards, and storing them on different machines. Each machine then holds, and answers for, only part of the data. It is how a database grows past what one machine can store or write.
Do not confuse it with replication. Replication keeps full copies of the same data on several machines, to add read capacity and survive failures. Partitioning gives each machine different data, which adds storage and write capacity. Real systems usually do both: each partition is itself replicated.
The hard part is not the splitting. It is choosing the partition key, the field that decides which partition a row belongs to. That choice decides which queries stay on one machine, whether the load is even, and how painful it is to add machines later. It is also hard to change once the data is large.
What you will be able to do
- Choose a partition key for a design, and say which queries it helps and hurts.
- Explain hash, range and directory partitioning, and when each one fits.
- Spot a hot key and spread it with caching or salting.
- Handle lookups by a second field, and queries that span every partition.
- Explain how data moves when you add a machine, and why fixed partitions move so little.
Key concepts
- Horizontal and vertical partitioning
- Partition key
- Hash partitioning
- Range partitioning
- Directory and lookup
- Hot keys and salting
- Scatter-gather
- Co-location
- Rebalancing
Interview questions
- How would you choose a partition key for a chat application?
- Hash or range partitioning: how do you choose?
- One user has 50 million followers and their partition is overloaded. What do you do?
- Your users table is sharded by user_id. How do you look a user up by email?
- What happens when you add a node, and how do you keep data movement small?
- Why are joins, unique constraints and transactions harder after sharding?
- When would you not partition at all?
- How is data partitioning related to consistent hashing and replication?
Sign up free to read the full topic
More System Design topics
- Core Banking System
- CRDTs: Conflict-Free Replicated Data Types
- Customer Onboarding and KYC
- Customer Support Chatbot System
- Data Labeling and Annotation System
- Data Lakes and Lakehouse Table Formats
- Data Pipeline Orchestration and Backfills
- Data Quality, Contracts and Lineage
- Data Warehousing and Dimensional Modeling
- Database Indexes
- Databases
- Dealing with Contention
Start free on Cariara · Coding problems · System design studio