Data Lakes and Lakehouse Table Formats
How Apache Iceberg, Delta Lake and Apache Hudi turn a folder of Parquet files in object storage into a table with atomic commits, snapshots, time travel and row-level deletes, and what each one does differently.
Overview
A data lake is a large store of files, usually in object storage such as Amazon S3, Google Cloud Storage or Azure Blob Storage, in open formats like Parquet. It is cheap, it grows without limit, and many engines (Spark, Trino, Flink, Python) can read the same files. The catch is that a folder of files is not a table. There is no commit, no schema anyone enforces, and no way to change one row.
A table format fixes that by adding a layer of metadata next to the data files. The metadata says exactly which files make up the table right now, and every change to the table is a new version of that metadata, made visible by one atomic step. Readers follow the metadata instead of listing folders. This gives a lake the things people expect from a database table: all-or-nothing writes, consistent reads, safe concurrent writers, schema changes, time travel and row-level deletes. A lake with such a layer is often called a lakehouse.
Three open formats do this: Apache Iceberg, Delta Lake and Apache Hudi. They share the core idea and differ in how they store the metadata and how they commit. This topic explains each one from its own specification and documentation. For how the data gets there, see the Batch vs Stream Processing topic; for a full design that versions training data with Iceberg snapshots, see the Training Data Lake problem.
What you will be able to do
- Explain what goes wrong with a plain folder of files, and why listing is not a commit.
- Draw how Iceberg, Delta Lake and Hudi each record a table version and commit it atomically.
- Describe optimistic concurrency between writers, and when a failed commit may be retried.
- Plan the upkeep: snapshot expiry, compaction of small files, and cleanup of orphan files.
- Choose between copy-on-write and merge-on-read for updates and deletes.
Key concepts
- Data lake on object storage
- Table format
- Snapshots
- Atomic pointer swap
- Optimistic concurrency
- Time travel and expiry
- Hidden partitioning
- Compaction
- Copy-on-write vs merge-on-read
Interview questions
- Why is a folder of Parquet files in S3 not a table?
- Walk me through what happens when Iceberg commits an append.
- How does Delta Lake stop two writers from both committing version 10?
- When can a failed commit be retried, and when must it fail?
- Copy-on-write or merge-on-read for a table fed by a CDC stream?
- What is hidden partitioning and why does it matter?
- What maintenance does a lakehouse table need?
- Iceberg, Delta Lake or Hudi: which would you choose?
Sign up free to read the full topic
More System Design topics
- Content Delivery Network
- Core Banking System
- CRDTs: Conflict-Free Replicated Data Types
- Customer Onboarding and KYC
- Customer Support Chatbot System
- Data Labeling and Annotation System
- Data Partitioning
- Data Pipeline Orchestration and Backfills
- Data Quality, Contracts and Lineage
- Data Warehousing and Dimensional Modeling
- Database Indexes
- Databases
Start free on Cariara · Coding problems · System design studio