Data Versioning and Dataset Management

Git-like versioning for large datasets and data lakes using DVC, Delta Lake, Apache Iceberg, and LakeFS.

Overview

Why data versioning matters

A model retrained weeks later can behave completely differently if its dataset changed silently. Datasets are gigabytes or terabytes, live in object storage and are mutated in place, so plain Git cannot track them. Data versioning tools separate a lightweight metadata pointer from the heavy payload.

DVC — Git companion for data files — .dvc pointers in Git, bytes in a remote — Reproducible pipelines with a content-hash cache
Delta Lake — Storage layer for Spark lakes — JSON transaction log in _delta_log/ — ACID writes, time travel, schema enforcement
Apache Iceberg — Open table format spec — Snapshots in a metadata tree — Hidden partitioning, partition evolution, many engines
LakeFS — Git semantics over object storage — Zero-copy branches and commits — Branch, validate, merge whole datasets

DVC

DVC replaces each tracked file or directory with a small .dvc pointer committed to Git. The bytes live in a remote (S3, GCS, Azure Blob, SSH or a local path), and dvc pull fetches the exact content-addressed version the pointer names.

Pipelines: dvc.yaml records each stage's inputs, outputs and command.
Smart reruns: dvc repro re-executes only stages whose dependencies changed, a content-hash build cache.
Result: the Git commit plus the DVC pointer fully determine a training run.

Delta Lake

Delta Lake brings ACID transactions to Spark data lakes. Every INSERT, UPDATE, DELETE or MERGE appends a JSON entry to _delta_log/, and readers and writers coordinate through that log for snapshot isolation without a central lock manager.

Time travel: SELECT * FROM events TIMESTAMP AS OF '2024-01-01' or VERSION AS OF 42 replays the log to an earlier snapshot.
Schema enforcement: rejects writes that would break downstream consumers.
Schema evolution: allows explicitly approved changes through MERGE schema options.

Apache Iceberg

Iceberg is a table format specification, not a storage engine. Its metadata tree makes features possible that Hive's directory scanning cannot support.

Hidden partitioning: partition values are stored with each data file, not in the directory path, so partitioning can change without rewriting data (ALTER TABLE ADD PARTITION FIELD).
Snapshot isolation: readers see a consistent view while writers commit new snapshots.
Compaction: rewrite_data_files merges small streaming files into larger ones for batch reads.

LakeFS

LakeFS adds Git-like branch, commit and merge directly on S3, GCS or Azure Blob without copying data. A branch is a zero-copy pointer, so creating one is instant at any size.

> [!TIP] LakeFS speaks the S3 API, so Spark, Pandas and DVC work against it unchanged.

Reproducing a training run

Pin three coordinates: the Git commit, the data version (DVC pointer or Delta/Iceberg snapshot ID) and the environment (Docker image digest or conda lock file).

> [!IMPORTANT] Teams that store all three in the experiment tracker can rebuild any run deterministically, which is essential for debugging production regressions and for regulatory audits.

Interview questions

Walk me through how you would make a model training run fully reproducible using DVC.
How does Apache Iceberg handle partition evolution, and why is it superior to the Hive partition model?
Describe a data branching strategy for a team running multiple parallel ML experiments using LakeFS.
What are the trade-offs between Delta Lake, Apache Iceberg, and Apache Hudi for ML use cases?

More MLOps & LLMOps topics

Start free on Cariara · Coding problems · System design studio