Monorepo Builds at Scale: Dependency Graphs, Test Selection and Change Impact

Build only what changed, test only what it can break, and keep main green

Design the build and test system for a monorepo that thousands of engineers commit to: hermetic builds, a dependency graph, remote caching and execution, running only the tests a change can affect, and a merge queue that keeps main green.

Overview

Thousands of engineers commit to one repository. It holds C++ services, Python tools and ML training code, Go and Rust services, and firmware cross-compiled for several chips. Running every test on every change takes hours and costs about as much compute as the rest of engineering combined, so most changes must run only a small slice of the tests. The hard part is choosing that slice without ever skipping a test the change could break.

The design rests on one idea: make the build a pure function. If every build step declares all of its inputs and produces the same output from the same inputs, then three things follow. A step's result can be cached under a hash of its inputs and shared by everyone (remote caching). Steps can run on any machine in a pool (remote execution). And the build system knows the full dependency graph, so it can answer exactly which targets a change can affect (change impact). Test selection, the merge queue and the safety nets all build on that graph.

The interviewer is listening for whether you see the trap in each speed-up. A cache that is fast but keyed on the wrong inputs serves wrong binaries. Test selection that is fast but misses a dependency lets main go red silently. A merge queue that is fast but treats flaky failures as real rejects good changes. Every shortcut needs a safety net that catches what it misses.

Scope it first

How many engineers and changes? — 4,000 engineers, about 4,000 merges a day, each change pushed about 3 times before merging: 12,000 CI runs a day
Which languages and targets? — C++, Python, Go, Rust services; Python ML code needing GPUs for some tests; firmware cross-compiled for 3 chip families
How big is the test suite? — 60,000 test targets, about 1 CPU-minute each on average: about 1,000 CPU-hours to run everything once
What must stay true about main? — Main always builds and passes its gating tests; a broken main blocks everyone
How fast must presubmit answer? — Our target: the median change gets its results in under 15 minutes
One build system already? — Assume we can require one hermetic build system with declared dependencies (Bazel, Buck2 or Pants) across the repository
Where does compute run? — A pool on our own hardware plus cloud capacity for peaks; GPU and firmware toolchains on separate worker pools

Key concepts

Hermetic builds
Action graph
Content-addressed cache (AC + CAS)
Remote execution (REAPI)
Cache poisoning
Reverse dependencies (rdeps)
Test selection
Flaky test quarantine
Merge queue batching
Sparse checkout

Functional requirements

Build and test any target from a clean checkout with the same result on any machine (hermetic, reproducible)
Cache every build and test action by a hash of its inputs and share results across engineers and CI
Run actions on a remote pool with separate platforms for CPU, GPU and each firmware toolchain
For each change, compute the affected targets and run every affected test in presubmit, or everything when impact cannot be computed safely
Gate merges through a merge queue that tests changes on top of the latest main, in batches, and isolates a failing change
Run all tests on main periodically, find the culprit of any new failure and offer a one-click revert
Detect flaky tests, quarantine them from the merge gate, and keep running them so real breaks are still found
Request reviews from the owners of every path a change touches

Non-functional requirements

Presubmit latency: median change results in under 15 minutes, 95th percentile under 45 minutes (our targets)
Correctness: a cache hit must equal what a fresh build would produce; any doubt means a miss, never a guess
Main stays green: fewer than one red main per week caused by something the gate should have caught (our target)
Merge throughput: the queue must sustain the peak of about 750 merges an hour with time to spare
Cost: presubmit compute grows with the size of changes, not with the size of the repository
Security: only trusted CI may write to the shared cache; developer machines read only

Interview questions

Why does everything here depend on hermetic builds?
Walk me through what happens when a build asks the remote system for one action.
How can the shared cache get poisoned, and how do you recover?
How do you decide which tests a change must run?
Graph selection still runs 36,000 tests for a base-library change. What then?
Design the merge queue. Why not just require branches to be up to date?
How do you handle flaky tests without hiding real failures?
What safety nets stop test selection from letting main go red unnoticed?