Build Performance and Caching for C++

Where C++ build time actually goes and how to get it back — measurement first, then ccache and sccache, distributed compilation, include-graph reduction, and the link step that refuses to parallelize.

Overview

How build performance work goes wrong

Someone reads that unity builds or precompiled headers are fast, enables them globally, breaks incremental builds for the whole team, and cannot say whether it helped because there was never a baseline. The discipline is the same as any performance work: measure, find the dominant cost, attack that, re-measure.

Measurement is easy now

-ftime-trace (Clang 9+) — Chrome-tracing JSON per object: frontend and backend time per source, per template instantiation, per included header
ClangBuildAnalyzer — Aggregates traces: slowest files to parse, most expensive instantiations, costliest headers overall
-ftime-report (GCC) — Coarser, but separates frontend from backend
ninja -d stats, ninja -d explain — Internal timings, and the definitive answer to "why did my incremental build rebuild that"

What measurement usually shows

The frontend dominates, with enormous redundant work across TUs — Compilation caching (ccache, sccache turn redundancy into a hash lookup) and include-graph reduction (include-what-you-use, forward declarations, pimpl for widely included types)
The link is a serial tail: waits for the last object, largely single-threaded in traditional linkers, exhausts memory — mold: a drop-in flag removing a bottleneck no build parallelism touches

Distribution: the most confusion

distcc, icecream, sccache distributed — Compilation of individual TUs — Remote toolchain matches local, or the toolchain is shipped
Bazel and Buck2 remote execution — Arbitrary actions from a hermetic graph, cached by action digest; subsumes distribution and caching — Hermeticity

> [!NOTE] Those build systems are covered in the Bazel and Monorepo Build Systems topics. The point here is not to conflate the two mechanisms.

Every intervention has a cost

Unity builds — Incremental speed for clean-build speed
Precompiled headers — A rebuild-the-world dependency for parse savings
LTO — Link time for runtime
Caching — Correctness risk for speed, if sloppiness is configured carelessly

> [!TIP] The interviewer probes method, not tool trivia: baseline, measurement, hypothesis, targeted change, re-measurement, and the trade-off of each. A candidate who presents any of these as free has not run one in production.

Key concepts

-ftime-trace: Clang 9 and later. Emits a Chrome-tracing JSON file per translation unit alongside the object file, breaking time into frontend, backend, per-template-instantiation, and per-header. The input ClangBuildAnalyzer consumes. GCC equivalent is the coarser -ftime-report.
ClangBuildAnalyzer: Aggregates -ftime-trace output across a whole build. Workflow is --start, build, --stop, --analyze, or --all followed by --analyze. Reports slowest files to parse and codegen, most expensive template instantiations, slowest functions, and most expensive headers with inclusion chains.
ccache direct mode: Hashes the source, its include files, and the compiler options, storing a manifest that maps them to a cached result. Avoids running the preprocessor. Falls back to preprocessor mode (compile with -E, hash the output) when direct mode cannot be trusted, notably when time macros are present.
CCACHE_BASEDIR and hash_dir: base_dir rewrites absolute paths under a given prefix to relative before hashing, so the same source in different checkout directories hits. hash_dir controls whether the current working directory is hashed; disabling it is the complementary knob. Neither fixes absolute paths already baked into debug info — that needs -fdebug-prefix-map or -fdebug-compilation-dir.
depend mode: ccache depend_mode (CCACHE_DEPEND) and its sccache analogue skip the preprocessor entirely and rely on compiler-generated dependency information from -MD or /showIncludes. Cheaper on a cache miss because the preprocessor never runs twice; lower hit rate, and -MD drags system headers into the dependency set.
Unity build: Concatenating several source files into one translation unit so the shared header closure is parsed once. CMake exposes it as the UNITY_BUILD target property with UNITY_BUILD_MODE (BATCH or GROUP), UNITY_BUILD_BATCH_SIZE, UNITY_BUILD_UNIQUE_ID for anonymous-namespace safety, and SKIP_UNITY_BUILD_INCLUSION to exclude a file. Destroys incremental granularity.
ThinLTO versus full LTO: Full LTO merges all bitcode into one module for whole-program optimization: best code, serial and memory-bound link. ThinLTO keeps per-module summaries, performs cross-module import decisions from those summaries, and parallelizes and caches the backend, delivering most of the runtime win at a small fraction of the link cost.
Link as the serial tail: The link cannot start until the last object file exists and traditional linkers are largely single-threaded, so it is unaffected by build parallelism. It is also the peak-memory phase. mold parallelizes aggressively; on a 16-core benchmark linking Clang 19 it takes 1.35 seconds against 5.20 for lld and 42.07 for GNU ld.

When to use it

Clean CI builds have grown past the point where a pull request gets feedback within the reviewer attention span
Incremental local builds recompile far more than the change warrants, which is a dependency-graph problem rather than a raw speed problem
Build machines swap or get OOM-killed during the link phase, which points at -j sizing rather than at the compiler
A cache exists but hit rate is unknown or low, so the money spent on it is buying nothing
Before adding LTO, unity builds, or precompiled headers, so there is a baseline the change can be judged against

Interview questions

Your clean build takes 45 minutes. Walk me through how you would find out where the time is going and what you would do first.
Explain how ccache decides on a hit, and name the things that defeat it along with the fixes.
Compare distcc, icecream, sccache distributed mode, and Bazel remote execution. What actually distinguishes them?
When would you use unity builds and precompiled headers, and what does each actually cost?
How do you choose -j, and why is the link phase special?