Bazel

Overview

Overview

Bazel is Google's open-source build and test tool, released in 2015 as the externalized version of Google's internal Blaze system. It targets monorepos where conventional build tools break down: many languages, hundreds of services, thousands of engineers, and test suites measured in hours.

Why the Google origin matters

Google's monorepo, approximately 2 billion lines of code per Potvin and Levenberg (CACM 2016), runs on the same underlying design as Bazel. Every design choice is informed by operating at that scale for decades.

• Strong correctness guarantees — Steep learning curve
• High operational capability — Significant initial configuration investment

How Bazel differs from conventional build tools

• Hermetic builds by default — Each action runs in a sandbox with only its declared inputs visible. A Makefile, npm script or Gradle task can read any file in the repository; a Bazel action cannot
• Multi-language, single build graph — Java, Go, Python, C++, TypeScript, Rust, Scala, Kotlin, Swift, any language with Bazel rules. A Python service that generates proto code and a Go service that imports it connect in one BUILD file with full incremental correctness
• Remote caching and remote execution — A shared remote cache (GCS bucket, S3 or a dedicated cache service) is populated by the first build; laptops, CI runners and other pipelines get cache hits. Remote execution dispatches actions to a compute cluster, parallelizing the graph to its critical path
• Incremental correctness — All inputs are tracked, so running bazel test //... on an unchanged repo after a full CI build should produce 100% cache hits, unlike heuristics-based "runs tests that changed" approaches

> [!IMPORTANT] Hermeticity is what makes Bazel's cache trustworthy: the same declared inputs produce the same outputs.

The three concepts that govern everything

• Targets and BUILD files — Every buildable artifact (binary, library, test, Docker image) is a target declared in a BUILD file. Rules define what a target type does; targets specify which sources and dependencies belong to it
• Hermetic sandboxing — Each action runs with only its declared inputs; the sandbox enforces the declaration at the OS level
• Content-addressed caching — Artifacts are keyed by a hash of all their inputs. Same inputs: cache hit, no rebuild. Different inputs: cache miss, new artifact. A shared remote cache means one build populates it for everyone

Who uses Bazel in 2026

• Google — Invented it
• Stripe — One of the most-discussed public Bazel migrations
• Dropbox — Migrated its Python monorepo around 2020
• Pinterest — Both mobile and backend
• Jane Street — Via OCaml rules
• Databricks, Spotify, Canva, LinkedIn, Uber, Square/Block — Large multi-language codebases

> [!NOTE] The size and engineering investment required create self-selection toward well-resourced engineering organizations.

When NOT to use Bazel

• Small teams and repos (below 20 services or 30 contributors) — The learning curve and BUILD-file maintenance cost exceed the benefit; Turborepo or Nx is the right default for JS/TS monorepos
• JavaScript-only monorepos — Turborepo or Nx provide 80% of Bazel's caching benefit with 10% of the configuration cost
• Short total build times (full CI under 10 minutes) — Incremental builds and remote caching add minimal value

> [!TIP] Invest in Bazel when build time is a genuine productivity drain.

Key concepts

Packages, targets, and BUILD files

A package is a directory containing a BUILD or BUILD.bazel file. A target is a named buildable or testable artifact declared inside a BUILD file. Rules define what a target type does; targets instantiate rules with specific sources and dependencies. Example BUILD file for a Go service: Target labels follow the pattern //package:target. //services/payments:server refers to the server binary. :payments_lib is shorthand for //services/payments:payments_lib within the same package. deps declares explicit build-time dependencies. Bazel traverses these to build the dependency graph. Any dep not declared here is invisible to the build action. If the Go code imports it, the build fails with a clear, actionable error. visibility controls which other packages can depend on this target. "//visibility:public" allows any package. Restrict to specific packages for internal-only libraries to enforce architectural boundaries.

Hermeticity and sandboxing

Hermeticity means a build action's output is fully determined by its declared inputs. Same declared inputs produce byte-identical outputs on every machine, every time. This is what makes Bazel's cache trustworthy. Bazel enforces hermeticity by running each action in a sandbox. On Linux, this is a symlink forest (a directory containing only declared inputs) combined with Linux namespaces to isolate the process. On macOS, similar isolation via sandbox-exec. What the sandbox prevents: - Reading files not declared as inputs: the file is simply not present in the sandbox filesystem. - Writing to the source tree: outputs must go to a declared output location. - Accessing the network: disabled by default with --sandbox_block_network. - Reading environment variables not explicitly passed: Bazel controls the environment. .bazelrc sandbox settings: Common hermeticity violations and fixes: 1. Reading /etc/hostname or /etc/passwd in tests. Fix: use a test fixture; mock the hostname. 2. Embedding git revision via a shell call inside an action. Fix: use --workspace_status_command. Bazel runs a workspace status script, stamps the values as a stable status file, and declares it as an input to stamped rules only. 3. Reading $HOME for config files. Fix: set HOME to a controlled path in the action environment. 4. Using time.Now() or date in generated code. Fix: inject time via BUILD_TIMESTAMP from workspace status. Non-hermetic actions produce wrong cache hits: Bazel caches an artifact, serves it from cache on the next run, but the artifact is wrong because a hidden input differed. The failure: tests pass on CI, fail locally, with no code changes. Diagnosing: run with --sandbox_debug to log what a sandboxed action actually accesses.

Remote caching — end-to-end

Bazel supports a remote cache via the Remote Execution API (REAPI). The cache stores build artifacts keyed by action hashes. Configuring remote cache with BuildBuddy (recommended for teams without GCS): Configuring with GCS: Cache hit mechanics: 1. Bazel computes the action key: SHA-256 of command, input file digests, environment, and platform. 2. Queries ActionCache.GetActionResult(action_digest) on the remote. 3. Hit: fetch artifact from ContentAddressableStorage by digest. Skip the action. 4. Miss: execute action locally. Upload outputs to CAS. Write ActionResult to ActionCache. --remote_download_minimal: only download final outputs the developer needs locally. Intermediate artifacts stay in the cache. Critical for large builds where downloading everything would negate the cache speedup. What breaks remote caching: - Non-hermetic actions: same hash, wrong output. Cache is poisoned. - Toolchain version drift: different developer machines have different compiler versions not declared in the toolchain. Hashes differ per machine. - Timestamps in outputs: generated files embedding build timestamps get unique hashes every build. - ACTION_ENV pollution: --action_env=PATH passes the host $PATH. $PATH differs per machine. Cache keys diverge. Use --incompatible_strict_action_env. Cache hit rate measurement: A healthy hit rate is 80-95% for an active monorepo. Below 70% means hermeticity issues or environment pollution.

Starlark -- the BUILD file language

Starlark is a Python-like language used for BUILD files and rule definitions. It is intentionally restricted: no imports, no network access, no filesystem I/O, deterministic execution, no mutable global state after module initialization. Restrictions vs Python: no while loops, no classes, no import statement, no recursion, no file I/O. A custom rule in Starlark: Key concepts in rule authoring: - ctx.actions.run: declares a single sandboxed subprocess with declared inputs and outputs. - ctx.actions.run_shell: shell script action. - ctx.actions.write: creates a file from string content (no sandboxing needed). - depset: an efficient, deduplicating collection of files supporting lazy traversal. Critical for large dependency graphs -- never call list() on a depset inside a rule. - cfg = "exec": the tool runs on the execution machine, not the target machine. Relevant for cross-compilation. Starlark's restrictions guarantee that BUILD files are deterministic and analyzable without executing arbitrary code. The Bazel analysis phase can run without side effects or network calls.

Bzlmod -- modern dependency management (MODULE.bazel)

Bzlmod replaces the legacy WORKSPACE file with a module-based system modeled on Go modules. Enabled by default since Bazel 7.0. The Bazel Central Registry (registry.bazel.build) is the package index. Problems with WORKSPACE: no version resolution (last load wins, conflicts are manual), opaque procedural execution, no package registry. MODULE.bazel replaces WORKSPACE: Version resolution: Bzlmod uses Minimum Version Selection (MVS, same algorithm as Go modules). Diamond dependency conflicts are resolved automatically to the highest required version. No manual conflict resolution. MODULE.bazel.lock: records the resolved dependency graph. Commit it for reproducibility. Migration from WORKSPACE: (1) Enable --enable_bzlmod in .bazelrc. (2) Translate http_archive calls to bazel_dep() entries using registry.bazel.build names. (3) Convert toolchain setup to use_extension calls. (4) Run bazel mod tidy to generate the lockfile and missing use_repo declarations. (5) Verify with bazel build //... and remove WORKSPACE entries incrementally. Private registries: host a Bazel registry as a static file server matching the BCR layout for internal modules. Add to .bazelrc:

Gazelle -- automatic BUILD file generation

Gazelle is a Bazel BUILD file generator maintained by the rules_go team. Without Gazelle, every Go import must be manually declared in a BUILD file -- a maintenance burden that makes Go monorepos impractical at scale. Gazelle also supports Python and JavaScript. Setup in root BUILD.bazel: Running Gazelle: What Gazelle generates for Go: - Walks the repo tree, reads Go source files, extracts import paths. - Writes go_library, go_binary, go_test targets for each package directory. - Infers deps from import statements, maps import paths to Bazel labels via go_deps. - Handles cgo, build constraints, testdata directories. Gazelle directives -- inline instructions in BUILD files: CI integration to prevent drift: Pre-commit hook:

bazel query, cquery, and aquery

Bazel provides three query tools for inspecting and debugging the build graph. bazel query: evaluates BUILD files without configuration. Use for target relationships. bazel cquery: configuration-aware query. Use when targets have multiple configurations (cross-compilation, coverage instrumentation). bazel aquery: action graph query. Shows the actual commands that would be executed. Affected-target computation for CI (run only tests for changed code):

When to use it

• Monorepo with 4+ languages needing a unified build graph (Java, Python, Go, TypeScript together)
• Total CI test suite exceeds 15 minutes and incremental rebuilds are needed for developer productivity
• Remote caching required across developer machines and CI to share build work without repeating it
• Cross-language dependencies (proto definitions, generated code) need explicit dependency tracking
• Compliance or security requirement for hermetic, reproducible, auditable builds with provenance
• Migrating from a Blaze-based internal system to open-source tooling

Interview questions

• Explain Bazel's hermetic build model. What does hermetic mean, why does it matter, and what are the most common ways it breaks?
• Walk me through setting up Bazel for a Go monorepo from zero.
• What is remote execution in Bazel? How does it differ from remote caching, and when does it justify the operational cost?
• When should a team choose Bazel over Nx or Turborepo for their monorepo?
• How does Bzlmod change dependency management compared to WORKSPACE, and what are the migration steps?
• What are the most common Bazel pitfalls in production, and how do you diagnose them?

More DevOps topics

• App of Apps Pattern
• Argo CD
• Argo CD at Scale — Sharding, Performance Tuning, Failure Modes and Mistakes
• Argo Workflows
• Backstage
• Bare-Metal Provisioning
• Bitbucket Pipelines
• Build Performance and Caching for C++
• Building MCP APIs and Servers
• Buildkite
• Buildpacks, Jib & ko
• C++ Dependency Management with Conan and vcpkg