Dagger

Dagger is a programmable CI/CD engine that lets you define pipelines as code using Go, Python, or TypeScript SDKs, backed by a BuildKit-powered DAG executor with automatic caching. It eliminates the gap between local development and CI by running the exact same containerized pipeline everywhere.

Overview

Dagger treats CI/CD pipelines as ordinary programs rather than configuration files. You write pipeline logic in Go, Python or TypeScript with a typed SDK instead of a provider-specific YAML DSL.

The SDK compiles to a GraphQL description of the build graph. The Dagger engine runs that graph in containers on BuildKit, so every step is reproducible and isolated regardless of the host.

Local equals CI

The gap between local development and CI is mostly a tooling problem. A YAML pipeline runs only on the provider that understands it; a function runs anywhere the Dagger CLI is installed.

> [!TIP] Debugging a failing pipeline no longer means pushing commits and waiting for a runner. You reproduce the exact failure locally in seconds.

The building blocks

Content-addressed caching — Every DAG node is hashed from its inputs; change one file in a monorepo and only nodes that transitively depend on it re-run
Modules and the Daggerverse — Reusable pipeline functions as versioned packages, shared like application libraries rather than YAML anchors and template repos
Services — Sidecars such as PostgreSQL scoped to a run: started when first needed, torn down automatically at the end
Dagger Cloud — A distributed cache: cache volumes and layer digests are pushed after a successful run and pulled by any runner

Caching versus path filters

How reuse is decided — Automatically, by input hashes — Manually specified
Failure mode — Re-runs exactly what changed — Easy to get wrong

> [!IMPORTANT] Services make database integration tests as easy as unit tests: declare the service, bind the test binary to its hostname, run. No Compose files, pre-provisioned databases or cleanup scripts.

With Dagger Cloud plus local-equals-CI parity, teams get fast pipelines without pre-warming runners or maintaining custom AMIs.

Key concepts

Dagger Engine and GraphQL API: The Dagger engine is a long-running daemon that exposes a GraphQL API over a Unix socket. Every operation a client SDK performs, mounting a directory, running a container, publishing an image, is expressed as a GraphQL query or mutation. The engine translates these operations into a BuildKit LLB graph and executes them. Because the API is GraphQL, the engine can resolve only the subgraph relevant to the requested result, automatically skip nodes with cached outputs, and fan out parallel branches without any explicit parallelism annotations from the caller. The dag import is the root client bound to the engine connection. Every chained method call appends a node to the local DAG description. No container actually runs until a terminal method like stdout or sync is awaited, at which point the entire accumulated graph is sent to the engine in one GraphQL call and executed with maximum parallelism.
Pipeline as a Dagger Function: In Dagger a pipeline is a regular function decorated with SDK-specific annotations. The Dagger CLI introspects these functions and exposes them as sub-commands. Callers invoke them with dagger call function-name --arg value. The function receives typed arguments, builds a DAG of container operations, and returns a typed result. This design means pipelines are testable, composable, and callable from any context: a developer terminal, a CI step, or another Dagger function in a different module. The function signature uses only Dagger types: Directory for a source tree, File for a build artifact, CacheVolume for a named persistent cache. The engine knows how to serialize and deserialize these types across module boundaries, enabling composition without shared filesystem state.
BuildKit-backed DAG and Caching: Dagger uses BuildKit as its execution engine. BuildKit represents every operation as a node in a directed acyclic graph where edges are data dependencies. Each node is identified by a content-addressed hash of its inputs: the base image digest, the command arguments, mounted file digests, and environment variable values. When you run a pipeline twice with identical inputs, BuildKit finds the cached result for every node and returns it without re-executing any container. Change one source file and only the nodes that depend on that file are re-executed. All sibling branches that do not depend on the changed file are served from cache instantly. The node_cache cache volume is keyed by name. Its contents persist in the BuildKit content store across pipeline runs. npm ci reads from node_modules and only fetches packages not already present, dramatically reducing test cycle time. In Dagger Cloud this volume is synchronized to a distributed store so any runner picks up the warm cache.
Dagger Modules and Daggerverse: Dagger Modules are versioned, self-contained packages of Dagger functions published to the Daggerverse registry or any Git repository. A module declares its own Dagger SDK dependency and exposes typed functions that other modules or CLI callers can invoke by name. You install a module with dagger install github.com/some-org/some-module@v1.2.3 and call its functions with dagger call function-name --arg value. Module functions can accept and return Dagger types like Directory, Container, Secret, and Service, enabling deep composition across module boundaries. The Daggerverse acts as a package registry where the community publishes modules for common tasks: building Docker images, deploying to Kubernetes, running security scans, publishing NPM packages. Teams can pin exact versions, fork and override individual functions, and compose modules without YAML template repositories or copy-paste.
Services for In-Pipeline Databases and Caches: Dagger Services are long-running container sidecars bound to a specific pipeline run. A service starts when the first dependent node is evaluated, stays alive while dependent nodes run, and is torn down automatically when the run ends. Services are referenced by a stable hostname that is only routable within the same pipeline execution. This makes it trivial to run integration tests against real database engines without pre-provisioned test environments or Docker Compose orchestration outside the pipeline. The WithServiceBinding call registers the postgres service under the hostname db. The test binary connects to postgres://postgres:test@db:5432/testdb exactly as it would connect to a real database server. The service is never exposed outside the pipeline and requires no cleanup step because Dagger tears it down when the run graph completes.
Secrets Management: Dagger has a first-class Secret type that represents sensitive values. Secrets are passed into pipelines from the host environment, from files, or from external secret stores, and they are scrubbed from all log output, GraphQL responses, and cache keys. A secret can be mounted into a container as a file or injected as an environment variable, but its plaintext value is never serialized into the DAG description sent to the engine. The env:REGISTRY_TOKEN syntax tells the CLI to read the value from the host environment variable and wrap it in a Secret object before sending it to the engine. The engine stores only the hash of the secret value for cache key computation. Log output that would contain the plaintext is automatically replaced with a redacted marker.
Cache Volumes and Local-Equals-CI Parity: Cache volumes are named persistent mounts that survive across pipeline runs within the same BuildKit content store or across runners when Dagger Cloud is active. They are declared with client.CacheVolume("name") and mounted into containers with WithMountedCache. The cache is keyed by the volume name and optionally by a sharing mode: shared (default), private (one writer), or locked (exclusive access). Local-equals-CI parity means the same dagger call command that a developer runs on their laptop produces identical results in a GitHub Actions runner because the containerized environment is always the same. The only variable is which nodes are cache hits. This eliminates an entire class of bugs caused by differences between the local environment and CI, such as different system library versions, missing tools, or different shell initialization. With Dagger Cloud the go-mod and go-build volumes are pushed to the distributed cache after each successful run. The next runner that picks up the workflow pulls the warm cache before executing any containers.

When to use it

You are building a complex multi-stage Go, Python, or TypeScript application where integration tests require real database or cache sidecars and you want those tests to run identically on developer laptops and in CI without separate Docker Compose files.
Your monorepo contains many independent services and you need fine-grained DAG-based caching so that changing service A does not re-run the full build and test suite for service B.
You are publishing reusable pipeline components for a platform engineering team to consume across dozens of repositories and you want versioned, typed, composable functions rather than copy-pasted YAML templates.
Your CI pipelines are slow because cache invalidation is coarse-grained and you want BuildKit-level content-addressed caching tied to actual input digests rather than manually maintained cache keys.
You want to run the exact same pipeline locally for debugging without setting up a local GitHub Actions runner or maintaining a separate local build script that diverges from CI over time.
You are adopting Dagger Cloud and want a distributed cache layer that warms any runner that picks up a pipeline without requiring sticky runner groups or pre-warmed AMIs.
You need to integrate secrets into pipeline steps without exposing them as environment variables that appear in logs, using Dagger's typed Secret type that is scrubbed from all output.

Interview questions

How does Dagger caching work and what causes cache invalidation?
How does Dagger compare to GitHub Actions for a Go service with database integration tests?
Explain the Dagger Module system and how Daggerverse enables pipeline composition.
How do Dagger Services work for in-pipeline databases and what are the lifecycle guarantees?
When should you choose Dagger over a YAML-based CI system and what are the tradeoffs?

Dagger

Overview

Overview

Local equals CI

The building blocks

Caching versus path filters

Key concepts

When to use it

Interview questions

More DevOps topics