Flaky CI/CD Pipeline Failures
Systematically identify and eliminate flaky tests and infrastructure failures that erode CI trust.
Overview
Overview
Flaky CI/CD pipelines do more damage than they appear to. Once developers learn that failures are sometimes spurious, they merge anyway, and the pipeline stops being a quality gate while its infrastructure, cost and time stay the same.
> [!IMPORTANT] Fixing flakiness is a correctness requirement for the CI investment to pay off, not a convenience.
Three categories
- Test flakiness — The test code is non-deterministic; same commit passes some runs and fails others — Fix the root cause in the test
- Infrastructure flakiness — The runner is unreliable: network timeouts, disk full, contention on shared runners — Isolate jobs, avoid shared global state
- Dependency flakiness — Tests call external services (third-party APIs, staging databases) that behave inconsistently — Remove the dependency from the test
Test flakiness root causes
- Shared mutable state — A test pollutes global state a later test depends on, so ordering matters when it should not
- Async race — The test does not await an async operation and asserts on intermediate state
- Hardcoded ports — Parallel tests conflict (port 5432 is already in use)
- Time-dependent assertions — Asserts a timestamp equals "now" but the clock has moved
- Unordered collections — Non-deterministic ordering of sets or maps in comparisons
Detection needs history
A test with a 90% pass rate over 100 runs is flaky, even though any single run suggests it is fine.
- JUnit XML exported by most CI systems — Raw per-test results per run
- A database or a simple script aggregating them — Per-test pass rates over time
- Buildkite Test Analytics, Datadog CI Visibility — Flaky test tracking built in
Infrastructure isolation
Parallel jobs sharing a Docker socket can create port conflicts, and shared network namespaces let one job's traffic affect another's.
Dependency isolation
- Unit tests — Mock external services
- Integration tests — Local containers via the Testcontainers library, not shared staging
- End-to-end tests against real external APIs — Separate suite, longer schedule, explicit failure tolerance
> [!TIP] External end-to-end tests are smoke tests, not quality gates.
Key concepts
- Flaky Test
- A test that produces different results (pass or fail) on the same code without any changes to the code under test. The defining characteristic is non-determinism in test outcome.
- Hermetic Test Environment
- A test environment that is fully self-contained — all dependencies are controlled and deterministic, with no reliance on external network resources, shared state, or real-world timing.
- Test Quarantine
- A CI practice of disabling a flaky test from blocking merges while tracking it as known-flaky. The test continues to run but its failure does not fail the build, preventing CI trust erosion while the flake is fixed.
- Testcontainers
- A library for Java, Go, Python, and other languages that starts real Docker containers for databases, message queues, and other dependencies during test runs, providing isolation without mocking.
- Test Pass Rate
- The percentage of runs where a test passes, measured over time. A test with a 95% pass rate that never failed before today is flaky and should be quarantined even though it usually passes.
When to use it
- Diagnosing CI pipelines where failures seem random and do not reproduce locally
- Explaining test isolation and hermetic testing principles in engineering interviews
- Designing CI infrastructure that maintains trust even as the test suite grows
Interview questions
- Your CI pipeline has a 70% success rate over the past month, with failures seemingly random. How do you systematically identify whether the cause is test flakiness, infrastructure issues, or real bugs?
Sign up free to read the full topic
More Troubleshooting topics
Start free on Cariara · Coding problems · System design studio