Canary Deployment Problems

Fix canary traffic routing, statistical validity, and monitoring gaps that make canaries unreliable.

Overview

Overview

Canary deployments cut deployment risk, but they need more care than most teams expect. A naive canary (5% of traffic to the new version, watch 10 minutes, promote) often fails silently: it passes, the full rollout proceeds, and the bug it was meant to catch hits 100% of traffic.

The statistical validity problem

Detecting a 1% error-rate increase with 95% confidence needs roughly 7,000 requests per version.

  • Canary weight — 1%
  • Total requests needed for 7,000 canary requests — 700,000
  • Service throughput — 1,000 requests per minute
  • Time to meaningful statistical power — 700 minutes, about 11 hours
  • Typical observation window — 10 to 30 minutes

> [!WARNING] With a window that short, the canary passes whether or not the new version is buggy.

  • Raise canary weight to 10 to 20% — Cuts observation time by 10 to 20x
  • Sequential testing (the Wald sequential probability ratio test) — Decides with fewer samples when the effect is large
  • Threshold of "no detected regression at all" — Conservative; needs no statistical significance

Cohort consistency

If a user's first request hits the canary and the second hits stable, they may see inconsistent state, and auth tokens from the new version may be invalid in the old one.

Compare canary to stable, not to a static threshold

  • Static threshold of 1% — 0.3% — Looks fine
  • Ratio to stable — 0.3% — 0.05% — 6x regression

> [!IMPORTANT] Alarm when canary divided by stable exceeds 2x or 3x; apply the same ratio logic to p99 latency.

The stuck canary

When the deployment system cannot decide (for example, the health check flaps), two versions run side by side indefinitely.

  • Session state — Users see two behaviors
  • Debugging — Which version caused this bug?
  • Database migrations — Schema must stay compatible with both

> [!TIP] Set a maximum canary duration: after X minutes with no decision, promote or roll back.

Key concepts

Statistical Significance in Canaries
The minimum sample size required to detect a given regression with a given confidence. For 1% error rate regression at 95% confidence, approximately 7,000 requests per variant are needed — often requiring hours at low canary weights.
Relative Error Rate Comparison
Comparing the canary error rate against the stable version's error rate (as a ratio), rather than against a static absolute threshold. Detects regressions even when baseline error rates are low.
Sticky Session Routing
A canary routing mechanism that ensures a user consistently sees the same version (canary or stable) throughout their session, preventing state inconsistency from mixed-version request handling.
Header-Based Canary
Routing specific requests (identified by header values) to the canary version, regardless of traffic percentage. Used for beta user groups or internal testing without affecting random traffic sampling.
Maximum Canary Duration
A deployment safety circuit breaker that forces a promotion or rollback decision after a maximum observation period, preventing indefinite mixed-version running states.

When to use it

  • Diagnosing why a canary that passed did not prevent a production regression
  • Explaining statistical sampling requirements for meaningful canary analysis in system design interviews
  • Designing a canary deployment system with automated promotion and rollback decisions

Interview questions

  • You run a canary at 5% traffic for 15 minutes and see no elevated error rates, so you promote to 100%. Immediately after full rollout, error rates spike to 2%. How could the canary have passed while the version was actually broken?
  • How do you implement a canary that is statistically meaningful for a service handling only 10 requests per minute?

Sign up free to read the full topic

More Troubleshooting topics

Start free on Cariara · Coding problems · System design studio