Blue-Green Deployment That Resulted in Split-Brain Sessions

Incomplete session store migration meant users were authenticated on green but not on blue, causing random logout loops during traffic shifting.

Overview

A blue-green deploy that also rotated the session signing key logged users out over and over as the load balancer bounced them between environments.

What happened

Blue signed session tokens with HMAC-SHA256 and a 32-byte key. Green used the same algorithm with a newly generated key, and the load balancer sent 50% of traffic to green.

Why it broke

Each environment trusted only its own key, so any request that crossed environments failed validation and destroyed the session.

Version — Old — New
Algorithm — HMAC-SHA256 — HMAC-SHA256
Signing key — Existing 32-byte key — Newly generated key (planned rotation)

The fix

Green was changed to dual-read: accept tokens signed with either key, instead of discarding the old one at once.

> [!IMPORTANT] Key rotation and a traffic split do not mix unless both sides accept both keys for the whole transition window.

Key concepts

Split-brain session: A state where different servers validate the same user session differently, causing inconsistent authentication behavior depending on which server handles a request.
Dual-read key validation: Accepting tokens signed by both the old and new signing keys during a transition window, maintaining backward compatibility while new tokens are issued with the new key.
Key rotation: Replacing a cryptographic signing key with a new one. In session systems, requires a transition period where both keys are accepted to avoid invalidating active sessions.

When to use it

Designing session management during major version deployments
Explaining dual-read/dual-write patterns for compatibility during transitions
Discussing blue-green deployment prerequisites

Interview questions

How do you safely rotate session signing keys in a blue-green deployment without logging users out?