Retail Platform: Five Nines Across Stores and Clouds

Checkout and in-store sales that never stop, even on Black Friday

Design HA, DR and failover for a retailer whose stores, AWS, Azure and Google Cloud keep checkout and in-store sales at 99.999%, through store outages, region loss, payment provider failures and Black Friday.

Overview

A large retailer sells in two places at once. In its stores, cashiers scan items at checkout lanes, a point-of-sale (POS) system prices the basket and the shopper pays by card or cash. Online, shoppers browse, search, fill a cart and check out on the web or an app, and pick the order up in a store or have it shipped. Behind both sit the same services: products and prices, promotions, inventory, orders, payments, fulfilment and a recommendation model.

This page sets a target of 99.999% (five nines) for two things: completing a checkout online, and completing a sale at a store lane. Five nines allows about 26 seconds of downtime in a 30-day month. No single region, cloud or network link is that reliable, and a cloud failover alone takes longer than 26 seconds. So the design has to keep selling while parts are down. Stores sell from local copies of prices and stock and send the sales up later. Checkout runs in more than one region and can move to a second cloud. A payment provider outage moves new payments to a second provider. Under heavy load, the platform turns off recommendations and other extras long before it slows checkout.

The hard parts are not throughput. They are the decisions that must stay correct while systems are cut off from each other: how much a store may sell by card while offline, how online and in-store sales share the same units without overselling, and how a retried checkout or payment never charges a customer twice.

This page is the retail case. The generic store (catalog, search, cart, checkout saga) is the E-Commerce Platform topic, a sale where a crowd arrives at one instant is the Flash Sale System topic, the general multi-cloud architecture is the Microservices at 99.999% on AWS, Azure and Google Cloud topic, and DR runbooks are the Business Continuity and Disaster Recovery Across Clouds topic.

Questions to ask first

What must reach 99.999%? — Completing an online checkout, and completing a sale at a store lane. Browsing and recommendations may degrade
How big? — 800 stores with 12 lanes each; 2,500 store sales per store a day; 300,000 online orders a day
How big is the peak? — Black Friday: store sales 3 times a normal day, online orders 10 times
Which clouds? — Checkout active in two AWS regions; a warm copy of the checkout core on Azure; analytics and the recommendation model on Google Cloud
Must stores sell without the cloud? — Yes, including card payments within limits the business sets
Can an item be oversold? — Not for store pickup; a small, measured oversell rate is tolerated for shipped orders and fixed by a backorder
Payments? — Two payment providers (PSPs); card data never touches our own servers

The building blocks on each provider

The design is provider-neutral; this table is one way to build each block. The text names real products, so the names are not translated for other clouds.

Global entry and failover — Route 53 failover records with health checks — Front Door priority routing with health probes — Cloud Load Balancing, Cloud DNS — A multi-provider DNS service
CDN and edge cache — CloudFront — Front Door caching — Cloud CDN — Akamai, Fastly
Checkout services — EKS — AKS — GKE — Kubernetes
Orders and stock pools — Aurora PostgreSQL global database — Azure Database for PostgreSQL with a read replica — Cloud SQL or AlloyDB for PostgreSQL — PostgreSQL with streaming replication
Event stream — Amazon MSK — Event Hubs (Kafka protocol) — Pub/Sub — Apache Kafka
Store hub — Outposts servers — Azure Local — Google Distributed Cloud connected — Two small servers running k3s
Keys and secrets — KMS, Secrets Manager — Key Vault — Cloud KMS, Secret Manager — HSM, Vault

What you will be able to do

Turn 99.999% into an error budget and show why stores need offline selling and checkout needs more than one region to meet it.
Design a store that keeps selling offline, with store-and-forward, offline card limits and idempotent upload.
Share stock between stores and online with pools, safety stock and idempotent reservations so pickup orders are never oversold.
Fail payments over between two providers without a double charge, and keep card data out of most systems.
Plan Black Friday: pre-scale, shed load in a fixed order, and fail checkout over to another region or cloud.

Key concepts

Error budget
Store hub and lane offline database
Store-and-forward
Offline card limits
Stock pools and safety stock
Idempotent reservations
Payment orchestration
PCI DSS scope reduction
Load shedding and degradation levels
Cells and warm standby

Functional requirements

Sell at every store lane by card, cash, gift card and loyalty, with the store's prices and promotions
Keep selling at a lane when the store network, the store hub or the cloud is unavailable, within offline limits
Forward sales made offline to the cloud exactly once, in order per lane
Check out online with cart, price, stock reservation, payment and order creation
Reserve stock for buy-online-pickup-in-store (BOPIS) and shipped orders without overselling pickup orders
Pay through either of two payment providers, switching when one fails
Turn features off and back on by degradation level during overload or incidents

Non-functional requirements

Online checkout success: 99.999% of minutes in a month without a checkout outage, about 26 seconds of downtime (a target we chose)
Store sale availability: 99.999% for completing a sale at a lane, counting offline sales as successes (a target we chose)
Checkout p99 under 2 s at the Black Friday peak, excluding the payment provider's own time (a target we chose)
Region failover for checkout in under 1 minute; failover to the second cloud in under 15 minutes (targets we chose)
Orders: RPO of a few seconds across regions, with every lost order recoverable from the client retry, the payment record or the event stream
No customer charged twice for one order; no pickup order confirmed without a unit behind it
Card data never stored or logged by the retailer's own services (keeps most systems out of PCI DSS scope)

Interview questions

How can a store keep selling when the internet is down?
What risk does an offline card payment carry, and how do you limit it?
How do you stop online pickup orders from overselling what is on the shelf?
PSP A times out. Do you retry on PSP B?
What turns off first on Black Friday when load is too high?
Your checkout region fails. What happens to the orders in flight?
Why not let the CDN fail checkout over to another origin?
Why a warm second cloud instead of active-active across clouds?
How does PCI DSS shape this design?