Microservices at 99.999% on AWS, Azure and Google Cloud

HA, DR and failover for N services across three clouds

Design N microservices for 99.999% across AWS, Azure and Google Cloud: the availability math, a global traffic layer, cells, data across clouds, static stability, tiering and failover.

Overview

The owner of a platform says: we run N microservices and want 99.999% availability, on AWS, Azure and Google Cloud. What do we deploy, and how do we fail over? This is a design question about failure, not about features. The interviewer wants to see that you can turn a number into an architecture.

Start with what the number means. 99.999% ('five nines') leaves about 5.26 minutes of downtime a year, or about 26 seconds a month. One slow DNS failover can spend that. So the answer cannot be 'add a standby and fail over'. It has to be: many independent copies of the critical path, all serving at once, with traffic moving away from a broken copy faster than a person can react.

Three ideas carry the whole design. Serial parts multiply: a request that needs a load balancer, a cluster and a database, each at 99.99%, is at best about 99.97%. Parallel parts combine: two independent copies that each fail 0.03% of the time are both down far less often. And independence is the hard part: two copies that share a DNS provider, a deploy pipeline, an identity service or a bad config change fail together, and the math stops applying. Multiple clouds are one way to buy independence. They are not the only way, and they are expensive, so a strong answer applies them only to the services that need them.

Scope it first

Five nines for what? Every service, or a user journey? — The critical journeys (sign in, browse, check out); measured per request, not per server
How many services, and how many on the critical path? — 40 services; 12 are tier 0 (needed for those journeys)
Traffic? — 30,000 requests a second at peak, about 10,000 on average
Must writes survive the loss of a whole cloud? — Yes for orders and payments; other data may lose a little
Any data residency rules? — Customer data stays in its home geography (an overlay, see below)
Is multi-cloud a requirement or an option? — A requirement for tier 0; we will say where it does not pay

The interviewer listens for four things. Do you do the math and show why one cloud's services in series cannot reach five nines? Do you tier the N services instead of making all of them active-active on three clouds? Do you treat data as the hard part and name the consistency you give up? And do you know how failover actually behaves: health check intervals, DNS caching, clients that hold old connections, and spare capacity in the surviving cells?

Key concepts

Error budget
Composite availability
Cell-based architecture
Shuffle sharding
DNS failover and TTL
Static stability
Workload identity federation
RTO and RPO tiers
Split brain and fencing
N+1 headroom

Functional requirements

Serve the tier-0 journeys (sign in, browse, check out) from any of the three clouds
Move traffic away from an unhealthy cell, region or cloud without a deploy
Keep accepted orders and payments when a whole cloud is lost
Deploy every service to every cloud from one source of truth
Give each workload a cloud identity without long-lived keys
Let operators drain a cell, fail over a tier and fail back on purpose

Non-functional requirements

Availability target (our choice): 99.999% of tier-0 requests succeed each month, measured at the edge
Tier 1 target: 99.99%; tier 2: 99.9% (our choice)
Traffic shifts away from a failed cell within seconds for new requests (retries at the edge), and within about 2 minutes for DNS-cached clients
Tier-0 writes: RPO 0 for orders and payments; RPO of seconds allowed for other data
No runtime dependency from one cloud's cells on another cloud's control plane
Each cell can lose one zone and stay inside its latency target

Interview questions

Why can't we reach 99.999% inside one cloud by picking services with high SLAs?
Walk me through what happens when a whole cloud region fails.
How do you tier N microservices, and what does each tier get?
How do you keep data consistent across clouds?
What is static stability, and how does it apply here?
How do you avoid split brain when failing over a database?
Is multi-cloud worth it?
How does the design change for manufacturing, AI, healthcare or insurance?