On-Premises and Hybrid Cloud at Five Nines

Two data centers, a witness site and three clouds at 99.999%

Run N microservices in your own data centers at 99.999%: a metro pair with synchronous data, a witness in a third site, redundant links to AWS, Azure and Google Cloud, and the clouds for DR and burst.

Overview

Some workloads cannot simply move to a public cloud. Patient or payment data may have to stay in a country or under the company's own control (data sovereignty). A factory line or trading system may need answers in a few milliseconds from machines next to it. A large steady estate of servers, a mainframe or a GPU cluster may already be paid for and cheaper to keep. The question is how to run N microservices on that estate at 99.999% availability, which leaves about 5.3 minutes of downtime a year, while still using AWS, Azure and Google Cloud where they help.

The cloud gives you zones, regions and instant capacity. On-premises, you build those yourself. A failure domain is a set of things that fail together: one rack shares a power strip and a top-of-rack switch, one data center shares a building, a utility feed and a flood plain, and one metro area shares an earthquake zone and a power grid. Five nines means no single failure domain below the metro may take a service down, and each failure the design absorbs must cost seconds. The 99.999% counts every failed tier-0 request, with no exclusions; a metro-wide disaster is the one failure this design does not absorb, so in the year it happens the target is missed (the Failover section shows by how much).

The design in this topic: two owned data centers in one metro area about 60 km apart by fiber (the metro pair), running every critical service active in both, with synchronous data between them; a small witness in a third site that decides which side keeps writing if the two lose sight of each other; redundant private links to all three clouds; and the clouds used for asynchronous disaster recovery copies, burst capacity for stateless services, and anything not bound to the premises.

Scope it first

Why on-premises? — Sovereign data and low-latency plant systems must stay in our sites; stateless front ends may burst to the cloud
How many services and how busy? — 40 microservices, about 4,000 cores at peak in total
Availability target, and for what? — 99.999% of tier-0 requests succeed, measured at the edge with no exclusions; 99.95% for internal tools
How many sites do we own? — Two data centers in one metro, plus space for a witness in a third location
RPO and RTO? — RPO 0 and RTO under 1 minute for losing one site; RPO under 5 minutes for losing the whole metro
Which clouds, for what? — AWS, Azure and Google Cloud: DR copies, burst capacity, and services with no residency rule
How long does new hardware take? — About 16 weeks from order to racked and in service

An interviewer listens for four things. Do you name the failure domains and the one each layer survives? Do you place sites by the round-trip time synchronous replication can afford? Do you add a third vote so a cut link cannot cause split brain (two sides both accepting writes)? And do you do the capacity math for a site that must carry everything alone, with hardware that takes months to arrive?

Key concepts

Failure domains
Metro pair and distant DR
Synchronous vs asynchronous replication
Quorum witness
Split brain and fencing
Redundant interconnects and BGP
Hybrid control planes
Capacity lead time
Cloud bursting
Error budget

Functional requirements

Run 40 microservices active in both owned data centers, with sovereign data stored only in approved sites
Keep serving with no data loss when any rack, room, power feed, network device or one whole data center fails
Choose one writer automatically when the two data centers lose contact, and stop the other side from writing
Keep an asynchronous copy of every critical data set outside the metro, in a cloud region the policy allows
Burst stateless services into AWS, Azure or Google Cloud when on-premises capacity runs short
Fail a whole site over and back on demand, as a planned drill, without user-visible errors beyond the budget
Track hardware, capacity headroom and every change against the error budget

Non-functional requirements

Availability: 99.999% of tier-0 requests succeed each year (our target), about 5.3 minutes of full downtime equivalent
Site failure: RPO 0 and RTO under 60 s inside the metro (our target)
Metro disaster: RPO under 5 minutes, RTO under 1 hour, declared by a person (our target); the downtime still counts against the 99.999%, so a metro loss misses that year's target by design
Latency: synchronous commits add at most about 1 ms, which caps the fiber route between the pair near 100 km
Capacity: each data center alone carries 100% of peak at no more than 70% utilization
Residency: no sovereign record is written outside the approved sites, including in DR and burst paths

Interview questions

Why can't two data centers alone give you five nines safely?
How far apart should the two data centers be?
How do you connect the sites to AWS, Azure and Google Cloud without a single point of failure?
Should the Kubernetes control plane for on-premises services live in the cloud?
How do you plan capacity when hardware takes four months to arrive?
Your RTO for a metro disaster is one hour. How is that five nines?
What uses up a five-nines error budget in practice, and what do you do about it?
Which data gets synchronous replication?