Healthcare Platform: Five Nines Across Clouds

HA, BCP and DR for clinical services on AWS, Azure and Google Cloud

Run a health system's clinical services on AWS, Azure and Google Cloud at 99.999%: tiered services, one writer per data domain, no lost orders, no duplicate charts, safe degraded modes and ransomware recovery.

Overview

A health system runs its clinical software as about 40 microservices: chart and results viewing, orders, medication administration, emergency department (ED) tracking, scheduling, imaging, clinical AI such as triage and imaging models, and FHIR APIs for partner hospitals, labs and apps. They run on AWS, Azure and Google Cloud. The board wants 99.999% availability and the privacy office wants patient data protected through every failure.

Five nines is about 26 seconds of downtime a month, or 5.3 minutes a year. No single cloud region promises that, and no person can notice a failure, decide and act inside it. So the design must keep working through a failure automatically, and it must be clear about which services the five nines covers. This page does not design the record itself; the Healthcare Platform (EHR) topic does. Here we take an EHR-like set of services and run it across clouds.

Three things make healthcare different from a generic multi-cloud design. First, a wrong answer can hurt a patient: two clouds that both accept writes can create two charts for one person, and a lost medication order is a safety event. Second, HIPAA adds rules: an agreement with every cloud that holds patient data, a contingency plan with backups, disaster recovery and an emergency mode, and audit controls that keep working in a failover. Third, the most likely disaster is not an earthquake but an attacker who encrypts everything, including the replicas.

Questions to ask first

Which services carry 99.999%? — Tier 0 only: reading charts, results, allergies and the medication list, the ED board, and accepting orders and medication events
What role does each cloud play? — AWS holds the writers; Azure serves reads all the time and can take over writes; Google Cloud is the witness, the immutable backup vault and the clean room
What counts as available? — A tier 0 read answered, or an order accepted and durable; read-only mode counts for reads, not for edits
RPO and RTO? — Orders and medication events: nothing acknowledged is lost. Other clinical writes: up to 5 s. Reads move in seconds; full write failover within 15 minutes (targets we chose)
Regulation? — HIPAA; a business associate agreement (BAA) with each cloud; patient data only in services the agreement covers, in US regions
Is ransomware in scope? — Yes: assume an attacker gets admin rights in one cloud and the identity provider

What you will be able to do

Say which services the 99.999% covers and show, with arithmetic, why reads must already be served from two clouds rather than failed over by DNS.
Keep one writer per data domain with a witness and fencing epochs, so a failover never leaves two clouds writing the same chart.
Make orders and medication events impossible to lose with client-made IDs, a quorum intake log and idempotent replay.
Plan the clinical side: read-only mode, downtime procedures, temporary patient IDs and back-entry after recovery.
Recover from ransomware into a clean room from locked backups, with keys and agreements in place in every cloud.

Key concepts

Service tiers
Error budget
Single writer per domain
Fencing epoch and witness
Idempotent order IDs
Read-only degraded mode
Downtime procedures
MPI split-brain
Customer-managed keys
Clean-room recovery

Functional requirements

Serve tier 0 reads (chart, results, allergies, medication list, ED board) from AWS or Azure at any time
Accept orders and medication administration events during any single-cloud failure, and lose none
Fail writes for each data domain over to Azure on a declared failover, and back again
Put each tier into the right mode (normal, read-only, write failover, downtime) and show it to users
Keep FHIR partner APIs, imaging and clinical AI working or degrading safely during a failover
Keep immutable backups outside the production clouds' admin reach and restore them into a clean room
Feed downtime viewers in every unit so care continues if every online path fails

Non-functional requirements

99.999% monthly availability for tier 0 reads and order intake (a target we chose); 99.99% for tier 1, 99.9% for tier 2
RPO 0 for acknowledged orders and medication events; RPO 5 s for other clinical writes (targets we chose)
Reads move to the other cloud in seconds without a person; write failover within 15 minutes (targets we chose)
Never two writers for one data domain, even during a network partition between clouds
Patient data only in services covered by each cloud's BAA, in US regions, encrypted with customer-managed keys
Every read and write audited, in every mode, into write-once storage replicated to a second cloud
Tier 0 restored from locked backups within 24 hours of a ransomware attack (a target we chose)

Interview questions

What exactly does 99.999% apply to, and how do you measure it?
Why not run active-passive with DNS failover between clouds?
How do you make sure an order is never lost in a failover?
How do you avoid split brain between AWS and Azure?
What happens to patient registration during a failover?
What does HIPAA require of this design?
How would you recover from ransomware that reached one cloud's admin accounts?
How does clinical AI fail over safely?