Insurance Platform: Five Nines Across Clouds

HA, BCP/DR and failover for an insurer on AWS, Azure and Google Cloud

Design HA, DR and failover for an insurer on AWS, Azure and Google Cloud, so reporting a loss, proving coverage and taking payments meet 99.999%, even in a hurricane while a cloud region is down.

Overview

An insurer already runs its products: quoting, binding, policy administration, billing, claims, fraud and pricing models, and the portals and APIs that agents, brokers and partners use. This problem is not about designing those features (the Insurance Platform: Policies and Claims topic does that). It is about keeping them running: about 60 microservices spread over AWS, Azure and Google Cloud, with a 99.999% availability target on the paths customers depend on, and regulators who expect the insurer to prove it can keep its most important services within limits through severe disruption.

Two facts make the problem hard. First, 99.999% allows about 26 seconds of failure in a 30-day month. A DNS failover alone takes longer, so the design cannot rely on noticing a failure and switching; the important paths must keep answering while a cloud fails. Second, insurance has its own worst day: a hurricane or hailstorm can make loss reports jump tens of times over (forty in this design's assumptions), in exactly the area where a nearby cloud region may also be impaired. And behind every path sits the policy system of record (often a mainframe or a vendor policy suite) that runs in one place and is nowhere near five nines.

Scope it first

Which paths need 99.999%? — Reporting a loss (first notice of loss, FNOL), proving coverage (ID cards, policy lookup), and taking payments. Not everything
Where does the policy system of record run? — On the insurer's own mainframe, with its own second data center; it is not moving
Which clouds, and why three? — AWS and Azure run the customer paths; Google Cloud holds data, models, backups and a tie-breaking vote
Where are the customers? — US home, auto and small business, plus a UK and an EU subsidiary whose data stays in their jurisdiction
What is the worst day? — A landfall day: about 100,000 loss reports instead of 2,500, with a cloud region near the storm impaired
Who moves card money? — A payment service provider (PSP); card numbers never enter our systems

Key concepts

Important business services
Impact tolerance
Severe but plausible scenarios
Error budget
Static stability
Active-active across clouds
Change data capture
Fencing epochs
Catastrophe surge
Concentration and exit risk

Functional requirements

Accept a loss report in either cloud and return a claim reference at once, whether or not the policy core is reachable
Show proof of insurance (ID card, coverage summary) from a copy of the policy kept in each cloud
Take premium payments and send claim payments exactly once, through a payment provider and a ledger
Keep quoting, binding and the broker and partner APIs available, with a slower recovery than tier 0
Move traffic away from a failing cloud or region, and back, under a controlled, recorded decision
Run nightly batch (billing, renewals, reporting) to completion even when a site is lost mid-run
Keep every jurisdiction's personal data inside its allowed regions, in every cloud

Non-functional requirements

99.999% of FNOL submissions, proof lookups and payment requests succeed, measured per request each month (a target we chose)
No single cloud provider, region or zone failure stops a tier-0 service: each active cell can carry 100% of tier-0 peak alone
No payment, refund or bind takes effect twice, through any retry, failover or failback
Impact tolerances chosen per important business service (for example FNOL intake: no disruption longer than 1 hour) hold in every severe but plausible scenario tested
Card numbers never reach our services; the PCI scope stays at the payment provider's hosted fields
The insurer can leave any one cloud provider with a tested, written plan

Interview questions

Why not run all 60 services at five nines?
What does 99.999% actually allow, and how does that change failover?
The policy system of record runs on a mainframe in one data center. How do you get five nines in front of it?
How do you make sure a payment is never made twice across two clouds?
A hurricane hits and the nearest cloud region fails at the peak. Walk me through it.
Why three clouds and not two?
What do regulators expect, and how does the design answer it?
How do fraud and pricing models fail over, and what if they are simply down?
How do you keep EU and UK data in place while failing over between clouds?