Business Continuity and Disaster Recovery Across Clouds

Fail over a region or a whole cloud, and fail back, without breaking a 99.999% SLA

The BCP and DR program for microservices on AWS, Azure and Google Cloud: tier services by business impact, pick a DR strategy per tier, then detect, decide, fail over a region or a cloud, and fail back.

Overview

The interviewer asks for business continuity and disaster recovery for a set of microservices spread over AWS, Azure and Google Cloud, with a 99.999% SLA. The trap is to answer with a diagram of three clouds and a load balancer. The real question is a program: which services matter how much, how much data and time each may lose, what kinds of failure you plan for, who presses the button, in what order things move, how you come back, and how you know any of it works.

People often mix up three words. High availability (HA) keeps a service running through small, expected failures, such as a lost server or zone, usually automatically. Disaster recovery (DR) restores service after a large failure that HA cannot absorb: a region, a whole provider, or data that is destroyed or corrupted. Business continuity planning (BCP) is the wider plan that keeps the business operating: it decides which processes matter, sets the targets that DR must meet, and covers people, communication and suppliers as well as systems.

Five nines is a very small budget: 0.001% of a year is about 5.26 minutes. No human can be paged, read a dashboard and approve a region failover inside that. So the design splits: the few services on the 99.999% path run active in several places at once and lose a region or a cloud as a traffic shift, while everything else gets a cheaper strategy and a target the business has agreed to.

Scope it first

Does every service need 99.999%? — No: only the checkout path (Tier 0); the rest are tiered by business impact
What must we survive? — A zone, a region, a whole cloud, a bad change pushed everywhere, data corruption or ransomware, an identity outage
How much data may we lose? — None for payments; up to 1 minute for Tier 1; more for lower tiers
Who may trigger a region or cloud failover? — A human incident commander, against a trigger agreed in advance; zone failover is automatic
Are the three clouds used at once? — Tier 0 is active in one region of each cloud; Tier 1 has a warm standby in a second cloud
Do we need to fail back? — Yes, as a planned, tested move with no data loss
Is there audit or regulation? — Yes: drills must leave evidence of the RTO and RPO achieved

The interviewer listens for four things. Do you tier before you design, instead of buying active-active for everything? Do you name failures that redundancy does not cover, such as a bad deploy or corrupted data that replication copies everywhere? Do you give a failover order and say who decides? And do you explain how you stop two regions from both accepting writes, and how you come back?

Key concepts

Business impact analysis
RTO and RPO
Service tiers
Backup and restore, pilot light, warm standby, active-active
Correlated failure
Fencing and epochs
Idempotency keys
DNS TTL and client caching
Failback
Game days and chaos engineering

Functional requirements

Keep the Tier 0 checkout path serving through the loss of a zone, a region or a whole cloud, without a manual failover
Fail a Tier 1 service over to its standby in another cloud within its RTO, with a human approval step
Restore any data store to a point in time before corruption, from backups that an attacker with admin rights cannot delete
Show, per service, its tier, targets, readiness (replication lag, reserved capacity, quota headroom) and last drill result
Fail back to the original region as a planned move with no data loss
Record every drill and real failover with the RTO and RPO achieved, as audit evidence

Non-functional requirements

Tier 0: 99.999% a year (target chosen), about 5.26 minutes of downtime
Tier 1: RTO 15 minutes, RPO 1 minute (targets chosen); Tier 2: RTO 4 hours, RPO 15 minutes; Tier 3: RTO 24 hours, RPO 24 hours
Failover uses only actions that work while the failed region or cloud is down: no step may depend on it
At most one writer per data set at any time, enforced by the data store, not by convention
Backups are immutable for their retention period and kept outside the production accounts and identity domain
Every Tier 0 and Tier 1 plan is drilled at least once a quarter (a policy we chose)

Interview questions

What is the difference between BCP, DR and HA?
How do you set RTO and RPO for 40 services?
Why can a 99.999% service not rely on failover?
Your data replicates to three clouds. Are you protected against ransomware?
Walk through failing over a Tier 1 service to another cloud.
How do you prevent split brain?
What happens to the writes that were not replicated?
How do you fail back?
How do you know the DR plan works?