High Availability System Design

HA eliminates single points of failure: Load Balancer → Multiple App Servers in different AZs → Primary DB + Replica + Backup/Monitoring/Failover. No single point of failure, multi-zone deployment, automatic failover, health checks. HA vs single server: shared traffic, multi-zone, DB replica, health checks trigger failover, better uptime.

Overview

Designed to survive failure

High Availability (HA) means designing a system to operate continuously with minimal downtime, even when individual components fail. Production systems are designed to SURVIVE failure, not avoid it.

Eliminate every single point of failure

Any component that, if it fails, takes down the entire system is a single point of failure (SPOF). HA means every critical component has redundancy.

Server crash (CPU, memory, OS) — Full outage
Deployment gone wrong — Full outage
Disk full — Full outage
Network partition — Full outage
Datacenter power event — Full outage

The five HA components

1. Load balancer — Distributes traffic across healthy instances; health checks remove unhealthy ones — Itself redundant: active-active pairs or a cloud-managed LB; AWS ALB, GCP GLBC, Azure Application Gateway are multi-AZ by default
2. App servers in multiple AZs — If one AZ has a power event, the others keep serving; the LB stops routing to the unhealthy AZ — Stateless, no local session state: any server can handle any request
3. Database primary + read replica — Primary handles writes; the replica handles reads and can be promoted on failure — Multi-AZ RDS fails over to the standby automatically, typically in 60–120 seconds; Aurora in about 30
4. Backup and point-in-time recovery — Automated daily snapshots plus transaction log shipping — Minutes-granularity recovery
5. Monitoring and alerting — Health checks at each layer; alerts trigger automatic failover or page on-call — CloudWatch, Datadog, Prometheus on CPU/memory, error rates, p99 latency, replication lag

> [!IMPORTANT] App servers must be stateless: session state on one server turns that server back into a SPOF.

Recovery objectives

RTO (Recovery Time Objective) — How fast must we recover?
RPO (Recovery Point Objective) — How much data loss is acceptable?

SLA math

99.9% — 8.7 hours
99.99% — 52 minutes
99.999% — 5 minutes

> [!TIP] Each extra nine requires increasingly redundant architecture, so ask what the business actually needs before designing.

Key concepts

Single Point of Failure (SPOF): A component whose failure causes the entire system to fail. Eliminating SPOFs is the primary task of HA design. Audit every component: if it fails, does the system go down? If yes, add redundancy.
RTO (Recovery Time Objective): Maximum acceptable time to restore service after a failure. Example: RTO = 30 minutes means the system must be back online within 30 minutes of any failure. Drives: how fast failover must be, whether warm standby or cold standby is sufficient.
RPO (Recovery Point Objective): Maximum acceptable data loss measured in time. RPO = 5 minutes means you can lose at most 5 minutes of transactions. Drives: backup frequency, replication lag limits, whether synchronous or asynchronous replication is required.
Active-Active vs Active-Passive: Active-Active: all instances serve traffic simultaneously; failure of one reduces capacity but system stays up. Active-Passive: one instance is hot standby, receives no traffic until the primary fails; faster failover than cold standby. Active-Active is preferred for web tier; Active-Passive for database Primary.
Health checks: Load balancers and orchestrators continuously probe instances (HTTP /health endpoint). Failed checks remove the instance from rotation. Kubernetes liveness/readiness probes are the K8s equivalent. Without health checks, traffic continues to a failed instance.
Availability Zone (AZ): Physically separate datacenter within a cloud region with independent power, cooling, and networking. Deploying across 3 AZs means a full AZ outage (which AWS/GCP/Azure SLA for) only takes down 1/3 of capacity, not the whole system.

When to use it

Designing any production system that must meet an uptime SLA
Architecture review of a system that currently has SPOFs
Interview questions about system design for reliability
Planning multi-AZ or multi-region deployments
Presenting infrastructure design to a CTO or technical leadership

Interview questions

Design a highly available web application on AWS.
What is the difference between RTO and RPO and how do they drive architectural decisions?