Single-Leader Replication

One node takes every write and streams its changes to read-only copies. How the log is shipped, what synchronous and asynchronous replication cost, and how failover goes wrong.

Overview

Replication means keeping a copy of the same data on several machines. The copies let you survive a machine failure, serve reads from more servers, and put data closer to users. The hard part is keeping the copies the same while the data changes.

Single-leader replication is the most common answer. One node, the leader (also called the primary), accepts every write. The other nodes, the followers (replicas or standbys), only apply the changes the leader sends them, in the same order. PostgreSQL streaming replication, MySQL source-replica replication and most managed read replicas work this way.

One node decides the order

Because only the leader accepts writes, there is one order of changes and no write conflicts between nodes. Clients send writes to the leader. Reads can go to the leader or to any follower. Adding followers adds read capacity, but never write capacity: every write still lands on one machine.

How changes reach the followers

The leader describes each change in a replication log and sends it to the followers. There are three common formats, and the choice decides how tightly the leader and followers are coupled.

Statement-based — The SQL statement itself — Nondeterministic statements (a random value, the current time, triggers) can give different results on a follower
Physical (write-ahead log) — The exact bytes changed in data pages — Leader and followers must run compatible storage versions
Logical (row-based) — Row-level changes, keyed by primary key — More data for bulk updates, but independent of storage layout

PostgreSQL streaming replication ships the physical write-ahead log (WAL). PostgreSQL logical replication sends row changes identified by the replication identity, usually the primary key. MySQL writes its binary log in row-based format by default and can also log statements or a mix of both.

Synchronous or asynchronous

The key question is when the leader tells the client 'your write is done'. With asynchronous replication the leader confirms after writing locally and sends the change afterward. With synchronous replication it waits until a follower confirms it has the change.

Asynchronous — Lowest — Recent confirmed writes can be lost — Leader is unaffected
Synchronous (every follower) — Waits for the slowest — Nothing confirmed is lost — Writes stop
One synchronous, rest async — Waits for one — At least two copies of each confirmed write — Another follower can take the synchronous role

Real systems use the middle ground. In PostgreSQL, synchronous_standby_names names the standbys that must confirm, either by priority (FIRST 1 (s1, s2)) or as a quorum (ANY 2 (s1, s2, s3)). The synchronous_commit setting then says how far the change must get before the commit returns.

off — Nothing: success can be reported before the local flush
local — The leader's own disk flush only
remote_write — Synchronous standbys have written it to their file system
on (default) — Synchronous standbys have flushed it to durable storage
remote_apply — Synchronous standbys have applied it, so queries there can see it

PostgreSQL streaming replication is asynchronous unless you configure synchronous standbys. If a required synchronous standby crashes and there is no other to take its place, commits wait. MySQL's semisynchronous replication makes the other trade: the source waits for at least one replica to log the change (the count is configurable), but after rpl_semi_sync_source_timeout (10 seconds by default) it falls back to asynchronous replication.

Adding a follower

A new follower cannot simply start reading the log from the beginning: old log segments are deleted. Instead it starts from a copy.

The same idea lets a follower recover after a restart. It remembers the last log position it applied and asks the leader for everything after it.

Failover

When the leader fails, a follower must become the new leader. That is failover. It has four parts, and each can go wrong.

PostgreSQL itself does not detect a failed primary. It provides promotion (pg_promote()) and pg_rewind, which turns a former primary into a standby of the new one. A separate tool, such as Patroni, does the detection. Patroni keeps a leader lock in a store such as etcd, Consul, ZooKeeper or Kubernetes. The lock has a time to live (ttl, 30 seconds by default), and the replicas start an election when it expires. Its maximum_lag_on_failover setting keeps a replica that is too far behind out of the election.

Timeout too short — A slow leader (load spike, long pause) is replaced for no reason — Tune the timeout; prefer false negatives over flapping
Timeout too long — Writes are down for the whole timeout — Accept a known outage window
New leader was behind — Writes the old leader confirmed are gone — Synchronous standby, lag limit on candidates
Old leader comes back — Two nodes accept writes (split brain) — Fencing: the old leader must be stopped or refused
Clients cached the old address — Writes go to a node that is no longer leader — Connection routing through a proxy or service discovery

Split brain and fencing

Split brain is two nodes both acting as leader and accepting writes. Their data diverges, and there is no single order left to repair it from. The PostgreSQL documentation calls the defense STONITH ('shoot the other node in the head'): before or during promotion, make sure the old primary cannot keep acting as one.

A general form of fencing uses an increasing number. Each new leader gets a higher term (Raft) or fencing token. Storage and followers remember the highest number they have seen and reject requests carrying a lower one. In Raft, a leader that sees a higher term steps down to follower.

What you will be able to do

Explain why single-leader replication has no write conflicts and scales reads but not writes.
Compare statement, physical and logical replication logs.
Choose between asynchronous, synchronous and one-synchronous setups, and name what each loses.
Walk through failover step by step and name the failure at each step.
Explain split brain and how fencing with an increasing term prevents it.

Key concepts

Leader (primary): The one node that accepts writes and decides the order of all changes.
Follower (replica, standby): A copy that applies the leader's changes in the same order. It can serve reads, which may be slightly stale.
Replication log: The ordered stream of changes the leader sends. It can carry SQL statements, physical WAL records or logical row changes.
Synchronous replication: The leader confirms a write only after a follower has it. Nothing confirmed is lost if the leader fails, but writes wait on that follower.
Asynchronous replication: The leader confirms after its own write and replicates afterward. Fast, but the newest confirmed writes can be lost on failover.
Log position (LSN): A point in the leader's log. PostgreSQL calls it the log sequence number. It tells how far each follower has got.
Failover: Promoting a follower to leader after the leader fails, then pointing clients and other followers at it.
Split brain: Two nodes both accepting writes as leader, so their data diverges.
Fencing: Stopping a former leader from acting as one, for example by rejecting requests that carry an older term or token.

When to use it

Explaining how PostgreSQL or MySQL read replicas work.
Choosing synchronous or asynchronous replication for a given durability need.
Walking through failover and its failure modes in a system design interview.
Explaining why adding replicas scales reads but not writes.

Interview questions

Walk through failover for a PostgreSQL primary with two replicas.
Synchronous or asynchronous replication: how do you choose?
What is split brain, and how do you prevent it?
Your read replicas cannot keep up. What are your options?
Why not just use statement-based replication? It sends less data.