Replication Lag and Consistency Anomalies

Reading from asynchronous replicas can show users old data. The three anomalies that lag causes (missing your own write, going back in time, effects before causes) and the guarantees that fix each.

Overview

With asynchronous replication, a follower applies each change a little after the leader. That delay is replication lag. The PostgreSQL documentation says it is typically under one second when the standby can keep up. Under heavy writes, a slow network or a busy replica it can grow to seconds or minutes, and nothing in asynchronous replication puts an upper limit on it.

If every copy eventually catches up once writes stop, the system is eventually consistent. That promise says nothing about what a user sees in the meantime. Three anomalies show up in practice, and each has its own guarantee and fix.

Missing own write — Saves a comment, reloads, the comment is gone — Read your writes
Going back in time — Sees a new reply, reloads, the reply is gone — Monotonic reads
Effect before cause — Sees an answer before the question it answers — Consistent prefix reads

Where lag comes from

Write bursts — The follower replays changes more slowly than the leader produces them
Network between sites — Changes take longer to arrive, especially across regions
Long queries on the follower — In PostgreSQL, replay can pause for a query that conflicts with it, up to max_standby_streaming_delay (30 seconds by default) before the query is canceled
Weaker follower hardware — Slower disk or CPU than the leader

Measuring lag

Measure lag in time and in bytes. In PostgreSQL, pg_stat_replication on the primary has one row per standby. write_lag, flush_lag and replay_lag show how long each stage took, and the gap between the primary's current WAL position and a standby's replayed position is the lag in bytes.

Managed services expose the same idea. Amazon RDS publishes a CloudWatch metric, ReplicaLag, in seconds. For PostgreSQL replicas it is computed from the time of the last replayed transaction. Alert on lag, not just on replica health: a replica that is up but ten minutes behind serves wrong answers.

Read your writes

The most visible anomaly: a user saves something, the next page load reads from a follower that has not replayed the change, and the save looks lost. Read-your-writes consistency (also called read-after-write) promises a user always sees their own earlier writes. It promises nothing about other users' writes.

Read own data from the leader — Profile pages and anything the user can edit go to the leader — Leader carries those reads
Leader for a short window after a write — For some seconds after a write, that user reads from the leader — The window is a guess; long lag breaks it
Track the log position — Store the commit position; use a replica that has replayed past it (pg_last_wal_replay_lsn() in PostgreSQL, WAIT_FOR_EXECUTED_GTID_SET() in MySQL) — App or proxy must carry the position
Wait for replay on commit — PostgreSQL synchronous_commit = remote_apply returns only after synchronous standbys have applied the change — Slower commits; only helps reads sent to those synchronous standbys
Show it from the client — The UI inserts the new item from local state at once — Hides lag on this screen only

It gets harder in two cases. If the user has several devices, the position must be stored on the server, not in one browser. If replicas are in several regions, every request from that user must reach the same region's leader or replica.

Monotonic reads

A user reads from a replica that is nearly current, then from one that is further behind, and sees data disappear. Monotonic reads promises that a user never sees older data after newer data. Their reads may still be stale, but they never go backward.

The usual fix is to send each user to the same replica every time, for example by hashing the user ID to pick one. If that replica fails, the user moves to another that may be further behind, so a stronger version also carries the last position the user saw, as for read your writes. Random load balancing across replicas breaks monotonic reads.

Consistent prefix reads

Some writes depend on others: a question and its answer, an order and its cancellation. Consistent prefix reads promises that a reader sees writes in an order that could have happened, with no effect showing before its cause.

On one leader with one log this happens on its own, because every follower replays the same log in order. It breaks when data is split into partitions that replicate independently. If the question lives in one partition and the answer in another, a reader can see the answer's partition catch up first.

Keep causally related writes in one partition (for example, a whole thread) — Only works when related data shares a key
Track dependencies and hold back a write until its causes are visible — More metadata on every write
Read from one consistent snapshot across partitions — Needs a system that provides it

What you will be able to do

Explain what replication lag is, what causes it, and how to measure it.
Recognize the three anomalies from a user's bug report.
Implement read your writes with leader routing or log positions, and say what each costs.
Keep a user's reads monotonic, and explain why random load balancing breaks it.
Explain why consistent prefix is automatic with one log and not with partitions.

Key concepts

Replication lag: The delay between a write committing on the leader and becoming visible on a follower. Usually small, with no upper limit under asynchronous replication.
Eventual consistency: If writes stop, every replica eventually holds the same data. Says nothing about how stale a read can be before then.
Read your writes: A user always sees the effects of their own earlier writes. Other users' writes may still be stale.
Monotonic reads: A user's successive reads never return older data than an earlier read did.
Consistent prefix reads: A reader sees writes in an order that could have happened: a prefix of the history, with no effect before its cause.
Session guarantees: Guarantees about what one client sees across its own reads and writes, such as read your writes and monotonic reads.
Bounded staleness: A promise that reads are never more than a set time or number of versions behind.
Sticky routing: Sending all of one user's reads to the same replica, usually by hashing the user ID.

When to use it

Designing read routing for a database with read replicas.
Explaining why a user does not see their own post after saving it.
Contrasting eventual consistency with the session guarantees users actually need.
Setting alerts and limits on replica lag.

Interview questions

A user posts a comment, refreshes, and the comment is missing. What is happening, and how do you fix it?
What is the difference between monotonic reads and consistent prefix reads?
Does pinning a user to one replica give read your writes?
How would you monitor and limit replication lag in production?
Is eventual consistency enough for a social feed?