System Design Replication and Read Freshness

Replication keeps copies of the same data on several machines. It buys you three things: survival when a machine dies, more capacity for reads, and copies closer to users in other regions. It costs you a question that every read now has to answer: which copy is current enough for this request?

This guide works through the leader-follower model used by PostgreSQL, MySQL, MongoDB replica sets and Redis, with numbers for commit latency, replication lag and failover loss.

Leader, followers and the log

In leader-follower (also called primary-replica) replication, one node accepts writes. It records each change in a log (PostgreSQL’s write-ahead log, MySQL’s binlog, MongoDB’s oplog) and streams that log to followers, which replay it in order to stay identical. Clients can read from followers, but writes always go to the leader.

The design choices are about timing. When does the leader tell the client “committed”? Before any follower has the change, after one has it, or after a majority do? And how far behind may a follower fall before reads from it become a problem?

Synchronous, asynchronous and the space between

With asynchronous replication, the leader commits locally, answers the client, and ships the change afterwards. Commits are as fast as a single machine. If the leader dies, anything not yet shipped is lost.

With synchronous replication, the leader waits for a follower to confirm it has the change (durably written, in most configurations) before answering. A committed write survives the loss of the leader. The price is a network round trip on every commit, and a follower that’s slow or unreachable stalls writes.

To see the price, a simulation models a 0.5 ms local commit and follower acknowledgements that take about 1.5 ms (a same-region, cross-zone round trip plus an fsync), with a realistic spread:

modes = {
    "async (no follower wait)":            lambda: 0.5,
    "sync, wait for 1 follower":           lambda: 0.5 + ack(),
    "sync, wait for both of 2":            lambda: 0.5 + max(ack(), ack()),
    "quorum, wait for first of 2":         lambda: 0.5 + min(ack(), ack()),
}

python

commit latency (ms)                p50    p99
  async (no follower wait)         0.50   0.50
  sync, wait for 1 follower        2.00   4.79
  sync, wait for both of 2         2.42   5.27
  quorum, wait for first of 2      1.67   3.18

Text

Waiting for one named follower quadruples median commit latency and puts its every hiccup into your p99. Waiting for both followers is worse, since you get the slower of two each time. The interesting row is the last one: wait for whichever of two followers answers first. The median drops below the single-follower case and the p99 falls by a third, because one slow follower no longer delays commits. PostgreSQL supports this with synchronous_standby_names = 'ANY 1 (replica_a, replica_b)', and it’s the same idea as a majority quorum in Raft-based systems.

Across regions the numbers change by an order of magnitude. A synchronous follower 70 ms away adds at least 70 ms to every commit, which is why cross-region replicas are almost always asynchronous unless the data truly can’t be lost.

Where lag comes from

An async follower is usually milliseconds behind. It falls further behind when it can’t apply changes as fast as the leader produces them. A common reason is that replay is often less parallel than the original writes: hundreds of connections write concurrently on the leader, while PostgreSQL’s standby replays WAL in a single process (MySQL can apply in parallel when configured to).

Here’s a replica that can apply 6,000 transactions a second, during a nightly batch job that pushes the leader from 3,000 to 8,000 a second for ten minutes:

replica lag during a 10-minute batch job (apply limit 6,000 tx/s):
  t= 0:00  primary  3000 tx/s  replica behind by         0 tx =    0 s of commits
  t= 5:00  primary  8000 tx/s  replica behind by     2,000 tx =    0 s of commits
  t=10:00  primary  8000 tx/s  replica behind by   602,000 tx =   75 s of commits
  t=14:59  primary  8000 tx/s  replica behind by 1,200,000 tx =  150 s of commits
  t=15:00  primary  3000 tx/s  replica behind by 1,197,000 tx =  150 s of commits
  t=20:00  primary  3000 tx/s  replica behind by   297,000 tx =   99 s of commits
  t=25:00  primary  3000 tx/s  replica behind by         0 tx =    0 s of commits

Text

During the batch the backlog grows by 2,000 transactions every second, reaching 1.2 million. Measured in time, the replica is serving data two and a half minutes old by the end of the job. Notice the 20-minute line: the backlog has shrunk to a quarter, but the replica is still 99 seconds behind, because it’s replaying the dense batch-era transactions. It takes ten minutes after the batch ends to catch up fully.

Other causes of lag worth naming: long-running queries on the replica that conflict with replay (PostgreSQL either delays replay, up to max_standby_streaming_delay, or cancels the query), network congestion between zones, and a replica on smaller hardware than the leader.

Routing reads by how fresh they must be

Lag only matters for reads that depend on recent writes. So route each read by its freshness requirement:

Catalog pages, search results, public profiles. A few seconds stale is fine. Read from any healthy replica.
Reports and analytics. Often fine minutes stale, which makes them ideal replica work. Just label the data with its as-of time.
A user’s own recent changes. After someone updates their address, their next page should show it. Read from the leader for that user for a short time, or better, carry the write’s log position and read from a replica only if it has replayed past it. Consistency and user expectations compares these approaches with a simulation.
Decisions on current state. Checking a balance before a transfer, or stock before a reservation, belongs on the leader inside the same transaction as the write.

Then protect yourself from the batch-job case: measure lag per replica (now() - pg_last_xact_replay_timestamp() in PostgreSQL, Seconds_Behind_Source in MySQL) and remove a replica from the read pool when it crosses a threshold such as 30 seconds. Serving two-minute-old data to readers who expect “basically current” is worse than spreading their load over fewer replicas.

A misconception worth correcting: replicas add read capacity, but not unlimited capacity. Every replica must replay the full write load, so on a write-heavy system much of each replica’s capacity is already spent applying changes.

Failover: promotion and the lost tail

When the leader dies, a follower is promoted. With async replication, the new leader is missing whatever it hadn’t received: at 3,000 transactions a second and 200 ms of lag, about 600 committed transactions. During the batch job above, a failover would lose up to 1.2 million. Tools like Patroni (PostgreSQL) or orchestrator (MySQL) usually promote the most up-to-date follower and can refuse automatic failover when lag exceeds a limit.

Two more things to get right. The old leader must be fenced so it can’t keep accepting writes if it was only partitioned, not dead. And when it comes back, it may have transactions the new leader never saw; it has to be rewound or rebuilt as a follower (PostgreSQL’s pg_rewind does this) rather than simply rejoining.

Interview practice: the reverting address

“After a nightly import was added, support gets reports between 2 a.m. and 2:30 a.m. that users change their delivery address, see a success message, then see the old address on the next page. Reads go to two async replicas, writes to the primary. What’s happening and how do you fix it?”

Answer

The timing points at replication lag caused by the import. The import pushes write volume above what the replicas can replay, so they fall minutes behind for the duration of the job and for a while after. The address update commits on the primary, but the next page reads from a replica that hasn’t replayed it, so the user sees the old address.

I’d fix it at three levels. For the user-visible bug, I’d give reads read-your-writes behaviour: after an address change, return the commit’s WAL position and route that user’s reads to a replica only if it has replayed past it, otherwise to the primary. For lag in general, I’d add a lag check to the read routing that takes any replica more than 30 seconds behind out of the pool. For the cause, I’d throttle the import (smaller batches with pauses, so its write rate stays under the replicas’ replay rate) or schedule it away from active hours. I’d also point out the failover risk: if the primary died during the import, an async replica minutes behind would lose every write in that window, so I’d watch lag during the job and block automatic failover to a replica that’s far behind.

System Design Replication and Read Freshness

Leader, followers and the log

Synchronous, asynchronous and the space between

Where lag comes from

Routing reads by how fresh they must be

Failover: promotion and the lost tail

Interview practice: the reverting address

Answer

Further reading

More in System Design

System Design Multi-Region Recovery and Data Risk

System Design Consistency and User Expectations

REST vs GraphQL: Differences, Trade-offs and When to Use Each