“We’re multi-region” is one of the most over-trusted sentences in system design. Running servers in two regions is the easy part. What decides whether a regional outage costs you five minutes or a day, and whether you lose data, is everything around those servers: how the data gets there, how quickly and how safely you switch, and whether anyone has actually tried it.
An advanced interview answer frames this with two numbers and then shows the arithmetic behind them.
RPO and RTO, defined by what they cost
Recovery point objective (RPO) is how much recent data you can afford to lose, expressed as time: “at most 5 seconds of writes”. Recovery time objective (RTO) is how long the service may be unavailable: “back within 30 minutes”.
Both are business decisions with a price attached. RPO zero means synchronous replication to another region, which adds a cross-region round trip (tens of milliseconds) to every commit. RTO of a few minutes means running a fully scaled second region all the time. A product catalogue might accept an RPO of minutes and an RTO of an hour; a payment ledger might need RPO zero. One design can use different targets for different data.
How much data async replication loses
Most multi-region databases replicate asynchronously: the primary region commits, acknowledges the client, and ships the change to the standby region shortly after. Whatever hasn’t shipped when the primary region dies is lost (or stranded until that region comes back). The amount is simply write rate times replication lag at that moment.
A simulation with 1,500 writes per second, sampling the lag at the moment of failure:
def lag_normal(): return 0.2 * math.exp(random.gauss(0, 0.5))
def lag_degraded(): return 5 * math.exp(random.gauss(0, 1.0))async replication, 1,500 writes/s:
failover while healthy: median 300 p99 956 committed writes lost
failover after network degradation: median 7,490 p99 76,611 committed writes lostWith healthy replication (about 200 ms of lag), a failover loses around 300 committed writes, under a thousand even in a bad case. The second line is more realistic for a real regional failure. Regions rarely vanish instantly; they degrade first. Packet loss and congestion on the cross-region link slow replication, so lag is often seconds when the failure finally happens. With lag around 5 seconds, a failover loses about 7,500 writes, and in the worst 1% of cases over 76,000.
That’s the trap in quoting “our replication lag is 200 ms” as your RPO. Your RPO is the lag at the worst moment, and the worst moment is correlated with the failure itself. Two things help: monitor lag and treat sustained growth as an incident in itself, and for writes you truly can’t lose, use synchronous replication or a quorum spanning three regions so that a commit is durable before it’s acknowledged.
Lost writes also need a plan after the fact. Payments that a customer saw confirmed but that never reached the standby must be reconciled, often from an external record such as the card processor’s settlement file.
A minute-by-minute recovery budget
RTO isn’t one number to promise; it’s a sum of steps, and each one can be measured and shortened. Here’s a plausible budget for a warm-standby setup:
recovery time budget (minutes):
detect (alarm on 3 failed 1-min checks) 3
decide (on-call confirms, approves) 10
promote replica, fence old primary 3
shift traffic (DNS TTL 60 s + stragglers) 5
warm standby at 100% capacity total 21
warm standby at 25%, scale out total 33
pilot light (DB only), build app tier total 56The figures are illustrative, but the shape is typical, and a few lines deserve comment.
Deciding is often the longest step. Failing over a region is disruptive and sometimes can’t be undone cleanly (writes since the last replication are stranded), so most teams require a human decision. Ten minutes for someone to be paged, look at dashboards and approve is optimistic at 3 a.m. Pre-agreed criteria (“if the primary region’s error rate is above 50% for 5 minutes, fail over”) shrink it.
DNS is slower than its TTL. A 60-second TTL bounds well-behaved resolvers, but some clients and resolvers cache longer, and long-lived connections stay put until they break. Anycast or a global load balancer that shifts traffic without waiting for DNS can cut this step.
Capacity decides the rest. A standby running at full size takes traffic immediately. A standby at 25% needs to scale out, and during a regional event every other customer of that cloud region may be trying to scale too, so capacity can be slow or unavailable. A “pilot light” (only the database replicated, the application tier built on demand) costs least and recovers slowest.
The general pattern in AWS’s disaster recovery guidance lines up with this: backup and restore (hours), pilot light (tens of minutes), warm standby (minutes), and multi-site active-active (near zero), each costing more than the last.
Fencing: never two primaries
The most dangerous failover bug isn’t slowness; it’s split brain. The primary region wasn’t dead, just unreachable from the monitoring system. You promote the standby. Now the old primary, still running, keeps accepting writes from clients that can reach it, and both regions diverge. Merging two diverged databases by hand afterwards is far worse than a longer outage.
Fencing prevents it. Before (or as part of) promoting the standby, make sure the old primary can’t accept writes:
- Remove it from routing so clients can’t reach it.
- Put the old database into read-only mode, or revoke the application’s write credentials, if you can reach it.
- Use an epoch or term number: each promotion increments it, every write carries it, and storage rejects writes from an older epoch. Consensus systems like etcd and Raft-based databases do this internally.
If you can’t confirm the old primary is fenced, the decision becomes a trade-off between availability and the risk of divergence, which is exactly why the human decision step exists.
Failback deserves the same care. After the old region recovers, its database holds stale data plus possibly some writes that never replicated. It has to be rebuilt as a replica of the new primary, and the stranded writes reconciled, before traffic can move back.
Active-active and its constraints
Running both regions as primaries removes the promotion step but adds write conflicts. The common approach is to give each piece of data a home region: a user’s profile and orders are written only in their home region, and other regions forward writes there. Truly global invariants, such as unique usernames or a single account balance, need either one owning region or a consensus protocol across regions, with the cross-region latency that implies. “Active-active everywhere with last-write-wins” quietly drops updates and isn’t an answer to give for anything involving money.
Interview practice: a ledger across two regions
“A payments company runs its ledger in one region with an async replica in a second region about 80 ms away. Regulators want an RPO of zero for ledger entries and an RTO of 15 minutes. What changes?”
Answer
Async replication can’t give RPO zero; at failover it loses whatever was in flight, and lag tends to be highest just before a regional failure. So ledger commits must be durable outside the primary region before they’re acknowledged. Synchronous replication to the second region adds at least one 80 ms round trip to every ledger commit. With group commit, throughput can stay high, but latency rises, so I’d confirm the product can accept that. A better shape is a quorum across three regions (or two regions plus a third witness), so a commit needs two of three acknowledgements and losing any one region loses nothing and still leaves a majority to elect a new leader automatically.
For the 15-minute RTO, I’d budget it: automated detection in 2 to 3 minutes, a pre-approved decision rule instead of an open-ended discussion, leader election handled by the quorum system itself with fencing built in through terms, and traffic shifted by a global load balancer rather than DNS alone. The application tier in the other regions must already run at enough capacity to take the load, because scaling during a regional event is unreliable. Finally, I’d rehearse it: a quarterly game day that fails over the ledger in production-like conditions, measuring each step against the budget.