A replica set is a group of mongod processes holding the same data. One is the primary and takes all writes; the others are secondaries that copy the primary’s oplog and apply it. If the primary goes away, the secondaries elect a new one. That’s the whole high-availability story, and the interview questions focus on its edges: how long it takes, why you need an odd number of members, and what happens to writes in flight.
Commands are mongosh for MongoDB 7 and 8. Outputs are illustrative; no replica set was running while this was written.
A three-member set
rs.initiate({
_id: "rs0",
members: [
{ _id: 0, host: "db-a:27017", priority: 2 },
{ _id: 1, host: "db-b:27017", priority: 1 },
{ _id: 2, host: "db-c:27017", priority: 1 }
]
})rs.status() shows each member’s stateStr (PRIMARY, SECONDARY, RECOVERING and so on), its optimeDate (how far it has applied the oplog) and its health. db-a has a higher priority, so it’ll be preferred as primary when it’s healthy and caught up.
Why majority, and why three
A member can become primary only if a majority of voting members vote for it. With three voting members, the majority is two. That rule is what prevents split brain: if the network splits the set into a group of two and a group of one, only the group of two can elect a primary. The lone member can’t get a majority and stays secondary.
Now consider two members. Majority of two is two. If one goes down, the survivor can’t form a majority, so there’s no primary and no writes. A two-member set gives you a backup copy but no automatic failover. That’s why three is the practical minimum, and why you add members in odd numbers: going from three to four voting members raises the majority to three without tolerating any extra failures. Three tolerates one failure; five tolerates two.
An arbiter votes but holds no data. It’s cheaper than a full member, but MongoDB discourages it in most production setups, partly because with a primary-secondary-arbiter (PSA) layout and one data-bearing member down, w: "majority" writes can’t be acknowledged by two data-bearing members and stall.
Timeline of a failover
Say db-a, the primary, loses its network connection.
- Members send each other heartbeats every 2 seconds.
- If
db-banddb-cdon’t hear from the primary forelectionTimeoutMillis(10,000 ms by default), one of them calls an election. - A candidate must be eligible (priority above 0, not hidden or delayed) and voters only vote for a candidate whose oplog is at least as up to date as theirs. That’s how the set avoids electing a member missing acknowledged majority writes.
- With two votes,
db-bbecomes primary. The election itself usually takes well under a second once called. - Meanwhile
db-a, if still running but isolated, notices it can’t see a majority and steps down to secondary on its own. It doesn’t keep accepting writes indefinitely.
So the gap without a primary is roughly the timeout plus election time: typically around 10 to 12 seconds with defaults. Lowering electionTimeoutMillis speeds detection but increases false elections over flaky networks. When the old primary rejoins, it syncs from the new primary; if db-a has higher priority, it’ll call an election once it’s caught up and take the primary role back.
Planned maintenance is gentler: rs.stepDown() makes the primary wait for a secondary to catch up, then step down, which usually completes in a couple of seconds.
What your application sees
During the gap, writes fail. Drivers surface errors like NotWritablePrimary (formerly NotMaster) or network errors for operations sent to the old primary. The driver’s server discovery notices the new primary within its heartbeat interval and routes new operations there.
Retryable writes, on by default in modern drivers (retryWrites=true), retry a supported write once after a network error or a “not primary” error. The server records each retryable write’s transaction number, so if the first attempt actually succeeded before the connection dropped, the retry doesn’t apply it twice. This covers single-document inserts, updates, deletes and findAndModify, and bulk writes of those, but not updateMany/deleteMany (multi-document writes aren’t retryable). Retryable reads do the same for reads.
One retry covers one failover. If your app needs to ride out longer outages, you need your own retry with backoff, and the operation needs to be idempotent at the business level.
Rollback: the part people miss
Writes that the old primary accepted but hadn’t replicated to a majority before it failed are on a branch of history the new primary never saw. When the old primary rejoins, it rolls back those writes to match the new primary, saving them to rollback files under the data directory (rollback/) for manual inspection.
If those writes were acknowledged to the client with w: 1, the client was told “done” for data that no longer exists. With w: "majority", the acknowledgement only comes after a majority has the write, and a majority-committed write can’t be rolled back, because any new primary must have it. That’s the real argument for majority writes: not “more copies”, but “an acknowledgement that survives failover”. Since MongoDB 5.0 the default write concern is majority for most deployments, so this mostly bites systems that set w: 1 for speed.
Interview practice
A three-member replica set runs in one cloud region across three zones. The app writes orders with w: 1 and the driver has retryWrites=false. During a zone outage the primary is lost. Afterwards, support finds two problems: some customers were told their order succeeded but it doesn’t exist, and others placed one order but have two. Explain both and fix them.
Answer
The missing orders are rollbacks. With w: 1, the old primary acknowledged those inserts before any secondary had them. When the zone went down, a secondary from another zone was elected without those writes, and when the old primary came back, its unreplicated writes were rolled back into rollback files. Customers were told “success” for writes that were never durable.
The duplicates come from application-level retries. With retryWrites=false, a network error during the failover surfaced to the app as a failure, the app (or the user clicking again) re-sent the order, and in some cases the first attempt had in fact succeeded and replicated.
Fixes: use w: "majority" for orders so an acknowledgement means the write survives a failover. Turn retryWrites back on so the driver retries once with server-side deduplication. For the app’s own retries and double clicks, make order creation idempotent: generate an idempotency key on the client, store it on the order with a unique index, and treat a duplicate key error as “already created, return the existing order”. Finally, recover the rolled-back orders from the rollback files and reconcile them with payment records.