Plenty of jobs should run on exactly one machine at a time. A scheduler that sends the nightly invoice emails, a partition owner that applies writes in order, the process that compacts a log: run two copies and you get duplicate invoices or corrupted data. Leader election is how a group of identical instances agrees which one does the work, and how they pick a new one when that instance dies.
Electing a leader is the easy half. The hard half, and the part interviewers dig into, is that an old leader doesn’t always know it has been replaced. Any design that ignores this has two leaders some of the time, which is often worse than having none.
Leases: leadership with an expiry date
The common building block is a lease. A node asks a coordination service for the lock “invoice-scheduler” with a time-to-live, say 10 seconds. If nobody holds it, the node gets it and becomes leader. While it’s alive it renews the lease every few seconds. If it crashes, it stops renewing, the lease expires, and another node’s next attempt succeeds.
The nice property is that nobody has to diagnose the failure. A crashed process, a dead VM and a network partition all look the same: renewals stop arriving, and after the TTL someone else takes over. The TTL trades failover speed against false alarms. A 2-second lease fails over fast but will flap on a slow network; a 60-second lease is stable but leaves the job unowned for up to a minute after a crash. Kubernetes’ client-go leader election, used by most controllers, defaults to a 15-second lease duration, a 10-second renew deadline and a 2-second retry period, which is a reasonable reference point to quote.
Where does the lease live? It needs a store that’s itself fault tolerant and gives you an atomic compare-and-set:
- etcd or ZooKeeper. Both run a consensus protocol (Raft and Zab respectively) across three or five nodes, so the lock state survives a node loss. etcd has leases built in; in ZooKeeper you create an ephemeral sequential znode and the lowest sequence number is leader.
- A Kubernetes
Leaseobject in thecoordination.k8s.ioAPI group, which is backed by etcd anyway. - Your own database with a conditional update:
UPDATE leases SET holder = 'node-b', expires_at = now() + interval '10 seconds' WHERE name = 'invoice-scheduler' AND expires_at < now(). If it updates one row, you won. DynamoDB conditional writes work the same way.
What not to use: a single Redis instance with SET NX PX for anything where correctness matters. If that Redis node fails over to a replica that hadn’t received the write, two nodes can hold the “lock” at once. That’s fine for “avoid doing duplicate work most of the time”, not for “never corrupt the ledger”.
How two leaders happen anyway
Here’s the scenario that breaks naive designs. Node A holds a 10-second lease. At t=2 it enters a long stop-the-world garbage collection pause. (Fifteen seconds is extreme but real on a badly tuned JVM with a large heap; a VM being live-migrated, or a process stopped with SIGSTOP, behaves the same way.) A can’t renew, so the lease expires at t=10. Node B acquires it at t=11 and starts writing. At t=17, A wakes up. From A’s point of view no time has passed: it was in the middle of writing a batch, it still has a lease object in memory, and it finishes the write.
Checking “do I still hold the lease?” right before writing doesn’t fix this. The pause can happen between the check and the write. Network delays do the same thing: a write A sent at t=9 can sit in a TCP buffer and arrive at t=12. No amount of care on the client side closes that gap, because the client can’t know how long it was frozen.
Fencing tokens: let the storage decide
The fix moves the decision to the resource being protected. Every time the lock service grants a lease to a new holder, it hands out a number larger than any before it: a fencing token. etcd’s revision numbers and ZooKeeper’s zxid work as tokens. The leader sends its token with every write. The storage remembers the highest token it has seen and rejects any write carrying a lower one.
This simulation in Node 22 plays out the GC pause, with and without fencing:
class LockService {
constructor(ttl) { this.ttl = ttl; this.holder = null; this.expires = 0; this.token = 33; }
acquire(node, now) {
if (this.holder && now < this.expires && this.holder !== node) return null;
if (this.holder !== node) this.token++; // new holder, new token
this.holder = node; this.expires = now + this.ttl;
return this.token;
}
}
class Storage {
constructor(fencing) { this.fencing = fencing; this.highest = 0; this.value = null; this.log = []; }
write(node, token, value, now) {
if (this.fencing && token < this.highest) {
this.log.push(`t=${now} REJECT ${node} token=${token} (highest seen ${this.highest})`); return false;
}
this.highest = Math.max(this.highest, token); this.value = value;
this.log.push(`t=${now} ACCEPT ${node} token=${token} value=${value}`); return true;
}
}
function scenario(fencing) {
const lock = new LockService(10), db = new Storage(fencing);
const tA = lock.acquire('A', 0); // A leads, lease until t=10
db.write('A', tA, 'batch-1', 1);
// A pauses for 15 s at t=2; the lease expires at t=10
const tB = lock.acquire('B', 11);
db.write('B', tB, 'batch-2', 12);
db.write('A', tA, 'batch-1-late', 17); // A wakes and finishes its write
return { fencing, final: db.value, log: db.log };
}
console.log(scenario(false));
console.log(scenario(true));Output:
{
fencing: false,
final: 'batch-1-late',
log: [
't=1 ACCEPT A token=34 value=batch-1',
't=12 ACCEPT B token=35 value=batch-2',
't=17 ACCEPT A token=34 value=batch-1-late'
]
}
{
fencing: true,
final: 'batch-2',
log: [
't=1 ACCEPT A token=34 value=batch-1',
't=12 ACCEPT B token=35 value=batch-2',
't=17 REJECT A token=34 (highest seen 35)'
]
}Without fencing, the stale leader’s write lands last and overwrites the real leader’s work. With fencing, storage has already seen token 35 from B, so A’s token 34 is refused, and A learns it’s no longer leader from the rejection. Note the comparison is <, not <=: the current leader makes many writes with the same token.
The catch is that the storage has to cooperate. A database can enforce it with a column and a conditional update (WHERE fence_token <= :token). An object store with conditional writes can do it with a version precondition. A third-party email API can’t, which is why exactly-once side effects on external systems also need idempotency keys.
What interviewers listen for
- Saying “use a distributed lock” and stopping there. Ask what happens when the holder pauses.
- Relying on synchronised clocks. Leases should be measured with a monotonic clock on each node, and the lock service should be the only judge of expiry. A node should also give up leadership a little before its lease ends, by its own clock, to leave margin for drift.
- Forgetting the leader’s unfinished work. When B takes over, it should resume from durable state (a checkpoint, the last committed offset), not from whatever A had in memory.
- Overusing election. If the work can be partitioned (each node owns a hash range of customer IDs), you may not need one leader at all, just an owner per partition, which spreads load and limits the blast radius.
Interview exercise: the duplicate invoices
Three instances of a billing service each run a cron that generates monthly invoices. To avoid duplicates, the team added a Redis lock with a 30-second TTL around the job. The job usually takes 20 seconds. Last month some customers got two invoices. What happened, and what would you change?
Reasoned answer
The likely failure is a job that ran longer than the TTL. A slow database or a GC pause pushed one run past 30 seconds, the lock expired, a second instance acquired it and started generating invoices while the first was still working. A Redis failover losing the lock key would produce the same symptom.
Fix it in three layers. First, renew the lease while the job is running (a heartbeat every 10 seconds) instead of guessing a TTL longer than the job, and stop work if a renewal fails. Second, add fencing: take a monotonically increasing token from the lock store (or use a row in Postgres with a version column) and include it in every invoice write, so a stale holder’s inserts fail. Third, make the outcome idempotent regardless of the lock: a unique constraint on (customer_id, billing_period) means a second invoice for the same month can’t be inserted, whatever the lock does. The lock then becomes an efficiency measure that avoids wasted work, and the constraint is what guarantees correctness.