An operation is idempotent when doing it twice has the same effect as doing it once. PUT /users/42/email with the same body is idempotent by nature: the second call overwrites the email with the value it already has. POST /charges is not. Call it twice and the customer pays twice.
The reason this comes up in nearly every system design interview involving money, orders or messages is that retries are unavoidable. Clients retry on timeouts, load balancers retry on connection resets, and message consumers get redeliveries. The interesting part of the answer isn’t “add an idempotency key”, which most candidates say. It’s what you store, when you store it, and what happens in the awkward cases.
Why a timeout leaves the client stuck
Picture the sequence. The mobile app sends POST /charges for $49.99. The server receives it, calls the card processor, writes the charge row, and starts sending back 201 Created. The phone switches from Wi-Fi to cellular at that moment and the response never arrives. From the client’s side, the request timed out.
The client now has three possibilities it can’t tell apart: the request never reached the server, the server failed partway through, or the server succeeded and the response was lost. Retrying is correct for the first two and harmful for the third. Not retrying is correct only for the third. Without help from the server, there’s no safe choice.
How often does this matter? A quick simulation makes it tangible. A toy payment service processes 1,000 checkouts. Each response is lost 20% of the time, and the client retries up to three times after a lost response, exactly as a reasonable HTTP client would.
without keys { orders: 1000, charges: 1282, duplicates: 282 }
with keys { orders: 1000, charges: 1000, duplicates: 0 }282 customers charged more than once. A 20% loss rate is far higher than you’d see in production, but even at 0.1% on a service doing a million payments a day, that’s a thousand duplicate charges and a thousand support tickets.
How an idempotency key works
The client generates a unique key per logical operation, usually a UUID, and sends it with the request, typically in an Idempotency-Key header (Stripe’s API popularised this; an IETF draft standardises the header name). The key belongs to the operation, not the HTTP attempt: every retry of the same checkout reuses the same key, and a new checkout gets a new one.
The server keeps a record per key with three things:
- the key itself, scoped to the caller (two different merchants could generate the same UUID, however unlikely, and they must never see each other’s responses);
- a fingerprint of the request, such as a SHA-256 of the canonical body plus method and path;
- the status and, once finished, the response: status code and body.
Here’s the core of the toy service from the simulation, in Node 22:
import { createHash, randomUUID } from 'node:crypto';
const charges = [];
const keys = new Map(); // key -> { fingerprint, status, response }
const fingerprint = (body) => createHash('sha256').update(JSON.stringify(body)).digest('hex');
function createCharge(body, idemKey) {
const fp = fingerprint(body);
const existing = keys.get(idemKey);
if (existing) {
if (existing.fingerprint !== fp) return { status: 422, body: { error: 'key reused with a different request' } };
if (existing.status === 'in_progress') return { status: 409, body: { error: 'request with this key is still processing' } };
return { ...existing.response, replayed: true };
}
keys.set(idemKey, { fingerprint: fp, status: 'in_progress' }); // atomic insert in a real store
const charge = { id: 'ch_' + (charges.length + 1), amount: body.amount };
charges.push(charge);
const response = { status: 201, body: charge };
keys.set(idemKey, { fingerprint: fp, status: 'done', response });
return response;
}
const k = randomUUID();
console.log(createCharge({ orderId: 7, amount: 4999 }, k));
console.log(createCharge({ orderId: 7, amount: 4999 }, k));
console.log(createCharge({ orderId: 7, amount: 5999 }, k));Output:
{ status: 201, body: { id: 'ch_1', amount: 4999 } }
{ status: 201, body: { id: 'ch_1', amount: 4999 }, replayed: true }
{ status: 422, body: { error: 'key reused with a different request' } }The second call returns the same charge, ch_1, with the same status code. That’s the important property: the client can’t tell a replay from the original, so its code doesn’t need a special path. (Stripe adds an Idempotent-Replayed: true header so you can see it in logs; the replayed flag here plays the same role.) The third call reuses the key with a different amount. That’s a client bug, and the server refuses it rather than guessing which request was meant.
The cases that trip people up
Two attempts at the same time. A client with an aggressive timeout can send a retry while the first attempt is still talking to the card processor. If both look up the key, both see nothing, and both proceed, you’ve charged twice anyway. The claim on the key must be atomic: INSERT ... ON CONFLICT DO NOTHING in Postgres, SET key value NX in Redis, or a conditional put in DynamoDB. Whoever wins the insert does the work. The loser gets 409 Conflict and retries later. In the in-memory code above, JavaScript’s single thread makes the get then set safe; across two servers sharing a database it would not be.
Where the key record lives. The cleanest design writes the idempotency record and the business row (the charge) in the same database transaction. Then either both exist or neither does. If the key lives in Redis and the charge in Postgres, a crash between the two writes leaves them disagreeing, and you need a recovery step that reconciles in_progress keys older than some timeout.
External side effects. The card processor is outside your transaction. If you crash after it approved the charge but before you recorded the result, retrying will call the processor again. The fix is to pass an idempotency key downstream too. Most payment processors accept one, so the processor itself deduplicates. Idempotency has to be end to end; a key that stops at your API layer only protects your database.
What to cache for failures. If the first attempt failed with a validation error (400), replaying the 400 is correct. If it failed with a 500 because the database was down, many designs delete the key so a retry can try again, since nothing happened. Be explicit about which errors are final.
Expiry. You can’t keep keys forever. Stripe keeps them for 24 hours. Pick a window comfortably longer than the client’s entire retry schedule, including offline retries from mobile apps that sync on reconnect. After expiry, the same key is treated as new, which is why keys must be random per operation, not derived from something that repeats.
GET doesn’t need this; PUT and DELETE might. Reads are naturally safe to repeat. PUT with a full replacement body is idempotent in effect. DELETE is idempotent in effect too, though the second call may return 404 instead of 204; decide whether that difference matters to your clients. It’s POST and partial updates like “add $10 to balance” that need a key.
Quiz yourself: duplicate stock reservations
An order service consumes OrderPlaced messages from a queue with at-least-once delivery. For each message it reserves stock in the inventory service over HTTP, then writes the order to its database. Occasionally customers see stock reserved twice for one order. Where is the duplicate coming from, and how do you fix it?
Answer
At-least-once delivery means the same message can arrive more than once: the consumer processed it but crashed, or was slow, before acknowledging, so the broker redelivered. Each delivery calls the inventory service again, which treats every POST /reservations as new.
There are two places to fix, and you want both. In the consumer, record processed message IDs in the same transaction as the order row, and skip a message whose ID is already there. That handles redelivery after a successful commit. But the HTTP call to inventory happens before that commit, so a crash between the two still causes a second call. To close that gap, send an idempotency key to inventory derived from the order (for example reserve:<orderId>, which is stable across redeliveries because it’s tied to the operation, not the attempt). The inventory service stores the key with its reservation and returns the original reservation on a repeat.
Then mention the edge cases an interviewer will push on: the inventory service must claim the key atomically, because two redeliveries can run concurrently on different consumer instances, and the key’s lifetime must exceed the queue’s maximum redelivery window. If the inventory service is owned by another team and can’t add keys, fall back to a natural unique constraint, such as one reservation per order ID.