If your service consumes events from Kafka, RabbitMQ, SQS or almost any other broker, it will sooner or later receive the same event twice. This isn’t a misconfiguration. Most messaging setups promise at-least-once delivery, and “at least once” includes “twice”.
A consumer that adds 5 euros to a wallet every time it sees a PaymentReceived event will add 10 euros when that event arrives twice. The fix is to make the consumer idempotent: processing the same event twice has the same effect as processing it once. This note shows where the duplicates come from, a version that breaks, and a version that holds up.
Where the second copy comes from
It helps to be able to name the mechanisms, because “the network is unreliable” isn’t a satisfying answer:
- Consumer crashes after processing, before acknowledging. The consumer updates the database, then dies before committing its Kafka offset (or acking the RabbitMQ message). On restart, the broker redelivers from the last committed position. The work is done again.
- Rebalances. In a Kafka consumer group, a partition can move to another instance mid-batch. The new owner starts from the last committed offset, so anything processed but not committed gets processed again.
- Visibility timeouts. SQS hides a received message for a timeout. If processing takes longer than that, the message becomes visible and another consumer picks it up while the first is still working.
- Producer retries. The producer sends, the broker stores it, the acknowledgement is lost, and the producer sends again. Two copies in the log.
- Outbox relays. A relay publishes an event and crashes before marking the outbox row as sent. When it restarts, it publishes the row again.
The first and last of these are by design. Acknowledging before processing would avoid duplicates but would lose events whenever the consumer crashes mid-processing. Most systems choose duplicates over loss, and push the job of handling duplicates to the consumer.
A wallet credited twice
Here’s a small model using Python’s built-in sqlite3. One event credits 500 cents to customer c-42. The broker delivers it twice, as it would after a crash before the offset commit.
event = {"id": "evt-8f3a", "customer_id": "c-42", "amount_cents": 500}
def naive(db, e):
db.execute("UPDATE wallet SET balance_cents = balance_cents + ? WHERE customer_id = ?",
(e["amount_cents"], e["customer_id"]))The deduplicating version adds a table keyed by consumer name and event id, and inserts into it inside the same transaction as the balance update:
db.executescript("""
CREATE TABLE processed_message (
consumer TEXT NOT NULL,
message_id TEXT NOT NULL,
processed_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (consumer, message_id));
""")
def deduplicated(db, e):
try:
db.execute("BEGIN")
db.execute("INSERT INTO processed_message (consumer, message_id) VALUES ('wallet-credit', ?)", (e["id"],))
db.execute("UPDATE wallet SET balance_cents = balance_cents + ? WHERE customer_id = ?",
(e["amount_cents"], e["customer_id"]))
db.execute("COMMIT")
return "applied"
except sqlite3.IntegrityError as err:
db.execute("ROLLBACK")
return f"skipped duplicate ({err})"Output from Python 3:
naive consumer, delivered twice: balance = 1000
dedup consumer, delivery 1: applied
dedup consumer, delivery 2: skipped duplicate (UNIQUE constraint failed: processed_message.consumer, processed_message.message_id)
dedup consumer, delivered twice: balance = 500The naive consumer paid the customer twice. The deduplicating consumer applied the credit once and rejected the second delivery at the primary key.
Why “same transaction” is the whole trick
Beginners often write the dedup check as three separate steps: look up the id, do the work, then record the id. Consider where a crash can land:
- Record the id after the work, in a separate transaction. Crash between the two and the work is done but the id isn’t recorded. The redelivery does the work again. You’re back to double credits.
- Record the id before the work, in a separate transaction. Crash between the two and the id is recorded but the work never happened. The redelivery is skipped as a “duplicate”. You’ve silently lost a payment, which is arguably worse.
- Look up first, then insert. Two consumers handling the same redelivered event concurrently both see “not processed” and both proceed. The race window is small but real under rebalances and SQS visibility timeouts.
Putting the insert and the update in one transaction removes all three. Either both commit or neither does. And because the dedup row is protected by a primary key, the database itself resolves the concurrent case: one transaction wins, the other gets a unique-constraint error and rolls back.
This only works when the dedup table lives in the same database as the data being changed. That’s one more reason each service should own its own database: the consumer’s dedup record and its business state can commit together.
Choosing the id to deduplicate on
The id must identify the business event, and it must be the same on every copy. Good choices:
- An event id the producer generates once (a UUID set when the event is created, ideally when the outbox row is written).
- A natural key that’s unique per business fact, like
payment-idforPaymentReceived.
Poor choices:
- The Kafka offset or the SQS receipt handle. A producer retry or an outbox re-publish creates a new message with a new offset, so offset-based dedup misses exactly the duplicates you’re worried about.
- A hash of the payload. Two genuinely different events can have identical payloads (a customer tops up 5 euros twice in a minute). You’d drop a real event.
Effects that can’t join the transaction
The pattern above covers changes to your own database. What if the consumer sends an email, calls a payment provider, or publishes another event?
You can’t put Stripe’s API inside your SQLite transaction. The options are:
- Pass an idempotency key downstream. Payment providers accept a key per request, and a retried request with the same key returns the original result instead of charging again. Derive the key from the event id.
- Write intent, not effect. In the transaction, insert a row into an outbox (“send confirmation email for order 991”). A separate process sends it and can itself deduplicate.
- Accept a rare duplicate. For a “your parcel shipped” email, sending two once in a while may be fine. Say so explicitly instead of pretending it can’t happen.
Some operations are naturally idempotent and need no table at all. UPDATE orders SET status = 'SHIPPED' WHERE id = 991 gives the same result run once or ten times. Increments, appends and “send” operations are the ones that need protection.
Keeping the table from growing forever
The processed-message table gains a row per event. Keep rows at least as long as a duplicate can realistically arrive: longer than the broker’s retention if consumers can be reset to an old offset, and longer than any replay window you support. A daily job that deletes rows older than, say, 14 days is common. If you replay a topic from the beginning for a rebuild, deduplication by a pruned table won’t protect you, so replays should go to a fresh read model or rely on idempotent operations.
Interview practice: the duplicate loyalty points
“A loyalty service consumes OrderCompleted events and adds points. Customers occasionally get double points. The consumer checks Redis for the event id, adds points in Postgres, then writes the event id to Redis with a 7-day TTL. What’s wrong?”
Answer
The dedup record and the points update are in different systems, so they can’t commit atomically. If the consumer crashes after the Postgres update and before the Redis write, the redelivered event passes the Redis check and the points are added again. There’s also a race: two consumers handling the same event concurrently (after a rebalance) can both miss in Redis and both add points.
I’d move the dedup record into Postgres, in a table with a primary key on (consumer, event id), and insert it in the same transaction as the points update. A duplicate then fails on the primary key and rolls back. I’d confirm the event id is a producer-assigned id rather than an offset, keep rows for longer than the 7 days if consumer offsets can ever be reset further back, and add a metric for skipped duplicates so a sudden spike (say, from a misbehaving relay) is visible.