An order goes through OrderPlaced, then OrderPaid, then OrderShipped. A shipping dashboard consumes those events and shows each order’s current status. If OrderPlaced arrives last, the dashboard shows a shipped order as “placed”, and it will stay that way until another event comes along.
“Kafka guarantees ordering” is the answer many candidates give, and it’s only partly true. Kafka orders messages within a partition. Across partitions there’s no ordering at all. Whether your events stay in sequence depends on how they’re assigned to partitions, how many consumers read them, and what happens when one of them fails and gets retried.
Three orders, three partitions, two strategies
The simulation below produces nine events for three orders, in the correct sequence, onto a topic with three partitions. Each partition has its own consumer, and the consumers run at different speeds (12 ms, 3 ms and 7 ms per event), as real consumers do when they’re on different machines or one is catching up after a restart. Each consumer applies events to a shared status view and records any transition that doesn’t make sense.
const PARTITIONS = 3;
const hashPartition = (key) => createHash('md5').update(key).digest().readUInt32BE(0) % PARTITIONS;
const next = { undefined: 'OrderPlaced', OrderPlaced: 'OrderPaid', OrderPaid: 'OrderShipped' };
async function run(label, partitionFor) {
const partitions = Array.from({ length: PARTITIONS }, () => []);
produced.forEach((e, i) => partitions[partitionFor(e, i)].push(e));
const state = {}, problems = [];
const perEventMs = [12, 3, 7]; // one consumer per partition, different speeds
await Promise.all(partitions.map(async (events, p) => {
for (const e of events) {
await sleep(perEventMs[p]);
if (next[state[e.orderId]] !== e.type) problems.push(`${e.orderId}: ${e.type} while ${state[e.orderId] ?? 'unknown'}`);
state[e.orderId] = e.type;
}
}));
// ...print partitions, problems and final state
}
await run('round-robin, no key', (e, i) => i % PARTITIONS);
await run('keyed by orderId', (e) => hashPartition(e.orderId));Kafka’s real default partitioner uses murmur2, not MD5, and modern clients use a “sticky” strategy for keyless messages rather than strict round-robin, but the effect is the same: without a key, related events land on whichever partition the producer chose. Output from Node 22:
round-robin, no key
p0: o-991:Placed o-992:Placed o-993:Placed
p1: o-991:Paid o-992:Paid o-993:Paid
p2: o-991:Shipped o-992:Shipped o-993:Shipped
problems:
o-991: OrderPaid while unknown
o-992: OrderPaid while unknown
o-993: OrderPaid while unknown
o-991: OrderPlaced while OrderShipped
o-992: OrderPlaced while OrderShipped
o-993: OrderPlaced while OrderShipped
final: {"o-991":"OrderPlaced","o-992":"OrderPlaced","o-993":"OrderPlaced"}
keyed by orderId
p0: o-992:Placed o-992:Paid o-992:Shipped
p1: o-991:Placed o-991:Paid o-991:Shipped
p2: o-993:Placed o-993:Paid o-993:Shipped
problems: none
final: {"o-991":"OrderShipped","o-993":"OrderShipped","o-992":"OrderShipped"}Without a key, each order’s three events were spread over all three partitions. The fast consumer on p1 applied OrderPaid before anything else. p2 applied OrderShipped, which looked like a valid transition at that moment. Then the slow p0 consumer applied OrderPlaced last, and every order ended up displayed as “placed”. Nothing crashed. The data is simply wrong, which is the dangerous kind of failure.
With orderId as the key, all events for one order land on the same partition, a single consumer reads them in sequence, and the final state is correct. Note what the keyed run does not promise: the final JSON lists o-993 before o-992. Events for different orders still interleave arbitrarily. That’s fine, because nothing depends on the relative order of two unrelated orders.
Choosing the key
The key should be the entity whose history must stay in sequence. Usually that’s the aggregate id: orderId, accountId, customerId. Questions to ask:
- Which events must be ordered relative to each other? If a
PaymentCapturedevent must be processed after theOrderPlacedfor the same order, both must use the order id as key, even though one is “about” a payment. - Is the key too coarse? Keying everything by
merchantIdkeeps all orders of one merchant in sequence, which you probably don’t need, and puts a large merchant’s entire traffic on one partition. One partition is consumed by one consumer at a time, so a hot key caps throughput. - Is it too fine? Keying by event id spreads load perfectly and orders nothing.
How ordering breaks even with a key
A key gives you ordering from one producer, into one partition, through one consumer. Each of the following quietly breaks one of those links:
- Adding partitions. The key-to-partition mapping is
hash(key) % partitionCount. Going from 3 to 6 partitions moves many keys to new partitions. Events produced after the change can be consumed before older events still waiting on the old partition. Plan partition counts up front, or drain before changing. - Producer retries with several in-flight requests. If batch 1 fails and is retried while batch 2 succeeds, they can land out of order. Kafka’s idempotent producer (
enable.idempotence=true, the default in recent clients) preserves order with up to 5 in-flight requests per connection. - Retry and dead-letter topics. A consumer that can’t process event v2 sends it to a retry topic and moves on to v3. When v2 comes back from the retry topic, it’s behind v3. This is the most common source of reordering in practice.
- Concurrent processing within a consumer. Pulling a batch and processing it on a thread pool reorders events unless you partition the work by key inside the consumer too.
- Multiple producers. If two services both emit events about an order, their events have no shared sequence.
Versions: detecting order instead of assuming it
Since a key alone can’t cover every case, robust consumers check. The producer stamps each event with the entity’s version (1, 2, 3 for one order’s lifecycle; the aggregate’s version column works well). The consumer stores the last version it applied per entity and compares:
const view = { status: null, version: 0 };
function apply(e) {
if (e.version <= view.version) return `ignored v${e.version} ${e.type} (already at v${view.version})`;
if (e.version > view.version + 1) return `parked v${e.version} ${e.type} (waiting for v${view.version + 1})`;
Object.assign(view, { status: e.type, version: e.version });
return `applied v${e.version} ${e.type}`;
}Feeding it v1, then v3 (early, because v2 went through a retry topic), then v2, then a redelivered v3:
version-guarded consumer, arrival order v1, v3, v2, v3(retry):
applied v1 OrderPlaced
parked v3 OrderShipped (waiting for v2)
applied v2 OrderPaid
applied v3 OrderShippedThe early v3 isn’t applied over a missing v2. It’s parked (kept in a small buffer or left for redelivery), v2 fills the gap, and v3 applies when it arrives again. An older event arriving after a newer one is ignored. The same check makes the consumer resilient to duplicates as a side effect.
Whether you need the “parked” branch depends on the consumer. A status view that only cares about the latest state can just ignore anything older than what it has and accept gaps. A consumer that needs every step (an audit log, a balance computed from deltas) must wait for the gap to close.
Interview practice: refunds before payments
“A finance service consumes PaymentCaptured and PaymentRefunded events from a topic keyed by paymentId. Occasionally a refund is processed before its capture, and the finance ledger rejects it as a refund of an unknown payment. The producer only publishes a refund after the capture is committed. How can this happen, and what would you change?”
Answer
With both events keyed by paymentId on one partition and a single producer, Kafka itself won’t reorder them. So something else does. The likeliest cause is the consumer’s own error handling: if processing the capture failed transiently (a ledger timeout), and the consumer sent it to a retry topic and continued, the refund behind it was processed first. Other candidates: the capture and refund are published by two different services or instances without a shared sequence, or partitions were added and the payment’s key moved between the two events.
I’d check the retry path first, because it’s the common one. The fix there is not to skip ahead within a key: either retry in place (block that partition briefly) or, if one key’s failures mustn’t block others, park all later events for that same paymentId until the earlier one succeeds. Then I’d add a per-payment version to the events, so the ledger can recognise a refund that arrives before version 1 and hold it rather than reject it. Rejection plus an alert is acceptable as a last resort, but it shouldn’t be the normal path for an ordering gap the system creates itself.