Microservices Transactional Outbox and Event Delivery

An order service has two jobs when a customer checks out: save the order in its database, and tell the rest of the system with an OrderPlaced event so payments, inventory and email can do their part. The obvious code does one after the other. That code has a gap in the middle, and sooner or later something falls into it.

The transactional outbox is the standard fix. It’s one of the most frequently asked microservices patterns in interviews because it tests whether you understand what a transaction can and can’t cover.

The gap between commit and publish

Whichever order you do the two steps in, there’s a crash point between them:

• Commit, then publish. The order is saved. Then the broker is unreachable, or the process is killed during a deploy. The order exists, but no event was ever sent. Payment is never requested, the warehouse never hears about it, and nothing retries because nothing remembers that an event was owed.
• Publish, then commit. The event goes out. Then the database commit fails (a constraint violation, a deadlock, a failover). Downstream services now act on an order that doesn’t exist.

You can’t wrap both in one transaction, because the database and the broker are separate systems. Two-phase commit across them (XA) exists in theory, but most brokers don’t support it, and where they do, it couples your availability to both systems at once. Kafka’s own transactions don’t help either: they make a set of Kafka writes atomic with each other, not with your Postgres commit.

This problem has a name, the dual write, and it’s worth saying that name in an interview.

Putting the event in the same transaction

The outbox trick is to stop needing two systems at the moment of the business change. Instead of publishing, the service inserts the event into an outbox table in its own database, in the same transaction as the order. Either both rows commit or neither does. A separate process, the relay, reads unpublished outbox rows and sends them to the broker.

Here’s a complete model in Python’s sqlite3, including the dual-write version for comparison:

db.executescript("""
  CREATE TABLE orders (id TEXT PRIMARY KEY, customer_id TEXT NOT NULL, total_cents INTEGER NOT NULL);
  CREATE TABLE outbox (
    seq          INTEGER PRIMARY KEY AUTOINCREMENT,
    event_id     TEXT NOT NULL UNIQUE,
    aggregate_id TEXT NOT NULL,
    type         TEXT NOT NULL,
    payload      TEXT NOT NULL,
    published_at TEXT);
""")

# --- the dual write: commit, then publish ---------------------------------
def place_order_dual_write(order_id, customer, total):
    db.execute("INSERT INTO orders VALUES (?, ?, ?)", (order_id, customer, total))
    publish({"type": "OrderPlaced", "orderId": order_id})

# --- the outbox: both rows in one transaction ------------------------------
def place_order(order_id, customer, total):
    db.execute("BEGIN")
    db.execute("INSERT INTO orders VALUES (?, ?, ?)", (order_id, customer, total))
    db.execute("INSERT INTO outbox (event_id, aggregate_id, type, payload) VALUES (?, ?, 'OrderPlaced', ?)",
               (str(uuid.uuid4()), order_id, json.dumps({"orderId": order_id, "totalCents": total})))
    db.execute("COMMIT")

def relay_once(crash_before_mark=False):
    rows = db.execute("SELECT seq, event_id, type, payload FROM outbox "
                      "WHERE published_at IS NULL ORDER BY seq LIMIT 100").fetchall()
    for seq, event_id, type_, payload in rows:
        publish({"eventId": event_id, "type": type_, **json.loads(payload)})
        if crash_before_mark:
            raise SystemExit  # process dies after the broker accepted the event
        db.execute("UPDATE outbox SET published_at = CURRENT_TIMESTAMP WHERE seq = ?", (seq,))

publish appends to a list standing in for Kafka, and raises ConnectionError when the “broker” is down. The script places one order with the dual write while the broker is down, places a second through the outbox while the broker is still down, then brings the broker back and runs the relay twice, crashing the first time right after a successful publish:

dual write: broker unavailable
  orders in db: 1 | events on broker: 0
outbox, broker down: broker unavailable
  unpublished outbox rows: 1
relay crashed after publishing, before marking the row
  events on broker: [('o-2', '23b39edd'), ('o-2', '23b39edd')]
  unpublished outbox rows: 0

(The event id is a random UUID, so the prefix differs between runs.)

Read it top to bottom:

1. The dual write saved order o-1 and lost its event. There’s no record anywhere that an event was owed.
2. With the outbox, the broker outage didn’t stop the order from being placed, and the event is safe in the outbox table. The relay failed, but nothing was lost; the row is still unpublished.
3. When the broker returned, the relay published the event and then crashed before marking the row. On restart, it found the row still unpublished and sent it again. The broker now has the same event twice, with the same eventId.

The outbox gives you at least once, not exactly once

That last step isn’t a bug in the example. It’s inherent to the pattern. The relay publishes to one system and records success in another, which is the same dual-write gap moved somewhere safer. Moving it to the relay means the failure mode changes from “lost event” to “duplicate event”, and duplicates are fixable.

That’s why the outbox row has a stable event_id generated at insert time. Every copy of the event carries it, and consumers keep a processed-events record keyed on it, inserted in the same transaction as their own changes. Any explanation of the outbox that promises exactly-once delivery is wrong, and interviewers listen for this.

Running the relay in production

The toy relay polls. Real implementations take one of two shapes:

• Polling publisher. A loop runs SELECT ... WHERE published_at IS NULL ORDER BY seq LIMIT 100 every few hundred milliseconds. Simple, works with any database, adds some load and some latency.
• Log tailing (change data capture). A tool like Debezium reads the database’s replication log (Postgres WAL, MySQL binlog) and turns outbox inserts into Kafka messages. Lower latency and no polling queries, but another component to run and monitor.

Either way, a few details matter:

• Ordering. Publish in seq order and use the aggregate id as the message key, so events for one order stay in sequence on one partition. If you run several relay instances for availability, make sure only one is active (a leader lock), or partition the work by aggregate. Two relays racing over the same rows can publish v2 before v1.
• Cleanup. Delete or archive published rows after a few days, or the table grows without bound and the polling query slows down.
• Monitoring. Alert on the age of the oldest unpublished row. If the relay dies silently, orders keep succeeding and events quietly pile up, which looks healthy until someone asks why no confirmation emails went out today.
• Trace context. Store the current traceparent in the outbox row, so the published message continues the request’s trace.

In Spring, the business code stays simple because both writes share the @Transactional boundary (illustrative):

@Transactional
public Order placeOrder(PlaceOrder cmd) {
    Order order = orders.save(Order.from(cmd));
    outbox.save(OutboxEvent.of("OrderPlaced", order.getId(), toJson(order)));
    return order;
}

No broker client appears in the method at all, which also makes it easy to test.

Interview practice: publishing from an event listener

“A teammate implemented event publishing with a Spring @TransactionalEventListener(phase = AFTER_COMMIT) that sends OrderPlaced to Kafka once the order transaction commits. They argue this solves the dual-write problem because the event is only sent after a successful commit. Do you agree?”

Answer

It solves half of it. Publishing after commit means the event never describes an order that rolled back, so the phantom-event case is gone. But the lost-event case remains. If the process crashes, is killed during a deploy, or Kafka is unavailable in the moments after commit, the listener never sends, and nothing records that an event was owed. In-memory retries don’t survive a restart.

I’d replace it with an outbox: insert the event row in the same transaction as the order, and have a relay (a polling job or Debezium) publish it. The listener approach has a subtler problem too: the order and the attempt to publish are separated by the process’s memory, so there’s no way to audit afterwards which orders are missing their events. With an outbox, a query for old unpublished rows answers that immediately. I’d also make sure consumers deduplicate by event id, because the outbox relay can publish twice.

Further reading

• Microservices interview questions
• Microservices MCQs
• Consumers and duplicate events
• Kafka exactly-once and transactions
• Pattern reference: Transactional outbox on microservices.io