System Design Reviews Through Concrete Failure Cases

A design review that walks through the architecture diagram box by box rarely finds the problems that cause incidents. Boxes and arrows describe what happens when everything works. Outages happen in the gaps: a response that never arrives, a message delivered twice, a database that’s two seconds behind, a deploy that runs old and new code side by side.

The most productive review technique is also the simplest. Pick a concrete request, follow it hop by hop with real numbers, and at every hop ask what happens if this step is slow, fails, or happens twice. This guide applies that to an order-placement design, the kind of design an advanced interview asks you to critique.

The design under review

Here’s the proposal, as it might appear in a design doc:

1. The mobile app sends POST /orders with the cart and a saved card. The app’s HTTP client times out after 5 seconds and retries once.
2. An API gateway forwards to the order service, with a 30-second timeout.
3. The order service inserts an order row (status = 'pending'), calls the payment provider to charge the card, updates the row to paid, then publishes OrderPaid to Kafka.
4. A fulfilment service consumes OrderPaid and reserves stock in the warehouse system.
5. Expected load: 300 orders per second at peak. The payment provider usually answers in about 600 ms; its status page admits that around 2% of calls take 6 to 12 seconds.

On a diagram it’s five boxes and four arrows, and it looks fine.

Tracing one slow order

Follow a single order where the payment call is in the slow 2%:

• t = 0 ms. The app sends the request. The gateway forwards it. The order service inserts the pending row.
• t = 150 ms. The order service calls the payment provider.
• t = 5,000 ms. The app’s timeout fires. The user sees a spinner, then “Something went wrong, retrying”. The gateway is still holding the original request; its timeout is 30 seconds.
• t = 5,010 ms. The app’s retry arrives as a brand-new POST /orders. The order service inserts a second pending row and calls the provider again.
• t = 5,770 ms. The second charge, a normal fast one, succeeds. Order 2 is marked paid, and the app shows “Order confirmed”.
• t = 8,400 ms. The first charge finally succeeds too. The order service marks order 1 paid and publishes OrderPaid. The response goes back through the gateway to a connection the app abandoned three seconds ago.

The customer sees one confirmation and gets two charges and two shipments. Nothing crashed. Every component did what it was designed to do. The bug is in the contract between them: the caller gives up after 5 seconds while the callee keeps working for up to 30, and the retry has no way to identify itself as a repeat.

Putting numbers on it

Is this an edge case or a daily event? A simulation of 100,000 orders with that latency profile (5-second app timeout, one retry, payments usually around 600 ms with 2% taking 6 to 12 seconds) answers it:

for (const total of attempts) {
  if (!(idempotencyKey && charges > 0)) charges++;     // with a key, the repeat returns the first result
  if (total <= CLIENT_TIMEOUT) { succeeded = true; break; }
  finishedAfterClientLeft++;                            // the server kept going and charged anyway
}

as designed (no key)           double-charged  2087   app showed an error  45   attempts that finished after the app gave up 2132
idempotency key on the charge  double-charged     0   app showed an error  45   attempts that finished after the app gave up 2132

About 2.1% of orders are charged twice. At 300 orders per second during the evening peak, that’s several double charges every second. The second line adds an idempotency key (generated once by the app, reused on its retry, and passed through to the payment provider) and the double charges disappear. The 45 orders where both attempts were slow still fail, which is correct: the app shows an error and nobody is charged twice. The model treats the retry as taking as long as a fresh call, which simplifies what a real in-progress replay would do.

Notice what the key doesn’t fix: 2,132 attempts still finish after the app has left. That’s wasted work, and it hints at the second fix. The order service should know how long its caller is willing to wait (pass the remaining budget as a deadline header) and stop waiting on the provider when the budget runs out, then reconcile the charge’s outcome through the provider’s status API instead of guessing.

Turning traces into a failure table

After two or three traces like that, write the findings down as a table. It’s the most useful artifact a review produces, because each row is concrete and has an owner:

The last three rows come from the same habit applied further down. “Update the row, then publish to Kafka” is two writes to two systems: if the process dies between them, the database says paid and no event ever goes out. Kafka consumers get at-least-once delivery, so fulfilment will see some events twice. None of this appears on the diagram.

Questions that find real gaps

You can’t trace every request, so a reviewer needs a handful of questions that reliably surface trouble. These work across most designs:

• For every network call: what’s the timeout, who retries, and is the operation safe to repeat? Retrying at several layers multiplies load during outages.
• For every pair of writes to different systems: what if the process dies in between? This finds dual-write bugs like the database-then-Kafka step.
• For every read: how stale can it be, and does anything make a decision on it? Reading stock from a replica for display is fine; reserving from it isn’t.
• What happens at 10x the expected load, and which component fails first? If the answer is “the database connection pool”, say what the service does when it can’t get a connection.
• How does an operator find out? If the only signal for stuck orders is customer complaints, the design needs a metric (orders in pending older than 5 minutes) and an alert.
• What happens during a deploy? Old and new versions run together; can both read each other’s messages and rows?

A review isn’t an interrogation. The goal is a short list of changes, and asking “what would we see if this failed?” invites the author to find the gap rather than defend against it.

Interview practice: review a notification design

“Here’s a design: when a user is mentioned in a comment, the comment service writes the comment to PostgreSQL, then calls the notification service over HTTP with a 2-second timeout. The notification service sends a push notification through a third-party provider and returns 200. If the call fails, the comment service logs the error. Review it.”

Answer

I’d trace three cases. A normal mention works. Now the notification service is slow, taking 3 seconds because the push provider is lagging. The comment service times out at 2 seconds and logs an error, but the notification service carries on and sends the push. If anything retries, the user gets two pushes; if nothing retries, the log says “failed” for a push that actually went out, so the log can’t be trusted.

Second case: the notification service is down for ten minutes. Every mention in that window is logged and lost. Nobody will replay error logs, so those users are never notified. Third case: the comment service crashes after committing the comment but before calling notifications, which also loses the mention silently.

The underlying problem is a synchronous call for work that doesn’t need to be synchronous, plus a dual write. I’d have the comment service write a mention_created row to an outbox table in the same transaction as the comment, with a relay publishing it to a queue. The notification service consumes from the queue, deduplicates on the mention id so redelivery doesn’t double-push, and retries the provider with backoff. Comment posting no longer depends on notification health, mentions survive outages, and a metric on queue age tells operators when notifications fall behind.

Further reading

• System design interview questions
• System design MCQs
• The transactional outbox and event delivery
• Deadlines across call chains
• Amazon Builders’ Library: Challenges with distributed systems