System Design Graceful Degradation and Correctness

A product page on a typical online shop calls six services: catalog, pricing, inventory, reviews, recommendations and personalisation. If the page fails whenever any of them fails, its availability is worse than the worst of them. If it renders without the optional ones, a recommendations outage costs you a carousel instead of the page.

That’s the idea behind graceful degradation: keep the core job working when parts of the system don’t. The hard part isn’t the code, which is mostly timeouts and try/catch. It’s deciding, for each dependency, what the product should do without it, and making sure the fallback doesn’t quietly tell users something false.

Availability multiplies

When a request needs every dependency to succeed, availabilities multiply (assuming failures are independent, which is optimistic). Here’s the product page with plausible numbers for each service:

deps = {  # name: (availability, hard dependency?)
    "catalog": (0.9995, True), "pricing": (0.9995, True), "inventory": (0.999, False),
    "reviews": (0.995, False), "recommendations": (0.99, False), "personalisation": (0.995, False),
}
all_hard = math.prod(a for a, _ in deps.values())
only_hard = math.prod(a for a, h in deps.values() if h)

page needs all six:          97.8166% available, ~943 min of failed pages per 30 days
page needs catalog+pricing:  99.9000% available, ~43 min per 30 days

Requiring all six gives 97.8%, about 15 hours of failed page loads a month, even though no single service is worse than 99%. Making only catalog and pricing required brings the page to 99.9%. Nothing got more reliable; the page just stopped depending on things it didn’t need.

That’s the first design step: list every dependency and mark it required or optional, with the product owner in the room. A page without reviews is still a product page. A page without a price isn’t something you can sell from.

Timeouts decide how bad a bad day gets

Availability isn’t the only cost of a dependency. A page that calls six services in parallel waits for the slowest one. Simulating each call with a 40 ms median and a realistic tail:

one call:                    p50 40 ms  p99 161 ms
wait for all six:            p50 84 ms  p99 231 ms
optional calls cut at 120ms: p50 84 ms  p99 187 ms  (12.7% of pages missing a section)
optional calls cut at 250ms: p50 84 ms  p99 231 ms  (0.4% of pages missing a section)

Waiting for six calls roughly doubles the median, because the page takes the worst of six draws. A tight 120 ms timeout on optional calls trims the tail, but look at the price: 12.7% of pages render without some section on a perfectly normal day, because 120 ms sits inside the dependencies’ normal latency range. A 250 ms timeout rarely fires on a normal day.

So why have it? Because of days like this one, where recommendations is overloaded and takes about four seconds to answer:

recommendations slow (~4 s), wait for all: p50 3995 ms  p99 8030 ms
recommendations slow (~4 s), cut at 250ms: p50 250 ms  p99 250 ms  (100.0% of pages missing a section)

Without a timeout, one sick optional service makes every product page take four to eight seconds, and it holds request threads and connections the whole time, which can take down the page service too. With a 250 ms cutoff, every page is missing its carousel and loads in a quarter of a second. Set optional timeouts just above the dependency’s normal p99 so they don’t fire on good days, and let them cap the damage on bad ones. A circuit breaker on top stops sending requests to a dependency that’s clearly down, which saves the 250 ms wait and gives the dependency room to recover.

Fallbacks must keep the data’s meaning

The tempting fallback is “return something”: a default, a cached value, a guess. The question to ask is what the user, or the next system, will do with it.

Walk through the product page:

“0 reviews” is a false statement: the product has reviews, you just can’t fetch them. “In stock” from a guess turns into cancelled orders and support tickets. The honest fallback admits the gap in a way the user can act on.

The same reasoning applies to decisions with risk, where the choice is between failing open and failing closed. If the fraud-scoring service is down, approving every order (open) keeps revenue flowing and exposes you to fraud; declining every order (closed) is safe and loses sales. Neither is right in general. The answer usually depends on the amount and the customer’s history, and it’s a business decision to make in advance, not one for an engineer at 3 a.m.

The fallback nobody tested

Amazon’s Builders’ Library has an essay arguing for avoiding fallback in distributed systems, and the argument is worth knowing for an advanced interview. A fallback path only runs when something is already wrong. It gets little testing, it may hit code and capacity that nobody has exercised in months, and it often shifts load somewhere unexpected. A classic example: the main database is slow, so the service “falls back” to a secondary data source that was never sized for full traffic, and now two systems are down.

Their preferred patterns:

• Make the normal path the resilient one. Instead of calling a config service on each request and falling back to a cached copy on failure, always read the local copy and refresh it in the background. The “fallback” is then the code that runs every time.
• Exercise degraded modes continuously. Kill switches that turn off recommendations or reviews should be flipped in production regularly, and fault-injection tests should time out each optional dependency on purpose.
• Watch the fallback rate. A metric for “pages served without inventory data” tells you when degradation becomes the normal state, which is a problem in itself.

Degrading on purpose under load

Degradation also works in the other direction: when the site is overloaded, turn off expensive optional features before the core suffers. During a flash sale, recommendations and personalisation might use half the page’s backend CPU. A load-shedding rule (“above 80% CPU, skip personalisation”) keeps checkout fast. That’s often called a brownout. It needs the same groundwork as failure fallbacks: features marked optional, a cheap way to skip them, and a page that still renders correctly without them.

Interview practice: checkout and the fraud service

“Checkout calls a fraud-scoring service before charging the card. It normally answers in 80 ms. Last week it was down for 25 minutes, and checkout failed for every customer. Product asks you to make checkout survive the next outage. What do you design?”

Answer

First I’d separate what checkout can’t do without from what it can. Charging the card is required; a fraud score is a risk control, and the business can choose how much risk to accept for a limited time. So I’d put a timeout on the fraud call around 300 ms (above its normal p99) and a circuit breaker so an outage stops costing 300 ms per order.

Then the fallback, which must be explicit and agreed with the risk team rather than “approve everything”. A reasonable policy: when the score is unavailable, approve orders under a value threshold from customers with a good order history, mark them fraud_check: pending, and score them asynchronously when the service recovers, holding shipment for any that score badly. Orders above the threshold, or from new accounts, get a clear “we couldn’t complete your order, please try again shortly” rather than a silent decline. That keeps most revenue flowing while bounding the exposure.

To keep this from rotting, I’d add a metric and alert on orders approved without a score, cap the total value approved in fallback mode, and run a monthly game day that forces the fraud call to time out, confirming that the pending-score queue drains and nothing ships before it’s scored.

Further reading

• System design interview questions
• System design MCQs
• Circuit breakers and failure isolation
• Backpressure and load shedding
• Amazon Builders’ Library: Avoiding fallback in distributed systems