Retries are the first resilience tool most people reach for, and for good reason. A dropped connection or a brief leader election in a database fails one attempt and succeeds on the next. A retry turns a visible error into a few extra milliseconds.
The problem is what retries do when the failure isn’t brief. A dependency that’s struggling under load gets every request two, three or four times. The extra load makes it slower, which causes more timeouts, which cause more retries. Retries designed to recover from failures end up sustaining them. The usual name for this is a retry storm, and the usual fix is a retry budget.
How retries multiply across layers
A typical request path: the gateway calls orders, orders calls inventory, inventory queries its database. Suppose each layer has a reasonable-looking policy of “up to 3 retries” (4 attempts). When the database is failing, how many database calls does one user request produce?
function layeredCalls(depth, attemptsPerLayer, bottomFails) {
let bottomCalls = 0;
const call = (level) => {
if (level === depth) { bottomCalls++; return !bottomFails; }
for (let a = 0; a < attemptsPerLayer; a++) if (call(level + 1)) return true;
return false;
};
call(0);
return bottomCalls;
}database calls caused by ONE user request while the database is failing:
1 attempt(s) per layer, 3 layers: 1
2 attempt(s) per layer, 3 layers: 8
4 attempt(s) per layer, 3 layers: 64Four attempts per layer gives 4 × 4 × 4 = 64. The database, already failing, receives 64 times its normal traffic from the same users. Even one retry per layer gives 8x. Nobody chose “multiply database load by 64 during outages” as a policy; it emerged from three teams independently picking sensible-sounding defaults. Add a client app that also retries and it’s 256.
The first rule follows directly: retry at one layer only. Usually that’s the layer closest to the failing dependency (it knows most about the error) or the outermost one (it knows the user’s deadline). Every other layer passes the failure up without retrying.
A budget instead of a fixed count
Even at a single layer, “up to 3 retries” means up to 4x load when the dependency is fully down. A retry budget changes the question from “how many times may this request retry?” to “what fraction of my traffic may be retries?”
The common formulation, described in the Google SRE book and implemented in Finagle, gRPC’s retry throttling and Envoy: retries may add at most 10% on top of first attempts. A token-bucket implementation is small:
class RetryBudget {
constructor(ratio = 0.1, minPerSec = 10) { this.ratio = ratio; this.tokens = minPerSec; }
onRequest() { this.tokens = Math.min(this.tokens + this.ratio, 1000); }
tryRetry() { if (this.tokens >= 1) { this.tokens -= 1; return true; } return false; }
}Every first attempt earns 0.1 tokens; every retry costs 1. While failures are rare, there are plenty of tokens and every failed request can retry. When failures become common, tokens run out and retries stop, so the load can’t exceed about 1.1x. (A production version would also refill a small minimum per second so a quiet service can still retry; that’s what minPerSec hints at, simplified here to a starting balance.)
Simulating 10,000 requests through one layer with up to 3 retries, with and without the budget, at three failure rates:
one layer, 10,000 requests, up to 3 retries:
1% failing, no budget load 1.01x success 100.0%
1% failing, 10% budget load 1.01x success 100.0%
30% failing, no budget load 1.41x success 99.4%
30% failing, 10% budget load 1.10x success 77.7%
100% failing, no budget load 4.00x success 0.0%
100% failing, 10% budget load 1.10x success 0.0%The three pairs tell the whole story, including the uncomfortable part.
1% failing (a blip): the budget is invisible. Every failure gets its retries, success is 100% (rounded), load is 1.01x. This is the case retries exist for.
100% failing (an outage): retries can’t help, since nothing succeeds. Without a budget, the dependency receives 4x its normal load at the worst possible moment, which can stop it from recovering even after the original cause is fixed. With the budget, it receives 1.1x.
30% failing (partial failure): here the budget costs something real. Without it, retries rescue almost everything (99.4%) at 1.41x load. With it, success drops to 77.7%. If the 30% failure is random (one bad instance behind a load balancer), retries were helping and the budget got in the way. If it’s because the dependency is overloaded, the extra 41% of load was making things worse.
That’s the honest trade-off to describe in an interview. A budget gives up some recovery during partial failures in exchange for bounded load during total ones. The ratio is a tuning knob: 10% is a common default, and services that commonly see random single-instance failures may choose 20%. What you shouldn’t do is have no ceiling at all.
Retrying well within the budget
A budget limits how many retries happen. Several other rules determine whether they’re useful:
- Retry only what might succeed. Timeouts, connection resets, 502, 503, 504. Not 400, 401, 403, 404, 409 or 422: the same request will get the same answer.
- Respect the server’s signal. A 429 or 503 with
Retry-After: 5means “not before 5 seconds”. gRPC servers can send pushback metadata telling the client not to retry at all. Overloaded services should use these signals, and clients should obey them. - Back off exponentially with jitter. Wait about 100 ms, then 200, then 400, each randomised. Without jitter, a thousand clients that failed at the same moment retry at the same moment, and the synchronised wave fails again. “Full jitter” (a random delay between zero and the exponential cap) spreads them out well.
- Stay inside the deadline. A retry with 30 ms left on a 1-second deadline costs the dependency real work and can’t help the caller.
- Only retry idempotent operations, or ones protected by an idempotency key. Retrying
POST /paymentsafter a timeout without a key is how customers get charged twice.
Resilience4j’s Retry in Spring handles backoff and exception filtering (illustrative):
resilience4j:
retry:
instances:
inventory:
maxAttempts: 3
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
enableRandomizedWait: true
retryExceptions:
- java.net.SocketTimeoutException
- org.springframework.web.client.HttpServerErrorExceptionResilience4j doesn’t ship a token-bucket retry budget, so teams either combine its retries with a circuit breaker (which stops retries once failures are clearly sustained) or enforce budgets in the service mesh, where Envoy supports a retry budget as a percentage of active requests.
Interview practice: the outage that wouldn’t end
“The catalog database had a 2-minute failover. Afterwards, the database was healthy, but the catalog service’s error rate stayed high for 25 minutes and only recovered when traffic was manually shed at the gateway. What do you think happened, and what would you change?”
Answer
This looks like a retry storm sustaining itself after the trigger was gone. During the failover, every layer retried: the web client, the gateway, the catalog service and its database driver. The database came back to a flood of multiplied traffic, plus a queue of retries scheduled during the outage, and couldn’t serve it fast enough. Requests timed out, which caused more retries, keeping load above capacity. Shedding traffic at the gateway broke the loop by cutting the input below what the database could handle.
I’d change the retry design across the path. First, decide on one retrying layer for catalog reads, probably the catalog service itself since it understands which database errors are transient, and turn retries off in the gateway and the client for this path, or have them honour Retry-After. Second, give that layer a retry budget of around 10%, so retries can never more than marginally increase load during a sustained failure. Third, use exponential backoff with full jitter so retries don’t arrive in synchronised waves after a failover. I’d also add load shedding in the catalog service itself (reject quickly when its in-flight count is too high), so recovery doesn’t depend on someone manually intervening at the gateway.