Every network call needs a timeout. That part is familiar. The less familiar problem is what happens when a request crosses four services and each one picks its own timeout without knowing what the others chose. The client stops waiting after one second, but the services behind it keep working for another few hundred milliseconds, or several seconds, on an answer nobody will read.
A deadline fixes this. The client decides when it will stop waiting, and that point in time travels with the request. Every hop knows how much time is left and gives up when it runs out. gRPC builds this in; with HTTP you pass it yourself.
Watching a call chain outlive its caller
The model is a request that goes client → orders → inventory → pricing. Orders does 200 ms of local work, inventory does 300 ms, and pricing is having a bad day and takes 800 ms. The client’s deadline is 1,000 ms.
In the first run, each hop calls the next with its own fixed 1,000 ms timeout, which is a very common setup. In the second, each hop inherits the caller’s deadline. The relevant pieces of the Node 22 script:
// Each hop builds the context for its downstream call.
function childContext(ctx, mode, fixedTimeoutMs) {
if (mode === 'fixed') return { signal: AbortSignal.timeout(fixedTimeoutMs), deadline: null };
const remaining = ctx.deadline - (performance.now() - t0);
return { signal: AbortSignal.timeout(Math.max(Math.floor(remaining), 0)), deadline: ctx.deadline };
}
// Before starting work, refuse requests whose deadline has already passed.
function checkBudget(name, ctx) {
if (ctx.deadline === null) return;
const remaining = ctx.deadline - (performance.now() - t0);
if (remaining <= 0) { log(`${name} rejected: deadline already passed`); throw new Error('DEADLINE_EXCEEDED'); }
}
async function pricing(ctx, mode) {
checkBudget('pricing', ctx);
log('pricing starts (slow today: 800 ms)');
await work('pricing', 800, ctx); log('pricing finished'); return 42.5;
}work is a sleep that listens to ctx.signal, standing in for a database query or computation that can be cancelled. The output:
every hop uses its own 1000 ms timeout:
504 ms pricing starts (slow today: 800 ms)
1001 ms client gave up (1000 ms deadline)
1307 ms pricing finished
1307 ms inventory finished
1307 ms orders finished
every hop inherits the caller's deadline:
503 ms pricing starts (slow today: 800 ms)
1001 ms client gave up (1000 ms deadline)
1004 ms pricing abandoned its work
1004 ms orders returned error: pricing abortedIn the first run, pricing started at 504 ms with a fresh 1,000 ms allowance from inventory. It had no idea the client only had 500 ms of patience left. It finished at 1,307 ms, inventory and orders dutifully passed the result up, and the client had already shown an error 300 ms earlier. All of that work was waste.
In the second run, pricing received the same request but knew the deadline was 1,000 ms. It stopped at 1,004 ms, the error propagated up, and every service freed its resources at almost the same moment the client gave up.
Why the waste matters more than it looks
Three hundred milliseconds of wasted work per request sounds minor. The problem is what happens under load, which is exactly when a dependency is slow. If pricing is overloaded and slow, every request that has already been abandoned upstream still occupies a pricing worker, a database connection and an inventory thread. Pricing gets slower because it’s doing work for clients who left, more clients time out, they retry, and the retries add more abandoned work. Deadlines cut that loop: work stops when nobody’s waiting.
There’s a quieter version of the same bug: timeouts that grow as you go deeper. Orders gives inventory 2 seconds, inventory gives pricing 5 seconds. An inner timeout longer than the outer one can never be useful, because the outer caller will have given up first. Deadline propagation makes that mistake impossible, since an inner hop can never get more time than its caller has left.
Passing the deadline between services
Inside one process, a deadline is just an object you pass along: Go’s context.Context, a Java gRPC Context with a deadline, or the AbortSignal above. Between processes it has to go on the wire. Two choices:
- Absolute time (
x-deadline: 2026-10-07T09:15:02.350Z). Simple, but it trusts that both machines’ clocks agree. If the receiving host’s clock runs 400 ms fast, it believes the deadline is 400 ms closer than it is. - Remaining duration (
grpc-timeout: 650m, meaning 650 milliseconds). Each receiver converts it to a local deadline using its own clock the moment the request arrives. Clock skew between machines doesn’t matter; only network transit time is unaccounted for, and that’s small.
gRPC uses the second form. In Java, a client sets it with stub.withDeadlineAfter(800, TimeUnit.MILLISECONDS), and the server sees it on the incoming Context; by default, calls made from inside that context inherit it. Go’s gRPC does the same with ctx. If you use HTTP, define a header with the remaining milliseconds, set it in your shared client library, and read it in a middleware so individual handlers don’t have to think about it.
Rules each hop should follow
Propagating the number isn’t enough. Each service needs to use it:
- Take the minimum. A hop’s timeout for a downstream call is
min(its own configured timeout, time remaining). The configured timeout still protects against a caller who sent no deadline, or a generous one. - Reserve time for yourself. If 300 ms remain and you still need 40 ms to build the response after the downstream returns, give the downstream 260 ms, not 300.
- Check before starting. A request that sat in a queue or a thread pool may already be past its deadline when a worker picks it up. Reject it immediately with
DEADLINE_EXCEEDED(gRPC) or a 504, ascheckBudgetdoes. This is especially valuable for message consumers and job queues. - Fit retries inside the budget. A retry with 50 ms remaining isn’t a retry, it’s a guaranteed failure that costs the downstream real work. Skip retries when the remaining time is below the downstream’s normal latency.
- Cancel what you started. Abort in-flight HTTP calls, cancel database statements (or set statement timeouts from the remaining budget), and stop iterating over a batch. A deadline that only stops waiting, without stopping work, saves the caller but not the system.
When not to propagate
Not every downstream call belongs to the request’s budget. If placing an order also emits an event that triggers an email, the email shouldn’t be cancelled because the HTTP client disconnected. Work that must complete regardless (writing an outbox row, finishing a payment that has already been submitted) should run without the request’s deadline, or be handed off to something asynchronous. Part of the design is deciding which calls are “for this response” and which are “because this happened”.
Interview practice: a deadline and three retries
“A checkout API has a 2-second SLA. It calls a fraud service with a 1-second timeout and up to 3 retries. During a fraud service slowdown, checkout p99 jumps to 4 seconds and the fraud service’s queue keeps growing even after checkout traffic drops. Explain both symptoms and propose changes.”
Answer
Checkout’s p99 of 4 seconds comes from the retry policy ignoring the overall budget. One attempt can take 1 second, and with retries, up to 4 attempts fit in 4 seconds. The client gave up at 2 seconds, so at least half of that time was spent on a request nobody was waiting for.
The fraud queue keeps growing because abandoned requests are still in it. Checkout’s timeout stops checkout waiting, but the fraud service doesn’t know, so it processes every queued request, including retries of requests that already timed out. Retries also multiply the load at the worst moment.
I’d propagate the deadline from the checkout request to the fraud call: each attempt’s timeout is the minimum of 1 second and the time remaining minus a reserve for checkout’s own work. Retries only happen if enough time remains for a realistic attempt, which in practice means at most one retry. The fraud service reads the remaining time on arrival and when a worker dequeues the request, and drops requests whose deadline has passed. I’d also decide what checkout does when fraud can’t answer in time (queue the order for review, or decline), so a deadline produces a deliberate outcome rather than a generic error.