Every service has a maximum rate at which it can finish work. When requests arrive faster than that, the extra ones have to go somewhere. They can wait in a queue, they can be rejected, or the producer can be told to slow down. Backpressure is the last of those: a signal that flows upstream from a saturated consumer. Load shedding is the second: deliberately refusing some work so the rest still succeeds.
What interviewers want to hear is that you understand why these matter, and the reason isn’t obvious until you put numbers on it. A queue feels like a safety net. Under sustained overload it’s the opposite: it turns a slowdown into a situation where the server is busy, the CPU is at 100%, and almost nobody gets a useful answer.
What a queue actually buys you
Take a service with 10 worker threads where each request needs 100 ms of work. Its capacity is 10 / 0.1 s = 100 requests per second. Nothing you do with queues changes that number. A queue only decides two things: how long accepted requests wait, and what happens to the ones that don’t fit.
Little’s law makes the waiting part concrete. Time spent waiting equals queue length divided by throughput. With 100 requests queued and 100 completed per second, a new request waits about one second before a worker even looks at it. With 3,000 queued, it waits 30 seconds. That’s why a sensible bound comes from latency, not memory: if your clients time out at two seconds, a queue longer than about 200 items is storing requests that are already dead.
A short burst is fine. If traffic spikes to 150 req/s for two seconds, a queue absorbs the extra 100 requests and drains them over the next couple of seconds. The trouble starts when the overload lasts longer than the queue can hide.
Simulating a sustained spike
To see the difference, here’s a small discrete simulation in Node 22. It ticks once per millisecond, sends 150 req/s at the 100 req/s service for 60 seconds, and treats any response that takes longer than 2 seconds as wasted, because the client has already timed out and probably retried.
function run({ limit, seconds = 60, rate = 150, workers = 10, serviceMs = 100, timeoutMs = 2000 }) {
const queue = [], busy = new Array(workers).fill(0), inflight = new Array(workers).fill(null);
let rejected = 0, ok = 0, late = 0, maxQueue = 0, acc = 0;
const lat = [];
for (let t = 0; t < seconds * 1000; t++) {
acc += rate / 1000;
while (acc >= 1) { acc -= 1; if (queue.length >= limit) rejected++; else queue.push(t); }
for (let w = 0; w < workers; w++) {
if (busy[w] > 0 && --busy[w] === 0) {
const l = t - inflight[w];
if (l > timeoutMs) late++; else { ok++; lat.push(l); }
}
if (busy[w] === 0 && queue.length) { inflight[w] = queue.shift(); busy[w] = serviceMs; }
}
maxQueue = Math.max(maxQueue, queue.length);
}
lat.sort((a, b) => a - b);
const p = (q) => lat[Math.floor(q * (lat.length - 1))];
return { limit, servedInTime: ok, servedTooLate: late, rejectedFast: rejected, maxQueue, p50: p(0.5), p99: p(0.99) };
}
for (const limit of [Infinity, 100, 20]) {
const r = run({ limit });
console.log(`limit ${limit}: inTime ${r.servedInTime}, tooLate ${r.servedTooLate}, rejected ${r.rejectedFast}, ` +
`maxQueue ${r.maxQueue}, p50 ${r.p50} ms, p99 ${r.p99} ms`);
}The output:
limit Infinity: inTime 580, tooLate 5410, rejected 0, maxQueue 3000, p50 1034 ms, p99 2000 ms
limit 100: inTime 5990, tooLate 0, rejected 2900, maxQueue 100, p50 1093 ms, p99 1094 ms
limit 20: inTime 5990, tooLate 0, rejected 2980, maxQueue 20, p50 293 ms, p99 294 msRead the first line carefully. The unbounded server did exactly as much work as the others (about 5,990 completions in 60 seconds, its physical limit), but only 580 of them reached a client that was still waiting. The other 5,410 were computed for nobody. The queue grew to 3,000 entries and was still holding 3,000 requests when the spike ended, so the service would stay slow for another 30 seconds after traffic returned to normal. Meanwhile clients saw timeouts, retried, and added even more load. This failure has a name: goodput collapse. Throughput looks healthy on the dashboard while useful throughput is close to zero.
With a bound of 100, the server rejected 2,900 requests immediately and served every accepted one in time. The p99 is about 1.1 seconds, which is just Little’s law again: 100 queued divided by 100 per second. Shrink the bound to 20 and the p99 drops to 294 ms, with only 80 more rejections. A rejection that takes one millisecond is far kinder to a client than a success that arrives after it stopped listening.
Telling the producer to slow down
Rejecting is load shedding. Backpressure is making the rejection carry information so the producer adjusts. The mechanism depends on the protocol:
- HTTP. Return
429 Too Many Requests(or503 Service Unavailablewhen the whole service is overloaded) with aRetry-Afterheader. A500is the wrong code, because many clients treat it as a transient fault and retry immediately, which makes the overload worse. - Reactive streams. Project Reactor, RxJava and Akka Streams use a demand signal: the subscriber calls
request(n)and the publisher may send at mostnitems. A slow consumer simply asks for less. - In-process queues. A bounded
BlockingQueuewithput()makes the producing thread wait. Java’sThreadPoolExecutorwith a bounded queue andCallerRunsPolicydoes something similar: when the queue is full, the submitting thread runs the task itself, which naturally slows it down. - Message brokers. With Kafka the consumer pulls, so it’s naturally paced; the backlog shows up as consumer lag instead of memory pressure. With push-based brokers like RabbitMQ, a prefetch limit (
basic.qos) caps how many unacknowledged messages a consumer holds. - TCP. The receive window is backpressure at the transport layer. If your application reads from a socket slowly, the sender eventually stops.
The common mistake is a signal nobody honours. A 429 sent to a client that retries in a tight loop without backoff is just extra load. Pair server-side shedding with client-side exponential backoff with jitter and a retry budget, so retries can’t multiply traffic during an incident.
Choosing what to drop
Under real overload you can’t serve everything, so decide in advance what goes first. A typical ordering for an e-commerce API:
- Keep: health checks, payment and checkout writes, authentication.
- Degrade: product pages can serve cached or simplified responses.
- Shed first: recommendation calls, analytics events, search-as-you-type, prefetch requests.
Put that ordering into code. Tag requests with a priority (from the route, or a header set by the gateway), give each priority its own bounded queue or concurrency limit, and reject low-priority work at a lower load threshold. Check deadlines too: if a request carries a deadline that has already passed while it waited, drop it before doing the work.
Watch out for where the check happens. Shedding is only cheap if it happens early, before you parse a large body, open a database transaction or call three downstream services. The AWS Builders’ Library article on load shedding makes the point that rejecting a request must cost much less than serving it, or the service will still fall over while “only” rejecting.
Fixed limits are a starting point. Adaptive concurrency limits (Netflix’s concurrency-limits library is the usual example) watch latency and lower the limit when it rises, which handles the case where a slow downstream dependency suddenly reduces your real capacity.
Interview exercise: the shared worker pool
A checkout service and an analytics ingestion endpoint run in the same Spring Boot application and share one Tomcat thread pool of 200 threads. During a marketing campaign, analytics traffic triples and checkout latency goes from 120 ms to 9 seconds. What do you change?
Answer and reasoning
The root problem is that both workloads compete for the same capacity with no limit on either, so the cheaper, less important one crowds out the critical one. Fix it in layers.
First, isolate. Cap analytics at, say, 40 concurrent requests with a semaphore or a separate bounded executor, so it can never hold more than a fifth of the threads. That’s a bulkhead. Checkout keeps the rest.
Second, shed analytics early. When the analytics limit is reached, return 429 with Retry-After: 5 before reading the request body. Analytics clients batch events, so a short delay costs nothing important. Checkout should get its own, higher limit and a 503 only when it truly can’t keep up.
Third, decouple analytics from the request path. Accept the event, write it to a bounded buffer or straight to Kafka, and acknowledge. Consumer lag then absorbs the spike instead of HTTP threads.
Finally, check the clients. If the analytics SDK retries without backoff, the 429s will be hammered. Add jittered backoff and verify with a load test past the limit that checkout p99 stays flat, rejections are fast, and memory doesn’t climb.