The name comes from ships. A hull is split into watertight compartments, so a breach floods one section instead of sinking the whole vessel. In a service, the “water” is a slow dependency, and the compartments are separate limits on threads, connections or concurrent calls.
The failure a bulkhead prevents is easy to describe and surprisingly common. A checkout service calls two things: a payment provider (critical) and a recommendations service (nice to have, shows “you might also like”). Recommendations starts responding in 2 seconds instead of 15 milliseconds. Every request handler that calls it now holds a thread or a connection for 2 seconds. The pool fills up with requests waiting on recommendations, and the payment calls, which are perfectly healthy, can’t get a slot. Checkout is down because of a widget.
Watching a shared pool starve a healthy dependency
Here’s a small bulkhead in Node: a counter of active calls, a cap, and a bounded wait for a free permit. If no permit frees up in time, the call is rejected immediately instead of queueing forever.
class Bulkhead {
constructor(name, maxConcurrent, maxWaitMs) {
Object.assign(this, { name, maxConcurrent, maxWaitMs, active: 0, waiters: [] });
}
async run(fn) {
if (this.active >= this.maxConcurrent) {
await new Promise((resolve, reject) => {
const w = { resolve, timer: setTimeout(() => {
this.waiters.splice(this.waiters.indexOf(w), 1);
reject(new Error(`bulkhead ${this.name} full`));
}, this.maxWaitMs) };
this.waiters.push(w);
});
}
this.active++;
try { return await fn(); }
finally {
this.active--;
const next = this.waiters.shift();
if (next) { clearTimeout(next.timer); next.resolve(); }
}
}
}
const recommendations = () => sleep(400); // degraded: normally 15 ms
const payments = () => sleep(10);The test sends 60 requests, one every 5 ms: 40 to recommendations and 20 to payments, interleaved. First all of them share one pool of 10 permits. Then recommendations gets 4 permits and payments gets 6. Same total capacity. Real output from Node 22:
one shared pool of 10
recommendations ok=10 rejected=30
payments ok=4 rejected=16
separate pools (recs 4, payments 6)
recommendations ok=4 rejected=36
payments ok=20 rejected=0With one pool, recommendations calls grabbed all 10 permits within the first 75 ms and held each for 400 ms. Only the 4 payments that arrived before the pool filled succeeded; the other 16 were rejected even though the payment provider answered in 10 ms. With partitions, recommendations still suffered (it was degraded, nothing will fix that), but every single payment succeeded.
That’s the whole idea. A bulkhead doesn’t make the slow thing faster. It decides who pays for the slowness.
What you can partition
Interviewers like it when you can name concrete resources, not just “isolate things”:
- Concurrent calls per dependency. The semaphore above. Resilience4j’s
SemaphoreBulkheaddoes exactly this. - Thread pools per dependency. Resilience4j’s
ThreadPoolBulkhead(and Hystrix before it) run each dependency’s calls on its own executor. Useful with blocking clients, at the cost of thread handoff. - Connection pools. One HTTP client and pool per downstream instead of one global pool. A database connection pool per workload: interactive queries separate from the reporting job that runs a 40-second aggregate.
- Instances. Route bulk or partner traffic to a separate deployment of the same service, so a partner’s batch import can’t saturate the instances serving your own app. Taken further, cell-based architecture gives each group of customers its own full stack.
- Queues and consumers. A separate topic or consumer group for low-priority events, so a backlog of analytics events doesn’t delay payment events.
In Spring with Resilience4j, a semaphore bulkhead per dependency looks like this (illustrative configuration):
resilience4j:
bulkhead:
instances:
payments:
maxConcurrentCalls: 6
maxWaitDuration: 20ms
recommendations:
maxConcurrentCalls: 4
maxWaitDuration: 0@Bulkhead(name = "recommendations", fallbackMethod = "noRecommendations")
public List<Product> recommendationsFor(String customerId) { ... }
private List<Product> noRecommendations(String customerId, Throwable t) {
return List.of();
}When the bulkhead is full, Resilience4j throws BulkheadFullException and the fallback returns an empty list. The page renders without recommendations, which is exactly the degraded behaviour you want for an optional feature.
Sizing a partition with Little’s law
“How many permits?” is the follow-up you should expect. Guessing 10 or 50 isn’t an answer. Little’s law gives a starting point: the average number of requests in flight equals the arrival rate times the time each one spends in the system.
Say checkout makes 80 payment calls per second at peak, and the payment provider’s p99 latency is 250 ms. In-flight calls ≈ 80 × 0.25 = 20. Add headroom for bursts, maybe 30 permits. If you set it to 8, you’ll reject healthy traffic at peak. If you set it to 500, you haven’t built a bulkhead: a slowdown to 5 seconds would let 400 requests pile up before the limit does anything.
Two consequences are worth saying out loud:
- The limit has to be checked against the resources behind it. Thirty permits are pointless if the HTTP client’s connection pool to that host allows 10; the other 20 will wait inside the client, invisible to your bulkhead.
- The sum of partitions should fit the real capacity. If the service has 200 request threads and you give five dependencies 60 permits each, they can collectively exhaust the threads again.
Limits drift as traffic grows, so put the permit count, active count and rejection count on a dashboard. A bulkhead that rejects 0.5% of healthy traffic every day at 9am is mis-sized, not protective.
Waiting is where most bulkheads fail
Notice maxWaitMs in the example: 50 ms. That’s deliberate. The common mistake is to let callers wait indefinitely (or for 30 seconds) for a permit. Then the waiting requests pile up in front of the bulkhead, each holding an upstream thread, and you’ve moved the shared pool from inside the service to just outside it. Same outage, different place.
Rejection should be fast and should map to something the caller understands: a fallback for optional data, a 503 with Retry-After for the client, or an error the circuit breaker counts. Fast rejection under overload is a feature. It keeps latency predictable for the requests you do accept.
The other misconception is that a bulkhead replaces timeouts. It doesn’t. Without a timeout, a call to a hung dependency holds its permit forever, and the partition eventually fills with stuck calls and rejects everything for that dependency permanently. Timeouts free permits; bulkheads cap how many can be held; circuit breakers stop sending calls once failure is clear. You usually want all three, layered so the timeout is shortest.
Interview practice: the reporting query that took down login
“A user service handles login and also serves an admin reporting endpoint. Both use the same database connection pool of 20 connections. Every Monday morning, analysts run reports that take 30 to 60 seconds each, and login latency spikes to timeouts. Adding connections doesn’t help much. What would you do?”
Answer
This is shared-pool starvation. A few long reporting queries hold most of the 20 connections for a minute each, and short login queries queue behind them until they time out. Adding connections helps briefly but the reports expand to fill them, and the database itself starts to struggle.
I’d split the workloads. Give reporting its own small pool, say 3 connections with a short acquire timeout, so at most three report queries run at once and the fourth analyst gets “try again shortly” instead of degrading login. Login keeps its own pool sized from its rate and latency. Better still, point reporting at a read replica or a reporting database, so the partition extends to the database CPU and I/O, not just the client-side pool.
I’d also set a statement timeout on reporting queries and add metrics for pool wait time per pool. If the reporting pool shows constant rejections, that’s a capacity conversation about reporting, and login stays unaffected while it happens.