Timeouts, circuit breakers, fallbacks and retries are all claims about how a system behaves when something fails. Most of those claims are never checked until a real incident checks them. Resilience testing means checking them on purpose: inject a specific failure, observe what the system does, and compare that with what you expected.
The phrase “chaos engineering” makes it sound like randomly breaking things in production. The discipline is closer to the opposite: a controlled experiment with a written hypothesis, a limited scope and a way to stop. Interviewers asking about this topic usually want to hear that structure, plus some sense of which failures are worth testing.
The shape of an experiment
Every useful failure test answers the same questions, in this order:
- What does normal look like? Pick steady-state metrics that reflect user experience, not CPU graphs: checkout success rate, product page p99 latency, orders per minute.
- What do we expect when X fails? Write it down before running anything. “If recommendations is slow or failing, product pages still load in under 300 ms at p99 and success stays above 99.5%, because recommendations has a fallback.”
- What’s the blast radius? One instance, one test environment, 1% of production traffic. Start small.
- When do we stop? An abort condition that’s checked automatically: “if checkout success drops below 99% for 2 minutes, remove the fault.”
- What happened after we removed the fault? Did the system return to steady state, and how long did it take?
Writing the hypothesis first matters. Without it, any outcome can be rationalised afterwards (“well, a few errors are expected”).
An experiment that passes on errors and fails on latency
Here’s a small version of that recommendations experiment. A fault injector wraps the dependency with configurable extra latency and error rate. The product page handler exists in two versions: v1 as shipped, and v2 after the experiment’s findings.
// Fault injection: wraps a dependency call with extra latency and errors.
const faults = { latencyMs: 0, errorRate: 0 };
async function recommendations(signal) {
await sleep(15 + faults.latencyMs, undefined, { signal });
if (random() < faults.errorRate) throw new Error('503 from recommendations');
return ['p-1', 'p-2'];
}
async function productPageV1() { // as shipped
const recs = await recommendations().catch(() => []);
return { recs };
}
async function productPageV2() { // after the experiment
const recs = await recommendations(AbortSignal.timeout(150)).catch(() => []);
return { recs };
}The test driver fires 200 concurrent requests per phase and reports success rate and latency percentiles. Output from Node 22:
hypothesis: p99 < 300 ms and success >= 99.5% while recommendations is degraded
v1 (no timeout) baseline: success 100.0% p50 17 ms p99 17 ms
v1 (no timeout) 30% errors: success 100.0% p50 17 ms p99 17 ms
v1 (no timeout) +2000 ms latency: success 100.0% p50 2018 ms p99 2019 ms
v2 (150 ms timeout) baseline: success 100.0% p50 16 ms p99 17 ms
v2 (150 ms timeout) 30% errors: success 100.0% p50 16 ms p99 16 ms
v2 (150 ms timeout) +2000 ms latency: success 100.0% p50 154 ms p99 154 msIf the team had only injected errors, v1 would have passed: the .catch(() => []) fallback works perfectly when recommendations fails fast. The latency phase tells a different story. Every product page waited the full two seconds, because nothing ever gave up on the slow call. The success column still says 100%, which is a lesson of its own: in production, a 2-second product page means abandoned sessions, upstream timeouts in the gateway, and request threads piling up. “No errors” isn’t the same as “healthy”, which is why the hypothesis includes latency.
v2 adds a 150 ms timeout, and the same fault produces a 154 ms p99. The hypothesis holds.
This is the most common finding in real resilience testing. Teams write fallbacks for errors and forget that the dangerous failure is a dependency that’s alive but slow.
Which failures to inject
A useful catalogue, roughly from most to least commonly overlooked:
- Latency. Add 500 ms, 2 s, 30 s to a dependency. Finds missing timeouts, timeouts longer than the caller’s, and thread-pool exhaustion.
- Partial failure. 10% or 30% errors rather than 100%. Finds retry storms (every caller retrying the failing 30% three times) and circuit breakers with bad thresholds.
- Dependency unavailable. Connection refused, DNS failure. Usually the best-handled case, but still worth confirming fallbacks return something sensible.
- Broker or database failover. Kill the primary. Finds clients that cache a dead connection, or consumers that don’t resume.
- Instance and zone loss. Terminate pods, drain a zone. Finds capacity that only works when everything is up.
- Resource exhaustion. Fill the disk, consume memory, exhaust file descriptors.
- Clock skew. Shift a host’s clock. Finds token validation and deadline bugs.
You don’t need production to start. In integration tests, a proxy such as Toxiproxy sits between the service and its dependency and adds latency or cuts connections on command. In Kubernetes, a service mesh can inject faults per route (illustrative Istio configuration):
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: recommendations
spec:
hosts: [recommendations]
http:
- fault:
delay:
percentage: { value: 10 }
fixedDelay: 2s
route:
- destination: { host: recommendations }Recovery is part of the evidence
The title of this note says “recovery evidence” for a reason. A system that degrades gracefully but never recovers has only half-passed. After removing the fault, check:
- Did the circuit breaker close again, and how long did it take?
- Did queued messages drain, and did consumer lag return to baseline?
- Did any instance stay broken (a connection pool full of dead connections, a cache full of fallback values with a 24-hour TTL)?
- Did autoscaling scale back down, or are you paying for 40 extra pods?
Measure time to recovery the same way you measure behaviour during the fault, against the steady-state metrics. A run that ends with “the fault was removed at 14:32, checkout success was back above 99.5% at 14:33, consumer lag at baseline by 14:36” is evidence. “It looked fine” isn’t.
Game days and turning findings into tests
A game day is a scheduled, team-wide version of the same thing: the team simulates a larger failure (a region outage, the payment provider down for 30 minutes) and works through it with real dashboards and runbooks. It tests people and procedures as well as code: does the on-call engineer know where the failover switch is, does the runbook still match the system, does the alert fire at all?
Every finding should end up as something automated. The v1/v2 example above belongs in CI as a test that injects 2 seconds of latency and asserts the page responds within 300 ms. Otherwise a future refactor removes the timeout and nobody notices until the real incident.
Interview practice: proposing a first experiment
“Your team runs an order service that depends on payments, inventory and a fraud-scoring API. There’s never been any resilience testing. Management agrees to a first experiment. What would you run, and how would you make sure it’s safe?”
Answer
I’d start in a staging environment with production-like traffic replay, not in production, and with the fraud-scoring API, because it’s an external dependency, it’s the one most likely to be slow, and I’d expect the order service to have a defined fallback (approve low-value orders, queue high-value ones for review).
The steady state would be order success rate and order API p99 latency, from the existing dashboards. The hypothesis: with fraud scoring adding 3 seconds of latency to every call, order p99 stays under 1 second and success stays above 99.5%, because the fraud call has a timeout and a fallback. I’d inject the latency with a proxy in front of the fraud API, for 10 minutes, and abort if order success falls below 98%.
Then I’d remove the fault and record how long it took for latency and success to return to normal. Whatever the result, I’d write up the findings, fix any gaps (very likely a missing or overly long timeout), and add the scenario to CI as an automated test. Only after a few staging experiments pass would I propose a small production experiment, limited to one instance or a small slice of traffic with automatic abort.