A flaky test is one that passes and fails on the same code. Nobody changed anything, the pipeline ran twice, and you got one green and one red. Every team has a few, and they’re expensive in a way that’s easy to underestimate. Each spurious failure costs someone a re-run and a few minutes of doubt. Worse, the team learns that red doesn’t mean broken, and the next real failure gets re-run instead of read.
Flakiness also isn’t always the test’s fault. Some flaky tests accurately report a race that exists in production too, where it shows up as a rare, unreproducible bug report. That’s the main reason to diagnose them instead of retrying until green.
Diagnosis is mostly evidence gathering. You want to turn “it fails sometimes” into a specific claim, like “it fails when it runs after test X” or “it fails when the loader takes longer than 50 ms”, that you can confirm and then fix.
Measure the failure rate before you guess
Start by putting a number on it. Run the test many times in a loop and count failures:
fails=0
for i in $(seq 1 100); do
node --test warm.test.js > /dev/null 2>&1 || fails=$((fails+1))
done
echo "$fails failures in 100 runs"The number does three jobs. It confirms the test is really flaky, not newly broken. It gives you a baseline, so after a fix you can show the rate dropped to zero. And how the rate behaves is itself a clue: a rate that rises when the machine is busy (run the loop while something CPU-heavy is going on) points at timing, while a rate of zero in a loop but non-zero in the full suite points at interaction with other tests.
Alongside the rate, collect what was different between passing and failing runs. Most CI systems keep this: the test order, which worker or shard ran it, the random seed if the runner shuffles, timestamps (failures clustered near midnight UTC or at the start of a month are a strong hint), the machine image, and whether other jobs shared the runner. Log the seed and the order on every run; it costs nothing until the day you need it.
Alone versus together: finding leaked state
The single most useful experiment is to run the failing test by itself. Here’s a pair of tests for a price formatter that reads a shared config object:
// config.js
export const config = { currency: 'USD', locale: 'en-US' };
// format.js
export function formatPrice(cents) {
return new Intl.NumberFormat(config.locale, { style: 'currency', currency: config.currency })
.format(cents / 100);
}test('EU stores show prices in euros', () => {
config.currency = 'EUR';
config.locale = 'de-DE';
assert.equal(formatPrice(1999), '19,99\u00a0€');
});
test('prices default to US dollars', () => {
assert.equal(formatPrice(1999), '$19.99');
});The whole file fails; the second test alone passes:
$ node --test --test-reporter=spec prices.test.js
✔ EU stores show prices in euros (37.992541ms)
✖ prices default to US dollars (1.041292ms)
+ '19,99 €'
- '$19.99'
$ node --test --test-reporter=spec --test-name-pattern="default to US dollars" prices.test.js
✔ prices default to US dollars (12.565958ms)Alone passes, together fails: some earlier test left state behind. In a big file, you find the culprit by bisecting: run the failing test with the first half of the tests before it, then with the half that fails, and so on. With 200 tests ahead of it, that’s eight runs at most. Here the culprit is obvious: the euro test sets two properties on a module-level object and never puts them back.
(The \u00a0 is real: German formatting puts a non-breaking space before the €, and an ordinary space fails.)
Why does this count as flaky rather than just broken? Because in this file the order is fixed, but in many setups it isn’t. Runners that shuffle order, sharding that splits files across machines, someone adding .only while debugging, or a change in how files are grouped can all reorder tests. Node’s runner, for example, runs each test file in its own process by default, so state leaks only within a file. Split these two tests into separate files and they pass. Run the same two files with --experimental-test-isolation=none, where all files share one process, and the dollar test fails again. The bug was there all along; the configuration decided whether you saw it.
The fix is to stop mutating shared state, or to restore it reliably. The cleanest version passes configuration in (formatPrice(cents, { locale, currency })), so there’s nothing global to leak. If that’s not possible right now, save and restore it in the test with t.after(() => Object.assign(config, saved)).
Timing: the sleep that’s “long enough”
The second big family is timing. This cache warmer starts loading products and returns immediately:
export function warmCache(cache, loadProducts) {
loadProducts().then((rows) => {
for (const row of rows) cache.set(row.id, row);
});
}Its test can’t await anything, so it sleeps and hopes. The loader here simulates a backend that answers in 0 to 60 ms, which is how a real service behaves under varying load:
const slowLoader = () =>
new Promise((resolve) => setTimeout(() => resolve([{ id: 1, name: 'Desk lamp' }]), Math.random() * 60));
test('warming fills the product cache', async () => {
const cache = new Map();
warmCache(cache, slowLoader);
await new Promise((resolve) => setTimeout(resolve, 50)); // "should be enough"
assert.equal(cache.size, 1);
});The loop above reported 23 failures in 100 runs, each one looking like this:
✖ warming fills the product cache (51.535ms)
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
0 !== 1The failure message doesn’t mention timing at all. It just says the cache was empty, which could mean the loader failed, the warmer has a bug, or the test checked too early. The duration is the clue: 51.5 ms, right after the 50 ms sleep.
Raising the sleep to 100 ms would make this pass locally and fail on the next overloaded CI runner. The fix is to give the test something real to wait for. Make the function return its promise:
export async function warmCacheAsync(cache, loadProducts) {
const rows = await loadProducts();
for (const row of rows) cache.set(row.id, row);
}With await warmCacheAsync(cache, slowLoader) in the test, the same 100-run loop reports 0 failures in 100 runs. Production callers that don’t care when warming finishes can still ignore the promise. The flaky test was really pointing at an API that gave callers no way to know when work was done.
The usual suspects, and the evidence for each
Most flaky tests fall into a handful of categories. Matching your evidence against them saves time:
- Leaked state between tests: passes alone, fails in the suite or only in a particular order. Look for globals, singletons, environment variables, database rows and files that aren’t cleaned up.
- Timing and races: failure rate rises under load, failures cluster around a sleep or timeout value, durations sit right at the limit.
- Time and dates: fails at certain times of day, at month or year boundaries, around daylight-saving changes, or only on machines in another timezone. Freeze the clock in the test.
- Unseeded randomness: generated test data occasionally hits an edge case. Log the seed so a failure can be replayed exactly.
- Unordered results treated as ordered: an SQL query without
ORDER BY, or results gathered from parallel work, compared to a fixed array. Passes until the database picks a different plan. - Shared resources and infrastructure: fixed ports, shared test accounts, containers that aren’t ready yet, disk or memory pressure on the runner. Failures correlate with particular machines or with parallel jobs.
- Real external services: a sandbox API that’s occasionally slow or down. The failure is real but not about your code; stub it at the boundary and keep one separate, clearly labelled check against the real service.
Retries and quarantine
Automatic retries (pytest-rerunfailures, Jest’s jest.retryTimes, CI-level re-runs) make the pipeline green and leave the cause in place. If the test is flaky because of a production race, a retry hides that bug in the place most likely to catch it. Treat retries as a temporary measure, and make sure retried passes are still reported so the flake rate stays visible.
Quarantine is the more honest version: move the test out of the blocking suite, keep running it in a non-blocking job so you keep collecting evidence, and give it an owner and a date. A quarantine list without owners turns into a graveyard of tests nobody runs. What you shouldn’t do is delete the assertion that fails, raise every timeout, or mark the test skipped without a record of why.
Interview practice
A UI test that checks an order confirmation page fails on CI about once a week. It always passes when developers run it locally. The failure is expected "Order #1042 confirmed" but found "Loading...". How would you approach it?
Answer
The message tells me the assertion ran while the page was still loading, so my first hypothesis is timing rather than wrong data. Before changing anything, I’d get evidence: pull the last few weeks of CI runs and check whether failures correlate with slower runners or busier times, look at the test’s duration on failing runs compared to passing ones, and rerun it in a loop on a CI machine to estimate the rate. Running locally under artificial CPU throttling or a slowed network (browser dev tools can do both) often reproduces this kind of failure.
Then I’d read how the test waits. If it uses a fixed sleep or checks the text right after the page opens, that’s the bug: it should wait for the confirmation element to appear, with a timeout. Most UI frameworks have auto-waiting assertions, such as Playwright’s await expect(page.getByText('Order #1042 confirmed')).toBeVisible(), which retry until the condition holds or the timeout expires. I’d also check whether the order number itself is fixed test data or depends on what else ran, since #1042 could change if other tests create orders. I’d confirm the fix by rerunning the loop on CI and showing the failure rate went to zero, rather than closing the ticket after one green run.