Code coverage tells you which lines ran during the tests. It doesn’t tell you whether any test would have failed if those lines were wrong. A test that calls a function and asserts nothing gives you the same coverage as one that checks every output.
Mutation testing measures the thing coverage can’t. A tool makes small, deliberate changes to your code, one at a time, such as turning >= into > or true into false, and runs your tests against each changed version. If a test fails, the change was caught. If every test still passes, you’ve found code whose behavior your tests don’t actually pin down. It’s a test of your tests.
The examples below use Stryker, the main mutation tool for JavaScript and TypeScript, run against Node’s built-in test runner. PIT does the same job for Java, mutmut and cosmic-ray for Python.
How a mutation run works
Each modified copy of your code is a mutant. The rules that produce them are mutators: an equality-operator mutator swaps < for <=, a boolean mutator flips true and false, a string mutator empties string literals, a block mutator deletes a function body. For every mutant the tool runs the tests and records an outcome:
- Killed: at least one test failed. Good.
- Survived: all tests passed. Your suite didn’t notice the change.
- No coverage: no test even executes that code.
- Timeout: the mutant caused an infinite loop or hang. Usually counted as killed.
The mutation score is killed (plus timeouts) divided by all valid mutants. Some survivors are equivalent mutants: changes that don’t alter behavior at all, so no test could ever kill them. Those are the main source of noise, and part of reading a report is recognising them.
A shipping rule with full line coverage
Here’s a small pricing function. Orders of 50.00 or more ship free; express shipping costs extra; negative subtotals are rejected.
export const FREE_SHIPPING_FROM = 5000;
export function shippingCost(subtotalCents, { express = false } = {}) {
if (subtotalCents < 0) throw new RangeError('subtotal cannot be negative');
if (subtotalCents >= FREE_SHIPPING_FROM) return express ? 999 : 0;
return express ? 1499 : 499;
}And a test file that looks reasonable at a glance:
test('small orders pay standard shipping', () => {
assert.equal(shippingCost(2000), 499);
});
test('large orders ship free', () => {
assert.equal(shippingCost(8000), 0);
});
test('negative subtotals are rejected', () => {
assert.throws(() => shippingCost(-1));
});Node’s built-in coverage (node --test --experimental-test-coverage) reports:
ℹ file | line % | branch % | funcs % | uncovered lines
ℹ shipping.js | 100.00 | 77.78 | 100.00 |Every line ran. A coverage gate set at 90% lines would pass this without comment.
Running Stryker against node:test
Stryker has dedicated runners for Jest, Vitest, Mocha and others. For Node’s built-in runner, the generic command runner works: it runs a shell command per mutant and treats a non-zero exit code as a failed test run.
{
"testRunner": "command",
"commandRunner": { "command": "node --test test/*.test.js" },
"mutate": ["src/**/*.js"],
"reporters": ["clear-text"],
"coverageAnalysis": "off"
}npx stryker run (Stryker 10.0) created 11 mutants and printed:
[Survived] StringLiteral
src/shipping.js:4:47
- if (subtotalCents < 0) throw new RangeError('subtotal cannot be negative');
+ if (subtotalCents < 0) throw new RangeError("");
[Survived] EqualityOperator
src/shipping.js:4:7
- if (subtotalCents < 0) throw new RangeError('subtotal cannot be negative');
+ if (subtotalCents <= 0) throw new RangeError('subtotal cannot be negative');
[Survived] EqualityOperator
src/shipping.js:5:7
- if (subtotalCents >= FREE_SHIPPING_FROM) return express ? 999 : 0;
+ if (subtotalCents > FREE_SHIPPING_FROM) return express ? 999 : 0;
File | total | covered | # killed | # timeout | # survived | # no cov | # errors |
shipping.js | 72.73 | 72.73 | 8 | 0 | 3 | 0 | 0 |100% line coverage, 72.73% mutation score. The eight killed mutants included flipping each if condition to always true or always false, and deleting the function body. Those are the crude changes that any test at all would catch. The three survivors are the interesting ones.
Reading the survivors
>= became > on the free-shipping check. With that change, an order of exactly 5000 cents pays 4.99 for shipping. No test used 5000, so nothing failed. This is the classic survivor, and it maps directly onto boundary-value testing: the tests checked 2000 and 8000, comfortably on either side, and never the edge. This one matters. Customers at exactly 50.00 are the ones the marketing email promised free shipping to.
< 0 became <= 0 on the guard. Now a subtotal of zero throws. Is that wrong? It depends on whether zero is a valid subtotal, which is a question for the product owner: a cart where every item is covered by a gift card might well total zero. Here, assume it’s valid and should still pay standard shipping. A survivor doesn’t always mean “add a test”; sometimes it means “find out what the rule is”, and the test comes after.
The error message became an empty string. The negative-subtotal test only checks that something is thrown. It would pass if the function threw RangeError(""), or a TypeError, or a typo in a variable name that throws a ReferenceError. That last case is the real risk: a bare assert.throws can be satisfied by a bug. Whether the message text itself is part of the contract is debatable. Checking the error type and a key word is a reasonable middle ground.
Strengthening the tests
Three additions, one per survivor, plus tests for express shipping (more on why below):
test('free shipping starts at exactly 50.00', () => {
assert.equal(shippingCost(4999), 499);
assert.equal(shippingCost(5000), 0);
});
test('an empty or fully discounted cart is a valid subtotal', () => {
assert.equal(shippingCost(0), 499);
});
test('negative subtotals are rejected with a RangeError that says why', () => {
assert.throws(() => shippingCost(-1), { name: 'RangeError', message: /negative/ });
});
test('express shipping costs 14.99, or 9.99 above the free threshold', () => {
assert.equal(shippingCost(2000, { express: true }), 1499);
assert.equal(shippingCost(5000, { express: true }), 999);
});All tests
✓ All tests (killed 11)
shipping.js | 100.00 | 100.00 | 11 | 0 | 0 | 0 | 0 |What a perfect score still misses
Look back at the first report. The original tests never called the function with express: true, so express pricing was completely untested, and Stryker said nothing about it. None of its 11 mutants changed the numbers 999 or 1499 or the express ? ... : ... choice, so there was nothing to survive. A score only covers the changes its mutators make. That’s why the express test was added from reading the code, not from the report. Treat survivors as precise leads, and a high score as evidence, not proof.
A few more practical limits:
- Cost. Every mutant means another test run. The command runner reruns the whole suite per mutant; runners with per-test coverage analysis only run the tests that reach the mutated code. On a large codebase, use incremental mode (
--incremental) or mutate only changed files in CI. - Equivalent mutants. In
if (items.length > 0) return sum(items); return 0;, turning> 0into>= 0changes nothing observable, becausesum([])is already 0. Stryker lets you suppress a specific mutator on a line with a// Stryker disable next-linecomment and a reason, which beats chasing an unkillable mutant. - Where to spend it. Mutation testing pays off most on code where small logic errors are expensive: pricing, permissions, date calculations, parsers. Running it on view templates or glue code mostly produces noise.
Interview practice
A team proudly reports 95% line coverage on their billing module. You run mutation testing and get a score of 61%, with most survivors being changed comparison operators and removed return statements inside if blocks. What does that tell you, and what would you do next?
Answer
It tells me the tests execute most of the billing code but assert on little of what it does. Survivors on comparison operators usually mean boundary values aren’t tested: the tests use comfortable middle values, so > and >= behave the same for every input they try. Survivors from removed returns inside if blocks suggest early-exit paths, often validation or special cases, run without any test checking their result, perhaps because tests only assert that no exception was thrown, or check one field of a larger result.
I wouldn’t set a mutation-score target and ask people to hit it; that tends to produce tests written to kill mutants rather than to describe behavior. I’d go through the survivors in the most expensive part of the module (likely invoice totals and proration) together with the team, and for each one ask what the rule at that boundary should be. Most will turn into boundary tests and stronger assertions. Some will turn out to be equivalent mutants and get suppressed with a reason. A few may reveal that nobody knows the rule, which is a finding in itself. Then I’d run mutation testing on changed billing files in CI, so new code doesn’t add new survivors.