A test report is a claim, and people make decisions on it. A release manager reads “all tests pass” and ships. A reviewer reads “coverage 85%” and approves. If the report was technically true but left out that the payment integration never ran, the decision was made on information nobody actually had.
Honest reporting doesn’t mean pessimistic reporting. It means the reader can tell the difference between “verified”, “not verified” and “verified only in a limited environment” without asking. Interviewers probe this with questions like “how do you know your tests are enough?” because it reveals whether a candidate treats test results as evidence with limits, or as a green light.
What a green run actually says
Here’s a small refund module: a pure function decides how much of an order can be refunded (paid, within 30 days, minus earlier refunds), and refund calls a payment gateway. A thin adapter talks to Stripe’s HTTP API. The test file has three unit tests that use an in-memory gateway, and one integration test that calls Stripe’s test mode, which only runs when a test key is configured:
test('refunds against the real Stripe test-mode API',
{ skip: !process.env.STRIPE_TEST_KEY && 'STRIPE_TEST_KEY is not set' },
async () => {
const gateway = stripeGateway(process.env.STRIPE_TEST_KEY);
const result = await gateway.refund({ chargeId: process.env.STRIPE_TEST_CHARGE, amountCents: 100 });
assert.equal(result.status, 'succeeded');
});On a machine without the key, Node 22’s runner reports:
✔ a paid order is refundable for 30 days (1.217667ms)
✔ a partial refund reduces what is left to refund (0.110583ms)
✔ refunding more than was paid is rejected before calling the gateway (0.260042ms)
﹣ refunds against the real Stripe test-mode API (0.042083ms) # STRIPE_TEST_KEY is not set
ℹ tests 4
ℹ pass 3
ℹ fail 0
ℹ skipped 1The exit code is 0. CI shows a green check. The JUnit XML that most CI dashboards consume (written with --test-reporter=junit --test-reporter-destination=junit.xml, alongside the spec reporter on stdout) records the skip honestly:
<testcase name="refunds against the real Stripe test-mode API" time="0.000042" classname="test">
<skipped type="skipped" message="STRIPE_TEST_KEY is not set"/>
</testcase>But that detail lives one click deep in a test tab, while the summary everyone sees is a green check. If the CI secret was rotated and nobody updated it, every pipeline from now on silently skips the only test that proves refunds work against the real provider.
Skipped is not passed
Skips are useful. Some tests need credentials, hardware or a platform that not every environment has. The problem is skips that nobody notices, so make them loud:
- Always give a reason.
{ skip: 'STRIPE_TEST_KEY is not set' }shows up in every reporter. A bareskip: truetells the reader nothing. - Decide where a skip is acceptable. On a laptop, skipping the Stripe test is fine. In the release pipeline it isn’t. A small step after the tests that parses
junit.xmland fails if any<skipped>element appears, or if a named test is among them, turns a silent gap into a red build. - Track skip counts over time. A suite that had 3 skipped tests last month and has 41 today has changed, even if it’s still green.
- Don’t use skip as a mute button for failures. A skipped flaky test needs an owner and a date, or it never comes back.
Reading a coverage number honestly
The same run with --experimental-test-coverage:
ℹ file | line % | branch % | funcs % | uncovered lines
ℹ refunds.js | 100.00 | 77.78 | 100.00 |
ℹ stripe-gateway.js | 13.33 | 100.00 | 0.00 | 3-15
ℹ refunds.test.js | 90.00 | 100.00 | 80.00 | 27-29
ℹ all files | 72.88 | 86.67 | 75.00 |72.88% looks respectable. Look at what it’s made of. The test file itself is in the report, and test files are almost entirely covered by definition, since running them is what produces coverage. Excluding them with --test-coverage-exclude='test/**':
ℹ refunds.js | 100.00 | 77.78 | 100.00 |
ℹ stripe-gateway.js | 13.33 | 100.00 | 0.00 | 3-15
ℹ all files | 55.17 | 80.00 | 66.67 |The honest line coverage of the application code is 55.17%. Adding --test-coverage-lines=80 makes the run fail with Error: 55.17% line coverage does not meet threshold of 80%. and exit code 1, which is how you make a coverage policy enforceable rather than aspirational.
The per-file rows say more than either total. refunds.js is fully covered by lines, though only 77.78% by branches. stripe-gateway.js, the code that sends real money instructions to a real provider, is at 13.33%, and its only test is the one that was skipped. A single “55%” average hides that the riskiest file is the least tested. When you report coverage, report the files that matter, not just the total.
And remember what coverage measures: which lines ran. refunds.js at 100% doesn’t mean every rule in it is checked by an assertion. A test that calls refund and asserts nothing would produce the same number. Coverage tells you where you definitely have no evidence; it can’t tell you where you do.
Writing the report a reviewer can trust
For a pull request description, a release note or a QA sign-off, a short structure covers what readers need:
- What ran: which suites, at which commit, in which environment.
- Results: counts of passed, failed and skipped, with links to the CI run.
- Not run, and why: skipped tests and suites outside the run.
- Known gaps: risks the tests don’t cover, stated plainly.
For this change, an honest version reads something like:
Ran the refunds unit suite locally on Node 22 at commit
a1b2c3d: 3 passed, 0 failed, 1 skipped. The Stripe test-mode integration test was skipped because no test key is configured on my machine, so the gateway adapter (stripe-gateway.js, 13% line coverage) has not been exercised against the provider in this change. Refund eligibility rules are covered by unit tests. Not covered: webhook handling for asynchronous refund failures. I’d like the integration test to run in the staging pipeline before merge.
Every claim in that paragraph can be checked, and the reader knows exactly what to ask for next.
Wording that overclaims
Small phrasing choices change what a report promises:
- “All tests pass” when some were skipped. Say “3 passed, 1 skipped (reason)”.
- “Tested on staging” when only a smoke test ran there. Say which tests ran where.
- “100% coverage” for one module, quoted as if it described the system.
- “No regressions” when the regression suite wasn’t run, only the tests near the change.
- “Works with Stripe” based on a test that used a stub of Stripe. Stubs prove your code handles the responses you imagined; only the integration test proves the provider sends them.
None of these are lies, exactly. They’re claims broader than the evidence, and the gap is where incidents come from.
Interview practice
Your manager asks for a one-line status before a release: “Are we good to ship?” The CI run is green. You know that the end-to-end suite was disabled two weeks ago because it was flaky, and the mobile app tests only run nightly and last night’s run hasn’t finished. What do you say?
Answer
I wouldn’t answer with just “yes, CI is green”, because the green check doesn’t cover two things the manager probably assumes it does. I’d give a short answer with the evidence and the gaps: “Unit and API tests are green on the release commit. Two gaps: the end-to-end browser suite has been disabled for two weeks because of flakiness, so the main checkout journeys haven’t been verified automatically in that time, and tonight’s mobile test run isn’t finished yet. I can run the five critical checkout scenarios manually in the next 30 minutes, and we’ll have the mobile results by 9 a.m. My recommendation is to ship after those two checks, or tomorrow morning if you’d rather wait for the full nightly run.”
That keeps the decision with the person who owns it, gives them the specific risks rather than a vague “mostly fine”, and offers a way to close the gaps. Afterwards, I’d raise the disabled end-to-end suite as its own problem: two weeks without browser coverage on checkout is a gap that should have an owner and a date, not a quiet flag in the CI config.