Most bugs that reach production aren’t on the happy path. The happy path gets exercised every time a developer clicks through the feature. The paths that fail in production are the ones nobody tried: the file with a blank line at the end, the price written as NaN, the disk that fills up halfway through a write. Testing error paths means choosing those situations on purpose and checking that the code fails the way you designed it to fail.
That last part is where many test suites fall short. They do test errors, but loosely: “something was raised”. This article works through a small price-file parser in Python with unittest, finds a real bug that a loose test hides, and then tests a failure’s side effects.
A parser with designed failures
The function reads sku,price rows (from a CSV upload, say) and returns Decimal prices. Its error contract is explicit: every problem raises PriceFileError, which carries the line number so the upload screen can tell the user exactly where to look.
import csv
from decimal import Decimal, InvalidOperation
class PriceFileError(ValueError):
def __init__(self, line, message):
super().__init__(f"line {line}: {message}")
self.line = line
def load_prices(lines):
"""Parse 'sku,price' rows into a dict of Decimal prices."""
prices = {}
reader = csv.reader(lines)
header = next(reader, None)
if header != ["sku", "price"]:
raise PriceFileError(1, f"expected header sku,price, got {header}")
for line_no, row in enumerate(reader, start=2):
if len(row) != 2:
raise PriceFileError(line_no, f"expected 2 columns, got {len(row)}")
sku, raw = row
try:
price = Decimal(raw)
except InvalidOperation:
raise PriceFileError(line_no, f"price {raw!r} is not a number") from None
if price < 0:
raise PriceFileError(line_no, f"price {raw} is negative")
if sku in prices:
raise PriceFileError(line_no, f"duplicate sku {sku}")
prices[sku] = price
return pricesA few design choices make it testable. The function takes an iterable of lines rather than a filename, so tests don’t need files on disk. Errors are a custom subclass of ValueError, so callers can catch exactly this failure without also catching unrelated bugs. And from None suppresses the chained InvalidOperation, since the custom message already says what went wrong.
Asserting the failure precisely
Here are tests for each designed failure:
import unittest
from decimal import Decimal
from prices import PriceFileError, load_prices
class LoadPricesErrors(unittest.TestCase):
def test_valid_file(self):
self.assertEqual(load_prices(["sku,price", "TEE-M,19.99"]), {"TEE-M": Decimal("19.99")})
def test_non_numeric_price_names_the_line_and_value(self):
with self.assertRaises(PriceFileError) as ctx:
load_prices(["sku,price", "TEE-M,19.99", "MUG,twelve"])
self.assertEqual(ctx.exception.line, 3)
self.assertEqual(str(ctx.exception), "line 3: price 'twelve' is not a number")
self.assertIsNone(ctx.exception.__cause__)
def test_negative_price(self):
with self.assertRaisesRegex(PriceFileError, r"^line 2: price -1.00 is negative$"):
load_prices(["sku,price", "MUG,-1.00"])
def test_duplicate_sku(self):
with self.assertRaisesRegex(PriceFileError, "duplicate sku MUG"):
load_prices(["sku,price", "MUG,8", "MUG,9"])
def test_missing_column(self):
with self.assertRaisesRegex(PriceFileError, "line 2: expected 2 columns, got 1"):
load_prices(["sku,price", "MUG"])
def test_empty_input(self):
with self.assertRaisesRegex(PriceFileError, "line 1: expected header sku,price, got None"):
load_prices([])python3 -m unittest -v on Python 3.14 runs all six in about a millisecond and reports OK.
Each test pins down three things: the exception type, the line number (directly or in the message), and enough of the message to prove the right check fired. The twelve test shows the most thorough form: the context manager captures the exception so you can assert on its attributes, not just its text. Use assertRaisesRegex when the message matters but a full equality check would be brittle.
How precise should message assertions be? Assert the parts a user or caller depends on. Here, the line number and the offending value are the contract, because the upload screen shows them. Exact punctuation of a log message nobody reads isn’t worth pinning.
The bug a loose assertion hides
Now the interesting part. What about prices that Decimal accepts but that aren’t real prices? Probing a few edge values directly showed:
'NaN' InvalidOperation [<class 'decimal.InvalidOperation'>]
'Infinity' {'X': Decimal('Infinity')}
'1e3' {'X': Decimal('1E+3')}
' 5' {'X': Decimal('5')}
'' PriceFileError line 2: price '' is not a numberTwo bugs. Decimal('NaN') parses fine, but then price < 0 raises a raw InvalidOperation, because comparing a NaN decimal is a signalling operation. And Infinity is accepted as a valid price.
Here’s how a loose test would treat the first one, next to the precise version:
class NonFinitePrices(unittest.TestCase):
def test_nan_is_rejected_loosely(self):
with self.assertRaises(Exception):
load_prices(["sku,price", "MUG,NaN"])
def test_nan_is_rejected_precisely(self):
with self.assertRaisesRegex(PriceFileError, "line 2: price 'NaN' is not a finite number"):
load_prices(["sku,price", "MUG,NaN"])
def test_infinity_is_rejected(self):
with self.assertRaisesRegex(PriceFileError, "line 2: price 'Infinity' is not a finite number"):
load_prices(["sku,price", "MUG,Infinity"])test_infinity_is_rejected ... FAIL
test_nan_is_rejected_loosely ... ok
test_nan_is_rejected_precisely ... ERROR
ERROR: test_nan_is_rejected_precisely
if price < 0:
^^^^^^^^^
decimal.InvalidOperation: [<class 'decimal.InvalidOperation'>]
FAIL: test_infinity_is_rejected
AssertionError: PriceFileError not raised(Output trimmed to the relevant lines.) The loose test passes. It sees “an exception happened” and declares success, even though the exception is an unhandled crash that the upload endpoint would turn into a 500 error with no line number. assertRaises(Exception) would also pass if the code had a typo that raised NameError. It’s one of the most common weak tests in real codebases, and interviewers who review code will look for it.
The fix is two lines after the Decimal parse:
if not price.is_finite():
raise PriceFileError(line_no, f"price {raw!r} is not a finite number")With that in place, all nine tests pass. Delete the loose test; it adds nothing the precise one doesn’t.
Where did the idea to try NaN come from? From thinking about what the parser accepts, not what a user would type. For numbers: zero, negative zero, NaN, infinity, exponents, leading whitespace, the empty string, very long values. For collections: empty, one item, duplicates. For text: Unicode, a byte-order mark at the start of a CSV exported from Excel. The boundary value and equivalence partitioning techniques give a structured way to pick them.
Testing what a failure leaves behind
An error path isn’t finished when the exception is raised. The other question is what state the failure leaves. Say the parsed prices are saved to a JSON file the storefront reads. If the disk fills up halfway through, a naive open(path, "w") has already truncated the old file, and the storefront now has no prices. A safer version writes to a temp file and renames it into place:
import json
import os
import tempfile
def save_prices(prices, path):
"""Write prices atomically: either the whole new file lands or the old one stays."""
directory = os.path.dirname(os.path.abspath(path))
fd, tmp = tempfile.mkstemp(dir=directory, suffix=".tmp")
try:
with os.fdopen(fd, "w") as f:
json.dump({sku: str(p) for sku, p in prices.items()}, f)
os.replace(tmp, path)
except BaseException:
os.unlink(tmp)
raiseYou can’t wait for a real full disk in a test, so inject the failure:
class SavePricesFailure(unittest.TestCase):
def setUp(self):
self.dir = tempfile.mkdtemp()
self.path = os.path.join(self.dir, "prices.json")
with open(self.path, "w") as f:
json.dump({"MUG": "8"}, f)
def test_failed_write_keeps_old_file_and_leaves_no_temp_file(self):
with mock.patch("save.json.dump", side_effect=OSError(28, "No space left on device")):
with self.assertRaises(OSError) as ctx:
save_prices({"MUG": Decimal("9")}, self.path)
self.assertEqual(ctx.exception.errno, 28)
with open(self.path) as f:
self.assertEqual(json.load(f), {"MUG": "8"})
self.assertEqual(os.listdir(self.dir), ["prices.json"])The test passes against the atomic version. Against the naive open(path, "w") version it fails with JSONDecodeError: Expecting value: line 1 column 1 (char 0) when reading the old file back, because the old file is now empty. That’s the bug this test exists to catch. The three assertions check the whole failure contract: the original error reaches the caller unchanged (errno 28), the old data survives, and no .tmp file is left behind.
The same pattern applies to databases (the transaction rolled back, no orphan rows), queues (the message wasn’t acknowledged, so it’ll be redelivered) and external calls (no duplicate charge after a timeout and retry).
Interview exercise: the swallowed exception
You’re reviewing this function from a payments service:
def refund(order_id, gateway, log):
try:
gateway.refund(order_id)
return True
except Exception as e:
log.warning("refund failed: %s", e)
return FalseIts only test calls it with a gateway stub that raises TimeoutError and asserts that it returns False. What’s wrong with the function and with the test, and what tests would you write?
Answer
The function treats every exception the same way. A timeout is ambiguous: the refund may have gone through on the gateway’s side, so returning False invites the caller to retry and refund twice. A TypeError from a bug in our own code is also turned into a quiet False plus a warning nobody reads, so programming errors become “refund failed” reports. The test only confirms that behaviour, so it would keep passing if the code were broken in either way.
I’d split the cases: catch the gateway’s specific declined exception and return a clear result, treat a timeout as “unknown” (raise or return a distinct status so the caller checks the refund’s status with an idempotency key before retrying), and let unexpected exceptions propagate. Tests then cover a declined refund (asserting the result and that the log records the order ID), a timeout (asserting the unknown status and that no second gateway.refund call happens, using the stub’s call count), and a stub that raises TypeError (asserting with assertRaises(TypeError) that it propagates). I’d also add an assertLogs check that the order ID appears in the warning, because the on-call engineer needs it.