“Is it fast?” isn’t a question a performance test can answer. “Does checkout keep its 95th-percentile latency under 300 ms at 150 requests per second, with a realistic product catalogue, on production-sized instances?” is. Performance testing is measurement under a stated workload, and most of the mistakes people make come from leaving part of that statement out: testing with ten rows instead of ten million, with one user instead of the real traffic shape, or on a laptop and quoting the numbers as if they were production.
Interviewers use this topic to see whether you think in distributions and workloads rather than single numbers. The worked example below shows the two behaviors you most need to recognise: a service hitting its ceiling, and what happens to latency when traffic keeps arriving past it.
Write the workload down first
Before running anything, write down what you’re simulating:
- Arrival pattern. Either a number of concurrent users with think time between actions, or a target arrival rate in requests per second. They behave very differently past capacity, as you’ll see.
- Request mix. Real traffic might be 70% product views, 20% searches, 8% add-to-cart and 2% checkout. Testing only the cheapest endpoint flatters the system.
- Data. Table sizes, cache hit rates and payload sizes like production. An index that doesn’t matter at 1,000 rows decides everything at 50 million.
- Duration and shape. A five-minute ramp finds the ceiling. A two-hour soak finds memory leaks and connection-pool exhaustion. A sudden spike finds autoscaling lag.
- Environment. Instance sizes, replica counts, and where the load generator runs. A generator on the same machine as the system under test competes with it for CPU.
- Pass criteria. Something like “p95 under 200 ms and errors under 0.1% at 150 req/s”, agreed before the test so results can’t be rationalised afterwards.
A service with a hidden ceiling
To make saturation visible, here’s a small Node HTTP service where each request needs one of four database connections and holds it for about 20 ms, which is a common shape for real services sitting in front of a connection pool:
const POOL_SIZE = 4;
let inUse = 0;
const waiting = [];
const acquire = () => (inUse < POOL_SIZE ? (inUse++, Promise.resolve()) : new Promise((r) => waiting.push(r)));
const release = () => { const next = waiting.shift(); if (next) next(); else inUse--; };
http.createServer(async (req, res) => {
await acquire();
try {
await sleep(20);
} finally {
release();
}
res.end('ok');
}).listen(8090, '127.0.0.1');Four connections that each serve one request per ~20 ms give a theoretical maximum of 200 requests per second. Nothing in the code says that; you’d find it by measuring. The load generator is a short Node script using fetch, with a one-second client timeout counted as an error. Both ran on the same laptop, so treat the absolute numbers as illustrative and the shapes as the lesson.
Ramping virtual users: where throughput stops growing
The first test uses a closed model: N virtual users, each sending its next request as soon as the previous one returns. Five seconds at each level:
1 users 42 ok/s mean 24 ms p50 24 p95 25 p99 26 errors 0.0%
2 users 85 ok/s mean 24 ms p50 23 p95 25 p99 25 errors 0.0%
4 users 174 ok/s mean 23 ms p50 23 p95 25 p99 26 errors 0.0%
8 users 190 ok/s mean 42 ms p50 42 p95 44 p99 46 errors 0.0%
16 users 192 ok/s mean 84 ms p50 85 p95 86 p99 88 errors 0.0%
32 users 195 ok/s mean 167 ms p50 169 p95 172 p99 173 errors 0.0%Up to four users, throughput grows in step with users and latency stays flat at about 23 ms: there’s a free connection for every request. From eight users on, throughput barely moves (190, 192, 195) while latency doubles with each doubling of users. That’s what saturation looks like: the extra requests aren’t being served faster, they’re waiting in a queue for a connection.
Little’s law makes the relationship exact: the number of requests in the system equals throughput times the time each spends there. At 32 users, 195 requests per second times 0.167 seconds is about 32.6, matching the 32 users who are always either waiting or being served. If you know a service’s throughput ceiling and the concurrency hitting it, you can predict latency without running the test, and if a measurement disagrees with Little’s law, the measurement is probably wrong.
Here every request does identical work, so mean and percentiles stay close together. Real services have slow outliers (cache misses, garbage collection pauses, lock waits), and the mean hides them. A mean of 80 ms can sit on top of a p99 of 2 seconds, and with dozens of calls per page load, most users hit that tail regularly. Report percentiles, not averages.
Fixed arrival rate: what happens past the ceiling
The closed model has a property that makes it comfortable and misleading: when the server slows down, the virtual users slow down too, because each waits for its response before sending the next. Real users don’t coordinate like that. Visitors arrive at whatever rate they arrive, regardless of how the server is doing.
An open model sends requests at a fixed rate and measures each one from its scheduled start time, so queueing delay is counted. Ten seconds at each rate:
100 req/s 100 ok/s mean 23 ms p50 23 p95 25 p99 29 errors 0.0%
180 req/s 180 ok/s mean 22 ms p50 22 p95 24 p99 25 errors 0.0%
220 req/s 177 ok/s mean 512 ms p50 512 p95 953 p99 991 errors 19.4%
300 req/s 44 ok/s mean 622 ms p50 621 p95 963 p99 994 errors 85.4%Below the ceiling, the service is fast, faster than the closed test at saturation. At 220 req/s, only about 10% above what the pool can handle, the picture changes completely. Every second, around 30 more requests arrive than can be served, so the queue grows for the whole run. Latency climbs until requests hit the one-second client timeout, and almost a fifth fail.
At 300 req/s it gets worse in a way that surprises people: successful throughput drops to 44 per second. The server doesn’t know the client has given up, so it keeps spending connections on requests whose callers already timed out, and the requests that could still succeed wait behind them. Throughput collapsing under overload, rather than leveling off, is one of the most important behaviors a performance test can reveal, and a closed-model test with a fixed number of users will never show it.
Note that the errors are client timeouts, not server errors. The server’s own logs would show every request completing with status 200. Measuring only on the server side, or only successful requests, misses the users who gave up.
Turning measurements into a test
A measurement becomes a test once it has pass criteria and runs regularly against a stable environment. Tools like k6, Gatling, Locust and JMeter all support thresholds. In k6, for example, a script can declare:
export const options = {
scenarios: {
checkout: { executor: 'constant-arrival-rate', rate: 150, timeUnit: '1s', duration: '10m', preAllocatedVUs: 200 },
},
thresholds: {
http_req_duration: ['p(95)<200', 'p(99)<500'],
http_req_failed: ['rate<0.001'],
},
};The constant-arrival-rate executor is k6’s open model; thresholds make the run fail if the 95th percentile exceeds 200 ms, the 99th exceeds 500 ms, or more than 0.1% of requests fail.
A few practices make the numbers trustworthy:
- Compare to a baseline. Absolute numbers drift with hardware. “p95 rose from 120 to 180 ms since last week’s run on the same environment” is actionable; “p95 is 180 ms” on its own is hard to judge.
- Warm up first. JIT compilation, connection pools and caches make the first minute unrepresentative. Discard it or ramp gently.
- Watch the server while you test. CPU, memory, garbage collection, pool wait time, database locks. When latency rises, these tell you why; in the example above, connection-pool wait time would have pointed straight at the pool size.
- Keep the generator honest. Run it on separate machines, check it isn’t itself CPU-bound, and measure from scheduled start times so a slow generator doesn’t hide server slowness.
Interview practice
Your team load-tested a new search API with 50 virtual users and reported “average response time 140 ms, no errors”. Two weeks after launch, during a marketing campaign, search times out for many users. What might the load test have missed, and how would you redesign it?
Answer
Several things, and the test report doesn’t let us rule any of them out. Fifty virtual users is a closed model: if search slowed down, those users slowed down with it, so the test could never push the system past its capacity. Real campaign traffic arrives at a rate that doesn’t care how fast responses are, which is exactly the overload behavior the test couldn’t produce. An average of 140 ms also says nothing about the tail: p99 could already have been several seconds. And I’d ask what data it ran against, since search performance depends heavily on index size and query variety. A test that repeats the same few queries mostly measures a warm cache.
I’d redesign it around an open model at arrival rates taken from expected campaign traffic, with a margin above, for example ramping from normal load to three times normal over 20 minutes using k6’s ramping-arrival-rate. I’d use a production-sized index and a realistic query mix drawn from logs, report p50, p95 and p99 and the error rate from the client side, and set thresholds agreed with the product owner. I’d watch search-cluster CPU, queue depth and thread pools during the run to find the bottleneck, then rerun after each fix to confirm the ceiling moved. Finally, I’d check what happens just past the ceiling: if throughput collapses rather than leveling off, the API needs load shedding or request timeouts on the server side, not just more capacity.