If you ask how to make service-to-service calls faster, “reuse connections” is near the top of most answers. Interviewers then push on the details: how much does a connection actually cost? How big should the pool be? What happens when a pooled connection has quietly died? Those follow-ups are where you show you’ve run a client in production rather than read about keep-alive once.
The numbers below come from Node 22 on localhost, using a self-signed certificate for the HTTPS tests.
What a new connection costs
This script sends 50 sequential HTTPS requests to a local server twice: once with an agent that opens a fresh connection per request, once with a keep-alive agent. The server counts completed TLS handshakes.
import https from 'node:https';
import { readFileSync } from 'node:fs';
let connections = 0;
const server = https.createServer({ key: readFileSync('key.pem'), cert: readFileSync('cert.pem') }, (req, res) => res.end('ok'));
server.on('secureConnection', () => connections++);
const ca = readFileSync('cert.pem');
const get = (agent) => new Promise((resolve, reject) => {
https.get({ host: 'localhost', port: 8444, ca, agent }, (res) => { res.resume(); res.on('end', resolve); }).on('error', reject);
});
async function run(label, agent) {
connections = 0;
const start = performance.now();
for (let i = 0; i < 50; i++) await get(agent);
console.log(`${label}: 50 requests, ${connections} TLS connections, ${Math.round(performance.now() - start)} ms`);
agent.destroy();
}
server.listen(8444, async () => {
await run('new connection each time', new https.Agent({ keepAlive: false, ca }));
await run('keep-alive agent ', new https.Agent({ keepAlive: true, ca }));
server.close();
});new connection each time: 50 requests, 50 TLS connections, 102 ms
keep-alive agent : 50 requests, 1 TLS connections, 10 msThat’s a tenfold difference with no network at all. On localhost the round trip is effectively zero, so all of that gap is handshake CPU: key exchange and certificate signing and checking on both sides. Over a real network you add the round trips: one for TCP and one for TLS 1.3 on every new connection. At a 30 ms RTT that’s another 60 ms per request, or three extra seconds across these 50 calls.
There are quieter costs too. Each closed connection leaves a TIME_WAIT entry behind on whichever side closed first, and a client churning through thousands of short connections can run out of ephemeral ports. New connections also start with a small TCP congestion window, so a large response on a fresh connection takes extra round trips that a warm one wouldn’t.
How keep-alive works in HTTP/1.1
HTTP/1.1 connections are persistent by default. After a response, the connection stays open for the next request unless either side sends Connection: close. curl shows the reuse when you give it two URLs on the same host:
curl -sv http://127.0.0.1:8083/a http://127.0.0.1:8083/b* Connected to 127.0.0.1 (127.0.0.1) port 8083
< Connection: keep-alive
< Keep-Alive: timeout=5
* Connection #0 to host 127.0.0.1 left intact
* Re-using existing connection with host 127.0.0.1
< Connection: keep-alive
< Keep-Alive: timeout=5The server logged a single new connection. Keep-Alive: timeout=5 is Node’s server announcing that it closes idle connections after five seconds, which matters later.
An HTTP/1.1 connection carries one request at a time, so a client that needs ten concurrent requests to one host needs ten connections. That set of open, reusable connections is the pool. HTTP/2 changes this by multiplexing many requests over one connection, which is why HTTP/2 clients usually keep a single connection per origin.
In Node 19 and later, the global HTTP agent has keep-alive on by default:
node -e "const a=require('http').globalAgent; console.log('keepAlive', a.keepAlive, 'maxSockets', a.maxSockets, 'timeout', a.options.timeout)"keepAlive true maxSockets Infinity timeout 5000The most common way to lose pooling is to create a new agent or client inside the function that makes each request. Every call then gets an empty pool. The same mistake exists in Java (a new HttpClient per call) and .NET (a new HttpClient per request, famous for exhausting sockets). Create the client once per destination and share it.
Sizing the pool
Pool size is a concurrency limit, and the limit cuts both ways. This test fires six concurrent requests at a server that takes 200 ms per request, with two different maxSockets values:
maxSockets=2: 6 requests in 634 ms, at most 2 in flight on the server
maxSockets=6: 6 requests in 212 ms, at most 6 in flight on the serverWith two connections, the other four requests waited in the client’s queue. The server saw a tidy two requests at a time and reported fast responses, while the caller saw 634 ms. Queue time inside a client pool doesn’t show up in the server’s metrics, so you have to measure it on the client.
To pick a number, use Little’s law: concurrent requests equal throughput times latency. A service sending 500 requests per second to a dependency that answers in 40 ms has 500 × 0.04 = 20 requests in flight on average. A pool of 20 is the floor; allow headroom for latency spikes, perhaps 30 to 40.
An unlimited pool, Node’s default, has the opposite problem. When the dependency slows down, in-flight requests pile up and the client opens more and more connections, adding handshake load to a server that is already struggling. Multiply by the number of client instances, too. Fifty pods with a limit of 100 each can open 5,000 connections to one database proxy. Set a ceiling that matches what the downstream can take, and put a timeout on waiting for a connection so a stalled dependency fails fast instead of queueing forever.
When a pooled connection has gone stale
A connection sitting idle in a pool can be closed by the server, a load balancer, a NAT gateway or a firewall, each with its own idle timeout. Usually the client sees the close and drops the socket. Sometimes the close and a new request cross in flight, and the request goes out on a connection that’s already gone.
This server simulates that moment by destroying any connection that receives a second request:
const server = http.createServer((req, res) => {
req.socket.requests = (req.socket.requests ?? 0) + 1;
if (req.socket.requests > 1) return req.socket.destroy();
res.end('ok');
});With a keep-alive agent, each request reports how it ended:
first GET : 200
second GET: ECONNRESET (socket hang up), reusedSocket=true
third GET : 200
POST : ECONNRESET (socket hang up), reusedSocket=trueThe second GET reused the pooled socket and failed; the third got a fresh one. Node’s request.reusedSocket flag tells you the failure happened on a reused connection, which is the classic signature of this race.
Retrying the GET is safe. The POST is the hard case: from the client’s side, you can’t tell whether the server received and processed it before the connection died. Retry only requests that are idempotent by method or protected by an idempotency key, and surface the rest as uncertain.
The structural fix is to make the client give up on idle connections before the server does. A well-known failure: a load balancer keeps idle connections to its Node targets for 60 seconds, but Node’s server.keepAliveTimeout defaults to 5 seconds. The balancer occasionally sends a request down a connection Node has just closed, and users see a scattering of 502s that never show up in the application’s logs. Raising the target’s keep-alive timeout above the balancer’s idle timeout fixes it. Whoever sends the requests should have the shorter idle timeout.
Interview practice
A checkout service calls a payments API over HTTPS at about 300 requests per second, and each call takes about 50 ms. Latency is higher than expected, and a few times an hour the logs show “socket hang up” on calls that create charges. How would you configure the HTTP client?
Answer
First I’d check that the service uses one shared client or agent for the payments host rather than creating one per call, and confirm reuse by comparing the number of TLS handshakes with the number of requests. That alone usually removes most of the extra latency.
For pool size, Little’s law gives 300 × 0.05 = 15 requests in flight on average, so I’d allow around 30 connections per instance, set a short timeout for waiting on a free connection, and record that wait as a metric. I’d set the client’s idle timeout below the server’s or load balancer’s (asking the payments team, or reading their Keep-Alive header) so the client always closes idle connections first.
The hang-ups on charge creation need care. A reset on a reused socket might mean the charge went through or might not, so I wouldn’t blindly retry. I’d send an idempotency key with every charge request, which the payments API uses to recognise a repeat. Then a retry after a reset is safe: either it creates the charge once or returns the original result.