Node.js Graceful Shutdown and In-Flight Requests

Every deploy, scale-down and node drain ends the same way for a running process: the orchestrator sends SIGTERM, waits a grace period, then sends SIGKILL. What happens in between decides whether users see a deploy. If the process dies the moment it gets the signal, every request it was in the middle of fails, and so does any database write or queue message it hadn’t finished.

Graceful shutdown is a favourite “how would you deploy this safely?” interview question for Node, partly because the default behaviour surprises people and partly because there’s an ordering problem with load balancers that a quick answer misses.

What the default does to an in-flight request

The experiment uses a server whose checkout endpoint takes 500 ms. A client script starts the server as a child process, sends a checkout request, sends SIGTERM 100 ms later, and then tries a new request:

import http from 'node:http';
import { setTimeout as sleep } from 'node:timers/promises';

let shuttingDown = false;

const server = http.createServer(async (req, res) => {
  if (req.url === '/ready') {
    res.writeHead(shuttingDown ? 503 : 200).end(shuttingDown ? 'draining' : 'ready');
    return;
  }
  await sleep(500); // a slow checkout request
  res.end('order placed');
});
server.listen(3200);

With no signal handler, the output on Node 22 was:

[server] listening (default)
[client] checkout failed: UND_ERR_SOCKET
[client] server exited code=null signal=SIGTERM
[client] new request failed: ECONNREFUSED

Node’s default action for SIGTERM is to exit at once. The socket closed under the half-finished checkout, and fetch (which uses undici) reported UND_ERR_SOCKET; other clients show ECONNRESET or socket hang up. The client can’t tell whether the order went through. If the handler had written to the database and was about to respond, the order exists but the user sees an error and probably tries again.

Draining with server.close

server.close() does two things. It stops the server from accepting new connections immediately, and it calls its callback once all existing connections have ended. That’s the core of a drain:

process.once('SIGTERM', async () => {
  console.log('[server] SIGTERM: draining');
  shuttingDown = true;
  const force = setTimeout(() => {
    console.log('[server] drain timeout, forcing exit');
    process.exit(1);
  }, 10_000);
  force.unref();

  server.close(async (err) => {
    // runs once every connection has closed
    console.log('[server] all connections closed');
    // await db.end(); await queue.close(); flush logs...
    process.exit(err ? 1 : 0);
  });
  server.closeIdleConnections(); // keep-alive sockets with no request in flight
});

Same client, graceful mode:

[server] listening (graceful)
[server] SIGTERM: draining
[client] new request failed: ECONNREFUSED
[client] checkout: 200 order placed
[server] all connections closed
[client] server exited code=0 signal=null

The checkout finished with a 200, the new request was refused, and the process exited cleanly with code 0 after the last connection closed.

A few lines in that handler deserve explanation, because each one fixes a real failure:

• The forced-exit timer. Some requests never finish: a long poll, a stuck upstream call, a client streaming slowly. Without a deadline, the drain waits until the orchestrator gives up and sends SIGKILL, which skips your cleanup entirely. Set the timer below the grace period (Kubernetes defaults to 30 seconds) so your own exit path runs. unref() stops the timer from keeping the process alive by itself if everything else finishes.
• Idle keep-alive connections. HTTP clients and load balancers keep connections open between requests. server.close() waits for every connection, including idle ones, so before Node 19 a single idle keep-alive socket could hold shutdown open until its timeout. Node 19 and later close idle connections as part of close(); calling closeIdleConnections() explicitly is harmless and makes the intent obvious. closeAllConnections() also exists, as the hammer for the forced path.
• Cleanup after the drain, not before. Close the database pool and message consumers after requests finish. Closing the pool first makes the in-flight requests you’re trying to save fail on their next query.
• process.once. A second SIGTERM (or an impatient Ctrl+C) shouldn’t start a second drain. Some services make the second signal force an immediate exit.

Background work needs the same treatment. A queue consumer should stop fetching new messages, finish or release the ones it holds (so the broker redelivers them), then exit. Timers and intervals that start new work should be cleared.

The load balancer race

Look at the graceful output again: the new request got ECONNREFUSED. Inside a test that’s fine. Behind a load balancer it means dropped traffic, because the load balancer doesn’t learn about the shutdown at the same moment the process does.

In Kubernetes, deleting a pod does two things in parallel: the kubelet sends SIGTERM to the container, and the endpoints controller removes the pod from the Service, which then has to propagate to kube-proxy, ingress controllers and cloud load balancers. That propagation takes from a fraction of a second to several seconds. If the process closes its listener instantly, requests routed during that window hit a closed port.

So the order is:

1. On SIGTERM, flip a flag so the readiness endpoint returns 503. Keep accepting and serving requests normally.
2. Wait a few seconds for the routing change to propagate. A preStop hook with sleep 5 in the pod spec does the same thing before the signal is even sent.
3. Then call server.close(), drain, clean up and exit, all inside the grace period.

The readiness flag in the example only helps if the process is still listening when the load balancer probes it; once close() has run, probes get refused anyway. Keep liveness and readiness separate here, too. A failing liveness probe makes Kubernetes kill the container, which is the opposite of what you want during a drain.

Making sure the signal arrives at all

A surprising number of “graceful shutdown doesn’t work” reports come down to node never receiving SIGTERM.

In a container, the signal goes to PID 1. If your Dockerfile says CMD npm start or uses the shell form CMD node server.js, PID 1 is npm or /bin/sh, which may not forward the signal to node. The container then sits until SIGKILL arrives 30 seconds later, and every deploy takes 30 seconds per pod with no drain. Use the exec form, CMD ["node", "server.js"], so node is PID 1. Since PID 1 also has special signal semantics in Linux (default handlers don’t apply), you need the explicit SIGTERM handler or a small init like docker run --init or tini.

Locally, Ctrl+C sends SIGINT, not SIGTERM. Handle both if you want the same behaviour in development. SIGKILL can’t be caught at all, which is why everything important has to fit inside the grace period.

Interview practice

A Node API runs in Kubernetes with the shutdown handler from this article: on SIGTERM it calls server.close(), waits for connections and exits. During every rolling deploy, the team still sees a short burst of 502 errors from the ingress, but no errors in the application logs. What’s going on, and how do you fix it?

Answer

The application is draining correctly, which is why its logs are clean. The 502s come from requests that never reached it. When the pod is deleted, SIGTERM and the endpoint removal happen concurrently, and the ingress keeps routing to the pod for a short time. Because the handler calls server.close() immediately, the listener is already closed, so those requests get connection refused or reset and the ingress returns 502.

The fix is to delay closing. On SIGTERM, mark the pod unready (readiness returns 503) but keep serving for a few seconds, or add a preStop hook that sleeps for about 5 to 10 seconds before the signal is delivered. Only then call server.close(), drain in-flight requests and close the database pool. The total (delay plus drain plus cleanup) must fit within terminationGracePeriodSeconds, with the forced-exit timer just under it. I’d also check the ingress’s keep-alive behaviour: if it reuses connections to the pod, responses during the drain should send Connection: close so it opens new connections to healthy pods.

Further reading

• Node.js interview questions
• Node.js MCQs
• Health endpoints: liveness and readiness
• EADDRINUSE: address already in use
• Official docs: server.close() in the Node.js HTTP module