Every rolling update, node drain, autoscaler scale-down and eviction ends the same way: Kubernetes deletes a Pod that is still serving traffic. If shutdown isn’t handled, each of those routine events produces a small burst of 502s, reset connections or half-finished jobs. The burst is short enough to be ignored in dashboards and frequent enough that users notice. Getting it right means understanding a race in the shutdown sequence, then closing it from both sides: Kubernetes configuration and application code.
Examples assume Kubernetes 1.30 or later. The Node.js server below was run locally and its output is real; Kubernetes output is illustrative.
What happens after kubectl delete pod
When a Pod is deleted (by you, a ReplicaSet scaling down, or the Eviction API during a drain), the API server doesn’t remove it immediately. It sets metadata.deletionTimestamp and deletionGracePeriodSeconds, taken from the Pod’s terminationGracePeriodSeconds (default 30). From that moment, two independent chains of events start.
On the node, the kubelet sees the deletion and:
- Runs each container’s
preStophook, if there is one, and waits for it. - Sends SIGTERM (or the image’s
STOPSIGNAL) to PID 1 in each container. - Waits for the containers to exit, up to whatever remains of the grace period.
- Sends SIGKILL to anything still running.
In the control plane and on every other node, the EndpointSlice controller marks the Pod’s endpoint as terminating (ready: false, terminating: true). Then kube-proxy on each node, ingress controllers, service-mesh sidecars and cloud load balancer controllers each notice that change on their own schedule and stop sending new traffic to the Pod.
Nothing orders these two chains. The kubelet doesn’t wait for routing to update before sending SIGTERM. On a busy cluster, kube-proxy on some node might take a second or two to rewrite its rules; an ingress controller might reload its config a few seconds later; a cloud load balancer’s target deregistration might take longer still. If the app exits the moment it gets SIGTERM, any request routed to it during that window hits a closed port: connection refused, which ingress controllers report as 502 or 503.
That race is the single most important thing to explain in an interview about graceful shutdown. Everything else is a fix for it.
Closing the race with preStop
The standard fix is to make the Pod keep serving for a few seconds after deletion begins, so routing catches up before the app stops accepting. A preStop hook runs before SIGTERM, so a short sleep there delays the signal:
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
namespace: shop
spec:
replicas: 4
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
terminationGracePeriodSeconds: 45
containers:
- name: api
image: registry.example.com/checkout:5.4.1
ports:
- name: http
containerPort: 8080
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
lifecycle:
preStop:
sleep:
seconds: 10The sleep action is built into the kubelet (beta and enabled by default since 1.30), so it works in distroless images with no shell. On older clusters the same thing was written as exec: {command: ["sleep", "10"]}, which needs a sleep binary in the image.
Budget the timeline explicitly. With a 45-second grace period:
- 0 to 10 s: preStop sleep. The app still serves normally while every load balancer removes the Pod.
- 10 s: SIGTERM. The app stops accepting and finishes in-flight requests.
- up to 45 s: the app must have exited, or it’s killed.
The preStop time counts against the grace period, not in addition to it. A 30-second preStop with the default 30-second grace period leaves the app zero seconds to drain after SIGTERM. If the hook is still running when the grace period expires, the kubelet grants a one-off 2-second extension and then kills the container.
How long should the sleep be? Long enough for the slowest thing that routes to you. In-cluster kube-proxy updates usually take a second or two. Ingress controllers and cloud load balancers vary: AWS target groups have a deregistration delay (300 seconds by default) during which they drain existing connections, while new connections stop sooner. Five to fifteen seconds is common; measure instead of guessing.
What the application has to do on SIGTERM
The app’s half of the contract: stop accepting new connections, finish what’s in flight, close idle keep-alive connections, and exit. Here’s a minimal Node.js version:
import http from 'node:http';
let shuttingDown = false;
const server = http.createServer(async (req, res) => {
if (req.url === '/readyz') {
res.writeHead(shuttingDown ? 503 : 200).end();
return;
}
// A slow request: 3 seconds of "work".
await new Promise((r) => setTimeout(r, 3000));
if (shuttingDown) res.setHeader('Connection', 'close');
res.end(`done ${req.url}\n`);
});
server.listen(8080);
process.on('SIGTERM', () => {
const t0 = Date.now();
console.log('SIGTERM received');
shuttingDown = true;
server.close(() => { // stop accepting, wait for in-flight requests
console.log(`drained in ${Date.now() - t0} ms, exiting`);
process.exit(0);
});
server.closeIdleConnections(); // drop keep-alive sockets with no request in flight
setTimeout(() => { // safety net, below terminationGracePeriodSeconds
console.log('drain timeout, exiting anyway');
process.exit(1);
}, 20_000).unref();
});To check it, start a slow request, send SIGTERM half a second later, then try a new connection:
$ node server.mjs &
$ curl -s localhost:8080/report & # in flight, takes 3 s
$ kill -TERM %1 # 0.5 s later
SIGTERM received
$ curl -s -m 2 localhost:8080/new || echo "new connection: curl exit $?"
new connection: curl exit 7
done /report
drained in 2558 ms, exitingThe in-flight request completed, a new connection was refused (curl exit code 7 is “failed to connect”), and the process exited 0 about 2.5 seconds after the signal, once the last request finished. In Kubernetes, that refusal is harmless because the preStop sleep already moved traffic elsewhere before SIGTERM arrived.
A few details in that code matter more than they look:
- Keep-alive connections.
server.close()stops new connections but leaves open sockets alone, and a client with an idle keep-alive connection would happily send its next request down it.closeIdleConnections()handles idle ones;Connection: closeon responses during shutdown handles busy ones. HTTP/2 and gRPC have their own equivalent, a GOAWAY frame, which most servers send on graceful stop. - The safety-net timer sits below the grace period (20 of the 35 seconds left after preStop) so the app can log and exit cleanly instead of being SIGKILLed mid-write.
- Failing readiness is optional here: once the Pod is terminating, its endpoint is already marked not ready. It still helps with anything that polls the app’s health directly.
When SIGTERM never arrives
A surprising number of “graceful shutdown doesn’t work” reports are really “the app never got the signal”. The kubelet signals PID 1 in the container. If the Dockerfile uses shell form, CMD node server.mjs, PID 1 is /bin/sh -c, which typically doesn’t forward SIGTERM to its child. The app keeps running for the full grace period and is then SIGKILLed, so every shutdown takes exactly 30 seconds and ends with exit code 137. The tell-tale is in kubectl describe pod: Last State: Terminated, Reason: Error, Exit Code: 137 with no OOM involved.
Fixes: use exec form, CMD ["node", "server.mjs"]; in an entrypoint script, end with exec "$@" so the app replaces the shell; or run a tiny init like tini (docker run --init does the same locally) that forwards signals and reaps zombies. If the app expects a different signal (nginx uses SIGQUIT for graceful stop), set STOPSIGNAL in the image.
Long-running work and node drains
Some workloads can’t finish in 45 seconds: a worker processing a 10-minute video, a WebSocket server with thousands of open connections. Options:
- Raise
terminationGracePeriodSecondsto cover the longest unit of work, accepting that rollouts and drains take longer. - Make work resumable: on SIGTERM, stop taking jobs from the queue, return unacknowledged messages, and let another worker pick them up.
- For WebSockets, tell clients to reconnect (send a close frame with a reconnect hint) and let them spread across other Pods, rather than holding sockets open.
kubectl drain respects all of this: it evicts Pods through the Eviction API, each Pod gets its normal grace period, and PodDisruptionBudgets limit how many go at once. A force delete (kubectl delete pod --grace-period=0 --force) skips it all and removes the API object immediately, even if the container is still running on the node, which is why it’s a last resort.
Interview practice
“During every deploy of checkout, the ingress logs show about 20 to 40 502 Bad Gateway responses over a couple of seconds, though the app logs no errors. The Deployment has a readiness probe and maxUnavailable: 0. What’s causing it and what would you change?”
Answer
maxUnavailable: 0 and readiness only govern new Pods joining; the 502s come from old Pods leaving. When the rollout deletes an old Pod, the kubelet sends SIGTERM at the same moment the endpoint is marked terminating. If the app exits immediately on SIGTERM (or the framework closes its listener at once), the ingress controller, which hasn’t yet reloaded its upstream list, keeps sending some requests to a port that’s gone. It gets connection refused and reports 502. The app logs nothing because those requests never reached it.
I’d add a preStop sleep of around 10 seconds so the Pod keeps serving until the ingress and kube-proxy drop it, raise terminationGracePeriodSeconds so preStop plus drain time fits comfortably, and make sure the app on SIGTERM finishes in-flight requests and closes keep-alive connections rather than exiting abruptly. I’d also confirm the signal reaches the app (exec-form CMD) by checking that terminated containers exit 0 rather than 137. Then I’d run a load test during a rollout and confirm the 502 count drops to zero.