A health endpoint looks like the easiest code in a service: return 200. The interesting part is what the caller does with the answer. In Kubernetes, the kubelet uses one probe to decide whether to restart a container and another to decide whether the Service should route traffic to it. A load balancer uses its health check to pull instances out of rotation. Answer the wrong question on the wrong endpoint and the platform will “heal” your service into a full outage.
That’s why this is an advanced interview topic despite the tiny amount of code. The questions are usually “What should a health check check?” and “What happens to your probes when the database goes down?”
Two questions, two endpoints
The common convention (used by Kubernetes components themselves) is /livez and /readyz. They answer different questions:
- Liveness: is this process broken in a way that a restart would fix? Deadlocked, stuck in an infinite loop, wedged with a corrupted in-memory state. If the answer is yes, killing the container and starting a fresh one helps.
- Readiness: should this instance receive traffic right now? It might be perfectly healthy but still warming caches, or shutting down, or unable to reach something it needs. The fix is to route around it, not restart it.
Kubernetes also has a startup probe, which holds off liveness checks until a slow-starting app is up. Without it, a strict liveness probe can kill a container that’s still running migrations, forever.
For a Node service, liveness should be close to trivial. If the event loop can run the handler and write a response, the process is alive. That check is meaningful on its own: a process stuck in a synchronous loop can’t answer at all, and the probe times out.
A service whose probes tell the truth
Here’s a server with both endpoints. Readiness considers three things: whether startup has finished, whether the process is draining for shutdown, and whether the database answers a ping within a deadline.
import http from 'node:http';
import { setTimeout as sleep } from 'node:timers/promises';
// Stand-in for a database client: ping latency we can change from the outside.
const db = { pingMs: 5, down: false };
async function pingDb() {
await sleep(db.pingMs);
if (db.down) throw new Error('ECONNREFUSED');
}
const state = { started: false, draining: false };
function withTimeout(promise, ms) {
let t;
return Promise.race([promise, new Promise((_, rej) => { t = setTimeout(() => rej(new Error(`timeout ${ms}ms`)), ms); })])
.finally(() => clearTimeout(t));
}
// Readiness result is cached so probes from many sources don't hammer the DB.
let lastCheck = { at: 0, result: null };
async function readiness() {
if (!state.started) return { ok: false, reason: 'starting' };
if (state.draining) return { ok: false, reason: 'draining' };
if (Date.now() - lastCheck.at < 1000) return lastCheck.result;
let result;
try { await withTimeout(pingDb(), 200); result = { ok: true }; }
catch (err) { result = { ok: false, reason: `db: ${err.message}` }; }
lastCheck = { at: Date.now(), result };
return result;
}
const server = http.createServer(async (req, res) => {
if (req.url === '/livez') return res.writeHead(200).end('ok'); // the process can answer
if (req.url === '/readyz') {
const r = await readiness();
res.writeHead(r.ok ? 200 : 503, { 'content-type': 'application/json' });
return res.end(JSON.stringify(r));
}
res.end('hello');
});A test script walked the service through its life: booting, started, the database refusing connections, the database hanging, and a shutdown signal. The real output from Node 22:
booting livez 200 ok readyz 503 {"ok":false,"reason":"starting"}
started livez 200 ok readyz 200 {"ok":true}
db refusing livez 200 ok readyz 503 {"ok":false,"reason":"db: ECONNREFUSED"}
db hanging livez 200 ok readyz 503 {"ok":false,"reason":"db: timeout 200ms"}
SIGTERM received livez 200 ok readyz 503 {"ok":false,"reason":"draining"}Liveness stayed 200 the whole time, because in none of these states would a restart help. Readiness went to 503 in each case where traffic should go elsewhere.
The design decisions in that code
The timeout on the dependency check. The “db hanging” row is the one people forget. A database that accepts the TCP connection but never answers (a full connection pool, a lock pile-up, a network black hole) makes an unbounded await db.query('SELECT 1') hang. The probe then times out on the kubelet’s side with no useful information, and if the endpoint shares code with liveness, the container gets killed. A 200 ms deadline turns a hang into a fast, explicit 503 with a reason. Keep the internal timeout comfortably below the probe’s timeoutSeconds, which defaults to 1 second in Kubernetes.
Caching the result. Probes arrive from the kubelet every few seconds, possibly from a load balancer too, and maybe from a monitoring system. With 50 pods, uncached readiness checks can add a steady stream of pings to a database that’s already struggling. Caching for a second keeps the check cheap while still reacting quickly.
Startup and draining are readiness concerns. Before the service has loaded its config, connected its pools and warmed whatever it needs, it should report unready so it doesn’t get traffic it will fail. During shutdown it should report unready first, so the load balancer stops sending requests before the server closes its listener. The graceful shutdown note covers that ordering in detail.
Reasons in the body. A 503 with {"reason":"db: timeout 200ms"} turns a vague “pods not ready” alert into a diagnosis. Keep it to non-sensitive summaries, though. Health endpoints are often reachable more widely than you’d expect, so don’t put connection strings, hostnames or stack traces in them.
The liveness probe that causes outages
Here’s the mistake interviewers are listening for. A team decides the health check should be “thorough” and makes liveness check the database too. Then the database has a five-minute failover.
Every pod’s liveness probe fails. Kubernetes restarts every pod. The restarted pods start up, try to connect, fail liveness again, and enter CrashLoopBackOff with growing back-off delays. When the database comes back, the pods are sitting in back-off for minutes longer, so the outage lasts longer than the database’s. Worse, all the pods reconnecting at once can overload the database just as it recovers. Restarting a process can’t fix a database, so the database had no business being in the liveness check.
The same reasoning applies to anything outside the process: caches, downstream APIs, message brokers. Liveness checks the process; readiness can consider dependencies.
Should readiness check shared dependencies?
This is the follow-up that separates good answers. If every pod’s readiness checks the same database, then during a database outage every pod becomes unready at once. The Service has no endpoints, and clients get a connection error or a 503 from the ingress instead of your application’s own error response. You’ve hidden the outage behind the platform, and you’ve also blocked endpoints that don’t need the database at all, like a cached product page or a status page.
There’s no single right answer, but the reasoning matters. Readiness checks are most useful for things that differ between instances: this pod’s warm-up, this pod’s connection pool, this pod’s draining state. For a dependency shared by all instances, many teams leave it out of readiness and let requests fail fast with a clear error (or serve degraded responses) while alerting on the dependency directly. Others include it deliberately because their load balancer can fail over to another region when all instances report unready. Saying which you’d choose and why is the point.
Interview practice
A Node service has a single /health endpoint used for both liveness and readiness. It runs SELECT 1 against PostgreSQL and calls the payments provider’s status API, and returns 200 only if both succeed. During a 10-minute payments-provider incident, the service’s error rate went to 100% for all endpoints, including ones that never touch payments. Explain what happened and redesign the checks.
Answer
Because /health drives liveness, the payments outage made every pod fail its liveness probe, and Kubernetes restarted them repeatedly. Restarts can’t fix a third party, so the pods ended up crash-looping, and while they were restarting, nothing was served, including endpoints unrelated to payments. Readiness failing at the same time also removed all pods from the Service. On top of that, the external call had no visible timeout, so slow responses from the provider would make probes time out rather than fail cleanly.
I’d split it. /livez returns 200 if the process can respond, with no external calls. /readyz reports startup, draining, and per-instance problems such as this pod’s own database pool failing a ping, with a short timeout and a cached result. I’d remove the payments provider from both probes: payment endpoints should fail fast with a clear error, ideally behind a circuit breaker, while the rest of the service keeps working, and the provider’s health should feed an alert or dashboard rather than a probe. I’d add a startup probe if boot takes more than a few seconds.