Kubernetes Startup, Readiness and Liveness Probes

Kubernetes has three probes, and they look almost identical in YAML: an HTTP path or a command, a period, a threshold. What differs is the consequence when each one fails, and that’s where outages come from. A liveness probe written like a readiness probe can restart every replica of a healthy service because its database had a slow minute. This guide starts from consequences and works back to configuration.

Examples assume Kubernetes 1.30 or later. The Node.js server below was run locally and the curl output is real; Kubernetes output is illustrative.

Same mechanics, three different consequences

All three probes are run by the kubelet on the Pod’s node, against the container. They support httpGet (any status from 200 to 399 counts as success), tcpSocket (the port accepts a connection), exec (the command exits 0) and grpc (the standard gRPC health checking protocol). Each has periodSeconds, timeoutSeconds (default 1), failureThreshold (default 3) and successThreshold.

The difference is what the kubelet does when the threshold is crossed:

Readiness is reversible and cheap: the Pod stays up and comes back into rotation when the probe passes again. Liveness is destructive: it kills the process, dropping in-flight requests and any warm cache. That asymmetry is the whole design principle. Use readiness for “not right now”, and liveness only for “never again without a restart”.

Liveness should be boring

A liveness probe exists for one situation: the process is still running but can’t make progress, and only a restart will fix it. A deadlock, an exhausted thread pool that never recovers, an event loop stuck in a tight loop. So the liveness endpoint should answer “can this process respond at all?” and do as little as possible.

The classic mistake is a /health endpoint that checks the database, the cache and two downstream APIs, used for both probes. Walk through what happens when the database has a 30-second failover:

1. Every replica’s /health starts returning 500.
2. Readiness fails, so all Pods leave the Service. That’s arguably fine; they can’t serve anyway.
3. Liveness also fails, three times in a row, so the kubelet kills every container.
4. All replicas restart at once, reconnect at once and warm up at once, right as the database comes back and is most fragile.

The database blip became an application outage, and the restarts made recovery slower. A restart can’t fix a database outage, so the database has no place in liveness.

Here’s a small server that keeps the two concerns apart. Readiness reflects warm-up and shutdown; liveness only proves the event loop is responding:

import http from 'node:http';

let ready = false;      // flips true once warm-up finishes
let draining = false;   // flips true on SIGTERM

const server = http.createServer((req, res) => {
  if (req.url === '/livez') {
    // Liveness: is this process able to answer at all? No dependency checks.
    res.writeHead(200).end('ok');
  } else if (req.url === '/readyz') {
    // Readiness: should this Pod receive traffic right now?
    const ok = ready && !draining;
    res.writeHead(ok ? 200 : 503).end(ok ? 'ready' : 'not ready');
  } else {
    res.writeHead(200).end('hello');
  }
});

server.listen(8080, () => console.log('listening on 8080'));

// Simulate a slow warm-up (loading a model, priming a cache).
setTimeout(() => { ready = true; console.log('warm-up done'); }, 2000);

process.on('SIGTERM', () => {
  draining = true;
  console.log('SIGTERM: failing readiness, finishing in-flight requests');
  setTimeout(() => server.close(() => process.exit(0)), 1000);
});

Running it and probing by hand shows each state:

$ node server.mjs &
listening on 8080
$ curl -s -w ' %{http_code}\n' localhost:8080/livez
ok 200
$ curl -s -w ' %{http_code}\n' localhost:8080/readyz
not ready 503
warm-up done
$ curl -s -w ' %{http_code}\n' localhost:8080/readyz
ready 200
$ kill -TERM %1
SIGTERM: failing readiness, finishing in-flight requests
$ curl -s -w ' %{http_code}\n' localhost:8080/readyz
not ready 503

During warm-up the process is alive but not ready, which is exactly the state you want Kubernetes to see: don’t send traffic, don’t restart. During shutdown, readiness fails on purpose while in-flight work completes.

Should readiness check the database, then? It depends. If a Pod without a database connection can still serve something useful (cached pages, a degraded mode), keep the database out of readiness too. If every replica shares the same database, readiness failing on all of them at once just empties the Service and turns errors into connection refusals. Many teams make readiness reflect only Pod-local conditions (warmed up, not overloaded, not shutting down) and handle dependency failures in the application with timeouts and circuit breakers.

Slow starts: what startupProbe is for

Before startupProbe existed, a JVM service that needed 90 seconds to boot forced an awkward choice: a large initialDelaySeconds on liveness (slowing detection of real hangs forever) or a short one (and the kubelet kills the app mid-boot, which is a common cause of CrashLoopBackOff with no error in the logs).

A startupProbe solves this. Until it succeeds, liveness and readiness don’t run at all. Once it succeeds, it never runs again for that container. The time it allows is failureThreshold × periodSeconds:

apiVersion: v1
kind: Pod
metadata:
  name: pricing
  labels:
    app: pricing
spec:
  containers:
    - name: app
      image: registry.example.com/pricing:4.2.0
      ports:
        - containerPort: 8080
      startupProbe:
        httpGet:
          path: /livez
          port: 8080
        periodSeconds: 5
        failureThreshold: 36
      livenessProbe:
        httpGet:
          path: /livez
          port: 8080
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /readyz
          port: 8080
        periodSeconds: 5
        failureThreshold: 2

That gives boot up to 36 × 5 = 180 seconds, checked every 5 seconds so a fast boot isn’t held back. After startup, a genuine hang is detected in about 30 seconds (3 failures × 10 seconds). Readiness reacts within about 10 seconds. Size the startup budget from measured boot times at the slow end, not the average, and remember that boot is slower on a busy node or with a CPU limit throttling the JVM’s class loading.

Note that the startupProbe reuses the liveness endpoint. That’s the usual choice: “has it booted?” and “is it responsive?” ask the same thing at different times.

Smaller details interviewers probe for

• timeoutSeconds defaults to 1. Under load, a 1-second HTTP timeout can fail an otherwise healthy app, and with liveness that becomes restarts under exactly the traffic you need capacity for. Give liveness generous timeouts.
• exec probes cost a process spawn each time. Running pg_isready every 2 seconds across hundreds of Pods adds up on a node. HTTP or gRPC probes are cheaper.
• A Pod without a readiness probe is Ready as soon as its containers start. Rolling updates and PodDisruptionBudgets both trust readiness, so missing it makes both unsafe.
• Probes go to the Pod IP from the kubelet, not through the Service, so NetworkPolicies don’t normally block them, and a probe can’t tell you whether the Service routing works.

Interview practice

“Every night at 02:00 a batch job saturates the orders database for about two minutes. During that window, the orders-api Pods restart repeatedly, and the morning shows 40 restarts per Pod. The probes are livenessProbe and readinessProbe, both on /health, which runs SELECT 1. What’s going on and how would you fix it?”

Answer

SELECT 1 waits behind the saturated database, so /health exceeds the 1-second default timeout or errors. Readiness fails, which is harmless enough, but liveness fails too, three periods in a row, so the kubelet restarts the container. The new container’s liveness probe fails the same way, so it’s restarted again, accumulating restarts and backoff. The API Pods weren’t broken; their dependency was slow.

I’d split the endpoints. Liveness moves to a /livez that only proves the process responds, with no database call and a few seconds of timeout. Readiness gets its own /readyz; whether it includes the database depends on whether the API can do anything useful without it, but even if it does, failing readiness only removes Pods from the Service rather than killing them. In the application I’d set a query timeout and return 503 quickly when the database is saturated, so clients can retry. Separately I’d raise the batch job’s priority problem with its owners, since the probe fix stops the restarts but doesn’t stop the contention.

Further reading

• Kubernetes interview questions
• Kubernetes MCQs
• Pod termination and connection draining
• Deployments and rolling updates
• Official docs: Configure Liveness, Readiness and Startup Probes