Docker Health Checks and Useful Probes

docker ps says your container is “Up 3 hours”. That means the process hasn’t exited. It doesn’t mean the web server inside is answering requests; it could be deadlocked, stuck in a retry loop against a database that moved, or serving 500s on every route. A health check is how you teach Docker the difference between “running” and “working”.

The examples use nginx:alpine with checks written as docker run flags so they’re easy to reproduce. The same settings exist as the HEALTHCHECK Dockerfile instruction and the healthcheck: key in Compose.

What Docker actually does

You give Docker a command. It runs that command inside the container, as a separate process via docker exec-style execution, every interval. The exit code decides the result: 0 means healthy, 1 means unhealthy (2 is reserved; don’t use it). If the command takes longer than timeout, that run counts as a failure.

docker run -d --name web \
  --health-cmd 'wget -q --spider http://127.0.0.1/ || exit 1' \
  --health-interval 2s --health-timeout 2s --health-retries 3 \
  nginx:alpine

The equivalent Dockerfile instruction:

HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD wget -q --spider http://127.0.0.1/ || exit 1

The || exit 1 matters. Some tools exit with codes other than 0 and 1 on failure (curl -f returns 22, for example), and normalising to 1 keeps you inside the contract.

Polling the status every second shows the transition:

1 s: starting
2 s: healthy
3 s: healthy

starting lasts until the first successful probe. docker ps appends the state to the status column: Up 6 seconds (healthy).

Watching it go unhealthy

Remove nginx’s index page, so / returns 403, and the probe starts failing:

$ docker exec web rm /usr/share/nginx/html/index.html
$ docker exec web sh -c 'wget -q --spider http://127.0.0.1/; echo probe exit $?'
wget: server returned error: HTTP/1.1 403 Forbidden
probe exit 1

After three consecutive failures (the retries value) the status flips:

$ docker ps --format '{{.Names}} {{.Status}}'
web Up 13 seconds (unhealthy)

docker inspect keeps the last five probe results with their output, which is the first place to look when a container is marked unhealthy:

docker inspect -f '{{json .State.Health}}' web

From that JSON, the status, the failing streak and the latest probe:

unhealthy 5
1 'wget: server returned error: HTTP/1.1 403 Forbidden\n'

Here’s the part that surprises people. This container was started with --restart unless-stopped, and after going unhealthy:

restarts=0 running=true

Nothing happened. On a standalone Docker Engine, restart policies react to the process exiting, not to health status. “unhealthy” is a label. What acts on it: Docker Swarm (it replaces unhealthy tasks), Compose’s depends_on with condition: service_healthy (it waits for health before starting dependents), load balancers or scripts that read the status, and tools like autoheal that restart unhealthy containers via the API. Kubernetes ignores Docker’s HEALTHCHECK entirely and uses its own liveness and readiness probes. Saying this clearly is a good way to stand out in an interview, because many candidates assume the health check is self-healing. See restart policies for what does trigger restarts.

Timing: start period, interval, retries

The defaults are an interval of 30 seconds, a timeout of 30 seconds and 3 retries, with no start period. That means a broken container takes up to about 90 seconds to be marked unhealthy, and a slow-starting one (a JVM that takes 45 seconds to warm up) might be marked unhealthy before it ever had a chance.

start_period handles the second problem. During it, failures don’t count toward retries, but a success immediately marks the container healthy. Size it to your slowest realistic start, not the average. Docker Engine 25 added start_interval, a faster probe rate used only during the start period, so a service can be marked healthy within a second or two of being ready without probing every second forever.

For the steady state, the trade-off is detection time against overhead. Every probe forks a process inside the container; a probe every second that runs a Python interpreter costs real CPU. Something like interval: 10s, timeout: 3s, retries: 3 gives roughly 30-second detection for cheap probes.

Writing a probe that tells the truth

The command has to exist in the image. curl isn’t in node:22-slim, Alpine has wget from BusyBox but not curl, and distroless images have no shell at all. Options: use the language runtime (node -e "fetch('http://127.0.0.1:3000/healthz').then(r=>process.exit(r.ok?0:1),()=>process.exit(1))"), ship a tiny static probe binary, or build the check into your app as a subcommand (myapp healthcheck). Installing curl only for the probe adds a tool attackers like.

What to check is the more interesting question:

• Check that this process can serve. A dedicated /healthz endpoint that confirms the server loop is responsive, maybe that a worker thread pool isn’t wedged. Fast, no side effects, no authentication.
• Don’t fail because a dependency is down. If every API container’s health check queries the database, a 10-second database blip marks every container unhealthy at once. Swarm replaces them all, the replacements also fail, and a brief incident becomes a full outage. Dependencies belong in readiness logic (stop taking traffic) or in alerts, not in the signal that kills containers.
• Probe on the interface you serve on, when practical. 127.0.0.1 works for most servers; just remember that a probe hitting loopback passes even if the app accidentally bound only to loopback and is unreachable from outside, as the port publishing note describes.
• Keep it within the timeout. A probe that does real work under load times out exactly when the system is busiest.

Interview practice

Your Compose file has api with depends_on: db: condition: service_healthy, and db (Postgres) has test: ["CMD", "pg_isready", "-U", "app"]. During a load test, api containers are fine, but you’ve also added a health check to api that runs SELECT 1 against the database with a 2-second timeout, and an autoheal container restarts anything unhealthy. Under load, all api containers restart within a minute of each other and the error rate spikes. What’s happening, and how would you redesign the checks?

Answer

Under load, the database’s response times rise. The api health check’s SELECT 1 waits behind real queries in the connection pool, exceeds 2 seconds, and fails. Every api replica shares the same database, so they all cross retries at roughly the same time, and autoheal restarts them together. Restarting drops in-flight requests, and the new containers reconnect to an already busy database, which makes it worse. The health check has turned database slowness into an API outage.

Redesign: make the api health check local, a /healthz endpoint that confirms the HTTP server and event loop respond, without touching the database. Handle database trouble inside the app (timeouts, circuit breaking, returning 503 for affected routes) and surface it through metrics and alerts. Keep the db health check (pg_isready) for start-up ordering, since that’s what service_healthy is for. If something must restart containers automatically, add a longer interval or more retries so a short spike doesn’t trigger it, and avoid restarting all replicas at once.

Keep going

• Docker interview questions
• Docker MCQ practice
• Docker Compose dependencies and readiness
• Docker restart policies
• Official reference: Dockerfile HEALTHCHECK