Docker Logs and Application Diagnostics

When a container misbehaves, docker logs is the first command most people type. Sometimes it shows exactly what went wrong. Sometimes it shows nothing at all, and the reason is almost never Docker losing data. It’s that the application wrote somewhere Docker doesn’t look, or hasn’t written yet. Knowing how log capture works lets you fix those cases in minutes and set up logging that won’t fill a production disk.

Examples were run with alpine:3.20, nginx:alpine and the Python 3 interpreter that ships in node:22-bookworm, on a daemon using the default json-file driver.

Docker captures two streams, nothing else

The container runtime connects the main process’s stdout and stderr to the daemon, which hands each line to a logging driver. That’s the whole mechanism. A process that writes to both streams and a file:

docker run -d --name l1 alpine:3.20 sh -c \
  'echo "to stdout"; echo "to stderr" >&2; echo "to file" > /var/log/app.log; sleep 300'

$ docker logs l1
to stderr
to stdout
$ docker logs l1 2>/dev/null
to stdout

The file line never appears. docker logs replays the two streams on your terminal’s stdout and stderr respectively, which is why redirecting 2>/dev/null filters out the stderr line. It also means docker logs l1 | grep error misses errors written to stderr; use docker logs l1 2>&1 | grep error.

Plenty of software logs to files by default: Apache, older Java apps with log4j file appenders, PHP-FPM, cron jobs. The official nginx image solves it with symlinks:

$ docker run --rm nginx:alpine ls -l /var/log/nginx/
lrwxrwxrwx    1 root     root            11 Sep  2 21:05 access.log -> /dev/stdout
lrwxrwxrwx    1 root     root            11 Sep  2 21:05 error.log -> /dev/stderr

nginx thinks it’s writing files; the bytes go to the container’s streams. For your own app, configure the logger to write to stdout (12-factor style) instead. The file approach also has a second cost: those log files grow inside the container’s writable layer, invisible to log tooling and fatal on a read-only root filesystem.

The empty-logs mystery: buffering

A Python worker prints a start-up line and then sleeps:

$ docker run -d --name py node:22-bookworm python3 -c 'import time
print("worker started")
time.sleep(300)'
$ docker logs py
$

Nothing. The process is running and has called print. But when stdout isn’t a terminal, Python switches to block buffering and holds output until about 8 KB accumulates or the process exits. Inside a detached container, stdout is a pipe, so a quiet worker can run for hours with its log lines sitting in a buffer, and if it’s then killed with SIGKILL they’re lost entirely. Exactly the lines you needed for the post-mortem.

$ docker run -d --name py2 -e PYTHONUNBUFFERED=1 node:22-bookworm python3 -c '...same...'
$ docker logs py2
worker started

Set ENV PYTHONUNBUFFERED=1 in Python images, or run python -u. Other runtimes vary: Node writes to pipes asynchronously but promptly, C programs using stdio buffer like Python (stdbuf -oL or setvbuf helps), and Ruby needs $stdout.sync = true. If docker logs is empty but the process is clearly alive, suspect buffering before anything else.

Where the bytes go: drivers and rotation

The default driver is json-file. Each line is stored as a JSON object with the stream name and a timestamp, in a file per container:

$ docker info --format '{{.LoggingDriver}}'
json-file
$ docker inspect -f '{{.HostConfig.LogConfig.Type}} {{json .HostConfig.LogConfig.Config}} {{.LogPath}}' l1
json-file {} /var/lib/docker/containers/b06d37e1f9cf.../b06d37e1f9cf...-json.log

The {} is the problem. With no options, json-file never rotates. A container logging 1 MB a minute writes about 1.4 GB a day to /var/lib/docker, forever, until the disk fills and every container on the host starts failing. Set limits per container or, better, as daemon defaults in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" }
}

Daemon defaults only apply to containers created after the daemon restarts; existing containers keep their old config until they’re recreated. With a small limit you can see rotation discard old lines:

$ docker run -d --name l2 --log-opt max-size=10k --log-opt max-file=2 alpine:3.20 \
    sh -c 'i=0; while [ $i -lt 2000 ]; do echo "line $i padding padding padding"; i=$((i+1)); done; sleep 300'
$ docker logs l2 | head -1
line 1854 padding padding padding
$ docker logs l2 | wc -l
146

Only the last 146 of 2,000 lines survive. Local logs are a short buffer for debugging, not your log archive. The local driver is a more compact alternative with rotation on by default.

For a real system, ship logs off the host. Drivers like fluentd, gelf, awslogs, gcplogs and syslog send directly; alternatively keep json-file and run a collector (Fluent Bit, Vector, the Datadog or Grafana agent) that tails the files. Since Docker 20.10, docker logs keeps working with remote drivers through a local “dual logging” cache. One more option worth knowing for interviews: --log-opt mode=non-blocking with a max-buffer-size. In the default blocking mode, if the driver can’t keep up (a slow remote endpoint), the application’s writes to stdout block, and a logging outage becomes an application outage. Non-blocking mode drops log lines instead when the buffer fills. Which failure you prefer is a real design decision.

Logs are one signal among several

Structured JSON lines make logs queryable once they reach a backend:

{"level":"error","msg":"payment failed","order_id":"A-1042","latency_ms":812}

But logs only tell you what the application said. For what happened to the container, use the other built-in sources:

• docker inspect -f '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}' app for how it ended. Exit 137 with OOMKilled: true means the kernel killed it, and the app logged nothing because it never got the chance.
• docker events --filter container=app for the lifecycle timeline: start, die, oom, health_status changes, restarts.
• docker stats for live CPU, memory and network per container.
• docker logs --since 10m --timestamps app to line application output up with those events.

When a container exits immediately, the combination of exit code and the last few log lines usually explains it.

Interview practice

A Celery worker container on a VM restarts several times a night (restart: always). docker logs after each restart shows only the start-up banner, with nothing about why the previous run ended. Separately, the VM’s disk alert fires every few weeks and the team clears /var/lib/docker/containers by hand. Diagnose both and propose a setup.

Answer

For the restarts, the logs are from the current container run; check how the previous one ended. docker inspect on the container shows RestartCount, and docker events --since 12h --filter container=worker shows oom or die events with exit codes. If it’s 137 with OOMKilled: true, the kernel killed the worker, which explains the missing error. Also check buffering: Python workers without PYTHONUNBUFFERED=1 can lose their last lines on a kill. Set it, and ship logs off-box so earlier runs remain searchable.

For the disk, the containers are using json-file with no max-size, so logs grow without bound. Deleting files under /var/lib/docker/containers by hand is risky (it can corrupt container state, and the daemon may keep deleted files open so space isn’t freed). Set log-opts with max-size and max-file in daemon.json, restart the daemon in a maintenance window, and recreate containers so they pick it up. Ship logs to a central system with a collector, and consider mode=non-blocking if the remote endpoint is ever slow.

Keep going

• Docker interview questions
• Docker MCQ practice
• Docker container exits immediately
• Docker no space left on device
• Official reference: Configure logging drivers