Docker Layers and Build Cache Invalidation

The first build of an image is slow, and that’s fine. What isn’t fine is the tenth build, after you changed one line in server.js, taking just as long because the whole dependency install ran again. That’s almost always a Dockerfile ordering problem, and fixing it requires understanding exactly how the build cache decides what to reuse.

The builds here use BuildKit (the default builder since Docker Engine 23) with node:22-bookworm. To make the expensive step obvious, the dependency install is padded with sleep 5.

How a build turns into layers

Each instruction that changes the filesystem (RUN, COPY, ADD) produces a layer: a tarball of the files that step added, changed or deleted. Instructions like WORKDIR, ENV and CMD mostly change the image config. docker history lists the result, newest first:

$ docker history fast --format '{{.CreatedBy}}\t{{.Size}}'
CMD ["node" "server.js"]	0B
COPY . . # buildkit	20.5kB
RUN /bin/sh -c npm install --offline --no-au…	2.7MB
COPY package*.json ./ # buildkit	12.3kB
WORKDIR /app	8.19kB
CMD ["node"]	0B

Layers are stacked, content-addressed and shared. Ten images built on the same node:22-bookworm store its layers once, and a registry push only uploads layers the registry doesn’t already have.

The cache rule: a change reruns everything below it

When BuildKit reaches a step, it computes a cache key from the step’s parent (the result of everything above) plus the step’s own inputs. If a matching key exists, it reuses the stored result and prints CACHED. If not, it runs the step, and because the parent of every following step has now changed, nothing below it can hit the cache either.

What counts as “inputs” differs by instruction:

• COPY and ADD use a checksum of the copied files’ contents and metadata like permissions. Changing a byte invalidates the step. Touching a file without changing it doesn’t, as the timestamp test below shows.
• RUN uses the command string. BuildKit doesn’t look at what the command does, so RUN apt-get update stays cached for months even as the package index moves on.
• ARG and ENV values that a step can see are part of its key, so changing a build arg invalidates the first step that uses it.

The first rule is where the slow Dockerfile goes wrong:

FROM node:22-bookworm
WORKDIR /app
COPY . .
RUN npm install --offline --no-audit --no-fund && sleep 5
CMD ["node", "server.js"]

Edit server.js and rebuild:

#6 [2/4] WORKDIR /app
#6 CACHED
#7 [3/4] COPY . .
#7 DONE 0.1s
#8 [4/4] RUN npm install --offline --no-audit --no-fund && sleep 5
#8 DONE 6.0s

COPY . . includes server.js, so its checksum changed, so the install step’s parent changed, so the install reran. In a real project that’s minutes of downloading packages for a one-line change.

Ordering for the common case

Split the copy into what changes rarely and what changes constantly:

FROM node:22-bookworm
WORKDIR /app
COPY package*.json ./
RUN npm install --offline --no-audit --no-fund && sleep 5
COPY . .
CMD ["node", "server.js"]

Same edit, rebuild:

#7 [3/5] COPY package*.json ./
#7 CACHED
#8 [4/5] RUN npm install --offline --no-audit --no-fund && sleep 5
#8 CACHED
#9 [5/5] COPY . .
#9 DONE 0.1s

The install only reruns when package.json or package-lock.json changes, which is exactly when it should. When you’re diagnosing a slow build, --progress=plain output is the tool: find the first step that isn’t CACHED, because that step’s inputs are what changed, and everything after it is collateral. The pattern is the same in every ecosystem: requirements.txt then pip install, go.mod/go.sum then go mod download, pom.xml then mvn dependency:go-offline, then the source.

Running touch server.js package.json (new timestamps, same content) and rebuilding again hit the cache on every step. BuildKit’s file checksums ignore modification time, which is why a fresh git clone in CI can still reuse cached layers. So when you see unexpected cache misses, timestamps aren’t the cause. The usual real causes are a file you didn’t mean to include (a .git directory, local node_modules, a log file), which is what a .dockerignore file is for.

Two consequences people miss

Stale RUN steps. Because RUN is keyed on its text, this pair is a known trap:

RUN apt-get update
RUN apt-get install -y curl

Add a package to the second line months later and the first line is still cached with an old package index, so the install may fail with 404s for package versions that no longer exist. Put them in one RUN (apt-get update && apt-get install -y ...) so they’re always cached or rebuilt together. When you need fresh packages for a security fix, build with --no-cache or --pull; the base image maintenance note covers scheduling that.

Deleting doesn’t shrink. Layers only add. A file created in one layer and removed in a later one is hidden by a whiteout but still shipped:

FROM alpine:3.20
RUN dd if=/dev/urandom of=/big bs=1M count=50
RUN rm /big

$ docker history sz --format '{{.CreatedBy}}\t{{.Size}}'
RUN /bin/sh -c rm /big # buildkit	4.1kB
RUN /bin/sh -c dd if=/dev/urandom of=/big bs…	52.4MB

That image’s content is 56.5 MB. Doing the download and the delete in the same RUN gives 4.1 MB, essentially just Alpine. This is why you see rm -rf /var/lib/apt/lists/* at the end of the same RUN as apt-get install, and it’s also why secrets copied in and deleted later remain recoverable (see secrets in builds).

Cache beyond one machine

The layer cache lives with the builder. A CI runner that starts empty every time gets no cache at all, no matter how well the Dockerfile is ordered. BuildKit can export and import cache: --cache-to type=registry,ref=... and --cache-from the same ref, or type=gha on GitHub Actions. Cache mounts (RUN --mount=type=cache,target=/root/.npm) go further by keeping the package manager’s download cache between builds even when the layer itself must rebuild. Both are covered in the BuildKit cache note.

Interview practice

A Python service’s Dockerfile looks like this, and every build in CI takes eight minutes, even for README changes:

FROM python:3.12-slim
WORKDIR /srv
COPY . .
RUN apt-get update
RUN apt-get install -y build-essential libpq-dev
RUN pip install -r requirements.txt
CMD ["gunicorn", "app:app"]

List the problems and rewrite it.

Answer

First, COPY . . comes before every expensive step, so any change to any file in the context, including the README, invalidates apt-get and pip install. Second, apt-get update is a separate step, so when it is cached and the install line changes, the install runs against a stale index. Third, the apt lists and build tools stay in the image. Fourth, if CI runners are ephemeral, there’s no cache to hit anyway.

FROM python:3.12-slim
WORKDIR /srv
RUN apt-get update \
 && apt-get install -y --no-install-recommends build-essential libpq-dev \
 && rm -rf /var/lib/apt/lists/*
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["gunicorn", "app:app"]

System packages first (they change least), then the requirements file and install, then source. Add a .dockerignore excluding .git, virtualenvs and test data. In CI, add --cache-from/--cache-to with a registry cache so the runner can reuse layers. A strong answer also suggests a multi-stage build so build-essential isn’t in the final image at all.

Keep going

• Docker interview questions
• Docker MCQ practice
• Docker BuildKit and cache mounts
• Docker multi-stage builds
• Official reference: Build cache invalidation