Docker Secrets in Image Layers: Why Deleting Them Doesn't Work

It is one of the most common container security mistakes we see in code reviews: a Dockerfile copies an .env file or an API key into the image, uses it during the build, and then runs rm on it a few lines later. The final container looks clean — ls shows nothing — so the secret must be gone, right?

It isn't. Docker images are made of stacked, immutable layers, and deleting a file in a later layer only hides it. Anyone who can pull the image from your registry, receives a docker save tarball, or has access to the host can open the layers and read the secret in seconds.

In this post we will reproduce the problem step by step in a terminal, recover a "deleted" secret, and then go through the correct ways to handle secrets in Docker builds and at runtime.

All output below is real, captured on a Debian box with Docker 27.3.1. The secret used is obviously fake: FAKE_API_KEY=sk-demo-123.

How Docker image layers work (and why rm doesn't erase anything)

Every COPY, ADD and RUN instruction in a Dockerfile produces a new filesystem layer: a tar archive containing only the changes from that step. ENV, ARG, CMD and friends produce no files but are recorded in the image config and history (JSON metadata shipped with the image).

When you delete a file in a later step, Docker doesn't modify the earlier layer — layers are content-addressed and immutable. Instead, the new layer contains a whiteout file (.wh.<filename>) that tells the union filesystem to hide the original at runtime. The original bytes are still there, in the earlier layer, inside every copy of the image.

So there are two leak paths:

1. Metadata leaks — ARG and ENV values end up in the image config and docker history.
2. Layer leaks — files added by COPY/ADD/RUN stay in their layer forever, even if removed later.

Step-by-step: leaking a secret into a Docker image

Step 1: Build a "leaky" image

Create a working directory with a fake secret file:

mkdir leaky-demo && cd leaky-demo
echo 'FAKE_API_KEY=sk-demo-123' > secret.env

And this (deliberately bad) Dockerfile, which makes all three classic mistakes — a secret in ARG, a secret in ENV, and a copied secret file that is "cleaned up" afterwards:

FROM alpine:3.20
ARG API_TOKEN
ENV FAKE_API_KEY=sk-demo-123
COPY secret.env /app/secret.env
RUN cat /app/secret.env > /dev/null && rm /app/secret.env
CMD ["sh","-c","ls -la /app; echo done"]

Build it, passing a token as a build argument:

docker build --build-arg API_TOKEN=sk-demo-arg-456 -t leaky-demo:1 .

#6 [2/3] COPY secret.env /app/secret.env
#6 DONE 0.0s
#7 [3/3] RUN cat /app/secret.env > /dev/null && rm /app/secret.env
#7 DONE 0.1s
#8 naming to docker.io/library/leaky-demo:1 done

 2 warnings found (use docker --debug to expand):
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "FAKE_API_KEY") (line 3)
 - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "API_TOKEN") (line 2)

Notice that recent BuildKit versions already warn you about ARG/ENV secrets (SecretsUsedInArgOrEnv). It does not warn about the copied file.

Step 2: The container looks clean

docker run --rm leaky-demo:1

total 8
drwxr-xr-x    1 root     root          4096 Nov  4 21:14 .
drwxr-xr-x    1 root     root          4096 Nov  4 21:15 ..
done

/app is empty. This is exactly the false sense of security that makes the mistake so common.

Step-by-step: recovering the secret from the image

From here on, imagine you are an attacker — or just a colleague — with read access to the image. That's anyone who can docker pull from your registry, or anyone you've handed a tarball to.

Step 3: docker history --no-trunc exposes ARG and ENV values

docker history --no-trunc --format '{{.CreatedBy}} | {{.Size}}' leaky-demo:1

CMD ["sh" "-c" "ls -la /app; echo done"] | 0B
RUN |1 API_TOKEN=sk-demo-arg-456 /bin/sh -c cat /app/secret.env > /dev/null && rm /app/secret.env # buildkit | 0B
COPY secret.env /app/secret.env # buildkit | 25B
ENV FAKE_API_KEY=sk-demo-123 | 0B
ARG API_TOKEN=sk-demo-arg-456 | 0B
CMD ["/bin/sh"] | 0B

Both values are in plain text. The build argument even shows up twice: in the ARG line and as the |1 API_TOKEN=... prefix of every RUN that follows it. And the 25-byte COPY layer is a big hint that something interesting lives there.

The ENV value is also baked into the runtime config of every container:

docker inspect --format '{{json .Config.Env}}' leaky-demo:1

["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","FAKE_API_KEY=sk-demo-123"]

Step 4: Export the image with docker save

docker save leaky-demo:1 -o leaky.tar
mkdir x && tar -xf leaky.tar -C x
ls x

blobs  index.json  manifest.json  oci-layout  repositories

manifest.json lists the layers in order (base image first):

cat x/manifest.json | python3 -m json.tool

"Layers": [
  "blobs/sha256/08bc4e534116aa76b16015484b82eac51f9a593416feae9296c8a2d4bb7aa4a2",
  "blobs/sha256/54825ac4899071c804581de60fa111204f4ea9b6bd95b6eb4d88fdf5f61ca047",
  "blobs/sha256/0fa53e7390794b066999f1f36e108e888398dfc3c6c8dd9c543891115ce28342"
]

Step 5: Find the layer that added the secret

Each layer is just a tar file. List the ones that mention our file:

for l in x/blobs/sha256/*; do
  tar -tf "$l" 2>/dev/null | grep -q secret && echo "$l: $(tar -tf "$l" | grep secret)"
done

x/blobs/sha256/0fa53e73...28342: app/.wh.secret.env
x/blobs/sha256/54825ac4...ca047: app/secret.env

There it is: layer 54825ac4… (the COPY step) contains the real file, and the later layer 0fa53e73… (the RUN rm step) contains only .wh.secret.env — the whiteout marker that hides it.

Step 6: Read the "deleted" secret

tar -xOf x/blobs/sha256/54825ac4899071c804581de60fa111204f4ea9b6bd95b6eb4d88fdf5f61ca047 app/secret.env

FAKE_API_KEY=sk-demo-123

That's it. No exploits, no root on your server — just docker save and tar. The same works on any image pulled from a registry, because the registry stores exactly these layer blobs.

The image config blob tells the same story without even unpacking a layer:

python3 -c "import json;c=json.load(open('x/blobs/sha256/<config-digest>'));[print(h['created_by']) for h in c['history']]"

ARG API_TOKEN=sk-demo-arg-456
ENV FAKE_API_KEY=sk-demo-123
RUN |1 API_TOKEN=sk-demo-arg-456 /bin/sh -c cat /app/secret.env > /dev/null && rm /app/secret.env # buildkit

Optional: do it visually with dive

dive is a popular TUI for exploring image layers. Run dive leaky-demo:1, select the COPY secret.env layer on the left, and /app/secret.env appears in the file tree on the right — highlighted as added in that layer and removed in the next one. It's a great way to audit your own images, and a reminder of how easy this is for anyone else.

The right way to use secrets in Docker builds

Use BuildKit secret mounts (RUN --mount=type=secret)

BuildKit (the default builder since Docker 23) can mount a secret into a single RUN step as a tmpfs file or environment variable. It's never written to a layer, never stored in the image config, and never shows up in history.

# syntax=docker/dockerfile:1
FROM alpine:3.20
# As an environment variable, only for this RUN:
RUN --mount=type=secret,id=api_key,env=FAKE_API_KEY \
    test -n "$FAKE_API_KEY" && echo "secret available during this RUN only"
# Or as a file at /run/secrets/<id>:
RUN --mount=type=secret,id=api_key \
    wc -c /run/secrets/api_key
CMD ["sh","-c","env | grep FAKE || echo 'no secret in runtime env'"]

printf 'sk-demo-123' > api_key.txt
docker build --secret id=api_key,src=api_key.txt -t safe-demo:1 .

#8 [stage-0 2/3] RUN --mount=type=secret,id=api_key,env=FAKE_API_KEY     test -n "****" && echo "secret available during this RUN only"
#8 0.041 secret available during this RUN only
#9 [stage-0 3/3] RUN --mount=type=secret,id=api_key     wc -c /run/secrets/api_key
#9 0.046 11 /run/secrets/api_key

BuildKit even masks the value (****) in the build log. Now repeat the attack:

docker history --no-trunc --format '{{.CreatedBy}}' safe-demo:1 | head -3
docker run --rm safe-demo:1
docker save safe-demo:1 | tar -xO --wildcards 'blobs/*' | grep -a -c 'sk-demo-123'

CMD ["sh" "-c" "env | grep FAKE || echo 'no secret in runtime env'"]
RUN /bin/sh -c wc -c /run/secrets/api_key # buildkit
RUN /bin/sh -c test -n "$FAKE_API_KEY" && echo "secret available during this RUN only" # buildkit
no secret in runtime env
0

Zero occurrences of the secret anywhere in the exported image. You can pass secrets from an env var too (--secret id=api_key,env=API_KEY), and Docker Compose supports them under build.secrets. For private Git dependencies, use RUN --mount=type=ssh with docker build --ssh default instead of copying SSH keys.

Caveat: a secret mount only protects the secret itself. If your RUN writes it somewhere (for example npm config set into ~/.npmrc, or a generated config file), that file is committed to the layer. Make sure the command consumes the secret without persisting it.

Multi-stage builds (useful, but read the caveat)

Multi-stage builds let you do the dirty work in a builder stage and copy only the final artifact into a clean runtime image:

FROM node:22 AS builder
WORKDIR /src
COPY . .
RUN --mount=type=secret,id=npm_token,env=NPM_TOKEN npm ci && npm run build

FROM node:22-slim
COPY --from=builder /src/dist /app
CMD ["node", "/app/server.js"]

The final image only contains layers from the last stage, so a secret that touched the builder doesn't ship. But: the builder stage still exists in your local build cache, can be pushed if someone builds with --target builder, and can leak via exported cache (--cache-to type=registry,mode=max) or CI caches. Multi-stage is a size and hygiene tool, not a secrets tool — combine it with secret mounts rather than relying on it alone.

Inject runtime secrets when the container starts

Most secrets (database passwords, API keys) are not needed at build time at all. Give them to the container at runtime:

• Env files: docker run --env-file ./prod.env myapp — simple, but values are visible via docker inspect on the host and inherited by child processes.
• Docker Compose secrets — mounted as files under /run/secrets/, never baked into the image:

services:
  web:
    image: myapp:1
    secrets: [db_password]
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_password
secrets:
  db_password:
    file: ./secrets/db_password

• Docker Swarm secrets (docker secret create) or, on Kubernetes, Secrets backed by an external manager (Vault, AWS/GCP Secret Manager via External Secrets Operator).

Many official images (Postgres, MariaDB…) support the *_FILE convention out of the box — this blog's own Postgres runs with POSTGRES_PASSWORD_FILE.

Keep secrets out of the build context with .dockerignore

Even without a COPY secret.env, a broad COPY . . will happily pull .env, keys and credentials into the image. Add a .dockerignore:

.env
.env.*
*.pem
*.key
secrets/
.git
.aws/
.ssh/

Scan images and repositories for leaked secrets

Make detection automatic, both in Git and in built images:

# Scan an image (layers included) for secrets and misconfigurations
trivy image --scanners secret,misconfig leaky-demo:1

# Scan the repo / Dockerfiles before they're ever built
gitleaks detect --source .

Run them in CI and fail the pipeline on findings. Also enable BuildKit's build checks (docker build --check .), which flag SecretsUsedInArgOrEnv, as we saw above.

Already leaked a secret? Rotate it

If a secret ever landed in an image layer that left your laptop, treat it as compromised. Rebuilding the image without it, deleting the tag, or even purging the registry is not enough — the image may be cached on CI runners, nodes, mirrors, or someone's machine.

1. Revoke and rotate the credential at the provider immediately.
2. Check access logs for usage of the old credential.
3. Fix the Dockerfile (secret mounts / runtime secrets), rebuild and redeploy.
4. Delete the old tags and digests from the registry and clean CI caches.
5. Add scanning so it doesn't happen again.

Key takeaways

• Docker image layers are immutable: rm in a later step only adds a whiteout, the file is still in the earlier layer.
• ARG and ENV values are stored in image metadata and visible with docker history --no-trunc and docker inspect.
• docker save + tar (or dive) is all it takes to recover a "deleted" secret.
• Use RUN --mount=type=secret for build-time secrets and runtime secrets (Compose/Swarm/Kubernetes) for everything else.
• Add .dockerignore, scan with trivy and gitleaks, and rotate anything that leaked.

FAQ

Does docker history show secrets?

It shows every ARG and ENV value and the full command of each RUN, so any secret passed through those instructions is visible with docker history --no-trunc. It doesn't show file contents, but docker save exposes those.

If I delete a file in the same RUN step, is it safe?

If the file is created and deleted within one RUN instruction, it never gets committed to a layer. But if it was added by COPY/ADD or an earlier RUN, it's already in a previous layer. A secret mount is still the cleaner solution, because the command line and build args can leak too.

Does docker build --squash fix the problem?

Squashing merges layers, so a deleted file disappears from the final image, but it's an experimental feature, breaks layer caching, and doesn't remove ARG/ENV values from metadata. Use secret mounts instead.

Are multi-stage builds enough to keep secrets out?

The final image won't include builder-stage layers, but the builder stage remains in build caches and can be pushed or exported. Combine multi-stage builds with RUN --mount=type=secret.

Can someone read my secrets from a running container?

Runtime secrets are meant to be readable by the container. Anyone with access to the Docker host (docker inspect, docker exec) can see env variables. Prefer file-based secrets under /run/secrets and restrict who can access the Docker socket.

How do I check whether my existing images contain secrets?

Run trivy image --scanners secret <image>, browse layers with dive, and check docker history --no-trunc for suspicious ARG/ENV values.

---

Related reading on CloudArch: Docker HEALTHCHECK guide, Monitoring Docker with Prometheus, and Leaving WordPress: self-hosted efficiency and security.