windmill/docs/docker-security.md

Docker base-OS security patching

The runtime images are built on debian:trixie-slim (Debian stable). The base runtime stages run apt-get update && apt-get upgrade -y && apt-get install … so that base-OS packages pick up Debian security and point-release fixes at build time, instead of staying frozen at whatever versions the base tag shipped.

Where the upgrade lives

apt-get upgrade -y is applied in the first apt block of the three stages that establish a runtime Debian layer:

• Dockerfile (the primary windmill / windmill-ee image)
• docker/DockerfileSlim (windmill-slim)
• docker/DockerfileSlimEe (windmill-ee-slim)

Every other runtime image inherits its base OS from one of these transitively, so patching here is sufficient:

• DockerfileFull, DockerfileFullEe, DockerfileCuda build FROM the primary windmill / windmill-ee image.
• DockerfileExtra builds FROM windmill-ee-slim.

The nsjail builder stages are throwaway (only the compiled nsjail binary is copied out), so they are intentionally not upgraded. The CLI, Caddy-L4, CUDA-only, and RHEL/dnf images are out of scope for this apt-based patching.

Why apt-get upgrade and not unattended-upgrades / pinning

Debian stable's archive only receives security updates and ABI-stable point releases (e.g. openssl 3.0.x → 3.0.x+deb12u2, same soname). It does not ship feature/major bumps, so a build-time apt-get upgrade cannot silently break a pinned runtime dependency the way it might on a rolling distro. The default debian.sources already includes the *-security suite, so a plain upgrade picks up security fixes without extra machinery. unattended-upgrades adds a package and config for no benefit in a build context (it does not run at build time), and a pinned base digest would freeze the CVEs in place.

Note the runtime apt installs were already unpinned (only the throwaway nsjail builder pins versions), so these images were never byte-for-byte reproducible in this dimension; upgrade moves the same already-floating packages to their patched versions rather than changing the reproducibility posture.

Caching and freshness

apt-get upgrade sits in the same RUN as apt-get update && install. Docker keys that layer on the command string plus the parent layer, not on package contents, so an unchanged build is a cache hit and the upgrade does not re-run. The layer is invalidated — and fresh patches are pulled — when the parent layer changes, primarily when the mutable debian:trixie-slim base digest moves on a Debian point release. That self-aligns: the cache refreshes when there is something new to pick up.

Because of apt-cache staleness, a security fix that lands between base-digest bumps will not be picked up by a cached build until the next bump. To close that gap, rebuild and republish the latest / patch tags:

• on each Debian point release (base digest bump), and
• on a periodic cadence (e.g. per Windmill release), rebuilding the base stages with --no-cache if you need to force a fresh apt-get upgrade regardless of the base digest.

Scan the published images (e.g. Trivy / Defender) after rebuilds to confirm the base-OS finding count stays low.

Verifying image signatures, SBOMs and provenance

Release images are signed and attested at publish time:

• cosign keyless signature on the pushed manifest digest (index and per-arch manifests), via GitHub OIDC — no long-lived signing key exists (.github/actions/sign-attest-image).
• SBOMs are generated at build time (sbom: true on the depot build step) and embedded in the image index as BuildKit attestation manifests — one SPDX document per platform. They are part of the signed index digest, so the cosign signature covers them. They are not sent to a transparency log: SPDX documents for these images run tens of MB, beyond what Rekor or GitHub attestations accept as payloads.
• SLSA build provenance recorded as a GitHub artifact attestation and pushed to the registry (actions/attest-build-provenance).

What is covered

Only images published from a release tag (v*) are signed: windmill, windmill-ee, windmill-ee-cuda, windmill-slim, windmill-ee-slim, windmill-full, windmill-ee-full (.github/workflows/docker-image.yml), windmill-cli (build_cli_image.yml) and windmill-extra (publish_extra.yml). The :latest and :main tags are repointed on every main push as well as on releases, so they resolve to a signed digest only until the next main build lands — verify a version tag or a digest, not :latest. Development images (:dev, branch builds, windmill-test), the dispatch-only RHEL/rpi images and the caddy-l4 image are not signed.

How to verify

Signatures are keyless: trust is anchored in the Fulcio certificate identity, which for these images is the calling workflow file at a v* tag ref in this repository. Verify a signature with cosign (v2.x):

cosign verify \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp '^https://github.com/windmill-labs/windmill/\.github/workflows/(docker-image|publish_extra|build_cli_image)\.yml@refs/tags/v' \
  ghcr.io/windmill-labs/windmill:<version>

Extract the embedded SBOM (per platform; verify the signature first — it covers the index these documents live in):

docker buildx imagetools inspect ghcr.io/windmill-labs/windmill:<version> \
  --format '{{ json .SBOM }}'

Verify SLSA provenance through GitHub's attestation API:

gh attestation verify oci://ghcr.io/windmill-labs/windmill:<version> \
  -R windmill-labs/windmill

Note for registry housekeeping: cosign stores signatures as extra sha256-<digest>.sig tags in the same ghcr package, and the pushed provenance attestations live there as referrer artifacts — any tag-retention automation must not prune them.