mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-18 16:02:10 +00:00

Files

T

Ruben Fiszel 7590b28108 feat(sandbox): pull/extract images with crane instead of podman (#9455 )

* feat(sandbox): pull/extract images with crane instead of podman (+ add to image)

The sandboxed container runtime (`# sandbox <image>`) only ever pulls + flattens an
image (nsjail does the run), so a full container engine is overkill — and podman was
never actually in any Dockerfile, so the merged feature couldn't run in the shipped
image. Switch to crane (google/go-containerregistry): a single ~25MB static binary,
no daemon/store/root/privileged.

- docker_v2.rs: crane export -> flattened rootfs tar, crane config -> OCI config,
  crane digest -> content-addressed rootfs+config cache (cross-job dedup + automatic
  freshness), crane manifest -> pre-download size guard. DOCKER_CONFIG authfile dir.
  Cache eviction prunes the rootfs-tar cache by mtime (LRU). Pull policy honored via a
  ref->digest cache (missing/never reuse without a registry hit).
- Dockerfile + docker/DockerfileSlim{,Ee}: install the crane binary (Full/FullEe and
  the EE image inherit it via FROM the base image).
- docs + UI text + instance-setting descriptions updated (download size is compressed;
  cache is the rootfs-tar cache).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): address CI review — digest-pinned fetch, size cap on every job, eviction race

Codex P1s:
- Fetch by the resolved digest (name@digest), not the mutable tag, so content can't
  diverge from the digest the cache is keyed under if a tag moves mid-fetch.
- Enforce the size cap on EVERY job via a cached {digest}.size sidecar (no registry call
  on cache reuse), so lowering the limit rejects already-cached oversized images.
- Eviction race: hardlink the cache tar into the job dir before tar -xf (pins the inode
  against concurrent eviction) and re-fetch if it was evicted first.
Claude P2s: atomic config sidecar (tmp+rename) + tolerate torn parse; soften the LRU
comment (mtime = creation order); sweep orphaned *.tmp.* and .size on eviction.
+digest_key/ref_key unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P1 cross-fs cache staging (EXDEV), Dockerfile arch fail-fast

CI re-review (Claude + Codex P1): the eviction-race hardlink crosses filesystems in the
shipped deployments — the cache is its own volume (/tmp/windmill/cache) while the job dir
is on the container fs — so hard_link returns EXDEV (not NotFound) and every sandbox job
fails. Fall back to tokio::fs::copy on a non-NotFound link error; copy reads through the
source inode so it still survives a concurrent eviction.
Also: Dockerfiles fail fast with a clear error on an unsupported arch instead of building
a 404 crane URL; ref->digest file written via tmp+rename (no torn read under missing/never).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(sandbox): say 'oldest by creation time' not 'LRU' for cache eviction

Codex P2: the code evicts by tar creation time (cache hits don't touch mtime), so the
user-facing docs + instance-setting text shouldn't claim true LRU.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-05 13:52:55 +00:00

5.6 KiB

Raw Blame History

Sandboxed container runtime (daemonless docker)

Windmill bash scripts can run a container image. There are two runtimes:

	legacy `# docker`	sandboxed `# sandbox <image>`
selected by	bare `# docker`	`# sandbox <image>`
runtime	dind / Docker daemon (bollard, `dind` feature)	daemonless: extract rootfs + nsjail-run
boundary	separate (daemon outside the jail)	the job's own nsjail sandbox
nsjail	not provided (trusted-tenant)	required — this is the sandbox
safety	trusted-tenant	sandboxed (untrusted-capable)
compat	full `docker run`/`-d`/API	run-a-command subset

The three bash annotations are distinct and don't overload each other:

# docker → legacy daemon docker (unchanged).
# sandbox → run the bash script under nsjail.
# sandbox <image> → run that image's command under nsjail (this runtime).

Using it

Put the image ref on a # sandbox annotation line; the rest of the script runs inside that image:

# sandbox python:3.12-slim
name="$1"          # windmill args bind positionally, like any bash script
python3 -c "import sys; print('hello', sys.argv[1])" "$name"

The body runs via the image's /bin/sh -c (so the image needs a shell).
An empty body runs the image's ENTRYPOINT + CMD.
Windmill args (declared x="$1", …) are appended to the command.
The image's Env, WorkingDir are applied; the windmill reserved variables (WM_TOKEN, BASE_INTERNAL_URL, …) are injected so wmill/API calls work.

How it works

Pull/extract (crane, no daemon/store/root): crane export <image> streams the image's flattened root filesystem to a tar (layers + whiteouts applied, like docker export) and crane config reads its OCI config. The tar + config are cached content-addressed by digest (crane digest) so unchanged digests reuse the cache; tar -x materializes the per-job {job_dir}/rootfs. crane is a single ~25 MB static binary — we never run the image with it (nsjail does), so a full container engine like podman isn't needed.
Run (the job's nsjail sandbox): nsjail binds each top-level entry of the rootfs in place (binding the whole rootfs at / trips nsjail's read-only remount of its base root in a rootless userns), mounts the standard pseudo-filesystems (/proc from the jail's pid namespace, a tmpfs /tmp, /dev nodes), maps uid/gid 0 inside → the worker user outside, and runs the command. The container is the jail.

# sandbox <image>  ─▶  crane export → digest-keyed rootfs cache → tar -x → {job_dir}/rootfs  ─▶  nsjail (chroot rootfs)
                       crane config (OCI config) ───────────────────────────────────────────▶  Env / Cmd / WorkingDir

Because the run is just the job's own nsjail with the image's filesystem as root, the container inherits exactly the job's confinement:

Filesystem: only the rootfs + the job's mounts are visible — no host /, no other job dirs, no dep cache. There is nothing to bind-mount escape to.
/proc: the jail's own pid namespace — the worker and other jobs aren't visible.
uid: a single-uid jail — an escape lands as the unprivileged worker user.
network: the job's network (same as any bash job).

Image storage, freshness & limits

Where pulls live: a content-addressed cache of flattened rootfs tars (+ OCI config sidecars) keyed by image digest, under {ROOT_CACHE_DIR}/sandbox_rootfs (persistent, dedups pulls across jobs). The per-job extracted rootfs lives in {job_dir}/rootfs and is removed with the job.
Freshness (SANDBOX_IMAGE_PULL_POLICY, default newer): the cache is keyed by digest, so a moving tag whose digest changed re-pulls automatically. newer (default) / always re-resolve the digest each job (one cheap crane digest manifest fetch); missing reuses a cached digest for the ref without hitting the registry; never only uses the cache (errors if absent). Pinning a digest (img@sha256:…) is immutable and never stale.
Per-image size cap (SANDBOX_IMAGE_MAX_SIZE_MB, default 0 = off): images whose compressed download size (crane manifest) exceeds the cap are rejected before any layer is downloaded.
Cache size cap (SANDBOX_IMAGE_CACHE_MAX_MB, default 0 = off): best-effort eviction — after a run, the oldest cached rootfs tars (by creation time) are removed until the cache is back under the cap.

Requirements

crane and tar on the worker for image pull/extract (a single static binary — no daemon, root, or privileged).
nsjail on the worker — required. If nsjail is absent, a # sandbox <image> job errors clearly (use a bare # docker + a daemon instead).

Limitations (by design — daemonless, run-to-completion)

No docker run -d + later exec/attach/logs -f, no docker build, compose, swarm, healthchecks.
No arbitrary -v host bind mounts, --privileged, --cap-add, --device, host namespace sharing.
Images that drop to a non-root uid or chown to arbitrary uids inside need a subuid range in the jail (single-uid only today — follow-up: newuidmap range mapping).
The script result is a completion message; capture output via stdout/logs.

Follow-ups

Subuid-range nsjail variant for multi-uid images.
Per-container isolated networking (slirp/pasta).
Support under the non-nsjail unshare isolation mode.

5.6 KiB Raw Blame History