starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-08-18 16:02:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,432 Commits 725 Branches 1,509 Tags
fix/hook-tests-worktree-root
Commit Graph

3 Commits

Author SHA1 Message Date
Ruben Fiszel d5095515ed fix: return result.json and stdout results from sandboxed containers (#10460) 
* fix: return result.json and stdout results from sandboxed containers

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: capture unmasked stdout-only last line, validate container result.json

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: reject image WorkingDir that escapes the container root

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: verify the whole result mount destination against the extracted rootfs

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: name the right skip reason and gate the symlink test to unix

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-03 12:35:12 +02:00
Ruben Fiszel 7590b28108 feat(sandbox): pull/extract images with crane instead of podman (#9455) 
* feat(sandbox): pull/extract images with crane instead of podman (+ add to image)

The sandboxed container runtime (`# sandbox <image>`) only ever pulls + flattens an
image (nsjail does the run), so a full container engine is overkill — and podman was
never actually in any Dockerfile, so the merged feature couldn't run in the shipped
image. Switch to crane (google/go-containerregistry): a single ~25MB static binary,
no daemon/store/root/privileged.

- docker_v2.rs: crane export -> flattened rootfs tar, crane config -> OCI config,
  crane digest -> content-addressed rootfs+config cache (cross-job dedup + automatic
  freshness), crane manifest -> pre-download size guard. DOCKER_CONFIG authfile dir.
  Cache eviction prunes the rootfs-tar cache by mtime (LRU). Pull policy honored via a
  ref->digest cache (missing/never reuse without a registry hit).
- Dockerfile + docker/DockerfileSlim{,Ee}: install the crane binary (Full/FullEe and
  the EE image inherit it via FROM the base image).
- docs + UI text + instance-setting descriptions updated (download size is compressed;
  cache is the rootfs-tar cache).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): address CI review — digest-pinned fetch, size cap on every job, eviction race

Codex P1s:
- Fetch by the resolved digest (name@digest), not the mutable tag, so content can't
  diverge from the digest the cache is keyed under if a tag moves mid-fetch.
- Enforce the size cap on EVERY job via a cached {digest}.size sidecar (no registry call
  on cache reuse), so lowering the limit rejects already-cached oversized images.
- Eviction race: hardlink the cache tar into the job dir before tar -xf (pins the inode
  against concurrent eviction) and re-fetch if it was evicted first.
Claude P2s: atomic config sidecar (tmp+rename) + tolerate torn parse; soften the LRU
comment (mtime = creation order); sweep orphaned *.tmp.* and .size on eviction.
+digest_key/ref_key unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P1 cross-fs cache staging (EXDEV), Dockerfile arch fail-fast

CI re-review (Claude + Codex P1): the eviction-race hardlink crosses filesystems in the
shipped deployments — the cache is its own volume (/tmp/windmill/cache) while the job dir
is on the container fs — so hard_link returns EXDEV (not NotFound) and every sandbox job
fails. Fall back to tokio::fs::copy on a non-NotFound link error; copy reads through the
source inode so it still survives a concurrent eviction.
Also: Dockerfiles fail fast with a clear error on an unsupported arch instead of building
a 404 crane URL; ref->digest file written via tmp+rename (no torn read under missing/never).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(sandbox): say 'oldest by creation time' not 'LRU' for cache eviction

Codex P2: the code evicts by tar creation time (cache hits don't touch mtime), so the
user-facing docs + instance-setting text shouldn't claim true LRU.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-05 13:52:55 +00:00
Ruben Fiszel 1727271e19 feat: sandboxed daemonless container runtime via '# sandbox <image>' (#9453) 
* feat: add sandboxed docker v2 runtime via '# docker <image>'

Run a container image as a subprogram of the job's own nsjail sandbox:
extract the image rootfs with podman (rootless) and run it chrooted inside the
job's nsjail, so the container inherits the job's confinement and is safe under
nsjail / for untrusted code. Selected by '# docker <image>'; a bare '# docker'
keeps the v1 (dind) path untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: default to daemonless docker (drop dind from compose, allow docker on cloud)

docker-compose no longer ships the dind sidecar (v2 is daemonless: podman + nsjail
in the worker); removed the dind service, DOCKER_HOST env, depends_on and volume.
Removed the language-picker guard that blocked Docker scripts on the multi-tenant
platform, now that v2 makes docker safe to run sandboxed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: select sandboxed container via # sandbox <image>; add pull policy + size guards

- Surface moved from '# docker <image>' to '# sandbox <image>' (groups under the
  sandbox annotation; '# docker' stays v1-only, '# sandbox' stays nsjail-bash).
- SANDBOX_IMAGE_PULL_POLICY (default 'newer') so moving tags don't go stale.
- SANDBOX_IMAGE_MAX_SIZE_MB rejects oversized images before extraction.
- SANDBOX_IMAGE_CACHE_MAX_MB best-effort LRU eviction of podman's image store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): support # volume, honor nsjail tmp instance settings, v2 docker template

- Thread shared_mount into the sandbox container nsjail config so '# volume' mounts
  (and the same-worker /tmp/shared folder) apply inside the container.
- Use resolve_nsjail_tmp_mount_block for the container's /tmp so it honors the same
  nsjail_tmp_backing / nsjail_tmpfs_size_mb instance settings as other nsjail jobs.
- docker-compose comment + the editor's Docker template now use '# sandbox <image>'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): make image size/cache/pull-policy UI instance settings

Convert SANDBOX_IMAGE_* from worker env vars to DB-backed instance settings
(sandbox_image_max_size_mb, sandbox_image_cache_max_mb, sandbox_image_pull_policy),
hot-reloaded via the same mechanism as nsjail_tmpfs_size_mb and configurable in
#superadmin-settings. No worker restart needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): windmill-managed registry — default registry + private auth

Two new instance settings:
- sandbox_image_default_registry: prepended to unqualified image refs (alpine ->
  <registry>/alpine); fully-qualified refs untouched.
- sandbox_registry_auth: docker/podman auth.json blob written to a per-job authfile
  (0600, removed with the job) and passed to podman --authfile for private registries.
Both hot-reloaded and configurable in #superadmin-settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): protobuf-safe proto_str escaper, atomic 0600 authfile, registry tests

Addresses local-review P2s: proto_str now emits valid protobuf octal escapes for
control/non-ASCII bytes (not Rust \u{..} that nsjail would reject); the registry
authfile is created 0600 atomically (no world-readable window); add a
registry_qualified table test + a non-ASCII proto_str case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 — deliver image env via nsjail envar:, never the launcher process env

CI review (P0): the image's OCI Env (attacker-controlled keys+values) was applied to
the nsjail launcher process via .envs(), so a hostile image could set LD_PRELOAD/
LD_LIBRARY_PATH/LD_AUDIT on nsjail itself and execute code as the worker outside the
jail. Now the image env is rendered as proto-escaped 'envar:' directives (child-only)
and nsjail's process env carries only windmill-trusted keys (reserved vars + proxy).
Also: warn instead of silently bypassing the size guard on inspect failure; reset the
eviction guard via a Drop guard (no stuck flag on panic/early-return). +render_envars test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 symlink-write escape via rootfs script; P1 redact registry-auth logging

CI review:
- P0 (Codex): the body was written into the image-controlled rootfs as
  .windmill_docker_main.sh via write_file (follows symlinks) — a hostile image could
  plant that path as a symlink to a host file and capture the worker's write before
  nsjail starts. Now the body is passed straight to 'sh -c <body> sh <args>'; no file
  is written into the rootfs at all.
- P1 (Codex): sandbox_registry_auth flowed through the generic setting loader which
  logs the value (raw auth.json credentials). Replaced with a secret-aware reload that
  loads directly and logs only a redacted 'configured=' message.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): redact sandbox_registry_auth in instance-settings write log too

The settings API also logs 'Set global setting <key> to <value>' via format_setting_value;
add sandbox_registry_auth to SENSITIVE_SETTINGS so the credential is redacted there as
well as on reload.

* fix(sandbox): don't silently disable cache eviction on podman images parse error

Re-review (cubic/Claude P2): serde_json::from_slice(...).unwrap_or_default() meant any
parse hiccup (e.g. podman omitting Size/Created via omitempty for a zero value, or
schema drift) silently degraded to an empty Vec and disabled eviction with no log.
Now Size/Created are #[serde(default)] (a missing omitempty key -> 0, not a whole-array
parse failure) and a real parse error warns + breaks instead of being swallowed.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-05 08:35:51 +00:00