feat: sandboxed daemonless container runtime via '# sandbox <image>' (#9453)

* feat: add sandboxed docker v2 runtime via '# docker <image>'

Run a container image as a subprogram of the job's own nsjail sandbox:
extract the image rootfs with podman (rootless) and run it chrooted inside the
job's nsjail, so the container inherits the job's confinement and is safe under
nsjail / for untrusted code. Selected by '# docker <image>'; a bare '# docker'
keeps the v1 (dind) path untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: default to daemonless docker (drop dind from compose, allow docker on cloud)

docker-compose no longer ships the dind sidecar (v2 is daemonless: podman + nsjail
in the worker); removed the dind service, DOCKER_HOST env, depends_on and volume.
Removed the language-picker guard that blocked Docker scripts on the multi-tenant
platform, now that v2 makes docker safe to run sandboxed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: select sandboxed container via # sandbox <image>; add pull policy + size guards

- Surface moved from '# docker <image>' to '# sandbox <image>' (groups under the
  sandbox annotation; '# docker' stays v1-only, '# sandbox' stays nsjail-bash).
- SANDBOX_IMAGE_PULL_POLICY (default 'newer') so moving tags don't go stale.
- SANDBOX_IMAGE_MAX_SIZE_MB rejects oversized images before extraction.
- SANDBOX_IMAGE_CACHE_MAX_MB best-effort LRU eviction of podman's image store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): support # volume, honor nsjail tmp instance settings, v2 docker template

- Thread shared_mount into the sandbox container nsjail config so '# volume' mounts
  (and the same-worker /tmp/shared folder) apply inside the container.
- Use resolve_nsjail_tmp_mount_block for the container's /tmp so it honors the same
  nsjail_tmp_backing / nsjail_tmpfs_size_mb instance settings as other nsjail jobs.
- docker-compose comment + the editor's Docker template now use '# sandbox <image>'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): make image size/cache/pull-policy UI instance settings

Convert SANDBOX_IMAGE_* from worker env vars to DB-backed instance settings
(sandbox_image_max_size_mb, sandbox_image_cache_max_mb, sandbox_image_pull_policy),
hot-reloaded via the same mechanism as nsjail_tmpfs_size_mb and configurable in
#superadmin-settings. No worker restart needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): windmill-managed registry — default registry + private auth

Two new instance settings:
- sandbox_image_default_registry: prepended to unqualified image refs (alpine ->
  <registry>/alpine); fully-qualified refs untouched.
- sandbox_registry_auth: docker/podman auth.json blob written to a per-job authfile
  (0600, removed with the job) and passed to podman --authfile for private registries.
Both hot-reloaded and configurable in #superadmin-settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): protobuf-safe proto_str escaper, atomic 0600 authfile, registry tests

Addresses local-review P2s: proto_str now emits valid protobuf octal escapes for
control/non-ASCII bytes (not Rust \u{..} that nsjail would reject); the registry
authfile is created 0600 atomically (no world-readable window); add a
registry_qualified table test + a non-ASCII proto_str case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 — deliver image env via nsjail envar:, never the launcher process env

CI review (P0): the image's OCI Env (attacker-controlled keys+values) was applied to
the nsjail launcher process via .envs(), so a hostile image could set LD_PRELOAD/
LD_LIBRARY_PATH/LD_AUDIT on nsjail itself and execute code as the worker outside the
jail. Now the image env is rendered as proto-escaped 'envar:' directives (child-only)
and nsjail's process env carries only windmill-trusted keys (reserved vars + proxy).
Also: warn instead of silently bypassing the size guard on inspect failure; reset the
eviction guard via a Drop guard (no stuck flag on panic/early-return). +render_envars test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 symlink-write escape via rootfs script; P1 redact registry-auth logging

CI review:
- P0 (Codex): the body was written into the image-controlled rootfs as
  .windmill_docker_main.sh via write_file (follows symlinks) — a hostile image could
  plant that path as a symlink to a host file and capture the worker's write before
  nsjail starts. Now the body is passed straight to 'sh -c <body> sh <args>'; no file
  is written into the rootfs at all.
- P1 (Codex): sandbox_registry_auth flowed through the generic setting loader which
  logs the value (raw auth.json credentials). Replaced with a secret-aware reload that
  loads directly and logs only a redacted 'configured=' message.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): redact sandbox_registry_auth in instance-settings write log too

The settings API also logs 'Set global setting <key> to <value>' via format_setting_value;
add sandbox_registry_auth to SENSITIVE_SETTINGS so the credential is redacted there as
well as on reload.

* fix(sandbox): don't silently disable cache eviction on podman images parse error

Re-review (cubic/Claude P2): serde_json::from_slice(...).unwrap_or_default() meant any
parse hiccup (e.g. podman omitting Size/Created via omitempty for a zero value, or
schema drift) silently degraded to an empty Vec and disabled eviction with no log.
Now Size/Created are #[serde(default)] (a missing omitempty key -> 0, not a whole-array
parse failure) and a real parse error warns + breaks instead of being swallowed.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/docker-v2-runtime.md

@@ -0,0 +1,106 @@
+# Sandboxed container runtime (daemonless docker)
+
+Windmill bash scripts can run a container image. There are **two** runtimes:
+
+| | legacy `# docker` | sandboxed `# sandbox <image>` |
+|---|---|---|
+| selected by | bare `# docker` | `# sandbox <image>` |
+| runtime | dind / Docker daemon (bollard, `dind` feature) | daemonless: extract rootfs + nsjail-run |
+| boundary | separate (daemon outside the jail) | the job's own nsjail sandbox |
+| nsjail | not provided (trusted-tenant) | **required** — this *is* the sandbox |
+| safety | trusted-tenant | sandboxed (untrusted-capable) |
+| compat | full `docker run`/`-d`/API | run-a-command subset |
+
+The three bash annotations are distinct and don't overload each other:
+
+- `# docker` → legacy daemon docker (unchanged).
+- `# sandbox` → run the bash script under nsjail.
+- `# sandbox <image>` → run that image's command under nsjail (this runtime).
+
+## Using it
+
+Put the image ref on a `# sandbox` annotation line; the rest of the script runs
+**inside** that image:
+
+```bash
+# sandbox python:3.12-slim
+name="$1"          # windmill args bind positionally, like any bash script
+python3 -c "import sys; print('hello', sys.argv[1])" "$name"
+```
+
+- The body runs via the image's `/bin/sh -c` (so the image needs a shell).
+- An **empty** body runs the image's `ENTRYPOINT` + `CMD`.
+- Windmill args (declared `x="$1"`, …) are appended to the command.
+- The image's `Env`, `WorkingDir` are applied; the windmill reserved variables
+  (`WM_TOKEN`, `BASE_INTERNAL_URL`, …) are injected so `wmill`/API calls work.
+
+## How it works
+
+1. **Pull/extract** (podman, rootless): `podman create --pull=<policy> <image>` +
+   `podman export | tar -x` materializes the image's flattened root filesystem
+   into `{job_dir}/rootfs`, and `podman inspect` reads its OCI config. podman's
+   image store dedups pulls across jobs.
+2. **Run** (the job's nsjail sandbox): nsjail binds each top-level entry of the
+   rootfs in place (binding the whole rootfs at `/` trips nsjail's read-only
+   remount of its base root in a rootless userns), mounts the standard
+   pseudo-filesystems (`/proc` from the jail's pid namespace, a tmpfs `/tmp`,
+   `/dev` nodes), maps uid/gid 0 inside → the worker user outside, and runs the
+   command. The container *is* the jail.
+
+```
+# sandbox <image>  ─▶  podman create+export ─▶  {job_dir}/rootfs  ─▶  nsjail (chroot rootfs)
+                       podman inspect (OCI config) ──────────────────▶  Env / Cmd / WorkingDir
+```
+
+Because the run is just the job's own nsjail with the image's filesystem as root,
+the container inherits exactly the job's confinement:
+
+- **Filesystem**: only the rootfs + the job's mounts are visible — no host `/`,
+  no other job dirs, no dep cache. There is nothing to bind-mount escape to.
+- **/proc**: the jail's own pid namespace — the worker and other jobs aren't
+  visible.
+- **uid**: a single-uid jail — an escape lands as the unprivileged worker user.
+- **network**: the job's network (same as any bash job).
+
+## Image storage, freshness & limits
+
+- **Where pulls live:** podman's rootless graph root (default
+  `$HOME/.local/share/containers/storage`) — persistent, dedups pulls across jobs.
+  The per-job extracted rootfs lives in `{job_dir}/rootfs` and is removed with the
+  job; the transient `rootfs.tar` is removed right after extraction.
+- **Freshness (`SANDBOX_IMAGE_PULL_POLICY`, default `newer`):** `newer` re-pulls
+  only when the registry digest changed (one cheap manifest check per job, no data
+  transfer if unchanged) — so moving tags like `:latest` don't go stale. `missing`
+  is fastest but tags can go stale; `always` re-checks every job. Pinning a digest
+  (`img@sha256:…`) is immutable and never stale.
+- **Per-image size cap (`SANDBOX_IMAGE_MAX_SIZE_MB`, default 0 = off):** images
+  whose on-disk size exceeds the cap are rejected before extraction.
+- **Cache size cap (`SANDBOX_IMAGE_CACHE_MAX_MB`, default 0 = off):** best-effort
+  LRU eviction — after a run, the oldest images are removed until podman's image
+  store is back under the cap. In-use images are never removed.
+
+## Requirements
+
+- `podman` (rootless) and `tar` on the worker for image pull/extract.
+- `nsjail` on the worker — **required**. If nsjail is absent, a `# sandbox <image>`
+  job errors clearly (use a bare `# docker` + a daemon instead).
+
+## Limitations (by design — daemonless, run-to-completion)
+
+- No `docker run -d` + later `exec`/`attach`/`logs -f`, no `docker build`,
+  `compose`, swarm, healthchecks.
+- No arbitrary `-v` host bind mounts, `--privileged`, `--cap-add`, `--device`,
+  host namespace sharing.
+- Images that drop to a non-root uid or chown to arbitrary uids inside need a
+  subuid **range** in the jail (single-uid only today — follow-up: `newuidmap`
+  range mapping).
+- The script result is a completion message; capture output via stdout/logs.
+
+## Follow-ups
+
+- Content-addressed rootfs cache keyed by image digest (today each job re-exports;
+  podman's image store still dedups the network pull).
+- Pre-pull size guard via `skopeo` manifest inspection (reject before download).
+- Subuid-range nsjail variant for multi-uid images.
+- Per-container isolated networking (slirp/pasta).
+- Support under the non-nsjail `unshare` isolation mode.