Remove the `docker_image_storage_size_mb` instance setting and the polling
storage monitor entirely. Now that `# docker` jobs are refused under nsjail
(the per-job podman daemon runs outside the sandbox), the asymmetry that
justified a docker-specific disk cap is gone: a normal job can already exhaust
host disk in non-nsjail modes, so a docker-only cap was inconsistent. Docker
jobs now use disk like any other job — bound it at the infra level.
Also scope the rootless slirp4netns network-backend override to the per-job
podman instance via a job-scoped `$HOME` containers.conf instead of a global
`/etc/containers` drop-in, so rootful podman elsewhere in the *-full image is
unaffected (flagged by cubic).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-job-podman bind-mount escape can read world-readable host state — including
world-readable /proc (process cmdlines/args via `-v /proc`), other job dirs'
world-readable files, and the dep cache — but NOT 0400 files like /proc/<pid>/environ
(env secrets stay protected on a root worker). Document this in the security notes
(README + compose + code comment) with the corollary: don't pass secrets as
command-line args (windmill uses env). Verified: uid-1000 container via `-v /proc`
reading a root process's environ -> 'Permission denied'; its cmdline -> readable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enabling nsjail is an explicit 'fully sandbox jobs' signal, but the per-job podman
daemon runs OUTSIDE the jail and a # docker script can bind-mount worker-visible
paths through it — silently providing it under nsjail would be a surprising hole in
a setup the operator hardened on purpose. So refuse instead: under nsjail, a
# docker job with no explicitly-provided Docker daemon errors with guidance to use
DOCKER_HOST (a network-reachable daemon; a mounted unix socket isn't reachable in
the jail) or a non-nsjail worker group. nsjail is off by default (DISABLE_NSJAIL
defaults true → default workers use unshare/none), so docker-via-podman keeps
working out of the box; this only gates the explicit-nsjail case. Docs updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the docker worker runs as root (the default for compose/helm workers), start
the per-job podman via `runuser` dropped to a non-root uid so it runs rootless.
This keeps the worker's secrets in /proc/<worker>/environ (DATABASE_URL etc.)
root-owned and therefore unreadable by a `docker run --pid=host` container, and
makes a container escape land unprivileged — without the fragile namespace
confinement (no unshare/mask/pid-ns, which conflicted with podman system service).
A non-root worker can't drop further (podman runs rootless as itself; a --pid=host
container shares its uid), so run the docker worker as root for the /proc protection.
The per-job podman + its containers are spawned in a new process group so teardown
and the storage-cap monitor kill the whole tree (podman is a runuser grandchild).
This does NOT confine the container's filesystem view — a `# docker` script can
still bind-mount worker-visible paths — so docker-capable workers remain a
trusted-tenant capability (documented). Reverses the earlier `user: 1000` advice:
keep the worker root; windmill drops podman itself.
e2e verified (bare-metal, worker as root): stock docker job runs (dropped rootless
podman); a uid-1000 --pid=host container reading the root worker's /proc/environ ->
'Permission denied' (DATABASE_URL/secret protected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- P2: hot-reload docker_image_storage_size_mb — handle the setting in
process_notify_event (main.rs), not just initial_load, so the instance-settings
UI change takes effect on running workers without a restart.
- P2: don't let a mounted host /var/run/docker.sock suppress per-job podman under
nsjail — the bash nsjail proto never mounts the host socket into the jail, so it's
unreachable there; only count the unix socket when !nsjail (DOCKER_HOST tcp still
counts in all modes since the network crosses the jail).
- P1: make PerJobPodman teardown non-blocking — offload `podman system reset` +
service kill + rmdir to spawn_blocking (bounded by `timeout 30`) instead of
blocking the Tokio worker thread in Drop; run inline only when off-runtime.
- Docs: note that the per-job podman daemon runs outside nsjail (Docker-API bind
mounts can reach worker-visible paths — trusted-tenant capability, prefer
dedicated docker workers on shared fleets), and that mixed base/full fleets must
route docker jobs to a *-full WORKER_TAGS=docker group rather than the default tag.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tag-only routing (no new ScriptLang — the bash executor handles the annotation):
a Bash script with the '# docker' annotation and no explicit tag is auto-tagged
'docker' at script-create (stored on script.tag, before no-op detection) and at
preview push. 'docker' is added to DEFAULT_TAGS so default workers serve it out of
the box; run a worker group with WORKER_TAGS=docker to route docker jobs to
dedicated/bigger workers. Mirrors the routing half of bunnative/nativets.
Soft size cap for the per-job rootless-podman image store via a new instance
setting docker_image_storage_size_mb (default 8GB; 0 = uncapped). A background
monitor polls the graphroot ('podman unshare du', robust to subuid-owned overlay
layers) and, past the cap, logs a clear error and tears the runtime down (kills the
service so an in-flight pull fails, 'system reset' to stop containers + free space).
This is the rootless-compatible enforcement: a uid-1000 worker cannot mount a sized
tmpfs even when privileged, so a kernel hard-cap isn't available; soft (overshoot
up to one ~2s poll). Without this, # docker jobs could fill the worker disk under
nsjail, unlike other languages capped by the nsjail tmpfs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment-only cleanup flagged in review:
- PerJobPodman struct doc: it's used for docker jobs in ANY sandbox mode (not just
nsjail), and teardown is `podman system reset` (removes images + subuid-owned
layers), not `rm -af` + kill service.
- connect_docker / handle_docker_job param comments: drop the "(nsjail path)" — per-job
podman runs in all modes now.
- DockerfileFull/FullEe: the container_runtime config was removed; podman is started
automatically when no Docker daemon is provided.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the per-worker-group container_runtime option entirely and decide the docker
runtime automatically: for a `# docker` job, if a Docker daemon is already provided
(DOCKER_HOST set or /var/run/docker.sock mounted) use it (backwards compatible,
unchanged); otherwise start a per-job rootless podman (its own ephemeral daemon,
torn down with the job). podman must be present (the *-full images) — else a clear
error.
Removed: container_runtime from WorkerConfigOpt/WorkerConfig/load_worker_config/
WORKER_CONFIG/monitor, the CE config allowlist entry, and the frontend "Container
runtime" toggle. docker-compose / README: the windmill_worker_docker example no
longer sets CONTAINER_RUNTIME (podman is automatic when no daemon is provided).
Verified e2e: a worker with no CONTAINER_RUNTIME and no DOCKER_HOST auto-runs a
# docker job via per-job podman; the provided-daemon (legacy) path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Generalize the per-job rootless podman instance to every docker job (any sandbox
mode) instead of only nsjail, and remove the long-lived shared service:
- Drop start_container_runtime_maybe() + its worker-startup call + the global
DOCKER_HOST mutation. Each docker job now spins up its own ephemeral per-job
podman (start_per_job_podman) and tears it down at job end — so no container
outlives any job, and there's no standing shared socket on the worker.
- per_job_podman is gated strictly on container_runtime=podman. The legacy path
(externally-provided DOCKER_HOST or a mounted /var/run/docker.sock, without
container_runtime=podman) is unchanged: connect_docker(None)/forwarded env,
and Windmill only manages the $WM_JOB_ID container as before.
- docker_host handed to the script: the in-jail /tmp/podman.sock under nsjail
(bind-mounted), else the per-job host socket directly (unshare/none).
- The "no Docker daemon reachable" hint no longer fires when
container_runtime=podman (per-job provides the daemon).
- monitor: drop the container_runtime killpill — with no startup service the
config is read live per job, no worker restart needed on change.
Verified e2e against rootless podman: docker jobs work + confine extra containers
under nsjail AND unshare; legacy host-socket jobs run against the host daemon
with no per-job podman and extras persisting (unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-job podman teardown used `podman rm -af` (containers only) + remove_dir_all.
But rootless podman stores image layers under remapped subuids, which the worker
user cannot delete — so any image the job pulled leaked its overlay layers under
the job dir (and blocked the job-dir cleanup too). Use `podman system reset
--force`, which removes containers AND images and deletes the subuid-owned layers
from inside the user namespace, so remove_dir_all then succeeds.
Verified e2e (nsjail + rootless podman): a job pulling an image now leaves no
subuid-owned files and no per-job podman dir after teardown; container confinement
unchanged (an extra detached container is still killed with the job).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Under nsjail (untrusted), a docker job now runs against its OWN rootless podman
instance instead of the shared per-worker service, so no container it spawns can
outlive the job — even extra/detached ones the script starts.
- start_per_job_podman(): per docker+nsjail job (when container_runtime=podman),
start a rootless `podman system service` with isolated --root/--runroot under
the job dir; its unix socket is bind-mounted into the jail ({DOCKER_SOCK_MOUNT}
in run.bash.config.proto) and handed to the script as DOCKER_HOST.
- connect_docker()/handle_docker_job() take an optional per-job docker_host so
bollard talks to the per-job socket.
- PerJobPodman Drop tears the instance down on every exit path (success, error,
cancel, timeout, killpill): `podman rm -af --force` (removes ALL containers in
the instance, not just $WM_JOB_ID — conmon reparents so process-tree kill is
insufficient) + kill the service + remove the dir.
- unshare (trusted) keeps using the shared service unchanged.
Verified e2e on native rootless podman + nsjail: a job that detaches an extra
`sleep 600` container completes successfully and the extra container is killed
when the job ends (no leak). Tradeoff: per-job storage re-pulls images (no shared
cache yet); a shared read-only image store is a follow-up optimization.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a per-worker-group `container_runtime` option. When set to "podman", the
worker starts a rootless, daemonless `podman system service` on a unix socket
and points DOCKER_HOST at it, so docker-mode jobs (`# docker` bash scripts) run
against a Docker-API-compatible daemon without a privileged dind sidecar or the
host Docker socket. bollard's existing handle_docker_job talks to it unchanged
(validated: inspect/wait/logs/stats/kill/stop/remove against podman 5.8.2).
- windmill-common: container_runtime field on WorkerConfigOpt/WorkerConfig,
loaded from group config or CONTAINER_RUNTIME env.
- windmill-worker: start_container_runtime_maybe() spawns the rootless podman
service once per process (i_worker==1), waits for the socket, sets DOCKER_HOST.
Respects an existing DOCKER_HOST (legacy dind) and degrades with a clear log if
the podman binary is absent (use a *-full image or an init script).
- monitor: restart the worker (killpill) when container_runtime changes.
- api-configs: allow container_runtime in CE config updates.
Requires rootless podman with cgroup v2 delegation for memory monitoring;
podman ships in the *-full images (next commit).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With the dind sidecar now opt-in, a docker-mode bash script on a worker
with neither DOCKER_HOST nor a mounted host socket fails with a generic
"cannot connect to the Docker daemon" error. Prepend a Windmill-specific
hint to the job logs in that case, pointing at the --profile dind opt-in.
Only fires when no daemon is configured, so it stays quiet for valid
dind/host-socket setups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): authorize single-job read endpoints by job/flow visibility
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(jobs): share read links + cached access checks for run visibility
- Cache the job read-access RLS probe (size-bounded LRU keyed by the caller's
authz-relevant identity + job id; no TTL since job-side inputs are immutable).
- Inherit visibility along the full parent_job chain so any flow you can see lets
you read its (deeply nested) steps.
- Share read links: GET /jobs/job_view_token/{id} mints a stateless
HMAC(workspace_key, job_id) token (only if the caller can read the job); the
token grants an authenticated member read of that job and its flow subtree via a
?view_token query param or X-View-Token header. Run page gains a Share button and
honors a ?view_token link.
- Denied-but-existing reads now return 403 with guidance to request a share link
(vs 404 for non-existent), and the run page renders that case with instructions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jobs): address PR review — scope-tag check on mint, constant-time view-token verify
- P1 (Codex): get_job_view_token now enforces the caller's if_jobs:filter_tags
scope before minting, so a tag-scoped token can't mint a transferable link for a
job outside its tags. Adds a scoped-token regression test (allowed + denied).
- Constant-time view-token verification (HmacSha256::verify_slice) instead of
comparing hex strings (Claude/Pi nit).
- get_completed_job_result: an authed reader passing an invalid suspended-secret
triple now falls through to the normal visibility gate instead of erroring out
(Claude nit); unauthenticated callers still rejected.
- Length-prefix the read-access cache key fields so no input values can collide.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(api): add job_view_token to openapi spec; use generated client in run page
Addresses Codex review nit: the new GET /jobs/job_view_token/{id} endpoint was
missing from openapi.yaml (the source the frontend client is generated from). Adds
the path + operationId getJobViewToken, and switches the run page's Share button
from a raw fetch to JobService.getJobViewToken.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): carry view_token on share-link downloads
Addresses Codex review: download actions bypass the request interceptor that adds
X-View-Token (downloadViaClient uses raw fetch; cookie-mode downloads use plain
hrefs), so a share-link viewer got 403 downloading logs/results/args. Append the
view_token query param to the job download paths (result/logs/args/flow-all-logs)
via a new appendViewToken() helper, covering both client-fetch and href modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jobs): enforce tag scope in require_job_read_access (view-token use side)
Addresses Codex P1: the view_token use-side bypassed if_jobs:filter_tags on
handlers that don't tag-filter their data query (result_by_id,
get_flow_job_debug_info, get_otel_traces) — a tag-scoped token could use someone
else's valid share token to read out-of-scope job data. Move the tag-scope check
into require_job_read_access (runs before any created_by/view_token/RLS grant), so
it applies uniformly to every gated handler; removes the now-redundant explicit
check in get_job_view_token. Adds a use-side regression test (scoped token + valid
out-of-scope view_token denied on otel/result_by_id; in-scope still allowed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): include workspace in share read link
Addresses Codex P1: the copied share URL omitted the workspace. The token is
signed with the run's workspace key and the logged layout only switches
$workspaceStore when the URL carries workspace=, so a recipient whose persisted
active workspace differs would open the link against the wrong workspace and the
token would fail validation. Pin workspace= alongside view_token in the link.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(jobs): authorize get_result_maybe get_started branch for queued jobs
Addresses Codex P1: get_completed_job_result_maybe only gated when a completed row
existed; with ?get_started=true a non-reader reached the fallback branch and got
started:true for a running private job. Now fetches created_by and authorizes
(created_by/view_token/RLS, or anonymous for unauth) before disclosing
running-state; a non-existent job still returns started:false (leaks nothing).
Adds a regression test with a queued (no completed row) private job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Zoom URL-validation challenge handler in `handle_challenge_request`
would HMAC-sign any arbitrary `plainToken` and return the result. Since
Zoom webhook verification checks `HMAC-SHA256(secret, "v0:{ts}:{body}")`,
an attacker could craft a `plainToken` in that format to obtain a valid
signature for a forged body, bypassing authentication on a later request.
Unlike the Twitch handler, the Zoom handler verifies no signature on the
challenge request (Zoom's protocol does not include one). Reject any
`plainToken` containing `:` or longer than 128 chars: legitimate Zoom
validation tokens are short random hex strings that never contain colons,
while the exploit requires the colon-bearing `v0:{ts}:{body}` format.
Fixes WIN-2008
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#9393 added `--compile-bytecode` to the uv pip install run inside the
python download nsjail. uv spawns a Python interpreter that compiles .py
files with parallelism scaling to the host CPU count, opening many file
descriptors at once. The download nsjail capped `rlimit_nofile` at 64,
which is exhausted on high-core machines, failing every install with
"Failed to bytecode-compile ... Too many open files (os error 24)".
Low-core VMs never hit the cap, so this surfaced only as a regression on
larger workers after upgrading.
Raise `rlimit_nofile` to 10000, matching the runtime configs
(run.python3 / run.ansible) that already use that value.
Fixes WIN-2009
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): route //native TypeScript previews to native workers
Previewing a TypeScript script carrying the `//native` annotation was
pushed with `language = bun` (what the editor sends), so the job was
tagged `bun` and routed to a regular bun worker. A native-mode worker
neither matches the `bun` tag nor accepts a non-native `script_lang`
(worker.rs rejects with "cannot execute non-native job with language
'bun'"), so previewing a `//native` script on a native-only worker setup
failed — even though the deployed version of the same script runs fine
as `bunnative` / tag `nativets`.
`push` now reconciles the preview language with the `//native`
annotation for `JobPayload::Code`, mirroring the deploy-time logic in
`worker_lockfiles`: `bun` + `//native` is promoted to `bunnative` (tag
`nativets`), and `bunnative` without `//native` is demoted back to
`bun`. This makes a preview run exactly like the deployed script would,
and covers every preview entry point (run_preview_script, inline
preview, codebase preview) since they all go through `JobPayload::Code`.
Adds regression tests asserting the queued job's `script_lang`/`tag` for
all four (declared language × annotation) combinations.
Fixes WIN-2007
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(backend): add sqlx cache for preview_native_tag test query
The regression test's `sqlx::query!` for `v2_job` (tag, script_lang) needs
a cached entry so `SQLX_OFFLINE=true` CI compiles it. Adds exactly one new
cache file; no existing (OSS or EE) caches removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(backend): trim preview native-tag tests to the essentials
Keep the core regression (bun + //native → bunnative/nativets) and the
guard that plain bun previews are unaffected. Drop the two bunnative-
declared cases, which only re-verified the mirrored demote logic and
weren't the reported issue. The shared query is unchanged, so the sqlx
cache stays valid.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): stop git-sync promotion deploys from dropping triggers/schedules
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: bump git-sync hub script to hub/28261 (windmill-cli 1.713.2)
Points LATEST_GIT_SYNC_SCRIPT_PATH at the republished sync-script-to-git-repo
that pins windmill-cli@1.713.2, which carries the promotion include-derivation
fix in this PR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#9400 (WIN-2003) added the ctrl_break() handler as a `#[cfg(windows)]`
branch inside the two Windows-path `tokio::select!` blocks in
shutdown_signal. tokio's `select!` macro does not accept `#[cfg(...)]`
attributes on individual branches, so windmill-common fails to compile
on Windows ("no rules expected this token in macro call").
This slipped through CI because the only job that builds the backend on
Windows is cli-tests.yml's `test-windows`, which triggers only on
`cli/**` changes — #9400 was backend-only.
Fix: define `ctrl_break()` for the whole `not(any(linux, macos))` scope
instead of just `windows`. On Windows it awaits the real CTRL_BREAK
signal; on other non-unix targets it is a never-resolving future, so the
branch is inert there. The select! branches become plain (no per-branch
`#[cfg]`), which the macro accepts.
Verified: the `#[cfg]`-on-branch form reproduces the exact macro error
against tokio 1.46.1, and the fixed form compiles clean.
Fixes WIN-2003 (Windows build regression)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The instance username-conflict resolver rewrote
workspace_runnable_dependencies.app_path to the new user path before the
app row itself was renamed, violating fk_workspace_runnable_dependencies_app_path.
That FK is ON UPDATE CASCADE, so renaming the app already propagates the new
path; the manual rewrite was redundant and mis-ordered. Any user owning an app
under u/<username>/ with a tracked runnable dependency hit HTTP 500 and could
not have their username conflict resolved.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
On Windows, shutdown_signal only registered ctrl_c() (CTRL_C_EVENT).
CTRL_BREAK_EVENT — the default kill signal sent by Nomad's raw_exec
driver on Windows — had no handler, so the worker terminated
immediately without graceful shutdown, interrupting running jobs.
Add a ctrl_break() helper (mirroring the Unix terminate() helper) and
register it as an additional branch in both Windows tokio::select!
blocks in shutdown_signal.
Fixes WIN-2003
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* oauth: add salesforce provider
Register Salesforce OAuth (Authorization Code) for Windmill resource connect.
Production uses login.salesforce.com; the sandbox block points at
test.salesforce.com (URL overrides only; scopes inherited) per #9358, so a single
canonical `salesforce` resource type covers both with separate `salesforce_sandbox`
instance credentials.
Paired with the hub integration: windmill-labs/windmill-integrations#131.
The Salesforce icon already exists in the frontend (SalesforceIcon.svelte).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix JSON syntax error in oauth_connect.json
* fix: add salesforce production tile to OAuth settings dropdown
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Python jobs under nsjail experience slow imports (~465ms for `import
requests`) because dependency directories are mounted read-only. Without
pre-compiled `.pyc` files, Python recompiles `.py` source to bytecode
in-memory on every import in every fresh nsjail process, paying the cost
repeatedly.
Add `--compile-bytecode` to both `uv pip install` invocations (the
Rust-driven install in python_executor.rs and the nsjail
download_deps.py.sh script) so `.pyc` files are generated at install
time and available at runtime even through read-only mounts. The files
are included in both the local cache and S3 piptar uploads.
The flag is supported by the uv version (0.9.24+) shipped in the
Dockerfile. The existing `__pycache__` skip only filters top-level dirs
for dedup logic, not subdirs within packages, so there is no conflict.
Fixes WIN-2001
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): handle multi-version scripts when removing granular ACL
remove_granular_acl built a CTE whose `old` subquery selects one row per
matching table row, then read it back via the scalar subquery
`(SELECT old_write FROM old)` in the RETURNING clause. For the `script`
table the PK is (workspace_id, hash), so a path with multiple deployed
versions yields several rows sharing the same (workspace_id, path). When
two or more versions carried the ACL key, `old` returned multiple rows and
PostgreSQL rejected the scalar subquery with "more than one row returned by
a subquery used as an expression", making it impossible to remove an ACL
entry from a script's permissions panel.
All versions share the same extra_perms value (the UPDATE applies to every
matching row), so any single row's old_write is representative. Add
`LIMIT 1` to the scalar subquery. Other kinds are unaffected because they
have a unique constraint on (workspace_id, path/name).
Introduced by b3603d872 (#7365). Add a regression test reproducing the
multi-version case at the SQL level.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove granular ACL regression test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: trim ACL fix comment
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cache): allow overriding hub base url via env in `cache` mode
The `windmill cache hubPaths.json` prebuild step (run in the Dockerfile) never
connects to the DB, so HUB_BASE_URL stays at its compiled default
(https://hub.windmill.dev) — unlike server/worker modes which load it from the
DB global setting. This made it impossible to point the prebuild cache step at
a private or staging hub.
Read HUB_BASE_URL from the environment at the start of cache_hub_scripts and
store it into the existing HUB_BASE_URL ArcSwap (the same static the hub fetch
functions read). No effect unless the env var is set and non-empty; server and
worker modes are unchanged (they still use the DB setting).
This also enables validating hub-script dependency changes end-to-end against a
local fake-hub before pushing to the real hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(security): re-pin cached hub scripts to patched versions
windmill-integrations#133 was merged and pushed to the hub, minting new
versions with regenerated (CVE-free) lockfiles. Bump the hubPaths.json pins so
the prebuild cache step (`windmill cache`) fetches the patched lockfiles instead
of the old vulnerable ones (the hub serves each version_id immutably, so the old
pins keep returning the vulnerable deps until bumped).
- slackErrorHandler 19741 -> 28241
- slackRecoveryHandler 9080 -> 28239
- slackSuccessHandler 28220 -> 28240
- smtpReport 9086 -> 28242
- appReport 28076 -> 28243 (puppeteer screenshot script)
- gitInitRepo 28219 -> 28229 (already-fixed hub version; pin was stale)
Validated end-to-end against the real hub: `windmill cache` with these pins
produces a clean cache_nomount/bun (axios 1.16.1, form-data 4.0.5,
follow-redirects 1.16.0, nodemailer 8.0.10, ws 8.21.0, svelte 5.55.8,
devalue 5.8.1; basic-ftp and ip-address no longer pulled). No vulnerable
versions remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): preserve step/subflow worker tags under a custom-tagged flow
A flow running on a custom worker tag force-propagates that tag to every
descendant step, script and nested sub-flow, overriding their own declared
tags. This made it impossible to route a specific step or sub-flow to a
different worker group. The new opt-in FlowValue.preserve_step_tags lets a
step that declares its own non-empty tag run on it; untagged steps still
inherit the flow tag. Defaults off to preserve existing behavior.
* chore: regenerate system prompts for preserve_step_tags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(flows): nest preserve_step_tags toggle under flow worker tag setting
The toggle only affects routing when the flow has a custom worker tag, so
show it as a sub-setting of the Worker Group tag picker, visible only once a
tag is set, instead of as a standalone option.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): allow step worker tag picker when preserve_step_tags is enabled
When a flow defines a worker tag, the per-step tag picker was replaced by a
read-only "Flow's WG" label. With preserve_step_tags enabled the step's own
tag is honored, so the picker must remain editable in that case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): propagate preserve_step_tags to branch and loop bodies
payload_from_modules built the synthetic RawFlow for branch/loop bodies with
a default FlowValue, dropping preserve_step_tags. Tagged steps inside a
branch or loop therefore still inherited the parent flow tag even with the
flag enabled. Thread the flag through to the synthetic FlowValue so the
behavior is consistent for nested containers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags when flow worker tag is removed
Avoids the flag lingering as invisible state after the flow tag (and its
toggle) are removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): repair preserve_step_tags propagation to branch/loop bodies
The previous commit added flow.preserve_step_tags at the payload_from_modules
call sites but the parameter and FlowValue field were not actually threaded
through (a failed edit left the function unchanged), so the crate did not
compile. This completes the change: payload_from_modules takes preserve_step_tags
and sets it on the synthetic FlowValue for branch/loop bodies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): complete preserve_step_tags propagation to branch/loop bodies
Previous two commits left windmill-worker uncompilable: payload_from_modules
received flow.preserve_step_tags at its call sites but the parameter and the
synthetic FlowValue field were not actually added. This adds the parameter,
sets preserve_step_tags on the synthetic FlowValue, and threads
flow.preserve_step_tags through all five call sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags whenever the flow worker tag is removed
The flag was only reset when the Worker Group toggle was switched off, not
when the tag was cleared directly in the picker (or via the YAML editor),
leaving preserve_step_tags=true as invisible state with the advanced badge
still reporting it active. Move the cleanup into the reactive block that
already tracks the flow tag so every clear path is covered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(apps): make public apps opt into cross-origin isolation via wm_coep
Public app pages served at /public/* and custom paths /a/* were not
getting the COEP/COOP/CORP headers, so they were blocked when embedded
as an iframe inside a cross-origin-isolated page (e.g. another raw app,
which sets Cross-Origin-Embedder-Policy: require-corp). A nested
document loaded into a require-corp context must itself set COEP for
the iframe to load.
Rather than applying the isolation headers to all public pages (which
would also force COEP on classic apps and break subresources without
CORP, e.g. external image URLs or embeds), public apps now opt in via
a `wm_coep` query param on the embed URL:
<iframe src="https://<domain>/public/<ws>/<secret>?wm_coep=on">
The app publish drawer gains a URL/Embed toggle: "URL" shows the plain
shareable link (param-free), "Embed" shows a ready-to-copy iframe
snippet with wm_coep baked in, so the flag is discoverable exactly when
embedding and absent otherwise.
`wm_coep` is consumed internally and stripped from the app `query`
context so it doesn't collide with app-defined params. Only params we
own are stripped (an explicit set), not the whole `wm_` prefix.
Fixes GIT-884
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* nit
* nit
* fix(apps): only bake wm_coep into embed snippet for raw apps
AppEditorHeaderDeploy is shared by the classic (AppEditorHeader) and raw
(RawAppEditorHeader) deploy drawers. The embed snippet unconditionally
appended ?wm_coep=on, which for a classic/low-code app forces COEP
require-corp on the document and breaks no-CORP cross-origin subresources
(external <img> in AppImage/AppStatCard/AppNavbar, {@html} embeds in
AppHtml, CDN import() in AppCustomComponent) — the exact regression the
opt-in design avoids.
Add a `rawApp` prop (default false); the raw header passes rawApp. The
flag is appended only for raw apps; classic apps get a plain iframe
snippet, and the wm_coep helper text is shown only for raw apps.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
CLI deploys (sync push, set-permissioned-as) went through the same
create/update endpoints as a UI "deploy from draft", which delete the
draft at that path. That silently wiped teammates' in-progress drafts on
every push. Add a transient skip_draft_deletion deploy flag (mirroring
deployment_message) that the CLI sets; the backend then skips the
DELETE FROM draft for scripts, flows, and apps. UI deploys are unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_token_using_oauth resolved the AI OAuth resource's token_url and
POSTed to it without any SSRF validation, while base_url is validated in
get_base_url. A workspace member with resources:write could point
token_url at an internal/metadata address (e.g. 169.254.169.254),
turning the server into an authenticated blind SSRF probe.
Validate the resolved token_url with validate_url_for_ssrf before the
request, gated behind the same ALLOW_PRIVATE_AI_BASE_URLS opt-in as
base_url so private AI deployments keep working consistently for both
URL fields. ALLOW_PRIVATE_AI_BASE_URLS is now pub so windmill-api can
reuse it instead of re-parsing the env var.
* fix: trigger git sync for re-encrypted secrets on encryption key change
When changing a workspace encryption key, the secret variables get
re-encrypted with the new key, but the git sync was only dispatched for
the encryption_key.yaml metadata file. Repos with Secrets sync enabled
were left with stale ciphertexts until the next per-variable deployment.
Now, after the transaction commits, we also dispatch a Variable git sync
event for each re-encrypted secret so the new encrypted values are
pushed to the configured repos. Errors are logged but don't roll back
the key rotation.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: batch encryption-key rotation into one git-sync job
Workspace encryption key rotation now re-encrypts every secret variable
and then dispatches a single batched git-sync job carrying the Key event
plus one Variable item per re-encrypted secret. Repos with Secrets sync
enabled receive every new ciphertext in one commit instead of nothing
(previously only `encryption_key.yaml` was pushed) — and instead of N
separate jobs the debouncer might or might not merge.
Wires through the new `handle_deployment_metadata_batch` entry point
added in the companion EE PR; OSS has a no-op shim so the build stays
green.
Adds an integration test (`workspace_encryption_key_git_sync`) asserting
that rotating the key with 3 secret variables in scope produces exactly
one deployment-callback job whose `items` array contains the Key event
+ all 3 variable entries and `skip_secret=false`.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref for git-sync helper simplification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover non-debouncing git-sync fallback on key rotation
Adds a regression test exercising a workspace whose sync script predates
hub version 28103: the rotation must still queue a legacy-format
deployment-callback job per item (encryption_key + each re-encrypted
secret) instead of silently skipping the repo. Bumps ee-repo-ref to the
EE fallback fix.
Addresses the P1 raised in the PR review (Codex/Pi/Claude): batch path
dropped git sync entirely for repos without sync-job debouncing support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: add sqlx offline cache for encryption-key git-sync test queries
The cargo_test CI job builds with SQLX_OFFLINE=true; the two new
sqlx::query!/query_as! calls in
windmill-api-integration-tests/tests/workspace_encryption_key_git_sync.rs
had no cached entries, failing the build with E0282. Regenerated and
added only the two new query caches (no EE/feature cache loss).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to updated EE companion PR (08e3b9b)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ai): disable redirect following on AI proxy client to close SSRF
The AI proxy validates the configured base_url against SSRF rules but the
shared HTTP client followed up to 10 redirects without revalidating the
hops, so a public base_url could 3xx the server into a private/internal
address (e.g. the Docker socket or cloud metadata). Disable redirect
following so the validated host is the only one the server connects to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): remove heavy redirect SSRF integration test
Drop the integration-test-level regression for redirect following; it
spins up a full API server + DB for a one-line client-config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): don't follow symlinks when reading service log files
Defense in depth on top of the existing `..` path-traversal check in
the get_log_file handler: reject the request if the final path
component is a symlink, so a planted symlink in the logs directory
cannot be used to read arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): authorize and harden the jobs_u get_log_file endpoint
The unauthenticated jobs_u get_log_file endpoint served any job's log
file to anyone who knew the job UUID, with no authorization. Gate it the
same way as get_job_logs: look up the job (the log directory name is the
job id) filtered by workspace and the caller's scope tags, and only allow
non-logged-in callers to read logs of jobs created by the anonymous user.
Also add defense in depth: refuse to read through a symlink so a planted
symlink in the logs directory cannot be used to exfiltrate arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fork-branch deployment callback runs the hub script
sync-script-to-git-repo-windmill, which imports `windmill-cli` as a pinned
npm dependency and runs it in-process (it does NOT shell out to a PATH wmill).
hub/28236 pinned windmill-cli@1.706.1, whose `git-deploy --only-create-branch`
path returns early without pushing — so the fork branch was checked out
locally but never published to the remote. #9366 fixed the CLI and shipped it
as windmill-cli@1.712.0, but without a hub-script bump the running callback
still used 1.706.1.
Bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28238, which is identical to 28236
except it pins windmill-cli@1.712.0 (content + lockfile). This fixes
test_workspace_fork_creates_branch and production fork-branch creation.
Also add backend/windmill-common/src/workspaces.rs to the git-sync-test
path-gate so future script-path bumps trigger the e2e (the bump alone is not
otherwise covered by the gate).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(oauth): support per-provider sandbox URLs in registry + instance settings
* fix(oauth): polish sandbox review nits (cc lookup, header label, ee ref)
* refactor(oauth): drop dead build_oauth_clients duplicate in windmill-oauth
* refactor(oauth): derive sandbox-capable provider list from registry
* chore(docker): copy oauth_connect.json into frontend build stage
* test(oauth): cover sandbox helpers (as_sandbox, canonical_name, resolve)
* chore: update ee-repo-ref to 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
This commit updates the EE repository reference after PR #595 was merged in windmill-ee-private.
Previous ee-repo-ref: 3ab3eca9ac15ebab6db991e7964bc5e48ce21f42
New ee-repo-ref: 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] fix(git-sync): publish fork branch on only_create_branch from the CLI
Fixes WIN-1997. Forking a git-sync-configured workspace must push a
`wm-fork/<branch>/<id>` branch to the repo, but the integration test
`test_workspace_fork_creates_branch` failed: the fork callback job
succeeded yet no branch appeared.
Root cause: the fork-branch callback runs the sync script with
`only_create_branch: true` and no items. The hub sync script delegates
branch checkout to `wmill sync git-deploy --only-create-branch` and runs
its own in-process commit+push ONLY for the `!only_create_branch` path
(`if (!only_create_branch) git_push(...)`). #9284 had moved commit+push
out of the CLI to the caller for the GPG-cache-warmth invariant
(WIN-1974) — but it also dropped the CLI's push for the branch-only
case. A branch-only publish has no commit, so no signing is involved and
the GPG concern does not apply; with neither the CLI nor the hub script
pushing, the empty fork branch was never published.
Restore the CLI push for the `only_create_branch` path (a bare
`git push --porcelain` of the checked-out branch ref). Adds a
deterministic CLI regression test that runs `git-deploy
--only-create-branch` for a fork workspace and asserts the branch
reaches the remote with no caller-side push.
EE companion: format the fork-branch commit message with Display instead
of Debug (no more `Some("...")` leak).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
This commit updates the EE repository reference after PR #597 was merged in windmill-ee-private.
Previous ee-repo-ref: 8b02336fcebdfae4b9d2795cbb74fa7046530bcb
New ee-repo-ref: a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] feat(queue): duration-weighted fairness admission atomic
Add the `WORKSPACE_FAIRNESS_ADMISSION_PPM` atomic that the EE
`workspace_fairness_ee::refresh_overloaded` writes on each refresh
(see companion EE PR). The atomic is read on every pull by
`should_admit_capped` to decide whether the dispatch goes down the
standard or fairness path. Defaults to 10_000 (= admit all) so the
pre-fairness behaviour is preserved until the first refresh fires.
OSS stub in `workspace_fairness.rs` continues to return `true`
unconditionally, so non-EE builds are bit-identical.
* docs(queue): consolidate full fairness algorithm into workspace_fairness.rs
Move the algorithm doc — what "overloaded" means in worker-seconds, the
duration-weighted admission derivation, coordinated refresh structure,
audit emission, the SQL perf constraints (no params CTE, drive running
side from v2_job_runtime), and EE gating — into the OSS surface module
where it is readable without EE access. The EE file becomes implementation
only.
Also bump ee-repo-ref to the EE commit that strips the duplicate doc.
* docs(queue): clarify ADMISSION_PPM default is "admit all", not count-based
Addresses CI review (claude[bot]): the `10_000` initial value is the
"admit all" no-op default that applies before the first refresh
classifies an overloaded set — not the count-based value (which would
be `target * 10_000`). The count-based form is the empty-bucket fallback
inside `compute_admission_ppm`, a different thing.
* chore(queue): point ee-repo-ref at EE main (fairness admission merged via #593)
* fix(queue): duration-weighted admission uses unclamped service-time window
Bumps ee-repo-ref to the EE fix (windmill-ee-private#596) that sources
D_c/D_u for the admission probability from a separate 60s service-time
window of true `duration_ms`, instead of the occupancy aggregation whose
per-job contributions are clamped to the 10s occupancy window. The clamp
truncated D_c for capped jobs longer than the window, under-admitting the
duration skew (true 34s jobs → ~86% effective share instead of the target
65%). Occupancy worker-seconds still drive overload classification.
Updates the algorithm doc in workspace_fairness.rs accordingly.
Note: ee-repo-ref points at the EE feature branch; re-point to EE main
once #596 merges.
`announce_server_started` writes a `server_heartbeat:{INSTANCE_NAME}` row
on each startup. INSTANCE_NAME is a fresh random string per process, so
the row is never updated again and a new row is inserted on every
restart, growing background_task_state unboundedly.
Add an hourly monitor task that deletes server_heartbeat:* rows older
than 7 days. Older rows cannot influence check_any_server_started (which
only considers heartbeats refreshed after the restart was initiated), so
they are safe to prune.
Fixes WIN-1990.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(queue): bump EE ref to include worker_ping fairness signal
The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge
commit on the asset-graph-view-ee branch) instead of ddc9e80, which
contains the workspace-fairness fix that switches the active-share
signal from v2_job_queue.running=true to worker_ping. As a result
cloud was still computing overload off the legacy signal, so a
workspace with many in-flight/suspended flows (lancom01-prod, with
799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397
running-true rows) was flagged as 95% of cluster activity despite
consuming zero worker slots.
Bumping to ddc9e80 picks up the worker_ping-based signal, which
naturally excludes (a) suspended jobs (no worker pinging them),
(b) zombie running-rows from dead workers, and (c) flow/flownode
orchestration rows that never run on a worker in the first place.
* test(queue): seed v2_job rows + realistic durations for fairness helpers
The new duration-weighted fairness algorithm joins v2_job_queue and
v2_job_completed to v2_job for the `kind` filter (excluding flow
bookkeeping) and reads `duration_ms` for the completed contribution.
Update the test helpers to mirror that schema:
* `insert_completed` now inserts a matching v2_job row (kind=script)
and writes `duration_ms = 1000` with a 1-second [started_at,
completed_at] interval, so each completed row contributes ~1
worker-second when fully inside the refresh window.
* `insert_queued` likewise pre-inserts v2_job, sets `started_at`
to NOW() - 1s when running=true (so running rows contribute ~1
worker-second by the time the refresh runs), and seeds
v2_job_runtime.ping so the running side accrues real-time worker
seconds (the algorithm bounds end-of-interval by ping).
The zombie/suspended insert helpers are intentionally left without
v2_job rows — the new algorithm's INNER JOIN excludes them, so they
still correctly contribute zero worker-seconds.
* chore(queue): bump EE ref to duration-weighted fairness algorithm
Companion to windmill-ee-private#<TBD>: switch the EE workspace
fairness aggregation from a count-based UNION (worker_ping snapshot
+ v2_job_completed count) to a worker-seconds aggregation sourced
directly from v2_job_queue and v2_job_completed, with kind/suspend
filters mirroring handle_zombie_jobs and per-row defenses against
zombie inflation on both halves.
* chore(queue): bump EE ref for fairness perf fix (inline window_start)
* chore(queue): bump EE ref for fairness perf rewrite (driver-side flip)
* update ee ref
`fairness_ignores_zombie_running_rows` and
`fairness_ignores_concurrency_suspended_rows` panic intermittently in CI
(both Linux and Windows runs). Mark them `#[ignore]` until the
underlying flakiness is resolved.
