When HTTP request tracing is enabled, the NO_PROXY injected into traced jobs
was built solely from the no_proxy_hosts instance setting, ignoring the
worker container's own NO_PROXY. Enabling tracing therefore silently dropped
every exclusion an operator had already configured at the container level,
funneling those hosts into the MITM proxy (and on to any upstream corporate
proxy). The upstream-relay side already honored the container NO_PROXY; this
makes the injected-into-jobs side symmetric by merging both sources.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A flow inline step whose id is a Python keyword (e.g. `in`) crashed with a
`SyntaxError`: the wrapper emits `from {pkg} import {step_id} as inner_script`,
and `from x import in as y` is invalid Python.
The codegen already prefixes `_` to path segments that start with a digit
(`1234` → `_1234`); this extends that guard to Python hard keywords (`in` →
`_in`) in `compute_python_module_dir` and on the leaf in `compute_py_codegen`
and `prepare_wrapper`. The relative-imports write path inherits it for free.
Fixes#8893
* feat(otel): propagate inbound W3C traceparent to job spans
Capture the inbound traceparent header at the run endpoints
(WebhookArgs::to_args_from_format) into a reserved _wm_traceparent arg key
(gated on OTEL_TRACING_ENABLED), riding the args jsonb like
_ENTRYPOINT_OVERRIDE. At pickup, create_span_with_name attaches a span link
from the job's worker span to the originating distributed trace, so a job
triggered by an instrumented service is connected to the caller's trace
while keeping its UUID-derived trace id (trace-by-job-id unaffected).
The link/parse logic lives in the EE otel modules; this OSS side only
captures the header and calls the (no-op outside EE) hook. Companion EE PR
required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to inbound-trace-propagation EE branch
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(agents): don't attribute work to specific customers in repo content
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(otel): relocate job + script spans into the inbound trace
Builds on the captured _wm_traceparent: the worker job span is re-parented on
the inbound caller context, the script subprocess's TRACEPARENT env is the
inbound context (so its spans join the caller's trace), and the context is
propagated to flow steps so the whole flow relocates. Carried to the worker via
a new LogContext.inbound_traceparent field. Non-inbound jobs are unchanged.
Adds a relocation integration test. Companion EE PR required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to inbound-trace-propagation relocate commit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(otel): harden inbound traceparent capture
Address review feedback:
- strip any caller-supplied _wm_traceparent from args/extra before stashing the
header-captured value, so the reserved key is Windmill-controlled only
- valid_w3c_traceparent: reject version ff and require lowercase hex, so we don't
forward an inbound header that downstream OTel parsers would reject
- clarify that the capture helper does not validate the W3C format (done at use)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 2c7964460327fab5e3a27c0f74b8d6f26ab7f79a
This commit updates the EE repository reference after PR #604 was merged in windmill-ee-private.
Previous ee-repo-ref: 8fc04fb105dc49769205f7174d551a0d134d1bec
New ee-repo-ref: 2c7964460327fab5e3a27c0f74b8d6f26ab7f79a
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(sandbox): pull/extract images with crane instead of podman (+ add to image)
The sandboxed container runtime (`# sandbox <image>`) only ever pulls + flattens an
image (nsjail does the run), so a full container engine is overkill — and podman was
never actually in any Dockerfile, so the merged feature couldn't run in the shipped
image. Switch to crane (google/go-containerregistry): a single ~25MB static binary,
no daemon/store/root/privileged.
- docker_v2.rs: crane export -> flattened rootfs tar, crane config -> OCI config,
crane digest -> content-addressed rootfs+config cache (cross-job dedup + automatic
freshness), crane manifest -> pre-download size guard. DOCKER_CONFIG authfile dir.
Cache eviction prunes the rootfs-tar cache by mtime (LRU). Pull policy honored via a
ref->digest cache (missing/never reuse without a registry hit).
- Dockerfile + docker/DockerfileSlim{,Ee}: install the crane binary (Full/FullEe and
the EE image inherit it via FROM the base image).
- docs + UI text + instance-setting descriptions updated (download size is compressed;
cache is the rootfs-tar cache).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): address CI review — digest-pinned fetch, size cap on every job, eviction race
Codex P1s:
- Fetch by the resolved digest (name@digest), not the mutable tag, so content can't
diverge from the digest the cache is keyed under if a tag moves mid-fetch.
- Enforce the size cap on EVERY job via a cached {digest}.size sidecar (no registry call
on cache reuse), so lowering the limit rejects already-cached oversized images.
- Eviction race: hardlink the cache tar into the job dir before tar -xf (pins the inode
against concurrent eviction) and re-fetch if it was evicted first.
Claude P2s: atomic config sidecar (tmp+rename) + tolerate torn parse; soften the LRU
comment (mtime = creation order); sweep orphaned *.tmp.* and .size on eviction.
+digest_key/ref_key unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): P1 cross-fs cache staging (EXDEV), Dockerfile arch fail-fast
CI re-review (Claude + Codex P1): the eviction-race hardlink crosses filesystems in the
shipped deployments — the cache is its own volume (/tmp/windmill/cache) while the job dir
is on the container fs — so hard_link returns EXDEV (not NotFound) and every sandbox job
fails. Fall back to tokio::fs::copy on a non-NotFound link error; copy reads through the
source inode so it still survives a concurrent eviction.
Also: Dockerfiles fail fast with a clear error on an unsupported arch instead of building
a 404 crane URL; ref->digest file written via tmp+rename (no torn read under missing/never).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(sandbox): say 'oldest by creation time' not 'LRU' for cache eviction
Codex P2: the code evicts by tar creation time (cache hits don't touch mtime), so the
user-facing docs + instance-setting text shouldn't claim true LRU.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: make C# dotnet target framework configurable via DOTNET_TARGET_FRAMEWORK
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: include dotnet target framework in C# binary cache key
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add sandboxed docker v2 runtime via '# docker <image>'
Run a container image as a subprogram of the job's own nsjail sandbox:
extract the image rootfs with podman (rootless) and run it chrooted inside the
job's nsjail, so the container inherits the job's confinement and is safe under
nsjail / for untrusted code. Selected by '# docker <image>'; a bare '# docker'
keeps the v1 (dind) path untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: default to daemonless docker (drop dind from compose, allow docker on cloud)
docker-compose no longer ships the dind sidecar (v2 is daemonless: podman + nsjail
in the worker); removed the dind service, DOCKER_HOST env, depends_on and volume.
Removed the language-picker guard that blocked Docker scripts on the multi-tenant
platform, now that v2 makes docker safe to run sandboxed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: select sandboxed container via # sandbox <image>; add pull policy + size guards
- Surface moved from '# docker <image>' to '# sandbox <image>' (groups under the
sandbox annotation; '# docker' stays v1-only, '# sandbox' stays nsjail-bash).
- SANDBOX_IMAGE_PULL_POLICY (default 'newer') so moving tags don't go stale.
- SANDBOX_IMAGE_MAX_SIZE_MB rejects oversized images before extraction.
- SANDBOX_IMAGE_CACHE_MAX_MB best-effort LRU eviction of podman's image store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sandbox): support # volume, honor nsjail tmp instance settings, v2 docker template
- Thread shared_mount into the sandbox container nsjail config so '# volume' mounts
(and the same-worker /tmp/shared folder) apply inside the container.
- Use resolve_nsjail_tmp_mount_block for the container's /tmp so it honors the same
nsjail_tmp_backing / nsjail_tmpfs_size_mb instance settings as other nsjail jobs.
- docker-compose comment + the editor's Docker template now use '# sandbox <image>'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sandbox): make image size/cache/pull-policy UI instance settings
Convert SANDBOX_IMAGE_* from worker env vars to DB-backed instance settings
(sandbox_image_max_size_mb, sandbox_image_cache_max_mb, sandbox_image_pull_policy),
hot-reloaded via the same mechanism as nsjail_tmpfs_size_mb and configurable in
#superadmin-settings. No worker restart needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(sandbox): windmill-managed registry — default registry + private auth
Two new instance settings:
- sandbox_image_default_registry: prepended to unqualified image refs (alpine ->
<registry>/alpine); fully-qualified refs untouched.
- sandbox_registry_auth: docker/podman auth.json blob written to a per-job authfile
(0600, removed with the job) and passed to podman --authfile for private registries.
Both hot-reloaded and configurable in #superadmin-settings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): protobuf-safe proto_str escaper, atomic 0600 authfile, registry tests
Addresses local-review P2s: proto_str now emits valid protobuf octal escapes for
control/non-ASCII bytes (not Rust \u{..} that nsjail would reject); the registry
authfile is created 0600 atomically (no world-readable window); add a
registry_qualified table test + a non-ASCII proto_str case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): P0 — deliver image env via nsjail envar:, never the launcher process env
CI review (P0): the image's OCI Env (attacker-controlled keys+values) was applied to
the nsjail launcher process via .envs(), so a hostile image could set LD_PRELOAD/
LD_LIBRARY_PATH/LD_AUDIT on nsjail itself and execute code as the worker outside the
jail. Now the image env is rendered as proto-escaped 'envar:' directives (child-only)
and nsjail's process env carries only windmill-trusted keys (reserved vars + proxy).
Also: warn instead of silently bypassing the size guard on inspect failure; reset the
eviction guard via a Drop guard (no stuck flag on panic/early-return). +render_envars test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): P0 symlink-write escape via rootfs script; P1 redact registry-auth logging
CI review:
- P0 (Codex): the body was written into the image-controlled rootfs as
.windmill_docker_main.sh via write_file (follows symlinks) — a hostile image could
plant that path as a symlink to a host file and capture the worker's write before
nsjail starts. Now the body is passed straight to 'sh -c <body> sh <args>'; no file
is written into the rootfs at all.
- P1 (Codex): sandbox_registry_auth flowed through the generic setting loader which
logs the value (raw auth.json credentials). Replaced with a secret-aware reload that
loads directly and logs only a redacted 'configured=' message.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sandbox): redact sandbox_registry_auth in instance-settings write log too
The settings API also logs 'Set global setting <key> to <value>' via format_setting_value;
add sandbox_registry_auth to SENSITIVE_SETTINGS so the credential is redacted there as
well as on reload.
* fix(sandbox): don't silently disable cache eviction on podman images parse error
Re-review (cubic/Claude P2): serde_json::from_slice(...).unwrap_or_default() meant any
parse hiccup (e.g. podman omitting Size/Created via omitempty for a zero value, or
schema drift) silently degraded to an empty Vec and disabled eviction with no log.
Now Size/Created are #[serde(default)] (a missing omitempty key -> 0, not a whole-array
parse failure) and a real parse error warns + breaks instead of being swallowed.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): early stop can include the stopping step's result in the raised error
When a step uses Early Stop with "Raise an error message if stopped", the
flow result was entirely replaced with a static error object
({"error": {"name": "EarlyStopError", "message": "..."}}), discarding the
stopping step's own output. This made it impossible to stop+fail a flow
while preserving the data the step produced (e.g. an API that returns
HTTP 200 with a userErrors payload).
Add an opt-in `error_include_result` flag on StopAfterIf. When enabled on
the raise-error path, the raised payload becomes
{"error": {...}, "result": <step result>} instead of dropping the result.
Default is false, so existing behavior is unchanged. The option is threaded
through the worker's stop-after-if handling (including stop_after_all_iters_if
for loops/branchall) and exposed in the flow editor's Early Stop panel.
Fixes WIN-2012
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(flows): cover early-stop error_include_result payload shaping
Add a regression test asserting that a step using Early Stop with a raised
error message and error_include_result=true fails the flow while preserving
the step output as {"error": {..}, "result": <step result>}, and that with
the flag off the result is the bare {"error": {..}} object.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(flows): nest early-stop step result inside the error object
Embed the stopping step's result under `error.result` rather than as a
top-level sibling of `error`. This keeps the flow result shape as
`{ "error": { .. } }` — identical to a normal error — so consumers that
key off the top-level shape (single `error` key) keep working, while the
data is still preserved for those that look inside the error object.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): always include the stopping step's result in early-stop errors
Drop the opt-in `error_include_result` gate. Since the step result is nested
inside the error object (`error.result`), the top-level result shape stays
`{ "error": .. }` — identical to a normal error — so consumers that detect or
parse failures by the top-level shape are unaffected. Gating it added schema
surface, plumbing, and a UI toggle for no real compatibility benefit.
Now, whenever a step early-stops with a raised error message, the flow fails
and the raised error embeds the stopping step's own result under
`error.result` (aggregated iteration results for loops/branchall). This
reverts the `StopAfterIf.error_include_result` field, its threading, the
OpenAPI/generated-client surface, and the editor toggle; the "Raise an error
message" tooltip now notes that the step result is included.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): gate early-stop result inclusion behind opt-in flag
Re-introduce the per-step `error_include_result` flag (default off) instead
of always embedding the step result. Although nesting the result under
`error.result` keeps the result *shape* backward-compatible, it does not
address data exposure: a failed flow's result is propagated to synchronous
webhook callers, the flow's failure module, and the workspace/global error
handler (commonly a Slack/email/outbound-webhook notifier). Always including
the step output would surface previously-redacted intermediate data to all of
those sinks for every existing error-stop flow.
Gating keeps the existing behavior (bare `{ "error": .. }`) as the default and
only embeds `error.result` when the flow author explicitly opts in, matching
the original issue's intent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(flows): omit error_include_result when false; refresh generated prompts
- Add `skip_serializing_if = "is_false"` to `StopAfterIf.error_include_result`
so serialized flows are byte-identical when the flag is off. Fixes the
`flowmodule_serde` round-trip test (cargo_test) and avoids churn on existing
flows.
- Regenerate `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`
for the new OpenFlow `error_include_result` property. Fixes check-freshness.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(flows): cover error_include_result for the loop "stop after all iters" path
Add a regression test for the stop_after_all_iters_if branch, where `nresult`
already holds the aggregated iteration results — confirming `error.result`
carries each iteration's output (distinct from the per-step fallback path).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: resolve workspace imports via /f/,/u/ not $f/,$u/ aliases
Keep the CLI managed tsconfig.wmill.json / `refresh tsconfig` / Deno
import-map QoL from #9378, but re-key it on the existing /f/,/u/ workspace
paths instead of the new $f/,$u/ specifiers. Verified /f/,/u/ resolves in
tsc, Bun, Deno, the in-app ATA editor, and the worker, so the $-prefixed
alias added no value. Drop the $f/,$u/ handling from the parser, dep-map,
deno_executor, bun loaders, ATA, relative_imports and monaco paths; revert
the windmill-parser-wasm-ts bump (1.714.0 -> 1.695.0). Also fold in the
cli/package-lock.json sync for the already-committed pg-gateway dependency.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: drop duplicate relative-path check and restore rustfmt formatting
Follow-up cleanups to the previous commit's full-file reverts, which
restored pre-#9378 state that main had since improved:
- relative_imports.ts: remove the redundant duplicate d.startsWith('/')
(pre-#9378 had it; #9378 had repurposed that line, so main has no dup).
- windmill-parser-ts/src/lib.rs: restore the multi-line new_source_file(...)
formatting required by backend/rustfmt.toml (the single-line revert would
fail `cargo fmt --check`). Now differs from main only by the $f//$u/ removal.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a dependency job for an app is triggered by a relative/workspace
import (e.g. an imported script was updated), handle_app_dependency_job
re-appended the version captured at job-creation time to the versions
array. On a git-sync/CLI push that deploys both the imported script and
the importing app in the same batch, the script's dependency job
snapshots the app's old version; the app push then creates a newer
version (uploading its bundle against that new version); finally the
relock runs and re-appends the old version, making it latest again.
For raw apps this is fatal: bundle_secret is computed from the latest
version, so the served HTML requests /apps_u/get_data/v/<secret>.{js,css}
for a version that has no stored bundle -> 404 and a white screen.
Manually redeploying fixes it until the next merge re-triggers the revert.
Two changes:
- Re-query the current latest version to relock (mirrors the flow
dependency handler, #8673), so we don't lock a stale snapshot.
- Guard the re-publish append with `versions[array_upper(...)] = $1` so
it is a single atomic, never-demoting statement: it can only re-append
the version that is already latest, never revert to an older one. A
relock never creates a new app_version, so there is never a version to
legitimately promote here.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: support $f/ and $u/ import path aliases for scripts
$f/ and $u/ are local-friendly aliases for the absolute workspace
import paths /f/ and /u/. Unlike the /-prefixed form (which local tools
treat as a filesystem-root path), the $-prefixed form is a bare specifier
that can be remapped via tsconfig paths / Deno import maps, so the same
import resolves on the Windmill worker and in a local editor.
- worker: recognize $f//$u/ in the Deno import map and both Bun loaders
- dep-map/parser: normalize $f/->f/, $u/->u/ for lockgen + dep tracking
- cli: emit $f/$u path aliases in generated tsconfig.json / deno.json
- frontend: ATA + Monaco paths resolve $f//$u/ type hints in the editor
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): split generated tsconfig into managed + user file with refresh command
Mirror the AGENTS.cli.md/AGENTS.md prompts model for the IDE tsconfig so the
recommended settings can evolve without ever clobbering user customizations:
- tsconfig.wmill.json: wmill-managed, always refreshed, holds recommended
compilerOptions incl. the $f/$u path aliases (Deno: import_map.wmill.json)
- tsconfig.json: user-owned, created once, just extends the managed file;
warn (never auto-edit) when an existing one doesn't reference it
- add 'wmill refresh tsconfig'; init generates it unconditionally (no longer
gated behind resource-type namespace / a bound workspace)
- regenerate CLI guidance docs for the new subcommand
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): address PR review on $f/ tsconfig generation
- handle existing deno.jsonc so we don't shadow it with a new deno.json
(P1 identified by cubic)
- fix the bun-types hint that pointed users at the managed do-not-edit
tsconfig.wmill.json; tell them to install + re-run 'wmill refresh tsconfig'
- document the .ts-extension-only local-resolution limitation (cross-flavor
.bun.ts/.deno.ts/.fetch.ts scripts won't resolve in a local editor)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): warn when a project's tsconfig isn't wired to tsconfig.wmill.json
Mirror the prompts freshness check for the managed tsconfig so users with an
existing setup actually discover they're missing $f//$u/ resolution:
- embed a version hash in tsconfig.wmill.json (excludes the env-dependent
bun-types 'types' entry so it doesn't false-positive)
- add warnIfTsconfigStale to the main.ts freshness hook, gated identically to
the prompts check (skips init/refresh/help/version). When a tsconfig.json
exists it warns one line (stderr) if the managed file is missing, not
referenced via extends, or out of date; silent for non-TS projects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cli): make tsconfig setup equivalent to prompts (auto-wire + stale-only)
Unify the two managed-file systems so they behave identically:
- auto-wire an existing unlinked tsconfig.json/deno.json on init/refresh
(add extends / importMap; merge into an array extends), instead of only
warning. Parses JSON and falls back to a warning when it can't round-trip
(JSONC comments, or a conflicting deno imports/importMap) — never corrupts.
- narrow warnIfTsconfigStale to stale-only, gated on the managed file
existing, exactly like warnIfPromptsStale: it no longer nags about a
missing or unlinked tsconfig.json, so a deliberately-custom/unlinked setup
stays silent and a not-yet-initialized project isn't bothered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): place tsconfig.wmill.json first in extends to preserve user base config
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(cli): migrate legacy tsconfig and require consent for custom configs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(cli): align prompts wiring to the same consent model
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(cli): bump windmill-parser-wasm-ts to 1.714.0 for $f/ $u/ aliases
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(worker): resolve $f/ and $u/ in deno lock generation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: narrow relative-imports lock-gen guard to deno import-map failure
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(cli): sync bun.lock with windmill-parser-wasm-ts 1.714.0
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): warn when a custom tsconfig's paths would shadow $f/ $u/ aliases
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): resolve MCP resource token via caller RLS + SSRF-guard url
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): clone user_db for oauth2 refresh and drop advisory ids from comments
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): disable redirects on MCP client to prevent SSRF bypass
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#9393 added `--compile-bytecode` to the uv pip install run inside the
python download nsjail. uv spawns a Python interpreter that compiles .py
files with parallelism scaling to the host CPU count, opening many file
descriptors at once. The download nsjail capped `rlimit_nofile` at 64,
which is exhausted on high-core machines, failing every install with
"Failed to bytecode-compile ... Too many open files (os error 24)".
Low-core VMs never hit the cap, so this surfaced only as a regression on
larger workers after upgrading.
Raise `rlimit_nofile` to 10000, matching the runtime configs
(run.python3 / run.ansible) that already use that value.
Fixes WIN-2009
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Python jobs under nsjail experience slow imports (~465ms for `import
requests`) because dependency directories are mounted read-only. Without
pre-compiled `.pyc` files, Python recompiles `.py` source to bytecode
in-memory on every import in every fresh nsjail process, paying the cost
repeatedly.
Add `--compile-bytecode` to both `uv pip install` invocations (the
Rust-driven install in python_executor.rs and the nsjail
download_deps.py.sh script) so `.pyc` files are generated at install
time and available at runtime even through read-only mounts. The files
are included in both the local cache and S3 piptar uploads.
The flag is supported by the uv version (0.9.24+) shipped in the
Dockerfile. The existing `__pycache__` skip only filters top-level dirs
for dedup logic, not subdirs within packages, so there is no conflict.
Fixes WIN-2001
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flows): preserve step/subflow worker tags under a custom-tagged flow
A flow running on a custom worker tag force-propagates that tag to every
descendant step, script and nested sub-flow, overriding their own declared
tags. This made it impossible to route a specific step or sub-flow to a
different worker group. The new opt-in FlowValue.preserve_step_tags lets a
step that declares its own non-empty tag run on it; untagged steps still
inherit the flow tag. Defaults off to preserve existing behavior.
* chore: regenerate system prompts for preserve_step_tags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(flows): nest preserve_step_tags toggle under flow worker tag setting
The toggle only affects routing when the flow has a custom worker tag, so
show it as a sub-setting of the Worker Group tag picker, visible only once a
tag is set, instead of as a standalone option.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): allow step worker tag picker when preserve_step_tags is enabled
When a flow defines a worker tag, the per-step tag picker was replaced by a
read-only "Flow's WG" label. With preserve_step_tags enabled the step's own
tag is honored, so the picker must remain editable in that case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): propagate preserve_step_tags to branch and loop bodies
payload_from_modules built the synthetic RawFlow for branch/loop bodies with
a default FlowValue, dropping preserve_step_tags. Tagged steps inside a
branch or loop therefore still inherited the parent flow tag even with the
flag enabled. Thread the flag through to the synthetic FlowValue so the
behavior is consistent for nested containers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags when flow worker tag is removed
Avoids the flag lingering as invisible state after the flow tag (and its
toggle) are removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): repair preserve_step_tags propagation to branch/loop bodies
The previous commit added flow.preserve_step_tags at the payload_from_modules
call sites but the parameter and FlowValue field were not actually threaded
through (a failed edit left the function unchanged), so the crate did not
compile. This completes the change: payload_from_modules takes preserve_step_tags
and sets it on the synthetic FlowValue for branch/loop bodies.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): complete preserve_step_tags propagation to branch/loop bodies
Previous two commits left windmill-worker uncompilable: payload_from_modules
received flow.preserve_step_tags at its call sites but the parameter and the
synthetic FlowValue field were not actually added. This adds the parameter,
sets preserve_step_tags on the synthetic FlowValue, and threads
flow.preserve_step_tags through all five call sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): clear preserve_step_tags whenever the flow worker tag is removed
The flag was only reset when the Worker Group toggle was switched off, not
when the tag was cleared directly in the picker (or via the YAML editor),
leaving preserve_step_tags=true as invisible state with the advanced badge
still reporting it active. Move the cleanup into the reactive block that
already tracks the flow tag so every clear path is covered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (WIN-1988)
`tokio_tungstenite::connect_async` opens a raw TCP socket and ignores
the standard outbound-proxy env vars, so deployments behind a forward
HTTP proxy can't reach the WebSocket endpoint and Test Connection
times out after 30s.
Add a small `proxy` module that resolves the right proxy URL for the
target host (HTTPS_PROXY for wss://, HTTP_PROXY for ws://, NO_PROXY
exclusions, ALL_PROXY fallback, lowercase variants), opens an HTTP
CONNECT tunnel when one applies, and hands the resulting TcpStream to
`client_async_tls_with_config` for the TLS + WS handshake. Direct
connect remains the default when no proxy env is set.
Unit tests cover NO_PROXY matching, proxy URL parsing (including IPv6
literals and basic-auth userinfo), and the CONNECT handshake itself
against an in-process fake proxy (success, basic-auth header, 407
rejection).
Fixes WIN-1988
* refactor(websocket-trigger): reduce blast radius and reuse existing logic
Follow-up to the proxy support change. Three things:
1. Skip the new code path entirely when no proxy is configured.
`connect_async_with_proxy` now checks the env-var snapshots up front
and delegates straight to `tokio_tungstenite::connect_async` if
neither `HTTP_PROXY` nor `HTTPS_PROXY` is set. Same fall-through
applies when proxy env is set but `NO_PROXY` excludes the host or
the proxy URL doesn't parse. Non-proxied deployments now exercise
exactly the previous code path.
2. Move the `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` env-var snapshots
from `windmill-worker::worker` into `windmill-common`. The worker's
`PROXY_ENVS` static now reads from there, and the websocket trigger
reads from the same source — one place reads the env, one source
of truth for both call sites.
3. Replace the hand-rolled proxy-URL parser with `url::Url::parse`
(already a workspace dep, used across the codebase). Half the LoC
and handles edge cases (userinfo percent-encoding, IPv6 literals,
path/query stripping) via the well-tested crate instead of by hand.
All 13 proxy unit tests still pass. `cargo check` is clean.
* fix(websocket-trigger): unbreak EE build + trim proxy tests
- Re-export `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` from
`windmill-worker::worker` (via `pub use windmill_common::...`) so the
EE `otel_tracing_proxy_ee` module's `use crate::{HTTPS_PROXY, ...}`
resolves like it did before. Fixes the `check_ee_full` / `cargo_test`
CI failures from the previous commit.
- Trim the proxy tests to one un-ignored canary
(`http_connect_tunnel_sends_well_formed_request_and_unwraps_stream`)
that exercises the actual on-wire CONNECT handshake plus byte-perfect
tunnel passthrough. The NO_PROXY-matching, URL-parsing, and edge-case
tunnel tests are kept under `#[ignore]` for manual debugging
(`cargo test -- --ignored`) since they're either delegated to
`url::Url::parse` or trivial string matching — low ROI on every CI run.
The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls
std::os::unix::fs::symlink directly, which doesn't exist on Windows
targets. Without a cfg gate, `cargo check --tests` fails on Windows
with E0433. Other symlink call sites in this crate (php_executor,
bun_executor, rust_executor, etc.) already follow this pattern.
Fixes WIN-1972
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(nsjail): optional disk-backed /tmp via instance setting
* test(nsjail): unit-test tmp mount resolver and narrow visibility
* refactor(nsjail): switch tmp backing to select + conditional UI
* ui(nsjail): make tmpfs the visible default in /tmp backing select
* fix(nsjail): refuse preexisting jail_tmp to block symlink escape
* fix(nsjail): allow jail_tmp reuse on sequential nsjail calls
Codex flagged that python/ruby/rust executors invoke nsjail twice per
job_dir (install then run). The previous resolver treated any preexisting
jail_tmp as hostile and silently fell back to tmpfs on the second call,
so disk-backed mode never reached the main script run for those langs.
Use symlink_metadata().is_dir() to distinguish a real directory left by
an earlier call in the same job_dir (safe to reuse) from a symlink or
other entity (still refused, as the codebase-tar escape requires).
Also loosen the frontend visibility predicate: only hide nsjail settings
when job_isolation is explicitly 'none' or 'unshare', so deployments
that enable nsjail via DISABLE_NSJAIL=false with no DB setting can
still see the controls.
* feat: add UV_PYTHON_INSTALL_MIRROR env and instance setting
Allows operators to point `uv python install` at a private mirror of the
python-build-standalone releases. Configurable via the
`UV_PYTHON_INSTALL_MIRROR` env var or the `uv_python_install_mirror`
instance setting, with the env var as the boot fallback and the instance
setting taking precedence at reload.
Fixes WIN-1966
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: hoist uv_python_install_mirror binding above sandboxing branch
The non-sandboxed uv pip install branch referenced a binding that was
only declared inside the sandboxed branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: neutral placeholder for uv_python_install_mirror
The previous placeholder was the default public URL the setting is meant
to redirect away from. A neutral example mirror URL is clearer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cli): add datatable list and run commands
* feat(cli): render datatable query results as a table
* feat(cli): serve datatables as a postgres-wire endpoint
* feat(cli): add 'datatable psql' to launch psql against the proxy
* feat(cli): route datatable serve by client-supplied database name
* override database list + password option
* fix: support extended queries in datatable serve
* fix: correct cloud size threshold log and parse CLI descriptions with parens/trailing comma
* refactor: extract raw_output envelope encoding into pg_raw_output module
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
* feat(nsjail): make tmpfs size configurable via instance setting
Adds a new `nsjail_tmpfs_size_mb` instance setting that overrides the
size of the `/tmp` tmpfs mount inside the nsjail sandbox across all
languages. When unset, the existing per-language defaults (500MB or
800MB) continue to apply, so no behavior change for existing
deployments.
The setting is exposed under Settings → Jobs and is read at job
execution time, so changes take effect on the next job without a
restart.
Fixes WIN-1963
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(nsjail): unify default tmpfs size to 800MB
Previously each executor passed its own per-language default (500MB or
800MB) to resolve_nsjail_tmpfs_size. Unify on a single
DEFAULT_NSJAIL_TMPFS_SIZE_BYTES constant (800MB) so the placeholder
behavior is consistent across languages.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(nsjail): resolve tmpfs size outside ruby download closure
The download.ruby config render runs inside a sync closure passed to
par_install_language_dependencies_seq, so `.await` on
resolve_nsjail_tmpfs_size() was a compile error under the `ruby`
feature. Resolve the size once before the closure and capture the
string instead.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(nsjail): rename resolver to *_bytes and clarify fallback
Addresses CI review feedback:
- Rename `resolve_nsjail_tmpfs_size` to `resolve_nsjail_tmpfs_size_bytes`
so the returned unit is unambiguous at the call site (cubic P2).
- Fix the `NSJAIL_TMPFS_SIZE_MB` doc comment that still said "per-language
default" — there is no per-language fallback anymore, all unset
values resolve to the unified 800MB `DEFAULT_NSJAIL_TMPFS_SIZE_BYTES`
(codex/pi P2).
- Expand the resolver doc to call out that `Some(0)` and negative values
also fall back, since the match arm is `Some(mb) if mb > 0`.
No behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: thread temp_script_refs into preview jobs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: resolve python preview relative imports from temp script refs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: use local relative imports in wmill script preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: use local relative imports in wmill flow preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add temp_script_refs to Preview and FlowPreview openapi schemas
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: pass temp_script_refs to bun lockfile gen for no-lock preview
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: route script preview through shared buildPreviewTempScriptRefs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: resolve local relative imports in wmill app dev inline scripts
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address cubic review — bundle cache key, preview-mode gate, error masking
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf: skip dep-tree build when previewed script has no relative imports; narrow old-backend classifier
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address review issues (preview-only gate, bundle preview, app dev cwd)
Three P1s flagged in repeated codex/pi reviews on PR #9233:
- Gate _TEMP_SCRIPT_REFS extraction on JobKind::Preview (bun + python
executors) and propagation in worker_flow on JobKind::FlowPreview. job.args
includes caller-controlled request args, so honoring this key on deployed
runs would let a caller swap import resolution to local content uploaded
via /raw_temp.
- run_bundle_preview_script now injects temp_script_refs into PushArgs.extra,
mirroring run_preview_script — closes the silent data drop for the bundle
preview path.
- wmill app dev chdirs to the wmill.yaml root before buildPreviewTempScriptRefs
and restores after, so the `cd <app>__raw_app && wmill app dev` invocation
(cwd is the raw_app folder, no app_folder arg) still walks sibling workspace
scripts like f/lib.ts.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 5b347d6 (handle_python_deps arity fix)
Picks up the EE arity fix so cargo_test + check_ee_full compile cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 52e273d (agent-workers bundle path arity fix)
Picks up windmill-ee-private 52e273d which adds the missing &None arg to
compute_bundle_local_and_remote_path in windmill-api-agent-workers/src/ee.rs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ee): bump ee-repo-ref to 2d6ffd3 (EE main merged in)
Previous bump pinned an older EE commit, missing the audit-log object-store
export module (EE PR #579, commit ec3cd35) and other EE main updates. The
CE backend's `crate::ee_oss::anchor_audit_logs_s3_checkpoint_env_var` and
`export_audit_logs_to_object_store` references need the new EE definitions.
Merged origin/main into the EE branch and pinned the merge commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b0c87b1272c25dca4aa9148c87fb024a9d9ef322
This commit updates the EE repository reference after PR #583 was merged in windmill-ee-private.
Previous ee-repo-ref: 2d6ffd32c99bd93e79cf78675cb89499a81b17e1
New ee-repo-ref: b0c87b1272c25dca4aa9148c87fb024a9d9ef322
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(snowflake): derive public key from private key when omitted (WIN-1959)
Snowflake key-pair auth needs a SHA256 fingerprint of the public key for
the JWT iss claim, but the public key is mathematically derivable from
the RSA private key. Other tools (e.g. Power BI) only require the
private key, so requiring users to supply both is redundant. When
public_key is missing, fall back to deriving it from private_key (PKCS#8
or PKCS#1 PEM) instead of erroring out.
Fixes WIN-1959
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(snowflake): treat empty public_key/private_key as missing
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: bind /etc resolver files read-only in nsjail sandboxes
* docs(nsjail): explain why per-file /etc resolver binds are load-bearing
The explicit /etc/hosts, /etc/resolv.conf and /etc/hostname binds look
like removable duplication of the read-only /etc bind above them. They
are not: on Kubernetes those files are separate kubelet bind-mounts on
top of /etc and nsjail's read-only remount is non-recursive, so without
these shadow binds they stay writable and a job can persist cross-tenant
DNS poisoning for the pod lifetime. Comment guards against a future
"dedup cleanup" silently reintroducing the vulnerability.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(nsjail): shorten the load-bearing-bind comment to 3 lines
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix: atomic bundle cache writes to prevent parallel cold-load race
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: trust-but-replace in atomic_publish_dir to never trust a stale partial cache dir
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: simplify atomic_publish_dir and add content-addressed rename-failure fallback
Revert the destroy-then-recreate dir swap (introduced concurrent-publisher
edge cases: spurious Err under a real herd, EACCES masking a stale partial)
back to a single atomic rename. Add the content-addressed exists-fallback to
atomic_write_file_bytes/atomic_copy_file so the loser of a publish race (and
Windows, where rename cannot replace an open/existing destination) treats the
already-published identical cache as success instead of failing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: aggregate wait time should target the true root job, not flow_innermost_root_job
* refactor: reuse get_root_job_id helper for wait-time aggregation
Instead of duplicating the root_job → flow_innermost_root_job →
parent_job fallback chain inline, call the existing get_root_job_id()
helper (the same one used by push_next_flow_job) and filter out the
self-id case so standalone scripts still skip aggregate insertion.
Behaviorally identical, single source of truth.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: windmill-internal-app[bot] <1429786+windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(python): preserve strings containing Infinity/NaN in result JSON
* test(python): add sanity checks for Infinity/NaN in results
* refactor(python): use string-aware regex callback for single-pass cleanup
* refactor(python): compact regex + handle backslash-escape parity
* perf(python): short-circuit cleanup when no Infinity/NaN/NUL in result
* fix(bun): pass --preserve-symlinks on unbundled execution
Bun 1.2/1.3 moved its global package cache to a content-addressed
layout and the installer now creates a single directory symlink from
node_modules/<pkg> to the cache entry. Without --preserve-symlinks,
Bun resolves modules from each file's realpath, so any require/import
inside an installed package walks up from cache_nomount/bun/... and
never finds the sibling deps living under <job_dir>/node_modules/.
This manifested as e.g.
ENOENT while resolving package 'zod/v3' from
'/tmp/windmill/cache_nomount/bun/@langchain/core@1.1.44@@@1/dist/...'
on //nobundling scripts that pull @langchain/core, even though zod is
correctly installed alongside it in node_modules.
The bundled execution path already had --preserve-symlinks since #4132
(needed because we symlink the cached bundle file into the job dir).
The unbundled path didn't, because at the time Bun installed via per-
file hardlinks and the realpath of node_modules entries was the job
dir itself. The Bun installer's layout change made the flag necessary
on the unbundled path as well.
Add the flag to all three unbundled `bun run` invocations:
- nsjail unbundled path
- non-nsjail unbundled path
- dedicated worker (always unbundled)
This also fixes a latent bug on the first run of any bun script that
imports a package whose internals reference siblings (the build_cache
path runs unbundled this round while it builds the bundle for next
time).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(bun): regression test for nobundling + transitive require resolution
Adds an integration test that mirrors the original failure: a //nobundling
script importing @langchain/core, which (in its CJS internals) does
require('zod/v3'). Before --preserve-symlinks was added to the unbundled
bun run invocations, this failed with:
ENOENT while resolving package 'zod/v3' from
'.../cache_nomount/bun/@langchain/core@<ver>@@@1/dist/runnables/base.js'
The test covers the non-nsjail unbundled path. Reproducibility of the
pre-fix failure depends on Bun's installer choosing the directory-symlink
layout for the node_modules entry (the default on Bun 1.2/1.3+ with the
new content-addressed global cache that produced the user's error).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: refine windmill ai refactor plan
* refactor: move ai sse plumbing to windmill-ai
* refactor: remove ai re-export shims
* fix: update ee ai memory ref
* chore: update ee-repo-ref to d3bc7fa85195b46b7a38d43c2f806520bf8b5454
This commit updates the EE repository reference after PR #560 was merged in windmill-ee-private.
Previous ee-repo-ref: ff35bf7cc198e13884b33654e1d6dbd8a8b314d3
New ee-repo-ref: d3bc7fa85195b46b7a38d43c2f806520bf8b5454
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The Python per-package dependency cache could persist an incomplete wheel
extraction with `.valid.windmill` set, then propagate that broken artifact
to every worker through the object store. Customer hit this on
argon2-cffi==25.1.0 (missing argon2/_utils.py), and previously on
botocore/httpx (truncated tars). Symptom is a runtime ImportError that
looks like a missing dependency declaration rather than a Windmill bug.
Three changes that together stop the propagation:
1. After `pull_from_tar`, parse the wheel's `<dist-info>/RECORD` and
confirm every listed path exists on disk before writing
`.valid.windmill`. On failure, wipe the directory and fall through
to a fresh local install — the next install also self-heals the
broken object-store entry by pushing a fresh tar.
2. After `uv pip install` succeeds, run the same RECORD check before
queuing the piptar upload or writing `.valid.windmill`. A bad install
never becomes the source of a broken tar in the object store.
3. Finalize the tar (`drop(tar.into_inner()?)`) before reading its bytes
for upload, so we never push an unfinalized archive (no end-of-archive
marker) to the object store.
Verified with a 60-package end-to-end integration test (first-fill →
clear-local-cache → re-pull-from-objectstore → corrupt-objectstore-tar
→ detect-and-self-heal). All 27 packages on the live test pulled cleanly,
and the deliberately corrupted argon2-cffi tar was caught with the exact
expected log line ("wheel RECORD lists files missing on disk: argon2/_utils.py")
and replaced with a fresh tar.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: parse windmill_failure field in job result to tag run as failure
* feat: preserve top-level fields when windmill_failure tags a run as failure
* fix: address review findings on windmill_manual_failure
* refactor: rename windmill_manual_failure to wm_failure and add wm_* aliases
* fix: prefer injected ManualFailure error over sibling name/message in OTel
Ruben Fiszel
Amey Pawar
hugocasa
centdix
Diego Imbert
windmill-internal-app[bot]