The CLI docs generator anchored on the first `new Command()` in a file and
never resolved locally-defined command groups passed as
`.command("name", localCmd)`. For datatable this flattened the nested
`migrate` group: it emitted `datatable new/up/down` plus a bare
`datatable migrate`, and mislabeled the datatable command with the migrate
group's description. jobs was broken the same way (its description was pull's,
and pull/push rendered empty).
Anchor block extraction on the `export default`ed command, recurse into
locally-defined `const x = new Command()` groups mounted as subcommands, and
render nested sub-subcommands. Regenerated docs now show
`datatable migrate new/up/down` and `jobs pull/push` with their real
options.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai): add Azure AI Foundry as a native AI provider
Adds `azure_foundry` as a new AIProvider variant wired through the AI
chat (copilot) and AI agent flow steps. Foundry's chat completions API
is OpenAI-compatible and uses Azure conventions (api-key header, Azure
URL building), so it reuses the existing OpenAI-compatible query builder
and proxy path via the shared `is_azure` helper (renamed from
`is_azure_openai`).
Backend (windmill-ai):
- New `AzureFoundry` enum variant (serde `azure_foundry`)
- `get_base_url` requires a resource base URL (like Azure OpenAI / Custom)
- `is_azure()` covers Azure OpenAI + Foundry (api-key auth, Azure URL)
- Added to OpenAI-compatible proxy support and HttpForward proxy mode
- New proxy URL unit test
Frontend (copilot):
- New provider entry, completion config, model-token handling, streamed
usage tracking, and reasoning registry (all model-id-gated, so a no-op
for Foundry's non-OpenAI catalog)
- Treated as a chat-completions provider, not the OpenAI Responses API
OpenAPI:
- `azure_foundry` added to AIProvider (openapi.yaml) and AIProviderKind
(openflow.openapi.yaml); regenerated CLI guidance
Note: the `azure_foundry` resource type (base_url + optional api_key) is
hub-managed and must be published to the Windmill Hub separately.
Fixes WIN-2122
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai): add azure_foundry to copilot flow Zod provider enum
The tracked copilot flow schema (openFlowZod.gen.ts and its openFlow.json
source) still carried the old AIProvider enum, so validateFlowModules /
validateSpecialFlowModule rejected AI-generated flow edits that create or
update an aiagent module with provider kind "azure_foundry" before they
could be saved. Add the value to both (preserving the generated single-line
format) and a regression test over the flow-module validation path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai): lead provider list with OpenAI, Anthropic, Google AI
Reorder AI_PROVIDERS so the three primary direct providers come first. The
AIProviderPicker renders the first three entries as quick-access buttons, so
these become the defaults (previously OpenAI, Azure OpenAI, Azure Foundry);
Azure OpenAI / Azure Foundry stay adjacent right after. No logic depends on
provider order (only per-provider defaultModels[0] is read).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scope SCD2 built-in data tests to current rows
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: add --partition to pipeline run and fix duckdb s3object upload binding
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: note filesystem storage type is dev-only in storage settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: use ISO week for weekly partition default in pipeline run
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(duckdb): auto-declare the partition arg for // partitioned scripts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): pipeline run --arg to pass plain run args to cascade scripts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(cli): correct misleading delete-fork command description
The `wmill workspace delete-fork` description claimed it deletes "a
forked workspace and git branch", but the implementation only deletes
the Windmill workspace via the backend API and removes the local
workspace profile. No git operations are performed, so the remote
branch is left untouched. Drop the "and git branch" clause and
regenerate the derived guidance/system-prompt files.
Fixes WIN-2120
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cli): permanently delete temp workspaces in folder test cleanup
The isolated-workspace test helper archived each temp workspace on
teardown. After #9865 added a CE cap of 1 archived workspace, the second
archive-cleanup is refused, so temp workspaces leak into the active set
and hit the 2-workspace CE cap — failing every subsequent create/fork
across the shared test backend.
Permanently delete the workspace instead (DELETE /api/workspaces/delete),
which frees the slot without occupying the archived quota.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat(pipeline): local development for data pipelines (CLI --local + pipeline dev preview)
Add the local edit→preview→run loop for data pipelines (folders of `// pipeline`
scripts), the analog of `wmill dev` / `wmill app dev`, usable from a code editor
or an agentic loop — without deploying.
No backend changes: full body inference comes from the same wasm the frontend
uses (windmill-parser-wasm-asset), which returns assets + pipeline annotations in
one call; local runs reuse runScriptPreview with _wmill_skip_asset_dispatch.
- localGraph.ts: wasm-backed working-tree → asset-graph builder (the enabler)
- pipeline show/run --local; new pipeline docs (PIPELINE.md/AGENTS.md) subcommand
- pipeline dev watcher + /pipeline_dev page (PipelineDevView) rendering the same
PipelineGraphEditor from the pushed local graph, run via preview
- cascadeRun.ts: reusable run primitives extracted from the route page
- regenerated CLI agent docs
See docs/pipeline-local-dev.md for the full design, test steps, and handoff TODOs.
The live `pipeline dev` browser preview is implemented but not yet stack-verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): improve local dev preview (run, activity, responsive)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): dev-preview args, multi-root run, ws auto-reconnect
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): connect managed-materialize producer in local dev graph
The CLI pinned windmill-parser-wasm-asset ^1.728.1, which predates managed-materialize support (added in 1.733.1); the frontend already pins 1.740.0. The CLI's wasm therefore never emitted `// materialize`, so the producer had no output edge and showed disconnected from its `// on` consumers. Bump the CLI to 1.740.0 (matching the frontend) and translate the parsed materialize target into the producer's write edge + materialize_target, mirroring frontend resolveGraph.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): harden local-dev CLI (bare-.sql crash, defaultTs, docs clobber)
Review fixes, complementary to the dev-preview/materialize/multi-root work already
on the branch (none overlap those commits):
- localGraph: a bare `.sql` (no dialect) made inferContentTypeFromFilePath throw and
abort the whole graph build — and wedge `pipeline dev` at startup. Skip the
unclassifiable file instead. Also map `bunnative` → parse_assets_ts and add
ruby/rlang/nu/powershell to the `#`-comment fallback.
- show/run/docs/dev: thread the resolved `wmill.yaml` defaultTs into the graph
builder so `.ts` infers under the workspace's runtime (bun vs deno) instead of
always bun — `opts.defaultTs` was always undefined (no such CLI flag).
- dev: wrap the startup graph build so a half-written file can't abort the watcher.
- docs: don't clobber a user-authored AGENTS.md/CLAUDE.md — only (over)write the
pointer when absent or already a generated `@PIPELINE.md` pointer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): bind dev WS to loopback + local-graph regression tests
- pipeline dev WS broadcast the folder's full script source (scripts[].content + temp_script_refs) unauthenticated on 0.0.0.0:3201 — bind 127.0.0.1 so it's not LAN-reachable (webview localhost + SSH/devbox port-forward still work).
- Add regression tests for the just-landed local-graph fixes: bare .sql is skipped (was a build/dev-startup crash), defaultTs threads into .ts runtime inference (bun vs deno), and #-comment languages (ruby) use the # annotation fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): --frontend flag for pipeline dev page origin
wmill pipeline dev opens <remote>/pipeline_dev, but that route only exists in this build's frontend, so it 404s against a remote whose deployed frontend predates it. --frontend <origin> points the page at a locally-run frontend (REMOTE=<remote> npm run dev) while the API/token still target the remote — enabling the live preview against a real backend before the PR is deployed. No behavior change when omitted. Regenerated CLI agent docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): WS session token + details-pane live-reload refresh
Addresses CI review (Codex/Pi/Claude):
- dev WS: a browser tab could open ws://localhost:<port>/ws and receive the folder's full source (browsers don't enforce same-origin on WS, loopback bind alone doesn't help). Gate the upgrade on an unguessable per-session token carried in the dev-page URL (verifyClient → 401 without it). Verified: no-token/bad-token connections get 401 with no bundle.
- details pane: scriptRes keyed on [workspace, selection, draftScript] didn't re-run on a pipeline dev live-reload (same selection), so the open pane showed stale source. Thread a localScriptsVersion (the pushed bundle) into the key. Verified: editing a selected node's file updates the pane source without reselect.
- docs/pipeline-local-dev.md: refresh the stale 'not yet exercised' status + done TODOs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): emit volume: annotation assets in local dev graph
Addresses CI review (Codex P1 / Pi P1): the wasm body parser doesn't surface `// volume: <name>` annotations — the frontend (infer.ts:parseVolumeAnnotations) and backend (asset_inference.rs) parse them separately and merge as rw volume assets. localGraph didn't, so a `# volume: cache` producer had no write edge and showed disconnected from its `// on volume://cache` consumer (and pipeline run --local wouldn't schedule downstream). Mirror the leading-comment-block scan (SQL excluded, matching both reference parsers) and merge into inferScriptAssets. Regression test added; verified producer -> volume://cache -> consumer connects.
Also (Codex P2): docs/pipeline-local-dev.md manual browser URL omitted the new ws_token param — without it the WS upgrade is rejected and the page sits disconnected. Doc now says to copy the URL the CLI prints (carries wm_token + ws_token) and recommends --frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): runAll excludes event roots + review polish
Addresses CI review (Codex P1, Claude P2/P3):
- pipeline run runAll: derive the whole-pipeline selection from validStarts + descendants instead of all runnables, so an unqualified 'pipeline run <folder>' no longer fires event-trigger roots (kafka/mqtt/…) with empty args/side effects. Verified: a kafka root is excluded from the plan.
- cascadeRun.ts runBoundedCascade: use buildLineageDownstreamMap (read-aware) so a pure-reader runs after its producer, and return cyclic — parity with the route page's bounded run (the file is meant to be THE shared correct primitive).
- PipelineGraphEditor: storedRightPaneSize starts at 0 so the orientation-aware default (55% stacked / 40% side-by-side) actually applies on first open.
- localGraph fallbackParse (go/bash): scan only the leading comment header (no body-comment phantom triggers) and strip key=value options from the asset URI; regression test added.
- docs: reject '..' in the folder arg (it writes files under f/<folder>).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): route local previews to the // tag worker
Addresses CI review P1: the local graph/bundle dropped the parsed `// tag`, so a node annotated `// tag gpu` ran on the default worker in both `pipeline run --local` and `/pipeline_dev`, while the deployed pipeline routes it to that worker tag. Carry the tag through LocalScript / the pushed bundle / LocalScriptContent and pass it to runScriptPreview at all three launch sites. Verified: a duckdb node tagged `bash` produces a job tagged `bash`; regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): add asset partitions/schemas routes to OpenAPI, use generated client
The ducklake asset panels (PartitionStatusGrid, SchemaHistoryPanel) hit /assets/partitions and /assets/asset_schemas via raw fetch with cookie-only auth, because those backend routes were never added to openapi.yaml so the generated client had no methods for them. On /pipeline_dev (token-via-URL, no session cookie) the raw fetches 401'd. Add both GET routes + MaterializedPartition/AssetSchemaVersion schemas to openapi.yaml and call them through AssetService, which injects the bearer token, types, and cancellation automatically. Verified: Partitions + Schema tabs load in /pipeline_dev. (backfill stays a raw fetch — it's an EE-only route not in the OSS spec — with the token added inline.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(cli): regenerate bun.lock for windmill-parser-wasm-asset
package.json / package-lock.json carry windmill-parser-wasm-asset@1.740.0 but the tracked bun.lock (the CLI installs/builds/tests via bun) was stale, so fresh bun installs would resolve a different graph than the committed lock. Regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): show asset producer + its runs in the dev-preview panel
Selecting a ducklake/asset node in /pipeline_dev showed 'No producer for this asset' because selectionProducers wasn't passed (it's derived from the deployed graph on the route page, absent here). Compute it from the local graph's w/rw write-edges (incl. the // materialize target) and pass it through, mirroring the route page — so the panel shows the producing script and its (preview) runs, including data-test failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): carry annotation metadata onto local-graph runnables
The local graph emitted only path/usage_kind/in_pipeline/materialize_target per runnable, so /pipeline_dev and pipeline show --local weren't the same surface as the deployed graph for annotated scripts — missing the badges/lineage the shared canvas renders. Map the wasm-parsed partition_kind, freshness, tag, retry, data_tests, column_lineage, and materialize_strategy (derived append/merge/replace) onto each runnable, mirroring the deployed AssetGraphRunnableNode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): exclude event handlers that are lineage descendants from runAll
The runAll guarantee ('never fires an event handler with empty args') only held for event ROOTS — validStarts excludes them, but runAll then unions in descendants(dag, start), so a kafka/mqtt/... handler that also reads an upstream pipeline asset (a lineage descendant of a valid start) still landed in the plan. Add eventTriggerScripts() and subtract it from the selection after the descendant union. +unit test.
Also: docs/pipeline-local-dev.md recipe used 'pipeline docs demo_pipeline' without --local (default queries the deployed graph → hits the empty hint); add --local.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): whole-pipeline run cuts at event handlers (drop their downstream too)
The prior runAll fix subtracted event handlers from the selection but left their downstream: for manual_root → asset_x → kafka_handler → asset_y → consumer, deleting only kafka_handler left consumer selected, and topoOrder then ran it as a root with missing/stale event-derived inputs. Replace the descendant-union+delete with reachableCutting(dag, validStarts, eventHandlers): traverse from valid starts but treat event handlers as cut points, so a node reachable ONLY through an event handler is dropped while one reachable via a non-event path stays. +unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): recover // tag in the go/bash annotation fallback
The wasm path carries out.tag, but the go/bash fallback (and the wasm-error degradation path) only recovered pipeline + on, so a // tag gpu on a bash/go node — or a temporarily-unparseable ts/py/sql node — silently routed the local preview to the default worker while the deployed pipeline routes to the tag. Scan for // tag in fallbackParse too. +test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): extract shared assetProducers helper
The 'who writes this asset' write-edge derivation was copied verbatim in PipelineDevView and the pipeline route page — two copies that would drift. Extract assetProducers(graph, selection) into graphTraversal.ts and use it from both, keeping the dev view and route page in lockstep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): only overwrite AGENTS.md/CLAUDE.md when it's the exact generated pointer
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): wire local-dev runs into the selected-node runs pane
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): exclude data_upload/webhook entrypoints from auto CLI runs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): --upload binds an object to a data_upload/webhook entry point
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): add "Run + downstream" to the dev preview detail form
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): cut non-autorun triggers on all run paths; multi-binding --upload
Address CI review: apply the data_upload/webhook/event barrier cut to the
single-root and bounded (--from/--to) paths, not just whole-pipeline; accumulate
repeatable --upload bindings per script (were overwritten); scope dev upload keys
by script+param to avoid basename clobbering; drop <script> from help text.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): reseed dev run form when a local edit changes the script's args
The read-only pane is keyed on script.path only, so in /pipeline_dev the selected
node re-resolves on every WS bundle without remounting; PipelineScriptView cloned
script.schema once, so adding/removing args left the run form on a stale schema
(could run with missing inputs). Extract PipelineRunForm (owns the SchemaForm
clone) and key it on the serialized schema: a real arg change reseeds the form,
an unchanged re-resolve keeps in-progress input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): don't cut a scheduled/manual root that also has a non-autorun trigger
Address Codex P1: the barrier set subtracted only --upload-bound scripts, so a
script with both `// on schedule` and `// on data_upload` resolved as the start
yet was also a barrier — reachableCutting skipped it, giving an empty run plan.
Subtract all valid starts (schedule/manual roots + bound handlers) from barriers:
a legitimately-scheduled root runs on its schedule path even if it also carries a
caller-input trigger; pure input-only roots stay cut. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): deployed non-autorun enrichment, s3:// storage, --to cut accounting, tag regex
Address CI review (Codex P1/P1/P2, Pi P2):
- Deployed `pipeline run` recovers marker-only data_upload/webhook/email triggers
from script bodies (like the `show` path) so input-only entrypoints are cut
instead of auto-run empty on the deployed graph.
- `--upload s3://<storage>/<key>` keeps the named storage (authority) instead of
folding it into the key, matching the S3Object round-trip convention.
- Bounded `--to` targets cut by a barrier are reported in droppedEnds (+warning),
not reachableEnds.
- fallbackParse `// tag` matches a single token (\S+), rejecting multi-word prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): header-only deployed marker scan, fail-closed enrichment, default-storage s3 keys
Address CI review (Codex P2, cubic P1/P1/P2):
- Deployed marker recovery scans the LEADING comment header only (shared
recoverHeaderMarkers helper, reused by the show enrichment too) so a body
comment `// on data_upload` can't inject a phantom trigger and over-cut.
- Deployed run enrichment fails CLOSED: a script-body fetch error aborts the run
instead of silently letting an input-only entrypoint run with empty args.
- Revert `--upload s3://` to default-storage whole-path keys (matching pipeline
`s3://` asset-URI semantics); named-storage authority-splitting broke nested
default keys like `s3://raw/2026/events.csv`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): reject trailing content on fallback native markers; trim s3:/// key
Address CI review (Codex P2, cubic P3):
- fallbackParse now requires a native marker (`// on data_upload`) to stand alone;
a line with trailing content (`// on data_upload f/foo`, `# on kafka topic`) is
rejected, matching the canonical parser and keeping local/deployed parity.
- s3UriKey trims a leading slash so the canonical empty-authority default form
`s3:///key` doesn't leak a leading slash into the object key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): persist dev WS token per-port so reconnect survives a CLI restart
Address Codex P2: the /pipeline_dev auto-reconnect reuses the ws_token from the
page URL, but `pipeline dev` minted a fresh random token each start, so a restart
on the same port left the open page rejected by verifyClient forever. Persist the
token per-port under the user-private config dir (0600) and reuse it on restart,
so an already-open page reconnects — matching the reconnect behavior's intent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): scope persisted dev WS token by workspace+folder+port
Address cubic P2: keying the persisted token by port alone let a stale browser
tab from a previous folder's session on the same port reconnect and receive a
different folder's source. Scope the token file by workspace+folder+port so a
same-session restart still reconnects, but a different folder on the same port
gets a distinct token that rejects stale cross-folder tabs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): caller args can't override skip-dispatch guard; hash the dev token key
Address CI review (Codex P1, cubic P2):
- makeLaunch / CLI run build args with `_wmill_skip_asset_dispatch` LAST (and drop
any caller-supplied copy) so a run-form/`--upload` arg can't re-enable backend
asset dispatch while the client orchestrates the cascade (double-run / running
deployed subscribers from a local preview). Adds a cascadeRun guard test.
- Dev WS token file key is a sha256 of NUL-delimited workspace+folder+port, so
different folders (`a/b` vs `a_b`) can't collide onto the same token file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): canonical s3://storage/key --upload parsing; scope dev token by remote+root
Address Codex P1/P1:
- Restore canonical S3Object URI parsing for `--upload` s3 sources, matching the
frontend's `parseS3Object` (`s3://<storage>/<key>`, empty authority ⇒ default,
`s3:///key`/`s3:///nested/key` for the default store). `s3://secondary/k.csv` →
`{ s3: "k.csv", storage: "secondary" }` so a named-storage object is read from
the right store. (This is the canonical convention; the default-storage nested
key is served by the `s3:///` form.)
- Scope the persisted dev WS token by remote+workspace+root+folder+port (was
workspace+folder+port), so two profiles on different remotes (or local checkouts)
with the same workspace/folder/port don't share a token — a stale tab can't
reconnect across a workspace/remote boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751)
A flow step could execute an unrelated (and in the reported case, destructive)
script at runtime even though every stored definition looked correct. A forensic
dump traced it to two issues:
- Deploy accepted absolute/local step paths. `wmill sync push` from a feature-
branch checkout under /tmp baked an absolute path
(`/tmp/.../ops/scripts/clean_device/...`) into a step's `value.path`. Persisted
verbatim, it mis-resolved to an unrelated script at runtime.
- The on-disk cache write was neither truncating nor atomic. `FsBackedCache::put`
used `write+create`, so a shorter overwrite left stale trailing bytes and
concurrent writers could interleave into a torn file — a corrupt cached blob
that a worker then scheduled from.
Fixes:
- Reject non-workspace flow step paths (must be u/, f/, g/ or hub/) in
`validate_flow_value` (covers create_flow + update_flow, recursively through
loops/branches/AI-agent tools) and early in the CLI `pushFlow`.
- Make `FsBackedCache::put` write a unique temp file (truncate + fsync) then
atomically rename it over the target, cleaning up on error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): validate failure/preprocessor module paths + sub-flow paths in CLI
Address PR review (cubic + claude):
- Backend `validate_flow_value` is the authoritative guard but only walked
`modules`; extend it to also validate `failure_module` and `preprocessor_module`
(which can themselves be sub-flows/loops/branches), so an absolute path there
can't be persisted.
- CLI preflight only collected `type: "script"` paths; now collects sub-flow
(`type: "flow"`) step paths too (recursively, incl. failure/preprocessor), so the
comment's claim matches the behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cli): include AI-agent tool step paths in flow path preflight
Address Codex review: collectStepPaths skipped aiagent tools, so a bad path in
a tool fell through to the API error instead of the local fail-fast. The backend
already validates these (traverse_modules walks AIAgent tools); this aligns the
CLI early-error with it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(flows): make failure/preprocessor path test key explicit
The test used `slot:` as a json! key. json! does interpolate an ident key to its
variable's value (json!({slot:1}) with slot="failure_module" => {"failure_module":1}),
so the test was correct and exercised the validation — but the behavior is subtle,
so build the key explicitly via serde_json::Map to remove ambiguity (review nit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cache): use a UUID temp name for atomic put (shared-volume safe)
Address Codex (P1): pid+counter temp names collide across container PID
namespaces on a shared cache volume (same pid, PUT_SEQ resets to 0 per process),
so two workers could truncate/clobber the same temp file before rename. Use a
random UUID suffix (matching worker.rs's atomic-write helpers) — globally unique,
so the cross-process temp-file hazard is closed. Also trims the comment to the
AGENTS.md <=4-line limit (Pi nit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(sdk): allow overriding worker tag when running jobs
Add an optional `tag` parameter to every job-running helper across the
TypeScript, Python, PowerShell and Rust client SDKs. When set, it is
forwarded as the `tag` query param on the `jobs/run/*` endpoints, which
the backend already honors as a worker-tag override.
The parameter is appended last and defaults to null/None everywhere, so
existing positional and keyword callers are unaffected. Rust has no
optional params, so its existing `run_script_async`/`run_script_sync`
signatures are left untouched and new `*_with_tag` variants are added.
Fixes WIN-2105
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(system_prompts): regenerate SDK docs for tag param
Regenerate auto-generated system prompts so the TypeScript/Python SDK
references (and the script skills that embed them) reflect the new
optional `tag` parameter on the job-running helpers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(powershell-sdk): preserve original RunScriptAsync/RunFlowAsync arities
PowerShell class methods dispatch by exact argument count and have no
default parameter values, so adding `$Tag` in place dropped the old
4-arg `RunScriptAsync` / 3-arg `RunFlowAsync` overloads — existing direct
class calls would fail with "Cannot find an overload". Re-add the
original arities as thin overloads that forward `$null` for `$Tag`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(system_prompts): generate prompts.d.ts to stop literal-content drift
prompts.d.ts was a tracked declaration file with string-literal types
baked in, but generate.py never regenerated it — only prompts.ts and the
hand-written index.d.ts. So every prompt change (e.g. the new SDK `tag`
param) left prompts.d.ts stale, and check-freshness didn't catch it
because generate.py never wrote the file.
Emit prompts.d.ts from generate.py as plain `export declare const X:
string;` declarations. The contents now live only in prompts.ts, so the
declaration file can't drift, and check-freshness covers it going forward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `script run command > runs a script and returns result` test runs a
trivial, deterministic bun script and asserts exit code 0. On CI it
intermittently fails when the standalone worker (notably on Windows)
transiently fails to execute the job — identical bun jobs complete
successfully elsewhere in the same backend session, so the failure is
environmental, not a regression.
Two problems made this both flaky and undiagnosable:
- `--silent` plus asserting only on `result.code` meant the job's actual
error never reached the CI log, so a flake left no trace.
- No test-level retry, so a single transient worker hiccup failed the run.
Add `retry: 2` to the two worker-executing tests in the block, and
include stdout/stderr in the assertion label so the next occurrence is
debuggable.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: self-host docs search for chat, mcp and cli; remove inkeep
Embed a vendored docs snapshot (llms.txt/llms-full.txt) in the backend and
serve ranking + page rendering from GET /api/docs/{search,page}. The AI chat,
the MCP searchDocs/readDocsPage tools, and 'wmill docs' all consume it, so docs
search works with no runtime egress and is no longer EE-gated. Removes the
inkeep proxy. EE companion deletes inkeep_ee.rs (ee-repo-ref bumped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: name read_docs_page param `url` instead of `path`
search_docs returns each hit's `Source` URL, so the read tool now takes a
`url` argument to match — the AI/MCP loop reads "search gives a Source URL,
read takes that url" rather than copying a `Source:` URL into a `path` slot.
A bare `/docs/...` path is still accepted and canonicalized before lookup.
Regenerated openapi-deref, the MCP endpoint tools, and the frontend client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: add scheduled workflow to refresh the vendored docs snapshot
The backend embeds docs_snapshot/*.gz at build time, so the in-product docs
corpus is otherwise only as fresh as the last manual fetch.sh run. This adds a
weekly (and manually dispatchable) job that re-runs fetch.sh, sanity-checks the
result against truncation/garbage, and opens a PR via the internal app when the
snapshot changed — so a human reviews the docs diff before it rides into the
next release build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: make docs tool-result strings caller-neutral
The search/page endpoints back three differently-named consumers (the AI chat
`read_docs_page` tool, the MCP `readDocsPage` tool, and the `wmill docs` CLI),
so the shared rendered text shouldn't name one of them. Refer to "the docs
page-reading tool" and its `url` argument instead, and add tests pinning the
caller-neutral follow-up guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: point ee-repo-ref at inkeep-removal companion rebased on EE main
The companion branch now carries only the inkeep_ee.rs deletion on top of EE
main (was based on the native-job-retry EE line, which polluted the EE PR diff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(docs): expose docs:read in token catalog; precompute lowercased corpus
Addresses two review nits on the self-hosted docs PR:
- docs:read was enforced (ScopeDomain::Docs) but missing from the token scope
catalog (token.rs ALL_SCOPES), so it couldn't be selected when creating a
standard scoped token in the UI — leaving scope-restricted CLI/MCP docs use
effectively ungrantable. Add a read-only "Documentation" group (no write
surface) and a test asserting it is exposed.
- search ran page.body.to_lowercase() on the whole corpus per query. Lowercase
body/title/description once at parse time (into the OnceLock corpus) and scan
the precomputed copies instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: update ee-repo-ref to 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
This commit updates the EE repository reference after PR #630 was merged in windmill-ee-private.
Previous ee-repo-ref: c7ec3a0c2fa38d4cb5e50bf0265eef4710de4860
New ee-repo-ref: 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: bounded-cascade selective execution for pipelines (UI + CLI)
Run a prefix of a pipeline cascade: from a schedule/manual root, fan
downstream but stop at chosen end node(s) — the path-between set over the
asset-graph lineage DAG. Exposed as a canvas 'Run downstream up to…' pick
mode and a 'wmill pipeline run <folder> --to' CLI command. No backend or
parser changes; reads the existing graph, tags, and triggers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: surface bounded-run on the run caret, trigger-node kebab, and Test button
Move 'Run downstream up to…' from the runnable kebab onto the play-button
caret popover (Edit mode, next to Run / Run + trigger N downstream); add it
to the trigger-node kebab so schedule/data_upload entrypoints expose it on
the View page; and to the ScriptEditor Test split caret for the open script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address CI review on bounded-cascade (cubic)
- Port CLI engine test from Deno to bun:test under cli/test/ (won't run under bun test otherwise).
- closure() now excludes the start node on a cycle back to it (descendants/ancestors contract); regression tests both engines.
- CLI 'pipeline run --to' rejects unresolved/ambiguous end tokens instead of silently running a different subset.
- Sort a copy in the runSelection order test so the launch-order assertions aren't invalidated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address standing review nits on bounded-cascade
Resolves the four recurring P1/P2 findings from the codex/pi/claude
reviews:
- UI gate (P1): the canvas/trigger-node "Run downstream up to…"
affordance was gated on the subscriber-only downstream map, so a valid
start whose only downstream is a pure reader had a non-empty bounded
set but no menu entry. Gate on the read-aware lineage downstream
(buildLineageDownstreamMap), matching the bounded engine.
- waitJob (CLI): a completed job without explicit success:true now
counts as a failure, mirroring the frontend waitJobTerminal — the
cascade only advances on a confirmed success.
- Comment fix (CLI): the unbounded `run` path uses the read-aware
lineage DAG (pure readers included); dropped the false "parity with
the canvas cascade" (subscriber-only) claim.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: expose bounded-run caret for pure-reader-only starts (codex P1)
The canvas wiring from the prior commit passed `onStartBoundedRun` from
the read-aware lineage map, but the leaf components still hid the popover
that holds the "Run downstream up to…" action behind a subscriber-only
gate:
- RunnableNode rendered the Run-button caret only when
`hasCascade = downstreamCount > 0` (subscriber-only). A valid start
whose only downstream is a pure reader got `onStartBoundedRun` but no
visible action. Now the caret opens when there's a cascade OR a
bounded-run start (`hasCaret`), and the "Run + trigger N downstream"
item is gated on `hasCascade` so it never reads "trigger 0".
- ScriptEditor's Test split button activated only when
`downstreamSubscribers > 0`, falling through to a plain Test button
(no caret) otherwise. Now it also activates when `onBoundedRun` is
set, with the "Test + trigger N" item gated on the count.
For a manual root (no trigger-node kebab fallback) with a pure-reader
downstream this was the only UI entry point, so it was previously
unreachable. Verified in-browser: a manual-root script writing an asset
read-only downstream now exposes "Run downstream up to…" on the
ScriptEditor Test caret with the cascade item hidden.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: gate ScriptEditor bounded-run on read-aware downstream; fix CLI asset-end warning (codex P2)
- Details-pane (ScriptEditor) bounded-run entry was gated only on
`validStartPaths`, broader than the canvas which also requires
read-aware downstream (`hasLineageDownstream`). An isolated start could
thus expose "Run downstream up to…" and enter pick mode with no
selectable end. Now gated on `lineageDownstreamPaths` (script paths with
a downstream in `buildLineageDownstreamMap`), matching the canvas.
- CLI dropped-end warning called `scriptPathOf(d)` unconditionally, which
slices `script:`-length chars off an asset id too — `datatable:main/raw`
printed as `le:main/raw`. Now prefix-checks like the JSON output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: correct --from error to exclude only row-backed event triggers (codex P2)
The bounded-start validation message listed `kafka/webhook/…` as event
triggers that can't start a bounded run, but webhook/data_upload are
rowless and read as manual roots (valid starts). Only the row-backed
native kinds (kafka/mqtt/nats/postgres/sqs/gcp/email — EVENT_TRIGGER_KINDS)
are excluded; the message now names those.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: surface dropped ends in CLI JSON; disambiguate shared-trigger bounded start (codex P2)
- CLI `run --json` silenced the dropped-end warning, and the JSON payload
echoed the originally-resolved `--to` list with no reachable/dropped
split — a resolved-but-unreachable end looked like a clean plan that
silently runs only the start. JSON now includes `reachableEnds` and
`droppedEnds` (shared `idLabel` helper, asset-id safe).
- Trigger nodes dedupe per (kind, ref), so a schedule shared across
scripts collapses to one node, but `recordSourceTrigger` kept only the
first target path — the bounded-run action then rooted at an arbitrary
script (or hid when only that first script lacked downstream). Now all
target paths are tracked and the action is offered only when exactly one
is a valid start with downstream; multi-eligible nodes suppress it
rather than guess.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: don't run hidden drafts in View-mode bounded cascade (codex P1)
launchCascadeScript unconditionally preferred drafts.get(path) over the
deployed script. In View mode with drafts hidden (displayGraph is
deployed-only), a bounded run started from a trigger-node kebab would
execute preview jobs from hidden local draft content instead of the
deployed scripts the user is looking at.
Gate draft execution on `mode === 'edit' || includeDrafts` — the exact
condition under which displayGraph includes drafts — so execution always
matches the displayed graph. No-op for scripts without a draft; the
edit-mode "Run + trigger N downstream" cascade is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: prevent variable push from corrupting is_secret variables
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cli): unit-test looksLikeWorkspaceCiphertext shape detection
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): scope is_secret downgrade to single-file push, not sync push
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): warn when variable push stores a secret value as already-encrypted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): route workspace-resolution and auth diagnostics to stderr
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(cli): rephrase comments to describe current behavior, not history
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ansible scripts previously lacked the AI assistant and the contextual
variable helper that other scripting languages expose in the script editor.
- Add 'ansible' to SUPPORTED_CHAT_SCRIPT_LANGUAGES so the AI chat button
shows in the editor toolbar and the AI chat opens in SCRIPT mode without
the "language not supported" warning.
- Add an Ansible system prompt (system_prompts/languages/ansible.md) plus a
LANGUAGE_METADATA entry, and regenerate the auto-generated prompts/skills
so the AI has tailored Ansible context.
- Show the contextual variable picker for ansible and insert references as
`{{ lookup('env', 'NAME') }}`, matching how Windmill exposes reserved
variables as environment variables to the ansible-playbook process.
Fixes WIN-2072
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Diego Imbert
Diego Imbert
Ruben Fiszel
hugocasa
centdix