* fix(s3_proxy): enforce CE 50MB upload cap on multipart uploads
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(s3): replace CE 50MB upload cap with 10GiB workspace storage quota
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): gate CE quota OSS stubs to not(enterprise) to match callers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): delta-aware CE storage quota + guard usage-load retry loop
Account for the overwritten object's size in the quota check so valid
same-size overwrites near quota are not rejected (Codex review), and stop the
storage-usage $effect from re-firing on persistent API errors (Pi review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): count chunked PUTs; revert overreaching volume quota copy
Volumes write to workspace storage via a separate worker-side path with its own
50MB-per-file cap that this PR does not change, so revert the drawer copy that
claimed they count toward the 10GiB quota (Codex review). Bump ee-repo-ref for
the chunked-PUT accounting fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): add SQLx cache for CE usage-bump/quota queries; exclude volumes
Regenerate the missing offline SQLx cache for the not(enterprise) bump and
remaining-quota queries so the private CE offline build compiles, and bump
ee-repo-ref for the volumes/-prefix exclusion from the counted quota (Codex
review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): always HEAD for CE upload delta so overwrites don't inflate usage
Bump ee-repo-ref for the fast-path overwrite-accounting fix (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): reserve volumes/ prefix on CE write surfaces to close quota bypass
Reject direct writes to the reserved volume prefix on the app-upload surface and
add the OSS stub; bump ee-repo-ref (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): refuse new multipart work when CE workspace is at quota
Bump ee-repo-ref for the multipart-initiate/part quota gate (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(s3): reserve in-flight multipart bytes against CE storage quota
Add workspace_multipart_inflight table + grants, SQLx cache for the reservation
queries, and bump ee-repo-ref. Bounds abandoned multipart uploads that the
list-based recount can't see (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): clear multipart reservation only after a successful complete
Add exclude-upload arg to the OSS quota stub/caller and the SQLx cache for the
updated remaining-quota query; bump ee-repo-ref (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(s3): per-part multipart reservation; commit only on part success
Per-part workspace_multipart_inflight schema (upload_id, part_id) so retries
replace rather than double-count; SQLx cache for the reworked queries; bump
ee-repo-ref (Codex review).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(s3): HEAD the multipart overwrite target once per upload, not per part
SQLx cache for the stored-credit lookup; bump ee-repo-ref.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to bea5a8b5120d6d69cab1ad4611ebe463559bd200
This commit updates the EE repository reference after PR #640 was merged in windmill-ee-private.
Previous ee-repo-ref: 6e6ff86f1939cf74736b7d435bf6851416437523
New ee-repo-ref: bea5a8b5120d6d69cab1ad4611ebe463559bd200
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: support workspace forks on cloud using parent workspace limits
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: clarify count_paid_seats approximates rather than mirrors billing seats
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: non-admin fork UI, attach cap, and fork-count for cloud forks
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: cloud fork billing cache on rename, usage display, attach cap edge
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: fork count in cloud quotas + fork billing points to parent
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: invalidate billing/fork caches on fork deletion for id reuse
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: gate fork usage remap on CLOUD_HOSTED, not just the cloud feature
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: note cloud feature vs CLOUD_HOSTED gating in backend guide
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reserve fork-cap slots for an attach candidate's whole subtree
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: invalidate team-plan cache on delete, raise fork depth cap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: cap fork nesting depth (MAX_FORK_DEPTH, default 5)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: fork count/height robust to cycles and deleted intermediates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): reset fork button loading state on creation error
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: invalidate billing cache for attached fork subtree; helper auth docs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: invalidate bun bundle cache on transitive relative-import changes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: do not memoize transient fetch errors in bundle-key import cache
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: use regular comment on lazy_static block (deny unused_doc_comments)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: align bundle-key import version selection with loader content endpoint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): show inline workspace name editor on general settings
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): guard rename and support enter-to-save on workspace name
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai): add Azure AI Foundry as a native AI provider
Adds `azure_foundry` as a new AIProvider variant wired through the AI
chat (copilot) and AI agent flow steps. Foundry's chat completions API
is OpenAI-compatible and uses Azure conventions (api-key header, Azure
URL building), so it reuses the existing OpenAI-compatible query builder
and proxy path via the shared `is_azure` helper (renamed from
`is_azure_openai`).
Backend (windmill-ai):
- New `AzureFoundry` enum variant (serde `azure_foundry`)
- `get_base_url` requires a resource base URL (like Azure OpenAI / Custom)
- `is_azure()` covers Azure OpenAI + Foundry (api-key auth, Azure URL)
- Added to OpenAI-compatible proxy support and HttpForward proxy mode
- New proxy URL unit test
Frontend (copilot):
- New provider entry, completion config, model-token handling, streamed
usage tracking, and reasoning registry (all model-id-gated, so a no-op
for Foundry's non-OpenAI catalog)
- Treated as a chat-completions provider, not the OpenAI Responses API
OpenAPI:
- `azure_foundry` added to AIProvider (openapi.yaml) and AIProviderKind
(openflow.openapi.yaml); regenerated CLI guidance
Note: the `azure_foundry` resource type (base_url + optional api_key) is
hub-managed and must be published to the Windmill Hub separately.
Fixes WIN-2122
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai): add azure_foundry to copilot flow Zod provider enum
The tracked copilot flow schema (openFlowZod.gen.ts and its openFlow.json
source) still carried the old AIProvider enum, so validateFlowModules /
validateSpecialFlowModule rejected AI-generated flow edits that create or
update an aiagent module with provider kind "azure_foundry" before they
could be saved. Add the value to both (preserving the generated single-line
format) and a regression test over the flow-module validation path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai): lead provider list with OpenAI, Anthropic, Google AI
Reorder AI_PROVIDERS so the three primary direct providers come first. The
AIProviderPicker renders the first three entries as quick-access buttons, so
these become the defaults (previously OpenAI, Azure OpenAI, Azure Foundry);
Azure OpenAI / Azure Foundry stay adjacent right after. No logic depends on
provider order (only per-provider defaultModels[0] is read).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): home New submenus fall back below hugging the right edge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): re-hug submenu on window resize even without a melt reposition
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix: revalidate workspace items cache on context picker open
The chat context picker (and workspace drill pickers) read from a
module-level cache that was only invalidated from two editor save paths,
so items created or deleted anywhere else stayed stale until a full page
reload. Make the loader do real stale-while-revalidate: cached items
render instantly and each workspace+kind is re-fetched once per picker
mount, keeping the state reference stable when nothing changed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: retry failed picker revalidation and handle its rejection
Mark a workspace+kind as revalidated only after the fetch succeeds, so a
failed revalidation is retried on the next ensureLoaded call instead of
stranding stale data for the rest of the mount. Catch the rejection
(callers fire-and-forget) and log it. Also dedupe the
stale-while-revalidate rationale to its canonical comment site in
workspacePicker.ts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A websocket trigger is permanently disabled (with a critical alert) when a
single connect attempt in get_consumer fails. Gateway endpoints fronted by
edge proxies (e.g. Discord behind Cloudflare) sporadically answer the
upgrade handshake with a transient 502/520, so a long-lived trigger that
reconnects frequently eventually catches one and dies until a human
re-enables it.
Retry transient failures (HTTP 5xx/429 handshake responses and IO errors)
up to 5 attempts with exponential backoff before surfacing the error, and
report retry progress through the trigger's error status. Permanent-looking
errors (bad URL, other 4xx, protocol/TLS mismatch) still disable immediately.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(pipeline): backfill a range of partitions from the asset drawer (ee)
* feat(pipeline): cancel in-flight backfill job and show cancelling state
* refactor(pipeline): move backfill range logic behind private feature
* fix(pipeline): close backfill cancel-launch race and record dispatch intent
* docs(openapi): producer_path also covers SDK write-edge producers
* chore: update ee-repo-ref to c3852ecb36bd0be1a74c63169e513888f3347850
This commit updates the EE repository reference after PR #641 was merged in windmill-ee-private.
Previous ee-repo-ref: 7c1450ef89fbc9e844a121b39cafe0d7235d704b
New ee-repo-ref: c3852ecb36bd0be1a74c63169e513888f3347850
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: scope SCD2 built-in data tests to current rows
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: add --partition to pipeline run and fix duckdb s3object upload binding
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: note filesystem storage type is dev-only in storage settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: use ISO week for weekly partition default in pipeline run
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): reserve fixed height for unsaved-changes banner to avoid content shift
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): remove border around reserved banner slot and shrink it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): tighten top padding under the unsaved-changes banner
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): give unsaved-changes banner buttons minimal vertical breathing room
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(frontend): drop redundant Metadata section title in trigger and script editors
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): tuck schedule editor labels under summary to match convention
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(frontend): keep Metadata section title in ScriptBuilder for a separate PR
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): drop leftover header-content margin on headless Section
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): drop top padding above resource editor first field
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(frontend): match variable editor bottom padding to resource
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): add Path label in new resource form to match edit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): reserve half the banner height to halve the idle gap
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): reserve a third of the banner height when idle
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): don't reserve banner slot or tighten top for new entities
Gate the reserved-height slot and the tight content top padding on the banner's baseline (bannerReserved) instead of merely on the banner snippet being present, so new-entity drawers keep normal top spacing and add no empty slot.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(frontend): trim banner comments to the 4-line invariant limit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(frontend): describe partial-reserve banner behavior accurately
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): keep pipeline graph layered when lineage has cycles
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): fit pipeline graph to visible canvas on initial load
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): style pipeline minimap so it reads as a minimap
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(parser): don't infer s3 reads from bare string-literal mentions in sql
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(duckdb): render temporal values as ISO strings in job results
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): key pipeline viewport fit on the loaded graph's folder
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(parser): treat list/named read-fn arguments as definitive s3 reads
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(frontend): surface pipelines in sidebar nav, index page and sql editor hint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(frontend): remove pipelines sidebar nav item
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(frontend): group compare & deploy items by folder
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(frontend): group folder items with their folder, fix disabled hint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(duckdb): auto-declare the partition arg for // partitioned scripts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(cli): pipeline run --arg to pass plain run args to cascade scripts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The cron schedule row placed the input next to a shrink-0 button group, so
on narrow layouts the buttons kept their width and squeezed the input to
near-zero. Make the row wrap and give the input a min width so the buttons
drop below it, keeping both visible.
Fixes WIN-2121
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fetching options for a `dynselect`/`dynmultiselect` input was inconsistent
between deployed scripts and deployed flows:
- scripts ran through `push_script_job_by_path_into_queue` — a `script` job
with the script's path, tag, lock and codebase resolution;
- flows ran their schema dyn-select code as an anonymous `preview` with no
path and no tag (always the language default), and reported access
failures as a raw `SqlErr: no rows`.
Deployed scripts are left exactly as they were (that path already handles
tag/lock/codebase/on-behalf-of correctly). The flow branch now:
- carries the flow path on the preview job,
- reads the flow's `tag` under RLS and routes the job to it (falling back to
the language default when unset), matching the script's worker group, and
- runs `check_tag_available_for_workspace` on that tag — the same gate a
normal flow run and the script path apply — so a caller who can read the
flow but is not allowed to use its (custom/scoped) worker tag is rejected
consistently.
The flow's tag read runs on every request, so it also serves as the
per-request access check, replacing the raw error with a clean
`NotAuthorized` / `NotFound`. Entrypoint-name validation now covers all
branches (it is interpolated into the generated wrapper). Inline is
unchanged: a `preview` with no path on the language default, blocked for
operators.
Fixes WIN-2118
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(forks): let partial-visibility users deploy the visible subset
The fork Compare & Deploy page hid the deploy button entirely whenever the
comparison reported any item not visible to the user (all_ahead/all_behind
flags), telling them to hand the deploy to someone with full access. But the
non-visible items are already filtered out of the diff list, and the UI already
supports deploying an arbitrary subset via per-item selection — so blocking
everything was inconsistent and, for stale/phantom rows, blocked on items that
don't even exist.
Show the deploy footer regardless; the user acts on the visible/selected items
(the per-item disabled conditions are unchanged). The hidden-items notice is
kept but downgraded to a non-blocking, direction-scoped banner that explains the
excluded items instead of removing the action.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(forks): surface hidden-item counts by kind + admin path list
WIP: expose items dropped by the visibility filter (hidden_ahead/hidden_behind
in the compare response): by-kind counts for everyone, kind+path only for admins.
* fix(forks): don't close the deployment request on a partial (hidden-items) deploy
Making the deploy button reachable in the partial-visibility case exposed a bug:
a clean merge-into-parent deploy unconditionally closed any open fork deployment
request as "merged" — marking its comments obsolete and notifying the requester
and assignees of a merge — even when hidden ahead changes were excluded from the
list and left undeployed. Only close the request as merged when the full ahead
set was visible (all_ahead_items_visible); otherwise leave it open (with a toast)
so someone with full access can finish it.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(cli): correct misleading delete-fork command description
The `wmill workspace delete-fork` description claimed it deletes "a
forked workspace and git branch", but the implementation only deletes
the Windmill workspace via the backend API and removes the local
workspace profile. No git operations are performed, so the remote
branch is left untouched. Drop the "and git branch" clause and
regenerate the derived guidance/system-prompt files.
Fixes WIN-2120
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(cli): permanently delete temp workspaces in folder test cleanup
The isolated-workspace test helper archived each temp workspace on
teardown. After #9865 added a CE cap of 1 archived workspace, the second
archive-cleanup is refused, so temp workspaces leak into the active set
and hit the 2-workspace CE cap — failing every subsequent create/fork
across the shared test backend.
Permanently delete the workspace instead (DELETE /api/workspaces/delete),
which frees the slot without occupying the archived quota.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
The blast-radius guard added in #9866 forced `all_ahead_items_visible` true for
any fork/target admin. But `filter_visible_diffs` keeps a modified/conflict row
(one that exists in the source AND the fork) only when the caller can see it on
both sides, so an ahead diff can be dropped for a source-side visibility gap even
when the caller is a fork admin. Forcing the flag on fork-admin alone then wrongly
reported "all ahead items visible", letting the UI enable deployment from an
incomplete comparison.
Gate the guard on admin of BOTH the source and the fork (superadmin satisfies
both), which is what actually guarantees full visibility of every item on every
side. Adds a regression test where a fork admin who is only a plain member of the
parent (no access to the item's folder) must still get `all_ahead_items_visible
= false`, plus the superadmin sanity path.
Also restores the SQLx offline cache entry for the phantom-trigger test INSERT
that #9866 landed without (CI/`SQLX_OFFLINE=true` builds failed on it), and adds
entries for the new test's all-literal queries.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deleting a trigger left a stale `workspace_diff` row: `delete_trigger` (the
generic TriggerCrud handler) was the only delete path that never called
`handle_deployment_metadata`, unlike every other kind. Because
`compare_workspaces` trusts a cached `has_changes=true` row for non-script/flow
kinds and the visibility filter then drops it (the trigger no longer exists), a
deleted trigger became a phantom "ahead" item that flipped
`all_ahead_items_visible` to false — hiding the deploy button and showing a
"changes not visible to your user" warning that even a superadmin could not
resolve (`reset_diff_tally` doesn't clear a `has_changes=true` row either).
- delete_trigger now re-tallies via handle_deployment_metadata, so the next
compare re-evaluates and corrects/removes the row (matches resource/variable/
folder/schedule deletes).
- compare_workspaces forces the visibility flags true per side for anyone who
sees that side in full: target/fork admin (or superadmin) for ahead items,
source/parent admin (or superadmin) for behind items. The flag is a pure
visibility guarantee — the deploy itself is authorized separately — so for
such users a dropped diff is provably a phantom, never a permission gap.
- Add a regression test asserting a phantom trigger diff row no longer blocks a
superadmin while still (conservatively) warning a partial-context user.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(workspaces): enforce CE workspace limit when unarchiving
Unarchiving a workspace re-activates a soft-deleted (deleted = true)
workspace, effectively bringing it back to the active set. On CE this
bypassed the 2-workspace cap that create_workspace enforces, letting a
user exceed the limit by archiving and re-unarchiving.
Run the same _check_nb_of_workspaces guard before flipping deleted back
to false. The workspace being restored is still deleted = true at that
point, so it is correctly excluded from the count.
Fixes WIN-2119
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(workspaces): cap CE archived workspaces at 1
Complements the unarchive-limit fix: without a cap on archived
workspaces, a CE user could stockpile many soft-deleted workspaces (each
of which still occupies its workspace id and can later be unarchived).
Refuse a new archive on CE when an archived workspace already exists,
mirroring the create/unarchive workspace-count guards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: use derived username instead of email for non-member superadmins
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address review - drop redundant username cache, guard whoami membership by email
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: use explicit non_member boolean instead of role string for superadmin banner
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: resolve email from password table for non-member superadmin permissioned_as
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: resolve non-member superadmin drafts via shared username->email resolver
Adds resolve_username_to_email (usr, then super_admin password fallback for both derived-username and email modes) and uses it in get_email_from_permissioned_as and the drafts get/list endpoints, so a non-member superadmin's drafts resolve and no email leaks into the drafts payload.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: superadmin-not-in-workspace schedule uses derived username as permissioned_as
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: resolve non-member superadmin identity in draft owner-circles, username_to_email, and home filter
Applies the password-fallback username resolution to the script/flow/app/draft owner-circle subqueries and the username_to_email endpoint (was an admins-workspace 'username == email' hack), and switches the home items-list user-folder filter to the non_member flag instead of the now-broken username-contains-@ heuristic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: backfill non-member superadmin favorites from email to derived username
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: propagate DB errors in username resolution instead of leaking email (CI review)
Addresses cubic-dev-ai P2: get_instance_username_or_fallback_to_email now returns Result and only falls back to the email for a genuine 'no derived username'; a query error propagates so callers fail closed rather than leaking the raw email as the acting username.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: clarify non-member superadmin popover (username used + admin permissions)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: keep username_to_email endpoint member-only to not disclose non-member superadmin email (CI review)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: forbid disabling automate_username_creation once usernames assigned (CI review)
Makes the setting effectively one-way once instance-wide usernames exist, so the global-uniqueness invariant that keeps stored u/<username> identities (schedules/triggers/drafts/superadmin ownership) unambiguous can never be dropped back to workspace-local uniqueness. Re-saving false on an already-disabled instance stays a no-op.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(offboarding): make global reassignment per-workspace and optional
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(offboarding): handle sole-member workspaces in workspace-level removal
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(offboarding): show close action instead of dead-end in reassign-only sole-member case
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(offboarding): prevent no-op success in global reassign-only with no reassignable workspace
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): add zoom and download to Mermaid graphs
Mermaid diagrams in the AI chat could only be viewed inline with
horizontal scroll. Add a download-as-SVG button and an expand button
that opens a fullscreen modal with pan/drag and zoom (mouse wheel plus
in/out/reset controls), reusing the existing `panzoom` dependency.
Fixes WIN-2117
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): guard panzoom action against close-before-import race
Address review nit: if the modal closes before the dynamic
import('panzoom') resolves, destroy() ran while instance was still
undefined (disposing nothing) and the late .then() built a leaked
panzoom on a detached node. A disposed flag makes the cleanup airtight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Folder owners use the format u/<username>, and usernames are frequently
email addresses containing `.` and `@`. The folder creation path bypasses
validate_owner(), so these owners get inserted successfully, but add_owner
and remove_owner both call validate_owner() and rejected any later
modification of email-style owners.
Extend the character allowlist to accept `.` and `@` (and update the error
message). SQL injection risk is already mitigated by the bind-parameter
queries introduced alongside this validation.
Fixes WIN-2116
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): local development for data pipelines (CLI --local + pipeline dev preview)
Add the local edit→preview→run loop for data pipelines (folders of `// pipeline`
scripts), the analog of `wmill dev` / `wmill app dev`, usable from a code editor
or an agentic loop — without deploying.
No backend changes: full body inference comes from the same wasm the frontend
uses (windmill-parser-wasm-asset), which returns assets + pipeline annotations in
one call; local runs reuse runScriptPreview with _wmill_skip_asset_dispatch.
- localGraph.ts: wasm-backed working-tree → asset-graph builder (the enabler)
- pipeline show/run --local; new pipeline docs (PIPELINE.md/AGENTS.md) subcommand
- pipeline dev watcher + /pipeline_dev page (PipelineDevView) rendering the same
PipelineGraphEditor from the pushed local graph, run via preview
- cascadeRun.ts: reusable run primitives extracted from the route page
- regenerated CLI agent docs
See docs/pipeline-local-dev.md for the full design, test steps, and handoff TODOs.
The live `pipeline dev` browser preview is implemented but not yet stack-verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): improve local dev preview (run, activity, responsive)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): dev-preview args, multi-root run, ws auto-reconnect
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): connect managed-materialize producer in local dev graph
The CLI pinned windmill-parser-wasm-asset ^1.728.1, which predates managed-materialize support (added in 1.733.1); the frontend already pins 1.740.0. The CLI's wasm therefore never emitted `// materialize`, so the producer had no output edge and showed disconnected from its `// on` consumers. Bump the CLI to 1.740.0 (matching the frontend) and translate the parsed materialize target into the producer's write edge + materialize_target, mirroring frontend resolveGraph.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): harden local-dev CLI (bare-.sql crash, defaultTs, docs clobber)
Review fixes, complementary to the dev-preview/materialize/multi-root work already
on the branch (none overlap those commits):
- localGraph: a bare `.sql` (no dialect) made inferContentTypeFromFilePath throw and
abort the whole graph build — and wedge `pipeline dev` at startup. Skip the
unclassifiable file instead. Also map `bunnative` → parse_assets_ts and add
ruby/rlang/nu/powershell to the `#`-comment fallback.
- show/run/docs/dev: thread the resolved `wmill.yaml` defaultTs into the graph
builder so `.ts` infers under the workspace's runtime (bun vs deno) instead of
always bun — `opts.defaultTs` was always undefined (no such CLI flag).
- dev: wrap the startup graph build so a half-written file can't abort the watcher.
- docs: don't clobber a user-authored AGENTS.md/CLAUDE.md — only (over)write the
pointer when absent or already a generated `@PIPELINE.md` pointer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): bind dev WS to loopback + local-graph regression tests
- pipeline dev WS broadcast the folder's full script source (scripts[].content + temp_script_refs) unauthenticated on 0.0.0.0:3201 — bind 127.0.0.1 so it's not LAN-reachable (webview localhost + SSH/devbox port-forward still work).
- Add regression tests for the just-landed local-graph fixes: bare .sql is skipped (was a build/dev-startup crash), defaultTs threads into .ts runtime inference (bun vs deno), and #-comment languages (ruby) use the # annotation fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): --frontend flag for pipeline dev page origin
wmill pipeline dev opens <remote>/pipeline_dev, but that route only exists in this build's frontend, so it 404s against a remote whose deployed frontend predates it. --frontend <origin> points the page at a locally-run frontend (REMOTE=<remote> npm run dev) while the API/token still target the remote — enabling the live preview against a real backend before the PR is deployed. No behavior change when omitted. Regenerated CLI agent docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): WS session token + details-pane live-reload refresh
Addresses CI review (Codex/Pi/Claude):
- dev WS: a browser tab could open ws://localhost:<port>/ws and receive the folder's full source (browsers don't enforce same-origin on WS, loopback bind alone doesn't help). Gate the upgrade on an unguessable per-session token carried in the dev-page URL (verifyClient → 401 without it). Verified: no-token/bad-token connections get 401 with no bundle.
- details pane: scriptRes keyed on [workspace, selection, draftScript] didn't re-run on a pipeline dev live-reload (same selection), so the open pane showed stale source. Thread a localScriptsVersion (the pushed bundle) into the key. Verified: editing a selected node's file updates the pane source without reselect.
- docs/pipeline-local-dev.md: refresh the stale 'not yet exercised' status + done TODOs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): emit volume: annotation assets in local dev graph
Addresses CI review (Codex P1 / Pi P1): the wasm body parser doesn't surface `// volume: <name>` annotations — the frontend (infer.ts:parseVolumeAnnotations) and backend (asset_inference.rs) parse them separately and merge as rw volume assets. localGraph didn't, so a `# volume: cache` producer had no write edge and showed disconnected from its `// on volume://cache` consumer (and pipeline run --local wouldn't schedule downstream). Mirror the leading-comment-block scan (SQL excluded, matching both reference parsers) and merge into inferScriptAssets. Regression test added; verified producer -> volume://cache -> consumer connects.
Also (Codex P2): docs/pipeline-local-dev.md manual browser URL omitted the new ws_token param — without it the WS upgrade is rejected and the page sits disconnected. Doc now says to copy the URL the CLI prints (carries wm_token + ws_token) and recommends --frontend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): runAll excludes event roots + review polish
Addresses CI review (Codex P1, Claude P2/P3):
- pipeline run runAll: derive the whole-pipeline selection from validStarts + descendants instead of all runnables, so an unqualified 'pipeline run <folder>' no longer fires event-trigger roots (kafka/mqtt/…) with empty args/side effects. Verified: a kafka root is excluded from the plan.
- cascadeRun.ts runBoundedCascade: use buildLineageDownstreamMap (read-aware) so a pure-reader runs after its producer, and return cyclic — parity with the route page's bounded run (the file is meant to be THE shared correct primitive).
- PipelineGraphEditor: storedRightPaneSize starts at 0 so the orientation-aware default (55% stacked / 40% side-by-side) actually applies on first open.
- localGraph fallbackParse (go/bash): scan only the leading comment header (no body-comment phantom triggers) and strip key=value options from the asset URI; regression test added.
- docs: reject '..' in the folder arg (it writes files under f/<folder>).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): route local previews to the // tag worker
Addresses CI review P1: the local graph/bundle dropped the parsed `// tag`, so a node annotated `// tag gpu` ran on the default worker in both `pipeline run --local` and `/pipeline_dev`, while the deployed pipeline routes it to that worker tag. Carry the tag through LocalScript / the pushed bundle / LocalScriptContent and pass it to runScriptPreview at all three launch sites. Verified: a duckdb node tagged `bash` produces a job tagged `bash`; regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): add asset partitions/schemas routes to OpenAPI, use generated client
The ducklake asset panels (PartitionStatusGrid, SchemaHistoryPanel) hit /assets/partitions and /assets/asset_schemas via raw fetch with cookie-only auth, because those backend routes were never added to openapi.yaml so the generated client had no methods for them. On /pipeline_dev (token-via-URL, no session cookie) the raw fetches 401'd. Add both GET routes + MaterializedPartition/AssetSchemaVersion schemas to openapi.yaml and call them through AssetService, which injects the bearer token, types, and cancellation automatically. Verified: Partitions + Schema tabs load in /pipeline_dev. (backfill stays a raw fetch — it's an EE-only route not in the OSS spec — with the token added inline.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(cli): regenerate bun.lock for windmill-parser-wasm-asset
package.json / package-lock.json carry windmill-parser-wasm-asset@1.740.0 but the tracked bun.lock (the CLI installs/builds/tests via bun) was stale, so fresh bun installs would resolve a different graph than the committed lock. Regenerated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): show asset producer + its runs in the dev-preview panel
Selecting a ducklake/asset node in /pipeline_dev showed 'No producer for this asset' because selectionProducers wasn't passed (it's derived from the deployed graph on the route page, absent here). Compute it from the local graph's w/rw write-edges (incl. the // materialize target) and pass it through, mirroring the route page — so the panel shows the producing script and its (preview) runs, including data-test failures.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): carry annotation metadata onto local-graph runnables
The local graph emitted only path/usage_kind/in_pipeline/materialize_target per runnable, so /pipeline_dev and pipeline show --local weren't the same surface as the deployed graph for annotated scripts — missing the badges/lineage the shared canvas renders. Map the wasm-parsed partition_kind, freshness, tag, retry, data_tests, column_lineage, and materialize_strategy (derived append/merge/replace) onto each runnable, mirroring the deployed AssetGraphRunnableNode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): exclude event handlers that are lineage descendants from runAll
The runAll guarantee ('never fires an event handler with empty args') only held for event ROOTS — validStarts excludes them, but runAll then unions in descendants(dag, start), so a kafka/mqtt/... handler that also reads an upstream pipeline asset (a lineage descendant of a valid start) still landed in the plan. Add eventTriggerScripts() and subtract it from the selection after the descendant union. +unit test.
Also: docs/pipeline-local-dev.md recipe used 'pipeline docs demo_pipeline' without --local (default queries the deployed graph → hits the empty hint); add --local.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): whole-pipeline run cuts at event handlers (drop their downstream too)
The prior runAll fix subtracted event handlers from the selection but left their downstream: for manual_root → asset_x → kafka_handler → asset_y → consumer, deleting only kafka_handler left consumer selected, and topoOrder then ran it as a root with missing/stale event-derived inputs. Replace the descendant-union+delete with reachableCutting(dag, validStarts, eventHandlers): traverse from valid starts but treat event handlers as cut points, so a node reachable ONLY through an event handler is dropped while one reachable via a non-event path stays. +unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): recover // tag in the go/bash annotation fallback
The wasm path carries out.tag, but the go/bash fallback (and the wasm-error degradation path) only recovered pipeline + on, so a // tag gpu on a bash/go node — or a temporarily-unparseable ts/py/sql node — silently routed the local preview to the default worker while the deployed pipeline routes to the tag. Scan for // tag in fallbackParse too. +test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): extract shared assetProducers helper
The 'who writes this asset' write-edge derivation was copied verbatim in PipelineDevView and the pipeline route page — two copies that would drift. Extract assetProducers(graph, selection) into graphTraversal.ts and use it from both, keeping the dev view and route page in lockstep.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): only overwrite AGENTS.md/CLAUDE.md when it's the exact generated pointer
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): wire local-dev runs into the selected-node runs pane
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): exclude data_upload/webhook entrypoints from auto CLI runs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): --upload binds an object to a data_upload/webhook entry point
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): add "Run + downstream" to the dev preview detail form
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): cut non-autorun triggers on all run paths; multi-binding --upload
Address CI review: apply the data_upload/webhook/event barrier cut to the
single-root and bounded (--from/--to) paths, not just whole-pipeline; accumulate
repeatable --upload bindings per script (were overwritten); scope dev upload keys
by script+param to avoid basename clobbering; drop <script> from help text.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): reseed dev run form when a local edit changes the script's args
The read-only pane is keyed on script.path only, so in /pipeline_dev the selected
node re-resolves on every WS bundle without remounting; PipelineScriptView cloned
script.schema once, so adding/removing args left the run form on a stale schema
(could run with missing inputs). Extract PipelineRunForm (owns the SchemaForm
clone) and key it on the serialized schema: a real arg change reseeds the form,
an unchanged re-resolve keeps in-progress input.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): don't cut a scheduled/manual root that also has a non-autorun trigger
Address Codex P1: the barrier set subtracted only --upload-bound scripts, so a
script with both `// on schedule` and `// on data_upload` resolved as the start
yet was also a barrier — reachableCutting skipped it, giving an empty run plan.
Subtract all valid starts (schedule/manual roots + bound handlers) from barriers:
a legitimately-scheduled root runs on its schedule path even if it also carries a
caller-input trigger; pure input-only roots stay cut. Adds a regression test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): deployed non-autorun enrichment, s3:// storage, --to cut accounting, tag regex
Address CI review (Codex P1/P1/P2, Pi P2):
- Deployed `pipeline run` recovers marker-only data_upload/webhook/email triggers
from script bodies (like the `show` path) so input-only entrypoints are cut
instead of auto-run empty on the deployed graph.
- `--upload s3://<storage>/<key>` keeps the named storage (authority) instead of
folding it into the key, matching the S3Object round-trip convention.
- Bounded `--to` targets cut by a barrier are reported in droppedEnds (+warning),
not reachableEnds.
- fallbackParse `// tag` matches a single token (\S+), rejecting multi-word prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): header-only deployed marker scan, fail-closed enrichment, default-storage s3 keys
Address CI review (Codex P2, cubic P1/P1/P2):
- Deployed marker recovery scans the LEADING comment header only (shared
recoverHeaderMarkers helper, reused by the show enrichment too) so a body
comment `// on data_upload` can't inject a phantom trigger and over-cut.
- Deployed run enrichment fails CLOSED: a script-body fetch error aborts the run
instead of silently letting an input-only entrypoint run with empty args.
- Revert `--upload s3://` to default-storage whole-path keys (matching pipeline
`s3://` asset-URI semantics); named-storage authority-splitting broke nested
default keys like `s3://raw/2026/events.csv`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): reject trailing content on fallback native markers; trim s3:/// key
Address CI review (Codex P2, cubic P3):
- fallbackParse now requires a native marker (`// on data_upload`) to stand alone;
a line with trailing content (`// on data_upload f/foo`, `# on kafka topic`) is
rejected, matching the canonical parser and keeping local/deployed parity.
- s3UriKey trims a leading slash so the canonical empty-authority default form
`s3:///key` doesn't leak a leading slash into the object key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): persist dev WS token per-port so reconnect survives a CLI restart
Address Codex P2: the /pipeline_dev auto-reconnect reuses the ws_token from the
page URL, but `pipeline dev` minted a fresh random token each start, so a restart
on the same port left the open page rejected by verifyClient forever. Persist the
token per-port under the user-private config dir (0600) and reuse it on restart,
so an already-open page reconnects — matching the reconnect behavior's intent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): scope persisted dev WS token by workspace+folder+port
Address cubic P2: keying the persisted token by port alone let a stale browser
tab from a previous folder's session on the same port reconnect and receive a
different folder's source. Scope the token file by workspace+folder+port so a
same-session restart still reconnects, but a different folder on the same port
gets a distinct token that rejects stale cross-folder tabs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): caller args can't override skip-dispatch guard; hash the dev token key
Address CI review (Codex P1, cubic P2):
- makeLaunch / CLI run build args with `_wmill_skip_asset_dispatch` LAST (and drop
any caller-supplied copy) so a run-form/`--upload` arg can't re-enable backend
asset dispatch while the client orchestrates the cascade (double-run / running
deployed subscribers from a local preview). Adds a cascadeRun guard test.
- Dev WS token file key is a sha256 of NUL-delimited workspace+folder+port, so
different folders (`a/b` vs `a_b`) can't collide onto the same token file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): canonical s3://storage/key --upload parsing; scope dev token by remote+root
Address Codex P1/P1:
- Restore canonical S3Object URI parsing for `--upload` s3 sources, matching the
frontend's `parseS3Object` (`s3://<storage>/<key>`, empty authority ⇒ default,
`s3:///key`/`s3:///nested/key` for the default store). `s3://secondary/k.csv` →
`{ s3: "k.csv", storage: "secondary" }` so a named-storage object is read from
the right store. (This is the canonical convention; the default-storage nested
key is served by the `s3:///` form.)
- Scope the persisted dev WS token by remote+workspace+root+folder+port (was
workspace+folder+port), so two profiles on different remotes (or local checkouts)
with the same workspace/folder/port don't share a token — a stale tab can't
reconnect across a workspace/remote boundary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MCP tool arguments were converted to URL query values via `value.to_string()`
+ `trim_matches('"')`. For string values containing JSON (e.g. the `args`/`result`
filters on job listing, `args` on schedule listing), `to_string()` JSON-encodes the
string and escapes inner quotes with backslashes; stripping the outer quotes leaves
`{\"k\":\"v\"}`, which the backend's `serde_json::from_str` then fails to parse,
falling back to `FALSE` and returning zero results.
Use `value.as_str()` to emit the raw string content for `Value::String`, falling
back to `value.to_string()` for non-string types (numbers, booleans). Adds
regression tests covering JSON-string, non-string, and plain-string params.
Fixes WIN-2114
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Same grant gap already fixed for notify_event (20260619091631),
script_trigger (20260619112847), and dispatch_event: tables created after the
one-time GRANT ALL in 20250205131523 rely on ALTER DEFAULT PRIVILEGES, which
only covers objects created by the role that set them. On deployments whose
migration runner is a different role, these tables end up ungranted, and writes
that run under the RLS role (a transaction opened via user_db.begin(&authed) ->
SET LOCAL ROLE windmill_user/windmill_admin) fail with "permission denied for
table <name>".
Audited every table created after 20250205131523: these three are the only
ones with a confirmed write on a user_db transaction that lacked a grant:
- workspace_diff: UPDATE in set_ws_specific (workspaces.rs)
- materialized_partition: INSERT via record_materialization (assets API);
sibling materialized_asset_schema was already granted
- debounce_stale_data: DELETE in resume_suspended_trigger_jobs (global_handler.rs)
GRANT is idempotent so re-application is a no-op.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): add managed SCD2 history materialize strategy
`// materialize ducklake://... key=<col> history [track=...]` (alias: `scd2`)
upgrades the keyed merge to SCD type 2: diff the current snapshot against live
rows, close changed versions (valid_to/is_current) and open new ones in one
transaction, keeping full history. Adds a consumer-convenience <dim>_current
view; effective-dated joins via native ASOF JOIN >= valid_from. Managed, so
// data_test and schema capture work (unlike manual mode). Non-partitioned v1,
soft-delete on absence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): document scd2 track= spacing, reserved _current suffix, schema-freeze
Addresses non-blocking CI-review nits on the new SCD2 public surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): null-safe scd2 key matching + create _current view inside txn
Addresses CI review: (1) Codex P1 — NULL natural keys were flagged as changed
but silently dropped because `key IN (...)` never matches NULL; close/open now
match with `IS NOT DISTINCT FROM` via correlated EXISTS. (2) cubic P2 — the
`<dim>_current` view was created after COMMIT and CREATE VIEW advances the
DuckLake snapshot, so the summary recorded the view's snapshot instead of the
data write; the view is now created inside the write transaction. Validated both
against a real DuckLake (NULL key materialized; one snapshot per run).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): create scd2 _current view with IF NOT EXISTS to keep no-change runs no-op
Addresses CI review (Codex P2): CREATE OR REPLACE VIEW advances the DuckLake
snapshot every run, so an unchanged rerun still minted/recorded a snapshot. The
view definition is static, so IF NOT EXISTS creates it once (folded into the
first data-write snapshot) and is a true no-op thereafter — verified an unchanged
rerun keeps max(snapshot_id) constant. Also softens the reserved-name collision:
IF NOT EXISTS skips silently instead of erroring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): add scd2 deletes=close (hard-delete-close)
Opt-in `deletes=close` closes the current version of a key that disappears from
the snapshot (dbt's hard_deletes=close); default stays soft-delete. Codegen adds
a vanished-key temp set (current keys EXCEPT snapshot keys) + a second null-safe
close UPDATE with no reopen; a reappearing key opens a fresh version (validity
gap = correct SCD2). Wired through both parsers with parity fixtures/tests, worker
derivation, unit + codegen tests, and docs. Verified end-to-end against a real
DuckLake incl. delete-close + reactivation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): align materialize deploy precedence warning with runtime (scd2>append>merge)
The deploy-time conflict warning only knew append>key, so
warned 'append wins' while the runtime (duckdb_executor) runs SCD2 (history wins).
Warn for history+append (history wins, append ignored) before the append+key case,
mirroring the runtime strategy precedence. (Pi review P2.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): register scd2 _current view as a produced asset for cascade dispatch
The docs present the companion <dim>_current view as a subscribable produced asset
(// on ducklake://.../<dim>_current), but deploy registered only the base table as
a write asset, so a subscriber on the view would never be dispatched (the cascade
fans out from deploy-time asset rows). Register <dim>_current as a produced write
asset when scd2 so those subscribers fire. (Codex review P1.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): don't register _current asset for manual+history (no view created)
Manual mode short-circuits before the scd2 codegen, so no <dim>_current view is
created; gate the produced-asset registration on !manual so a contradictory
// materialize manual ... history doesn't leave a false write edge dispatching
subscribers on a nonexistent view. (Codex review P2.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dispatch_event table (migration 20260523055641) was created relying on
ALTER DEFAULT PRIVILEGES to reach windmill_user/windmill_admin. Those default
privileges only apply to objects created by the role that set them
(20250205131523), so deployments whose migration runner is a different role
leave dispatch_event ungranted. Direct writes then run as the invoking role and
fail with "permission denied for table dispatch_event" -- notably the DELETE in
delete_jobs (windmill-common/src/jobs.rs) that reaps a job's side rows on
schedule disable, and the dispatcher insert in asset_dispatch.rs.
Grant explicitly, same fix as notify_event (20260619091631) and script_trigger
(20260619112847). GRANT is idempotent so re-application (squash, or an operator
who already granted manually) is a no-op.
Fixes WIN-2112
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ansible): support repo-provided ansible.cfg in delegate_to_git_repo
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ansible): accept colon delimiter and collections_paths alias in cfg parser
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [ee] test(audit): de-flake S3 export end-to-end test under parallel tests
Bumps the EE ref to pull in the companion fix for the flaky
`audit_export_end_to_end` test (`ee::audit_s3_export`). Postgres XIDs and the
snapshot xmin are cluster-wide, so under `--test-threads` a neighbor test's
in-flight transaction can hold the global xmin between this test's row xids,
deferring a committed row to a later export tick (`id 7 must be exported: got
[3,4,5,6]`). The EE change models successive ticks (drain until exported) and
waits for pre-existing rows to settle before anchors that must exclude them.
Test-only; no production code changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to bdd4ba4dfd05c8ca7db365e530812dc914517d7f
This commit updates the EE repository reference after PR #639 was merged in windmill-ee-private.
Previous ee-repo-ref: 70c4c61257bda9263c158ef0ac58eb3aa9c55fa8
New ee-repo-ref: bdd4ba4dfd05c8ca7db365e530812dc914517d7f
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: enforce tls verification for postgres verify-ca/verify-full sslmode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: make PG_ACCEPT_INVALID_CERTS value-based and keep cache key well-formed
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: grandfather existing postgres resources via per-resource trust_cert flag
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scope pg trust_cert migration to resource data, drop schema patch
Hub resource type sync (windmill cache-rt + startup SYNC_CACHED_RT) only touches the admins workspace and is opt-in, so the schema is left to the hub; the migration just grandfathers existing resource values so the upgrade is non-breaking.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: replace pg trust_cert with verify-*-scoped accept_invalid_certs, drop migration
Per-resource accept_invalid_certs (default false for new resources) replaces the trust_cert flag and grandfather migration. It only applies to verify-ca/verify-full; unset falls back to legacy behavior (verify only when a root cert is present) so existing and git-synced resources are not broken on upgrade.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: warn in job logs when a verify-* postgres resource skips cert verification
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* [ee] feat(licensing): enforce offline license seat cap
Companion to windmill-ee-private. Aligns the offline-license seat count
with the billing model and adds real-time enforcement when usage exceeds
the cap. OSS side carries the ee_oss stubs, the reactivation cap-check
call site, the regenerated SQLx cache, and the EE ref bump.
- Exclude instance-disabled users (password.disabled) and service
accounts from the seat count. Deactivating a user now frees a seat.
- Service accounts no longer consume seats (no check at creation).
- Hard-block reactivation when it would exceed the cap.
- Invalidate the license (halting jobs) when seat usage exceeds the cap,
mirroring CU-cap enforcement; recovers when usage drops back under or a
higher-cap key is loaded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [ee] fix(licensing): bump EE ref for reactivation seat-check fixes
Points to the EE companion commit that fixes reactivation double-counting
and preserves the original seat alert tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [ee] fix(licensing): reactivation seat delta includes pending invites
Bumps the EE ref and drops the now-orphaned usr-only cache entry; the
reactivation check reuses the existing usr ∪ workspace_invite query.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* [ee] test(licensing): bump EE ref for offline seat-cap tests
Adds #[sqlx::test] coverage for the offline seat counting and cap-check
logic; EE-only (runtime queries, no cache change).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: update ee-repo-ref to f814c3f75308c1ef1e4526d8d0eeb360ce16abe4
This commit updates the EE repository reference after PR #637 was merged in windmill-ee-private.
Previous ee-repo-ref: b2622e3afc2fe1fe3e2ec978ca46cf9decf91b82
New ee-repo-ref: f814c3f75308c1ef1e4526d8d0eeb360ce16abe4
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* [ee] fix(s3_proxy): preserve URL-encoding on forward re-sign for Hive-partition keys
Bump ee-repo-ref to pull the EE fix for SigV4 SignatureDoesNotMatch on
DuckLake Hive-partition writes through the S3 proxy. The forward re-sign
leg rebuilt the upstream URI from the decoded object key (literal `=`)
instead of the still-encoded request path (`%3D`), diverging from how
S3/minio canonicalizes the key. Companion EE commit c6b110f.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 1a98119b0b8b8548601983c9e5ab091150f0b180
This commit updates the EE repository reference after PR #638 was merged in windmill-ee-private.
Previous ee-repo-ref: c6b110fd3b3591a5c3f09952c388c42bd5766188
New ee-repo-ref: 1a98119b0b8b8548601983c9e5ab091150f0b180
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: add dev workspaces paired with a lockable prod workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: gate dev-workspace prod-lock on admin and prevent attach cycles
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: redirect locked-prod edits into the dev workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: make dev-workspace settings tab available on CE (was EE-gated)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: lock prod against forking too and funnel edits to the dev workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: open dev item page on edit and tailor dev-workspace lock messages
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: prevent nested dev workspaces and hide dev option when one exists
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: drop the redundant already-has-dev hint on the fork form
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: badge dev workspaces and sort them ahead of forks in the tree/switcher
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: label dev workspaces as 'Dev workspace of X' instead of 'Fork of X'
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: label edit as 'Edit in <dev>', cover editor headers, auto-expand dev in tree
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: split prod lock into separate block-deploy and prevent-forking toggles
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: make resources/variables workspace-specific from compare page
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: steer AI-chat sessions to the dev workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: refine session fork options and lock guidance for dev/prod
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: session picker reads prod's real rules, default to current ws
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: copy members into forks and clarify dev-workspace root labeling
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: place the workspace id field under the fork name
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address dev-workspace review findings and harden fork detection
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: regenerate sqlx offline cache
Restores entries dropped during the origin/main merge and adds the
dev-workspace queries (is_dev_workspace, ws_specific, has_parent).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address second-round dev-workspace review findings
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address Pi and Codex review findings on dev-workspace endpoints
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: gate locked-dev git-branch fork on admin and validate ws_specific path
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: clear prod dev-lock when deleting an attached dev workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: consolidate dev-workspace migration and scope all-group join to attach
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: restore dev-workspace CHECK into consolidated migration and scope all-group join
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: drop copy_members from the dev-workspace attach path
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: dev-workspace lifecycle/auth fixes from Codex review round
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: explicit create-in-other for workspace-specific items
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: make create-in-other strictly create-only (never overwrite target)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: return 403 (not 401) for dev-workspace permission denials
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: allow attaching a same-family fork as a dev workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: emphasize the go-to-dev action in the no-direct-deploy alert
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: seed a resource's linked variables when creating it in the other workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: judge workspace deploy/fork locks against the user's identity in that workspace
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* style: clarify create-in help text in workspace-specific panel
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: admin-gate dev-workspace creation and harden lock/seed edges
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: preserve a staged fork's source on picker create-mode re-entry
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: clear dev flag on archive and check dev existence server-side
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: make create-in-other atomically create-only via direct create
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: create-only resource insert, ws-specific list scopes, archive lock guard
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: reserve the dev_workspace_lock protection-rule name from the public API
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: reattach create_protection_rule doc comment to its function
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: make dev-archive pairing teardown atomic with the archive
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: follow deploy_to on root rename; show dev pairing to non-member prod admins
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: copy creator metadata on fork; invalidate fork routing cache on rename
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: accept g/ paths in set_ws_specific; gate copy_members to dev workspaces
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
build_gcs_client always called `.with_service_account_key(...)`, so an
absent key (the settings UI stores "no key" as the empty JSON object `{}`)
was handed to the builder and failed to parse instead of falling through
to the object_store crate's InstanceCredentialProvider. Skip the call when
the key is blank so GCS uses the instance's ambient credentials (GKE
Workload Identity / the GCP metadata server).
"Blank" (empty/whitespace/`{}`/`null`) is centralized in a shared
`gcs_service_account_key_is_blank` predicate so the build path and the
non-super-admin connectivity-test SSRF guard (`validate_object_storage_test`)
agree on what counts as "no key" — otherwise a blank key would bypass the
guard yet still trigger the ambient-credential fallback, letting an
untrusted caller probe arbitrary buckets with the server's instance role.
Also clarify the settings UI hint that the key may be left empty for
ambient credentials, and add regression tests for the blank-key build path
and the guard.
Fixes WIN-2110
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MermaidDisplay only showed the rendered SVG, hiding the raw source once
rendering succeeded. Add a copy button in the showSvg branch mirroring the
pattern in HighlightCode.svelte so the diagram source can be extracted.
Fixes WIN-2109
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(gcp): require token verification for authenticated push delivery
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2
This commit updates the EE repository reference after PR #636 was merged in windmill-ee-private.
Previous ee-repo-ref: 8c63d487c486002baf09c77ab937fd77a91765eb
New ee-repo-ref: 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(pipeline): AI chat tools to build pipeline nodes with diff/approval
Add a data-pipeline AI chat experience modeled on the flow editor and
surfaced through the dev-gated global chat (no new chat panel).
The /pipeline editor registers PipelineAIChatHelpers on the AIChatManager;
while it is open the global mode layers pipeline tools, a pipeline prompt
section, and the helpers on top of the full global tool set (behavior is
unchanged when no pipeline editor is open).
New tools (frontend/src/lib/components/copilot/chat/pipeline/core.ts):
- get_pipeline_graph / read_pipeline_node — read the live graph and bodies
- build_pipeline_node / edit_pipeline_node — stage changes as AI-pending drafts
- remove_pipeline_node — drop a staged proposal
- test_pipeline_node — preview-run a node (requires confirmation)
Tools never deploy: they stage drafts flagged aiPending, rendered on the
canvas with an accent ring and reviewed via Accept all / Reject all (the
flow editor's GlobalReviewButtons). Accept commits the drafts; Reject reverts
to a pre-AI snapshot, preserving earlier accepted drafts. Auto-accept is gated
on the chat autonomy mode.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): teach the global/session chat to author data pipelines
Without an open /pipeline editor the session chat had no pipeline concept, so
"create a data pipeline" loaded flow instructions and built a flow. Add a
first-class pipeline authoring path:
- system_prompts/base/pipeline-base.md — what a data pipeline is (a DAG of
annotated scripts wired by storage assets, NOT a flow) and how to author the
// pipeline / // on / // materialize annotations; wired through generate.py as
getPipelinePrompt() (regenerated prompts.ts/index.ts).
- global/core.ts — new get_instructions subject "pipeline", and a global-prompt
rule disambiguating data pipelines from flows so the model routes correctly.
- ai_evals/cases/global.yaml — two global cases (single node, two-node chain)
asserting pipeline-annotated script drafts and forbidding write_flow, guarding
the pipeline-vs-flow conflation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline): show & build pipelines in the AI session preview
Add a 'pipeline' session preview target so the session AI can show the
data-pipeline graph for a folder and build nodes in-pane:
- open_preview now accepts kind="pipeline" (path = folder); SessionTarget /
EDITOR_TARGET_KINDS widen accordingly. The slot/codec load model stays
flow|script|raw_app — pipeline bypasses it with its own fetch/draft state.
- New PipelineEditorView.svelte mounts in the session pane: fetches the
folder graph, overlays AI drafts, renders AssetGraphCanvas + the
Accept/Reject review buttons, and registers PipelineAIChatHelpers on the
*session-scoped* manager (via getAiChatManager) so build_pipeline_node /
edit_pipeline_node + the diff/approval work inside the session too.
- System prompt nudges the model to open the pipeline preview and use the
staging tools while building.
Verified end-to-end with a real model: the session AI called open_preview,
the graph mounted in the side panel, then build_pipeline_node staged a node
on the session canvas with its schedule trigger and ducklake output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): share the AI editor logic between route page and session
Consolidate the duplicated data-pipeline AI logic onto a single shared layer so
the route editor and the in-session preview behave identically and the session
gains the full code editor.
- New pipelineAiHelpers.ts: createPipelineAiHelpers(deps) owns the propose/edit/
remove/accept/reject/test staging + the per-turn snapshot bookkeeping that
powers Reject. Callers inject accessors for their own draft Map and graph.
- Route page (/pipeline/[folder]) drops its ~250-line inline AI-helper block and
wires the shared factory via deps (folder/workspace/graph/drafts + focus,
ensureEditable, run-started). Its shell — persistence, navigation guard,
activity, cascade, trigger drawers — is untouched.
- Session PipelineEditorView uses the same factory and now renders the real
AssetGraphDetailsPane (code editor + live overlays + test), so a node built in
a session opens with its source, matching the route editor.
Verified: route page hydrates/renders drafts unchanged; in a session the AI
opened the pipeline preview, built a node, and its code showed in the details
pane. check:fast clean, 197 unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): externalize editor state into PipelineEditorState (step 1)
Introduce PipelineEditorState — the data-pipeline analogue of the flow editor's
flowStore. It owns the draft Map, the live editor overlays, and the selection,
with callback-safe methods (handleDraftPersist / handleAnnotationsChange / … ),
so a single editor can be rendered by both the route page and the session.
This commit lands the store and points the in-session PipelineEditorView at it
(no behaviour change — the session already had these inline). Next steps move the
route page onto the store and a shared <PipelineGraphEditor>.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): point the route editor at PipelineEditorState (step 1)
Move the route page's draft Map, live editor overlays, selection, and the
draft-persist / live-change handlers onto the shared PipelineEditorState (`pe`),
referencing them as `pe.*` in place. No behaviour change — persistence, graph
resolution, run dispatch, AI staging, and deploy all stay on the page and now
read/write the externalized state.
This is the data-pipeline analogue of the flow editor's flowStore: the route
page and the in-session preview now share one source of editor truth, setting up
the shared <PipelineGraphEditor> in the next steps.
Verified: the page hydrates its DB draft, renders the overlay graph, the toolbar
counts (Save all (N)) track pe.drafts, and selecting a node opens it in the
details pane. check:fast clean, 84 unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): render the route editor via shared PipelineGraphEditor (step 2)
Extract the canvas + details-pane editor body into PipelineGraphEditor.svelte,
the data-pipeline analogue of FlowBuilder. The route page now delegates its
Splitpanes block to it, passing the externalized PipelineEditorState plus its
run/cascade/trigger/deploy callbacks; the component owns pane sizing,
selection/details-open derivation, and the canvas+details rendering.
Root-caused the earlier ts2769 "$props() No overload" to a prop named `state`
colliding with the `$state` rune (`let x = $state(...)` parsed as a store
auto-subscription on the prop) — the prop is now `editor`.
Net: the route page sheds ~310 lines of template/state; behaviour preserved.
Verified: the page hydrates its DB draft, renders the graph, opens the draft in
the details pane (live code editor + Test), pane sizing works. check:fast clean,
24 pipeline tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): move draft autosave into PipelineGraphEditor (step 3)
Fold the per-user `data_pipeline` DraftService bundle autosave (hydrate +
debounced persist + localStorage crash mirror) into PipelineGraphEditor, gated
by a `persistDrafts` prop — FlowBuilder's parameterized-autosave shape. The route
page passes `persistDrafts` + `folder` and reads `editor.loadedFromDbDraft` for
its AutosaveIndicator; the in-session preview will leave persistence off.
Also restores the `untrack(...)` wrapping on the pane-sizing $effect (dropped
when the editor body was extracted in step 2). Without it the Pane `bind:size`
feedback loops the effect and pegs the main thread when the details pane is
closed — a latent hang in the step-2 commit.
check:fast clean, 24 pipeline tests pass. Note: browser revalidation was not
possible this session (the Playwright MCP browser was reset); the autosave is a
verbatim port and the untrack fix is the original working form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(pipeline): render the session preview via shared PipelineGraphEditor (step 4)
Point the in-session PipelineEditorView at the shared PipelineGraphEditor instead
of its own inline canvas + details pane. The session now renders the exact same
editor body as the route page — gaining the full details/code pane — while opting
out of persistence (persistDrafts=false) and the run/cascade/trigger/bounded
affordances (their callbacks are omitted, so those controls hide). Building nodes
+ the Accept/Reject diff still work via the AI helpers.
Also fixes issues surfaced by a full `svelte-check` while wiring this up:
- PipelineGraphEditor: edit mode opened the details pane unconditionally (a step-2
regression); restored the route's "open only on selection/draft" behaviour.
- Route page passed an `isOperator` prop the component doesn't accept (step-2;
caught only by full check, not check:fast).
- SessionItemNotFound: narrow its `kind` to exclude `pipeline` (pipeline targets
never slot-load, so they can't 404 through it) — closes the SessionTarget-widen
fallout.
- PipelineEditorView: cast the resolveGraph base to AssetGraphResponse.
Full `svelte-check` now clean across all pipeline/session files; 137 unit tests
pass. (Browser revalidation still pending — Playwright MCP was unavailable this
session; see the smoke-test note on the PR.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): stop an infinite microtask loop when persisting a no-output draft
handleDraftPersist short-circuits when the open draft's content + inferred writes
are unchanged. The writes check compared `d.outputAssets?.length === writes.length`,
but a no-output draft has `outputAssets: undefined` (so `?.length` is `undefined`)
while the details pane infers an empty `writes: []` (length 0). `undefined === 0`
is false, so it never short-circuited: every persist re-wrote the drafts Map with
an equivalent object, which gave `activeDraft.script` a new identity → the pane
re-emitted its overlays → the graph re-derived → persist fired again. A self-
sustaining microtask loop that pegged the renderer and froze the tab on any
pipeline carrying a no-output draft (e.g. hydrating one from the saved
data_pipeline draft on load). It hangs rather than throwing effect_update_depth_
exceeded because it cycles across microtasks, not within one reactive flush.
Fix: coalesce the undefined length to 0 so "no outputs" compares equal to an empty
inferred-writes list. Adds pipelineEditorState.test.ts covering the idempotency
(fails without the fix) plus the change/no-change cases.
Root-caused by instrumenting the reactive churn: every iteration reassigned
drafts/liveContent/liveBodyAssets/liveAnnotations/displayGraph with identical
values — pure reference churn off the drafts re-write.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): make the agent open the pipeline editor before building nodes
In a session, the GLOBAL system prompt only *advised* opening the pipeline preview
("show its graph with open_preview ... prefer those tools once it is open"), so
the agent routinely skipped it: on a plain "build a data pipeline" request it
reached for write_script and staged plain script drafts, and the canvas editor
never opened. build_pipeline_node / edit_pipeline_node are only registered once
the preview is open, so skipping open_preview also loses the canvas-staged
Accept/Reject diff-approval flow entirely.
Make the guidance imperative: open_preview(kind="pipeline", path=<folder>) is the
FIRST step before creating any node (an empty or not-yet-created folder is fine —
create_folder first if needed), and pipeline nodes go through build_pipeline_node
/ edit_pipeline_node, never write_script. This also clears the agent's "the folder
might not exist" hesitation that pushed it toward write_script.
Verified live (same plain prompt, before/after): before it used write_script with
no editor; after, the agent opens the editor first and stages a canvas-highlighted
node with Accept all / Reject all. The guidance is gated on previewTools
(session-only), so it doesn't affect the non-preview global eval cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipeline): preserve in-session pipeline drafts across editor hide/show
The session preview's PipelineEditorState lived in the PipelineEditorView
component with persistDrafts=false. Hiding the editor sets editorVisible=false,
which makes `hasEditor` false and the `{#if hasEditor}` block unmount the view —
discarding its component-local store. Showing it again remounted a fresh, empty
one, so the pipeline the AI had built in the session vanished.
Move the PipelineEditorState onto the per-session SessionRuntime (like the flow /
script / raw_app editors, which already host their state there and take {runtime}),
so it survives the pane unmount on hide and across session switches. The runtime
is keyed by session id and only dropped on session deletion.
Because the instance is now reused, guard against a retarget to a different
folder: PipelineEditorView resets the state when `path` changes to a new folder
(a same-folder remount keeps the drafts). Adds `folder` + `reset()` to the store.
Verified: build a node in a session → Close editor → Show editor → the staged
node, its wiring, the details-pane code, and Accept/Reject all re-appear. Full
svelte-check clean; 139 pipeline tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipeline-ai): clearer diff + persistent review banner on the canvas
The AI review affordance had two problems on the pipeline canvas:
- The floating Accept-all / Reject-all bar sat bottom-center, where it
collided with the minimap once the canvas narrowed on node selection —
reading as "the buttons vanished when I select a node".
- Every staged draft rendered with the same blue ring, so it wasn't clear
what the review would actually change (a plain manual draft looked the
same as an AI proposal).
Replace the floating bar with a top-left review banner (z-30, clear of the
controls and minimap) that stays put regardless of selection and spells out
the pending counts. Color the diff per node: a proposal that adds a node
that isn't deployed rings green with a "new" chip; one that edits an
already-deployed node rings amber with an "edited" chip. Plain manual
drafts keep the neutral gray dashed border, so only the green/amber nodes
read as part of the Accept/Reject set.
aiPendingKind is resolved in resolveGraph (deployed runnable present →
modified, else added) and forwarded through the canvas to the node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): persist in-session pipeline proposals across reload/switch
Staged AI proposals lived only in the per-session runtime's in-memory
PipelineEditorState (persistDrafts=false), so a page reload — and an
LRU-evicted runtime on session switch — dropped them, leaving the canvas
and the Accept/Reject review empty even though the chat still showed the
nodes as staged.
Enable the same per-folder DB-draft persistence the route page uses for the
in-session editor. To keep hide/show cheap and race-free, hydration is now
gated per editor instance (PipelineEditorState.hydratedFromDb) rather than
per component mount: the runtime-hosted instance hydrates ONCE when fresh
(reload / evicted runtime) and then keeps its in-memory drafts across the
editor pane unmounting on hide — re-reading the DB on every remount would
race a not-yet-flushed autosave and drop a just-staged draft. A folder
retarget resets the flag so the new folder re-hydrates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): make Reject all work for rehydrated proposals
rejectAll only reverted paths tracked in the in-memory aiSnapshots map,
which is rebuilt empty on each editor mount. After a reload (or session
switch into a fresh runtime) the proposals are restored from the persisted
draft but have no snapshot, so Reject all was a no-op on exactly the nodes
it should discard. Sweep any still-pending draft without a snapshot and
discard it (revertPath with no snapshot deletes the path; for an edit of a
deployed node that correctly falls back to the deployed body). Adds unit
coverage for accept/reject including the no-snapshot case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): keep proposals visible while the graph reloads on switch
The session editor pane is LRU-capped (MAX_WARM_EDITORS), so returning to a
session whose pane was evicted remounts PipelineEditorView with a fresh
graphRes resource (loading=true, current=undefined). The deployed-graph
loading spinner gated the whole canvas, so the staged proposals and the
Accept/Reject review banner vanished until the re-fetch resolved — read as
"the proposal disappears when I switch sessions".
Only show the loading/error placeholder when there are no drafts to display.
When the runtime already holds staged drafts, render the editor immediately:
resolveGraph overlays them on an empty base so the proposals + banner stay
visible, and the deployed nodes fill in when the fetch completes. Verified
with a 4s-delayed graph fetch — proposals render through the load with no
spinner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(pipeline-ai): apply AI node edits directly as drafts, no approve/reject
The canvas-level Accept all / Reject all review (aiPending proposals, the
green/amber diff ring + "new"/"edited" chips, and the review banner) didn't
fit the pipeline editor. Match the flow/script editor instead: build/edit
apply directly as ordinary unsaved drafts on the canvas, which the user then
deploys — there is no separate approval step.
Removed across the surface:
- aiPending / aiPendingKind on the runnable node + resolveGraph seeding +
canvas forwarding; AI-built nodes now render with the existing plain
unsaved-draft dashed styling.
- the review banner, count derivations, and hasAiPending/onAccept/onReject
props from PipelineGraphEditor and both consumers (route page + session
view).
- acceptAll/rejectAll/hasPending and the per-turn snapshot bookkeeping from
the shared helpers; removeProposedNode now just discards the unsaved draft
at a path (undo a build). acceptAllProposals/rejectAllProposals/
hasPendingProposals dropped from the PipelineAIChatHelpers interface and
the manager's auto-accept hook.
- accept/reject language from the tool descriptions, return messages, and the
system-prompt section.
Tests updated; pipeline + AssetGraph suites pass (142).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(drafts-diff): support data_pipeline diffs + fix blank empty-summary row
Two issues in the session "Drafts" diff drawer (DraftDiffDrawer):
- Clicking a `data_pipeline` bundle row threw "Draft diff not supported for
kind data_pipeline" (utils_draft_deploy.ts) — there was no handler for the
kind, so it fell to the OVERLAY_GETTERS lookup and errored. The bundle has
no deployed counterpart (each node deploys individually as a script), so
diff it node-by-node: surface each node's draft body keyed by path, folding
in the deployed body as the "before" when a node edits a deployed script.
- A draft row whose summary is an empty string (e.g. the app draft) rendered
with no title at all: WorkspaceItemRow's single-line branch used
`summary ?? secondary`, and `??` doesn't treat '' as absent, so it showed
the empty summary instead of the path. Use `||` so an empty summary falls
back to the path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(drafts-diff): explode data_pipeline bundle into per-node subitems
A data_pipeline draft is a bundle of node-script drafts, so a single row
diffed the whole thing as one blob. Explode it in DraftDiffDrawer into one
script row per node, nested under the bundle's `…/data_pipeline` folder so
they read as the pipeline's subitems — each with its own path and a proper
script Content/Metadata code diff. The node's draft body is the "after"; its
deployed body (when the node is already deployed) is the "before", so edits
show as line diffs and new nodes as added. A single bundle row (via the
getDraftDiffValues data_pipeline fallback) is kept only for the case where
the bundle can't be read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor(pipeline-ai): simplify — drop vestigial approve/reject scaffolding & redundant field
Review pass over the PR, removing complexity left from the approve/reject
removal and the shared-component refactor (all behavior-preserving):
- Inline the `acceptPendingEdits` pass-through into `acceptPendingFlowEdits`
and revert the now-inert `autoAcceptEditsAvailable` GLOBAL+pipeline widening
(pipeline edits are direct drafts — nothing to auto-accept).
- Fix the global system prompt: pipeline tools "apply directly as unsaved
drafts (no accept/reject)", not "proposals the user Accepts or Rejects".
- Collapse the redundant `outputAsset` (singular) into `outputAssets`,
removing a whole resolveGraph fallback tier; simplify propose/editNode.
- Drop the single-field `PipelineAiHelpersHandle` wrapper (callers just
destructured `{ helpers }`); inline the misleading `isoNow()` helper.
- Remove the now-unreachable `data_pipeline` branch in getDraftDiffValues
(the drafts drawer explodes bundles per-node; an unreadable bundle is
skipped) and the "Step N consolidation" drafting narration.
- Un-export internal-only types; reuse `storageKey`; refresh stale comments
that still referenced proposals / the review banner / diff-approval.
svelte-check clean; 141 unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(pipeline): tooltip clarifying the Create/Save button deploys
The accent button in the asset-graph details pane ("Create" for a new script,
"Save" for an existing one) is really a deploy, but had no tooltip explaining
that. Add a title — "Deploy this new script to the workspace" / "Deploy your
changes to this script" — keeping the create-vs-update label distinction while
making clear both deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(pipeline-ai): document the `materialize` annotation in the pipeline prompt
The model invented "materialize run" because the prompt only mentioned
`// materialize <uri>` in passing. Spell out what it is in both the in-app
pipeline prompt (getPipelinePromptSection) and the base prompt
(pipeline-base.md, regenerated): a MANAGED output where the runtime writes the
table around a single SELECT (no manual CREATE/INSERT); replace (default) vs
`append` vs `key=<col>` strategies; `manual` to opt out (track-only); and its
pairing with `// partitioned …` (runs once per partition, `{partition}` token
substituted at run time). Explicitly: materialize is an output declaration,
not a command — there is no "materialize run".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(pipeline-ai): trigger drawers in the AI session preview
Bring the route page's native-trigger affordances to the in-session pipeline
editor by reusing the shared <PipelineTriggerEditors> (no duplication of the
drawer UI). Clicking a "Schedule · Missing — no trigger row" node (or
edit/delete on an attached trigger, webhook, data-upload) now opens the same
drawers the full editor uses, instead of doing nothing. Draft nodes get the
same "save the script first" guard (a trigger row needs a deployed script).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(pipeline-ai): run buttons + live run state in the AI session preview
Wire the per-node Run button and live run-state badges into the in-session
pipeline editor, reusing the shared folder-scoped job poll
(useActiveRunnableIds) the route page uses — node badges, the event log, and
the zero-latency "running" hint all come from it. The session runs one node at
a time (preview for an unsaved draft, the deployed version otherwise),
skipping the route page's cascade/deploy-queue machinery the AI-session UX
doesn't need. Verified: a node's Run button dispatches a job and the badge
updates live from the poll.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(pipeline): label the node deploy button "Deploy" (was Create/Save)
Users read "Create" and asked whether it deploys. It does — and the main
script editor's DeployButton already says "Deploy", so this is the consistent
term. Use "Deploy" for both the new-script and existing-script cases; the
new-vs-changes nuance stays in the button's tooltip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(home): surface data pipelines as units, including bundle-phase drafts
Treat a data pipeline as one home entry instead of scattering its member
scripts:
- The home "Pipeline · f/<folder>" entry now also covers bundle-phase
pipelines — a folder that so far only exists as a `data_pipeline` draft —
not just deployed ones, so a pipeline shows up the moment its first node is
drafted (union listPipelineFolders + data_pipeline draft folders).
- Pipeline-member scripts (`auto_kind='pipeline'`) are filtered out of the
individual scripts list; they're represented by their pipeline's entry.
- Tree view injects pipeline folders so they (and their "Pipeline" entry)
still appear when their only scripts are hidden members or they have none
deployed yet.
Verified in both list and tree view: app_groups (deployed member folded) and
a draft-only nyc_transit both show as pipelines; the member script no longer
lists individually.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(scripts): compute auto_kind for draft-only pipeline nodes
A never-deployed pipeline node (a script draft starting with `// pipeline`)
had no script row, so list_scripts synthesized it with `auto_kind: None` — and
the home page therefore couldn't tell it was a pipeline member, listing it
individually instead of folding it into its pipeline. Parse the draft content
the same way the create path does (`parse_pipeline_annotations(...).in_pipeline`)
and set `auto_kind = "pipeline"` on the synthesized draft-only row, so draft
nodes fold into their pipeline like deployed members.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(search): hide pipeline-member scripts from global search
The Ctrl+k global search listed pipeline-member scripts (`auto_kind='pipeline'`)
individually. Filter them out — they're reached through their pipeline, matching
the home page. Deployed members carry auto_kind from the script row; draft-only
members now do too (computed from draft content in list_scripts), so both are
excluded here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline): address PR review findings
Session run dispatch (the one real bug):
- runNode now passes `_wmill_skip_asset_dispatch: true` for a single-node run
of a deployed node unless the user chose "run + downstream" (cascade) —
previously a single Run could fan out to downstream deployed scripts via the
backend asset dispatcher and fire side-effecting production runs.
- onRunProducer guards `kind === 'script'`; onTestStateChange only clears the
run hint for the script the pane finished (not a different in-flight node);
clear the hint on folder retarget; gate the background poll on isActiveSession
so hidden warm panes don't poll; note the PipelineTriggerEditors workspace
coupling.
Home page pipeline surfacing:
- Fold pipeline-member folders into `pipelineFolders` (captured in loadScripts)
so a members-only / draft-only-`// pipeline` folder still shows its pipeline
entry instead of vanishing; and don't render the empty-state when only
pipelines remain (they aren't part of the text filter).
- Insert injected tree folders in name order instead of prepending.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(pipeline-ai): make clear `// materialize` is DuckDB + DuckLake only
The model put `// materialize` on a python3 node, which deploy rejects ("only
supported for DuckDB scripts"). The prompt only implied SQL ("write the body
as a single SELECT") without stating the hard constraint. Spell it out in both
the in-app prompt and pipeline-base.md: `// materialize` is DuckDB-only and its
target must be a DuckLake table; for python3/bun/postgresql nodes, write the
output via the SDK instead and let it be inferred — reach for duckdb when a
node should materialize a DuckLake table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(pipeline-ai): fix stale comment — session now wires run + trigger affordances
Addresses review: the comment still claimed the session 'opts out of the
run/cascade/trigger/bounded affordances', but run buttons + trigger drawers
were wired in. Describe the current state (wires run + triggers; omits only
cascade/bounded/add-script).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline): address Codex review — test_pipeline_node dispatch + tree search
- [P1] testNode (the test_pipeline_node tool) ran a deployed node via
runScriptByPath without `_wmill_skip_asset_dispatch`, so previewing one node
could fan out to downstream deployed subscribers and run side-effecting
scripts. Add the skip flag (test is always single-node) + a regression test.
- [P2] Home tree view injected pipeline folders — and rendered their Pipeline
row — even during a text search, surfacing unrelated pipelines. Gate both the
TreeViewRoot injection and TreeView's hasPipeline on `!isSearching`, matching
the list view which hides pipeline rows on a query.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): keep the pipeline prompt after update_user_instructions
rebuildGlobalSystemMessage (called by the update_user_instructions tool)
rebuilt only the base Global prompt, dropping the pipeline-editor section that
configureGlobalMode appends. So after the chat remembered an instruction, the
next GLOBAL turn lost the active /pipeline/<folder> context + direct-draft/
materialize guidance while pipeline tools stayed registered. Re-append the
pipeline section here when a pipeline editor is registered.
Addresses Codex review [P2].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(home): gate pipeline entries by kind/archived/owner filters
Codex review [P2]: pipeline rows/folders rendered independently of the item
filters, so a pipeline still showed under the Flows/Apps tabs, in the archived
view, and outside a selected owner. Add `visiblePipelineFolders` applying the
same gates the items get (kind ∈ {all, script}, not archived, owner-prefix
match) and route the list rows, tree injection, and empty-state check through
it. Pipelines are always `f/<folder>`, so the user-folder toggle and kind=script
keep including them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline): address review — route folder-switch state, AI node guards, diff identity
claude[bot] [P1]: the route page's in-app folder switcher navigates same-route
(no remount), but nothing reset PipelineEditorState — so folder A's drafts
displayed under B and autosave persisted them into B's bundle, and B never
hydrated. Reset pe on folder change (mirror the session retarget), and guard the
shared hydrateDrafts against a stale folder result landing after a retarget.
codex/claude [P2]: build_pipeline_node (proposeNode) only checked drafts.has —
now rejects a path outside the open folder and one colliding with an existing
deployed node (model should edit_pipeline_node). + 3 regression tests.
codex/claude [P2]: exploded pipeline-node diff rows shared `script/<path>` with a
standalone script draft at the same path, colliding in the {#each} key + value
cache. Add an explicit unique `key` (the distinct bundle-nested path) on DiffRow;
pipeline nodes set/look up by it while `path` stays the real edit target.
claude [P2]: session AI test_pipeline_node now arms the live run badge
(onRunStarted), matching the route page.
nit: pipelineAiHelpers.test uses afterEach(restoreAllMocks) instead of an
unreachable inline mockRestore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline): harden AI node mutations + close home label-filter / rename gaps
Codex [P1] (AI mutations trust model paths) — fully scoped now:
- editNode validates the open folder too (proposeNode already did), via a shared
assertInFolder; an edit_pipeline_node for f/other/* no longer persists an
unrelated script into the current folder's data_pipeline bundle.
- both build_pipeline_node and edit_pipeline_node now require the `// pipeline`
annotation (assertPipelineAnnotation) so a staged draft is definitionally a
pipeline member, not a silently-non-member script. + tests.
(proposeNode's folder + deployed-collision guards landed in the prior commit.)
Codex [P2] home label filter — visiblePipelineFolders ignored labelFilter, so a
label selection still showed every pipeline (and the empty-state fell through to
render pipeline rows). Pipelines carry no labels, so a label filter hides them.
Codex [P2] session rename — PipelineEditorView now wires onScriptRenamed
(repoint selection + refetch), matching the route page; a persisted-script
rename no longer leaves the canvas on the old path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(pipeline-ai): language-specific comment prefix for annotations
Codex [P2]: the tool schema and prompt told the model to write `// pipeline` /
`// on` / `// materialize` regardless of language, and pipeline-base.md grouped
SQL with `#`. A `//` (or `#`) annotation line is invalid in a DuckDB/Postgres
node — it passes the frontend parser (which strips `//`/`--`/`#`) but is a SQL
syntax error at deploy/run. Make the guidance language-specific everywhere:
`--` for SQL (duckdb/postgresql), `#` for python3/bash, `//` for bun/TS — the
`//` in examples is the TS form to translate. Regenerated the prompt outputs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): re-scope Global prompt on folder switch + language-aware base prompt
Codex [P2] x2:
- The route page resets editor state on an in-app folder switch, but the Global
chat's system message kept the old `/pipeline/<folder>` scope (the helper
methods read the reactive folder, but the prompt string is only rebuilt on
Global-mode reconfigure). Rebuild it on folder change so the next turn targets
the new folder.
- The pre-editor base Global prompt (seen before open_preview/get_instructions)
still showed TS-only `// pipeline` / `// on`. Make it language-aware (`--` SQL,
`#` Python/Bash, `//` TS) so the model can't draft invalid DuckDB/Postgres
nodes before the pipeline tools are registered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): authoritative new-node probe + SQL-correct eval checklist
Codex [P2] x2:
- build_pipeline_node's collision check relied on the resolved graph, which can
be empty while the session preview races open_preview (a build could shadow a
deployed node before the graph loads) and only covered pipeline runnables, not
a non-pipeline script at the same path. Add an authoritative backend probe
(ScriptService.getScriptByPath): any deployed script at the path → reject with
"use edit_pipeline_node". + regression test (empty graph, deployed script).
- The DuckLake eval judgeChecklist required the exact `// pipeline` annotation,
which would penalize the now-correct `-- pipeline` SQL output (or reward
invalid DuckDB syntax). Make both cases syntax-aware.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): rebuild Global prompt on session preview folder retarget
Codex [P2]: open_preview(kind="pipeline", path="B") can retarget an existing
pipeline preview from folder A to B without remounting. The retarget effect
resets editor state and the helper methods read the new path, but the
registration effect only depends on isActiveSession, so the Global system
message stayed scoped to /pipeline/A. Mirror the route-page fix: rebuild the
global system message on retarget (gated on isActiveSession — only the active
session's helpers are registered; a hidden session reconfigures when it next
becomes active).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipeline-ai): edit_pipeline_node preserves deployed script metadata
Codex [P1]: editNode kept only the deployed script's language and staged a fresh
makePipelineScript draft with empty hash/summary/description/tag/schema/settings.
Deploying that edit from the pane (auto_parent) would update the script while
wiping its metadata, and the route "Save all" path (no parent_hash) could hit
the backend path-conflict branch on the occupied path. Base the draft on the
existing draft's / deployed script object and replace ONLY content (+ inferred
output assets), preserving hash and metadata. + regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): redesign create-new popover and home header
Replace the home page "Home" title with a hover-driven "New" popover
(CreateActionsMenu) listing Script / Flow / Workflow-as-Code / Apps with
a description pane. Workflow-as-Code offers a Python / TypeScript choice;
other entries are created by clicking the list row. Move CLI/MCP to the
header far right, add a Hub link button, and drop the Workspace/Hub tab
switcher so the home page shows only the workspace list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): keep Home title, move New to the right, swap popover panes
Reintroduce the "Home" header title on the left and place the New popover
on the right alongside the Hub and CLI/MCP buttons (top-aligned, with extra
gap before New). Swap the popover panes so the description is on the left
and the option list on the right; the menu opens leftward again.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): make Hub, CLI/MCP and New header buttons uniform md size
Set all three header buttons to unifiedSize="md" (New keeps the accent
variant to stand out) and re-center the right group now that heights match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): remove divider between popover panes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): badge Workflow-as-Code as Advanced and low-code App as Legacy
Add inline pills (Advanced / Legacy) next to the option label and in the
description header, and widen the option list so the longest label plus
badge fits without truncating.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): reorder create options and share App icon
Order is now Script, Flow, App (full-code), Workflow-as-Code, App (low-code).
Full-code App reuses the low-code App dashboard icon (distinguished by accent).
Broaden Option.icon to also accept the BarsStaggered (Flow) component.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): add import / pipeline secondary actions to popover detail panels
Surface the previous create-menu extras in the matching detail panel:
Flow → Import flow + Pipeline (alpha); Workflow-as-Code → Import
Workflow-as-Code; App (full/low-code) → Import full/low-code app. A shared
YAML/JSON import drawer parses the pasted source into the relevant store
(or sessionStorage for the full-reload apps_raw route) and navigates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): add Data pipelines editor option (alpha) to create popover
Add a first-class "Data pipelines editor" entry right after Workflow-as-Code
(indigo accent, Workflow icon, emerald Alpha badge) routing to /pipeline, and
drop the now-redundant "Pipeline (alpha)" secondary action from the Flow panel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): emphasize import buttons and group the badged options
Render the detail-panel import actions as default (bordered) buttons with an
import icon instead of subtle text, and add a separator in the option list
between the three plain options and the three badged (Advanced/Alpha/Legacy)
ones.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): increase the y gap around the option-group separator
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): rename Data pipelines editor option to Data pipelines
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): click-open create popover with import submenu and toggleable docs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(home): keyboard-navigable create popover via melt dropdown with looping
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Guilhem Lemouel <guilhemlemouel@gmail.com>
* fix(audit): don't read pg_authid from an elevated context in S3 export migration
Migration 20260626132251 aborted instance startup on managed Postgres
(e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not
allowed in elevated context": the audit S3 export "oldest in-flight
xact_start" floor probe calls pg_has_role(...), which reads pg_authid,
and managed providers forbid that read from an elevated context. The
migration ran the probe inline in its UPDATE, so the whole migration —
and the instance boot — failed.
Extract the probe into a shared SQL function
audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight
xact_start (when cluster-wide stats are visible) or NULL otherwise. The
pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction,
so a pg_authid failure returns NULL (callers fall back to a conservative
7-day window / reject) instead of aborting. is_superuser (a GUC, no
catalog read) is checked first to short-circuit. The migration's trigger
and UPDATE, the OSS backfill try_start, and the EE exporter/startup
anchor (companion windmill-ee-private PR) all route through it.
Because 20260626132251 already shipped, it is added to the
potentially_stale list in windmill-api/src/db.rs: on startup the stale
_sqlx_migrations row (checksum mismatch) is deleted and the fixed,
idempotent migration re-applies, so already-migrated instances upgrade
without a checksum-mismatch boot failure.
Fixes WIN-2108
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802
This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private.
Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c
New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The /ws_debug debugger WebSocket gated JWT signature verification on inline
`code` being present (`if (code && REQUIRE_SIGNED_REQUESTS)`), so a
`program`-mode launch (naming an arbitrary server-side file path that is read
and executed) skipped verification entirely — even with
REQUIRE_SIGNED_DEBUG_REQUESTS=true. The WS handshake also performed no Origin
check, allowing cross-origin (CSWSH) drive-by from a malicious page.
- Enforce signing on every launch in both handlers (Python + Bun/TS): reject
program-mode outright and require+verify a token for inline code.
- Add opt-in DEBUG_ALLOWED_ORIGINS allowlist enforced at the WS handshake.
- Default docker-compose REQUIRE_SIGNED_DEBUG_REQUESTS to true.
- Update THREAT_MODEL T8/EP15 to reflect the root cause and mitigation.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(apps): add labels input to app editor deploy drawer
The labels feature (c4c9ef5fd) wired LabelsInput into the script, flow,
schedule, resource and variable editors but left the app editor out: it
had no labels state and createApp/updateApp never sent labels, so apps
could not be labeled from the UI despite full backend support.
Thread the deployed app's labels from the edit page through AppEditor
into AppEditorHeader, render LabelsInput in AppEditorHeaderDeploy after
the summary field (matching ScriptBuilder/FlowSettings), and include
labels in the create/update request bodies, the savedApp snapshot, and
the diff/deploy comparison values. The raw-app editor shares the deploy
drawer, so it is wired symmetrically (createAppRaw/updateAppRaw + the
raw page loader) to avoid leaking a non-functional input there.
Fixes WIN-2107
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(apps): drop redundant labels cast in app edit restore path
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(apps): include labels in deploy-drawer Diff current value
The Diff button inside the deploy drawer built its current value without
labels, so the approval preview could hide label changes that would be
deployed. (Identified by cubic.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(apps): reset raw-app labels on new-draft seed
The raw-app edit route keeps labels as route-level state and the
?new_draft=true seed-template branch never cleared it. Since the route is
reused across raw-app navigations, opening a labeled raw app then creating
a fresh one could remount RawAppEditor with the previous app's labels and
deploy them via createAppRaw. Reset labels with the other bleed-prevention
resets at the top of the new-draft branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: redeploy older app version from deployment history
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: apply restored app version to low-code editor on redeploy
Redeploying an older app version from Deployment History fired the
restore callback (toast shown) but the canvas kept displaying the
current version, and Deploy then shipped that current value.
AppEditor seeds its working state from `appDraftHandle.draft ?? app`,
preferring the per-path autosave over the freshly restored `app` prop.
The remount triggered by the restore therefore re-read the stale
pre-restore draft. `reloadDeployed` already clears the draft before
remounting for the reset-to-deployed flow; `onRestore` was missing the
same step.
Drop the autosave in `onRestore` so the remounted editor seeds from the
restored value. Raw apps are unaffected: RawAppEditor binds `files`
directly (no draft precedence), and `extractRawApp` mutates that bound
state in place.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(raw-apps): convert savedNewAppPath event forwarding to a callback prop
`svelte-check` (CI `npm check`) failed with one error: forwarding the
`savedNewAppPath` createEventDispatcher event through the runes-mode
RawAppEditor → RawAppEditorHeader chain types as "not assignable to
never". This is the same legacy-forwarding-through-runes pattern already
removed for `restore` in this PR — `on:savedNewAppPath` would likewise be
dropped at runtime, breaking navigation to the new path after a deploy
that renames the app.
Replace the `on:savedNewAppPath` forwarding with an `onSavedNewAppPath`
callback prop threaded page → RawAppEditor → RawAppEditorHeader, matching
`onRestore`. The header now invokes the callback instead of dispatching,
and its now-unused createEventDispatcher is removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: column-level lineage for ducklake pipelines via // column annotation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: auto-derive column lineage from DuckDB SQL AST (annotation as override)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: clarify column-lineage inference is server-side; drafts use annotations
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): surface inferred column lineage in live pipeline drafts
Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through
ScriptEditor -> details pane -> page -> resolveGraph, merged with // column
annotations (annotation wins) so the live preview matches the deployed graph.
Takes effect once windmill-parser-wasm-asset is republished with the inference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: column-lineage inference now runs live (WASM) too, merged with annotations
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): transitive column-lineage trace (impact analysis)
Stitches every producer's column_lineage into a pipeline-wide column graph
(columnLineageGraph.ts) and replaces the single-hop diagram with an
interactive ColumnLineageTrace: select an asset to see its columns' full
upstream/downstream lineage across scripts; click any column to highlight its
complete transitive impact set (forward + backward) and dim the rest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address CI review on column lineage (parse-fallback, node-id, perf, leak)
- backend: DuckDB SQL parse failure now falls back to `// column` annotation
lineage instead of dropping it (Codex P1)
- columnLineageGraph: collision-proof JSON node ids; deterministic first-write
output anchoring when a producer has multiple ducklake writes (cubic P2 ×2)
- pipeline page: gate buildColumnGraph to a ducklake-asset selection so it
doesn't rebuild on every editor keystroke (cubic P2)
- ScriptEditor: clear inferredColumnLineage on parse error so it can't leak
across a script switch (cubic P2)
- AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: resolve JOIN inputs + anchor column lineage to // materialize target
Addresses the second Codex review pass (two P1s):
- SQL inference now walks JOINed tables: build_from_maps maps every FROM entry
AND its joins into the alias map, and single-table attribution requires no
joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped).
- The column graph anchors a producer's lineage to its declared // materialize
target (surfaced on the runnable node) instead of guessing a ducklake
write-edge, which is unordered for deployed graphs and ambiguous for
multi-output scripts. Falls back to a write-edge when no materialize target.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: gate column-lineage badge to the // materialize target write-edge
The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output
producer showed the same column mapping on every ducklake write-edge. Use the
same materialize-target anchor as buildColumnGraph: the badge lands only on the
declared output's edge, falling back to the ducklake write-edge when there's no
materialize annotation. (Codex P1)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: build column trace from displayGraph so View hides draft lineage
The transitive column trace was built from graphWithDraft regardless of mode, so
in View with drafts hidden it could surface draft `// column` lineage the
deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas
actually renders) so the trace matches: draft overlays in edit / show-drafts,
deployed-only in plain View. (Codex P2)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: don't infer column lineage for local/temp staging CTAS
A CTAS into a local/temp staging table isn't the materialized output, but its
projection was inferred and (flat) column_lineage anchored to the script's
// materialize target — so staging columns showed up as the final asset's. Gate
inference to the actual output: a top-level managed-materialize SELECT, or a
CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: scope inferred column lineage to one output asset
Inference accumulated columns from every output-producing query into one flat
list, all anchored (frontend) to the script's // materialize target — so an
auxiliary CTAS into a different asset showed its columns on the materialized
one. Tag each inferred entry with its output asset and, in parse_assets, scope
the list to the // materialize target (keeping untagged top-level-SELECT
entries); with no declared target, drop inference when entries span multiple
output assets rather than attribute them to an arbitrary one. Parser-internal —
no wire change. (Codex P1)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE
A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp)
before being registered local, so a final SELECT reading it invented
`final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a
phantom asset. track_table_definition now registers any temporary table/view as
local up front, bypassing active-asset resolution; CreateTable/CreateView pass
their `temporary` flag. (Codex P1)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill
The S3/GCS audit-log export's steady-state query filters by `age(xmin)`
(unindexable), so the only scan bound is the timestamp floor. On a fresh
enable the floor was epoch, and on a re-enable the cursor resumed from its
pre-disable position — either way the first run scanned the whole
`audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan
never completes: the cursor never advances, nothing is exported, and the
repeated full scans saturate the database.
Re-anchor on enable (EE companion, windmill-ee-private#634):
- New trigger migration records a recent timestamp floor instead of the epoch
sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on
re-enable, so the export always resumes from ~now and never rescans history.
Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade.
Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints):
- Exports a chosen `[from, to)` window on demand, scanning strictly by
`timestamp` (the partition key) in bounded keyset pages — each query is an
index scan capped at one page (verified via EXPLAIN: later partitions
`never executed`, ~11ms/page), so it stays well under any statement timeout
regardless of window size. Writes alongside the steady-state objects under
logs/audit/, without touching the xmin cursor.
- POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise),
GET /settings/audit_logs_s3_backfill_status.
Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a
backlog" (the cursor is capped and catching up), and updates the setting
description to point operators at the backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): heartbeat backfill lease per object; bump EE ref
Address review (cubic): persist progress (refreshing the lease heartbeat) after
every object PUT in the backfill page loop, not only once per page, so the gap
between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads
and another replica can't re-claim mid-page and run a concurrent backfill.
Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain
regression into the single audit e2e test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject unstable backfill windows; bump EE ref
Address review (P1): the backfill keyset-pages over rows visible at scan time
and declares completion when the scan runs dry, but a row's `timestamp` is its
inserting transaction's `xact_start`. A window whose upper bound is recent or in
the future could silently omit a transaction that started inside `[from, to)`
but commits after the scan passed that timestamp. `try_start` now rejects any
`to` newer than the oldest in-flight `xact_start` (everything strictly older
than the oldest running transaction is committed and stable), using the same
trustworthy stats gating as the exporter's floor (restricted role / 2PC → a
7-day-old cutoff).
Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor
Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts
to now()-7d while leaving the old last_xmin in place. On an instance that
enabled export on the old code >7 days ago and got stuck before the first
successful batch, the next run would filter post-enable rows older than 7 days
out via `timestamp >= ts_floor` while still advancing last_xmin over the
interval — silently dropping them (the same floor-vs-cursor loss class fixed
elsewhere in this PR), and contradicting the "nothing committed after enabling
is skipped" guarantee.
A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be
arbitrarily old, so any recent floor prunes rows the cursor then skips, and an
epoch floor reintroduces the full scan). Re-anchor it to the migration's current
snapshot xmin instead — exactly like a fresh enable — so the export resumes
cleanly from ~now and the never-exported pre-upgrade window is recovered via the
opt-in backfill rather than silently dropped. Reword the setting description so
it no longer implies the disabled/legacy window is covered by the cursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(audit): end-to-end integration tests for the object-store backfill
The backfill previously had only SQL-level/EXPLAIN validation. Add real
integration tests (in-memory object store, sqlx::test) exercising the public
path:
- backfill_exports_window_in_pages: with the page size forced to 2 rows, a
settled 3-day window is exported across multiple keyset pages; asserts every
in-window row lands exactly once, rows outside [from,to) are excluded, a day
that straddles a page boundary yields more than one object, progress counts
match, and a re-run is idempotent (deterministic keys overwritten, no dupes).
- backfill_rejects_unstable_window: a future/live `to` is rejected as unstable,
a window safely in the past is accepted.
Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a
handful of rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(audit): note backfill scope is audit_partitioned only
Make explicit that, like the steady-state export, the backfill reads only
audit_partitioned; the pre-partitioning `audit` table is intentionally out of
scope (not a missed case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill windows before the partitioned boundary
Address review (Codex P1): the backfill reads only audit_partitioned, but
pre-partitioning history lives in the legacy `audit` table (still read by audit
list/get via UNION ALL, and retained for the configured period — 365 days by
default on EE). Since the setting text points operators at this API for
"pre-existing history", a window overlapping legacy rows would report completion
while silently omitting them.
Per the decision to not export the legacy table, reject instead of silently
omit: try_start now rejects a `from` earlier than the oldest audit_partitioned
timestamp (every legacy row predates the partition cutover, so a `from` at/after
that boundary can never overlap them). Reworded the setting text to scope the
backfill to the partitioned era. Added a regression test, plus an RAII guard
(cubic P2) so the test-only globals are restored even if an assertion panics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): backfill object keys per-window; require trustworthy settled cutoff
Address review (two P1s):
- Object-key overwrite loss: keys were `dt=<day>/audit_backfill_<min_id>.ndjson`.
A narrower, overlapping backfill can start a day's page at the same first row
(same min_id) but hold fewer rows, and `put` would overwrite a broader run's
object — silently dropping the rows only that object held. Include the
requested window in the key so different ranges write disjoint objects (same
window re-runs stay idempotent; consumers dedupe overlapping rows by id). New
regression test (verified red→green).
- Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role
lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old
now()-7d fallback could still let an old transaction commit rows inside an
accepted window after the scan, so a "complete" backfill silently missed them.
Since a backfill asserts completeness, reject in those cases instead of
falling back. (The continuous exporter keeps its 7-day fallback — it only
claims bounded lag.)
Also makes the tests robust under the parallel runner: run_backfill takes the
store as a param, so tests pass a local in-memory store (no global
OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment
Address review (1 P1 + 2 P2):
- Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when
audit_partitioned is empty, so an upgraded instance with legacy `audit` rows
but no partitioned rows yet would accept a window and complete with zero rows,
silently omitting the legacy rows. Check the legacy `audit` table directly:
reject any window that overlaps a legacy row (subsumes the boundary check and
covers the empty-partitioned case). Test updated accordingly.
- openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!)
so /openapi.{yaml,json} expose the new backfill endpoints.
- Migration comment (P2): trim the PR-history narration to the durable
constraints (why a recent floor and a monotonic cursor are required), per
AGENTS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d
This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private.
Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89
New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: capture managed-materialize output schema as asset metadata (#2a)
After a managed `// materialize` run, capture the producer's output schema
via a DESCRIBE folded into the existing one-row summary read (no extra
round-trip) and persist it in a new versioned `materialized_asset_schema`
sidecar table. This is the producer-side capture that pipeline parity gap
#2b (save-time consumer-ref contract enforcement) will read back.
- materialized_asset_schema sidecar (asset-level grain), versioned: a new
version row is inserted only when the captured column set changes.
- output_schema column added to the materialize summary codegen.
- worker extracts + records the schema on a successful materialize.
- /assets/asset_schemas read endpoint exposing the evolution history.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address CI review on schema capture (partition col, order, status gate)
- exclude the synthetic `_wm_partition` column from the captured schema for
partitioned assets, so the recorded contract is the producer's logical
output, not Windmill's storage detail (claude/cubic P1).
- make the captured column list explicitly ordered (`row_number()` over the
DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder
columns and spuriously bump the schema version (cubic P2).
- gate the API `record_materialization` schema upsert on a `Materialized`
status, so a failed/running write (or a client attaching a schema to one)
can't advance the schema history (cubic P2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: address Codex review (manual-mode schema gate + auth contract docs)
- gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a
`// materialize manual` run — whose result is the user's own query output —
can't persist a caller-shaped `output_schema` into materialized_asset_schema
(Codex P2). Verified e2e: a manual run returning a fabricated
`output_schema:[{injected,EVIL}]` records the partition but writes no schema
version, while the managed path still captures normally.
- document the authorization contract on the new public `record_asset_schema`
and `list_asset_schemas` helpers: they perform no access control (mirroring
the materialized_partition siblings) and require callers to pass a
workspace-authorized executor (Codex P1).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(frontend): schema-history tab on the ducklake asset node (#2a)
Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema
versions persisted by the materialize run. Master-detail (mirrors the History
tab): the version list (newest first, newest auto-selected) shows column count +
snapshot + capture time; selecting a version renders its column/type table.
Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling
PartitionStatusGrid convention (these materialization endpoints are not in the
generated client).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: schema tab is strategy-aware (history vs fixed schema)
Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns
run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema
table, so their schema is pinned at first materialize and the "history" framing
is degenerate (always one version).
- backend: surface the managed `materialize_strategy` (`replace`/`append`/
`merge`) on the asset-graph runnable node, alongside the existing
`partition_kind` (same parse-from-annotation path).
- frontend: the pipeline page derives `schemaCanEvolve` for the selected asset
from its write-producer (`replace` && not partitioned) and threads it to the
Schema tab. Evolvable → master-detail version history; fixed → a single
current-schema table with a short "schema is fixed" note. Unknown defaults to
evolvable so real history is never hidden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: schemaCanEvolve fails open on unknown producer strategy
Previously a producer present but missing `materialize_strategy` (e.g. a
draft-overlay runnable, synthesized without the field) fell through to
canEvolve=false, hiding captured history behind the fixed-schema view —
contradicting the "unknown defaults to evolvable" intent.
Now the fixed view shows only when *every* producer is a known insert-style
write (append/merge, or partitioned replace); any producer with unknown
(missing) strategy is treated as evolvable, so real history is never hidden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Global-mode chat could reference the user's existing folders in the system
prompt but had no way to create a new one, so for shared work where no
existing folder fit it would dead-end on "ask the user" or invent a
non-existent f/<folder>/… path (which fails at deploy).
- create_folder: dedicated, confirmation-gated tool for the immediate
(non-draft) folder mutation; the creator becomes an owner. Mirrors the
backend name validation client-side and returns a minimal { success } result.
- Folder path guidance now steers the model to create a folder only when the
user explicitly asks for one, and otherwise to ask which folder to use for
shared intent rather than guessing or inventing a path.
- ai_evals: in-memory create_folder mock + a create-folder case (global-path5);
path3 maxTurns bumped to give room to ask.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: skipped suspend step no longer parks the flow forever
A flow step that declares a `suspend` (approval) but is skipped via
`skip_if` was leaving the flow stuck waiting for a resume that would
never arrive.
Suspend gates the *next* step: before pushing step N, `needs_resume`
checks whether step N-1 declared a non-zero `suspend` and finished as
`Success`. A step skipped via `skip_if` is also recorded as
`FlowStatusModule::Success` (with `skipped: true`), so `needs_resume`
treated a skipped approval gate as a real one and parked the flow
waiting for an event that nothing ever sends — until the suspend
timeout (up to 24h).
The skip is most visible when the skipped suspend step is followed by a
branch/subflow: the flow appears stuck on the *following* predicate node
with a generic resume button, while none of the branch/subflow steps
ran.
Fix: honor the `skipped` flag in `needs_resume` and do not gate the next
step on a suspend that was skipped.
Adds regression test `skipped_suspend_step_does_not_block_next_step`
(times out without the fix, completes with it).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: reword regression test comment as a current invariant
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751)
A flow step could execute an unrelated (and in the reported case, destructive)
script at runtime even though every stored definition looked correct. A forensic
dump traced it to two issues:
- Deploy accepted absolute/local step paths. `wmill sync push` from a feature-
branch checkout under /tmp baked an absolute path
(`/tmp/.../ops/scripts/clean_device/...`) into a step's `value.path`. Persisted
verbatim, it mis-resolved to an unrelated script at runtime.
- The on-disk cache write was neither truncating nor atomic. `FsBackedCache::put`
used `write+create`, so a shorter overwrite left stale trailing bytes and
concurrent writers could interleave into a torn file — a corrupt cached blob
that a worker then scheduled from.
Fixes:
- Reject non-workspace flow step paths (must be u/, f/, g/ or hub/) in
`validate_flow_value` (covers create_flow + update_flow, recursively through
loops/branches/AI-agent tools) and early in the CLI `pushFlow`.
- Make `FsBackedCache::put` write a unique temp file (truncate + fsync) then
atomically rename it over the target, cleaning up on error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(flows): validate failure/preprocessor module paths + sub-flow paths in CLI
Address PR review (cubic + claude):
- Backend `validate_flow_value` is the authoritative guard but only walked
`modules`; extend it to also validate `failure_module` and `preprocessor_module`
(which can themselves be sub-flows/loops/branches), so an absolute path there
can't be persisted.
- CLI preflight only collected `type: "script"` paths; now collects sub-flow
(`type: "flow"`) step paths too (recursively, incl. failure/preprocessor), so the
comment's claim matches the behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cli): include AI-agent tool step paths in flow path preflight
Address Codex review: collectStepPaths skipped aiagent tools, so a bad path in
a tool fell through to the API error instead of the local fail-fast. The backend
already validates these (traverse_modules walks AIAgent tools); this aligns the
CLI early-error with it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(flows): make failure/preprocessor path test key explicit
The test used `slot:` as a json! key. json! does interpolate an ident key to its
variable's value (json!({slot:1}) with slot="failure_module" => {"failure_module":1}),
so the test was correct and exercised the validation — but the behavior is subtle,
so build the key explicitly via serde_json::Map to remove ambiguity (review nit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(cache): use a UUID temp name for atomic put (shared-volume safe)
Address Codex (P1): pid+counter temp names collide across container PID
namespaces on a shared cache volume (same pid, PUT_SEQ resets to 0 per process),
so two workers could truncate/clobber the same temp file before rename. Use a
random UUID suffix (matching worker.rs's atomic-write helpers) — globally unique,
so the cross-process temp-file hazard is closed. Also trims the comment to the
AGENTS.md <=4-line limit (Pi nit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The global AI chat @-mention picker only listed flows and scripts; the
whole `app` kind was excluded, so raw (code-based) apps never appeared.
Add raw apps as a `workspace_app` reference, gated to GLOBAL mode and
filtered to `raw_app === true` so visual apps stay out.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an update_user_instructions tool to the global-mode AI chat so the
user can ask it to remember a preference or change/stop a behavior, and
it persists the change to the user-level Global custom prompt.
- update_user_instructions tool: append a new instruction, or find/replace
to edit/remove existing text (reuses the shared findAndReplace helper);
enforces the 5000-char cap and echoes current text on a failed match.
- GlobalToolHelpers gains getUserInstructions/setUserInstructions; the
manager wires them to the localStorage user-prompt store and rebuilds
the system message so the change applies on the next chat-loop iteration.
- Render workspace vs user instructions under distinct headers in the
global system prompt (getCustomPromptParts) so only the user block is
presented as editable.
- Keep the tool result lean: return a short confirmation, not the full
instructions (already re-injected into the system prompt next turn).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The GCS service account key JSON contains the secret private_key and was rendered in plain text in the settings editor on every page load (unlike S3 secret_key / Azure accessKey, which use password inputs). When a key is already configured, hide the editor behind an explicit "Show sensitive values" reveal; the editor (and thus the private_key) is only rendered on opt-in. bucket_config keeps the real key untouched while hidden, so saving round-trips correctly.
Fixes WIN-2106
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Default" node of a branch-one is built with branchIndex -1 and is the
structurally-required else branch (stored separately from the branches array),
so it cannot be removed. Its delete button still rendered, and clicking it
called deleteBranch with index 0, which in removeBranch became
branches.splice(-1, 1) — destructively removing the LAST explicit branch.
Gate the delete button on branchIndex >= 0 so it only appears on explicit
branches.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(sdk): allow overriding worker tag when running jobs
Add an optional `tag` parameter to every job-running helper across the
TypeScript, Python, PowerShell and Rust client SDKs. When set, it is
forwarded as the `tag` query param on the `jobs/run/*` endpoints, which
the backend already honors as a worker-tag override.
The parameter is appended last and defaults to null/None everywhere, so
existing positional and keyword callers are unaffected. Rust has no
optional params, so its existing `run_script_async`/`run_script_sync`
signatures are left untouched and new `*_with_tag` variants are added.
Fixes WIN-2105
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(system_prompts): regenerate SDK docs for tag param
Regenerate auto-generated system prompts so the TypeScript/Python SDK
references (and the script skills that embed them) reflect the new
optional `tag` parameter on the job-running helpers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(powershell-sdk): preserve original RunScriptAsync/RunFlowAsync arities
PowerShell class methods dispatch by exact argument count and have no
default parameter values, so adding `$Tag` in place dropped the old
4-arg `RunScriptAsync` / 3-arg `RunFlowAsync` overloads — existing direct
class calls would fail with "Cannot find an overload". Re-add the
original arities as thin overloads that forward `$null` for `$Tag`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(system_prompts): generate prompts.d.ts to stop literal-content drift
prompts.d.ts was a tracked declaration file with string-literal types
baked in, but generate.py never regenerated it — only prompts.ts and the
hand-written index.d.ts. So every prompt change (e.g. the new SDK `tag`
param) left prompts.d.ts stale, and check-freshness didn't catch it
because generate.py never wrote the file.
Emit prompts.d.ts from generate.py as plain `export declare const X:
string;` declarations. The contents now live only in prompts.ts, so the
declaration file can't drift, and check-freshness covers it going forward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On Windmill Cloud the instance-level database is not supported for data
tables; users must point them at an external PostgreSQL resource. The
database-type picker previously labelled the "Instance" option only as
"Superadmin only", which is misleading on cloud where it can never be
enabled.
On cloud: disable the "Instance" option (subtitle "Not available on
cloud") and surface an info alert explaining that an external PostgreSQL
resource (e.g. Supabase, Neon) is required. Off-cloud behaviour is
unchanged.
Fixes WIN-2104
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Removes the "Pipelines" navigation button from the assets page header
along with its now-unused NetworkIcon and base imports.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: ping job during volume setup to prevent false zombie restarts
Volume mount setup (S3 lease acquisition wait + download) runs synchronously
before the language executor spawns the child process and its ping loop,
leaving the job ping frozen. A slow lease wait or cold S3 download could
exceed ZOMBIE_JOB_TIMEOUT (default 60s) and get the job falsely restarted as
a zombie. Heartbeat the job ping throughout volume setup.
EE companion: windmill-labs/windmill-ee-private#633
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 7b92c8e0de4cfc6d986499d60a5f79cd1c6b9d0b
This commit updates the EE repository reference after PR #633 was merged in windmill-ee-private.
Previous ee-repo-ref: 32e6b9a25f4ec3ea87f429b3d6279f9287a24de7
New ee-repo-ref: 7b92c8e0de4cfc6d986499d60a5f79cd1c6b9d0b
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* perf: eliminate dual-connection DB pool contention across worker, queue, and api
Reuse the held transaction (or move pool reads before begin()) instead of
checking out a second pool connection while a tx is open, extending the
fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus
several server-pool API handlers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: pass owned pool to get_email_from_permissioned_as in http trigger handler
The generified signature takes impl PgExecutor; the http trigger handler
passed &db where db is already &DB, yielding &&Pool which does not impl
PgExecutor (only surfaced under the full feature set in CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint
Addresses review of the dual-connection sweep:
- worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The
caller swallows the error and keeps using tx, so a DB read failure must
not leave the outer transaction aborted (it would fail the later commit).
Matches the previous pool-read semantics.
- Revert reads that were moved onto an RLS (user_db) transaction back to the
non-RLS pool, since RLS row-visibility/role context can change results:
push_scheduled_job (email/tag/settings lookups; reachable with a user_db
tx from api-schedule/api-flows), push_inner native-retry dedicated_worker
routing (RLS isolation variants), resources.rs app-namespace folder
auto-create (non-admins must not be blocked), and the script archive/delete
UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read
Adds a worker-crate failpoints feature and a data-driven hook: when the
stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs
SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still
completes (iteration marked failed) — which only holds if the savepoint keeps the
outer status-update transaction committable. Without the savepoint the abort would
poison the outer tx and the job would never complete.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `script run command > runs a script and returns result` test runs a
trivial, deterministic bun script and asserts exit code 0. On CI it
intermittently fails when the standalone worker (notably on Windows)
transiently fails to execute the job — identical bun jobs complete
successfully elsewhere in the same backend session, so the failure is
environmental, not a regression.
Two problems made this both flaky and undiagnosable:
- `--silent` plus asserting only on `result.code` meant the job's actual
error never reached the CI log, so a flake left no trace.
- No test-level retry, so a single transient worker hiccup failed the run.
Add `retry: 2` to the two worker-executing tests in the block, and
include stdout/stderr in the assertion label so the next occurrence is
debuggable.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add `GET /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}` as a
JSON alternative to `get_flow_all_logs`. It returns the same flow log
tree as an array of per-job entries (job_id, label, kind, step path,
depth, parent module type, sibling index/count, and resolved logs)
instead of a single delimited text blob, so callers can render or
process logs per-step without parsing the `=== ... ===` markers.
The shared auth, recursive-CTE query, and label-building logic is
extracted into `collect_flow_log_entries`; the existing text endpoint
now formats those entries and produces byte-identical output.
Fixes WIN-2102
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): never supersede a running debounce survivor
Companion to the windmill-ee-private change in upsert_debounce_key.
With debounce_args_to_accumulate + a concurrent_limit, a message arriving
while its debounce survivor is already running was marked completed/skipped
("Debounced Running by ...") and the running survivor deleted from the
queue, silently dropping accumulated elements. A slow step + concurrent
limit keeps the survivor running for a long window, so any arrival during
it was lost. The fix leaves a running survivor untouched and starts a fresh
debounce window for the late arrival.
Adds regression coverage in windmill-queue/tests/debounce_test.rs (push,
flow post-preprocessing, no-accumulation, committed-running, and
max-count-window cases) and refreshes the SQLx cache for the changed
upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): add missing SQLx cache for test-only running-flag query
The cargo_test CI job compiles the test target with SQLX_OFFLINE=true; the
new regression tests use `UPDATE v2_job_queue SET running = true ...` which
was not in the offline cache (the library-only `cargo sqlx prepare` skipped
test targets). check_oss/check_ee passed because they don't build tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): harden running-survivor guard against concurrent arrivals
Companion to windmill-ee-private: switch the running-state check to a
correlated EXISTS on the post-conflict-lock holder so two late arrivals
racing after a survivor started running can't both spawn independent
windows (the row lock serializes them; the second debounces into the
first's fresh window).
Adds a concurrent regression test
(test_debounce_concurrent_arrivals_after_running_survivor) asserting
exactly one late arrival survives and the other is debounced, and refreshes
the SQLx cache for the updated upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): serialize upsert per key (simpler, race-free)
Companion to windmill-ee-private: the running-survivor guard and batch
chaining are now protected by a per-key advisory lock instead of
snapshot-sensitive single-statement SQL. This closes a concurrent-arrival
data-loss race where a debounced late arrival's args could be dropped
because the batch lookup couldn't see the predecessor's just-committed
batch row.
Extends test_debounce_concurrent_arrivals_after_running_survivor to pull the
survivor and assert its accumulation includes BOTH racing late arrivals
(shared batch), and refreshes the SQLx cache for the rewritten queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic upsert robust to concurrent pull-time key deletion
Companion to windmill-ee-private: keep upsert_debounce_key a single atomic
INSERT ... ON CONFLICT DO UPDATE so a chaining push cannot fail when the
worker pull path concurrently deletes the holder's debounce_key (the prior
read+UPDATE split could hit "no row updated"). Adds
test_debounce_push_races_key_deletion_by_pull (races a chaining push against
the key deletion 50x, asserts the push never errors) and refreshes the SQLx
cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(debounce): claim-based exactly-once batch consumption
Eliminates the rare duplicate/loss when two survivors land on one debounce
batch (a narrow push/pull race), without locking the worker pull hot path.
- migration: v2_job_debounce_batch gains consumed_at + consumed_by.
- pull side (maybe_apply_debouncing): instead of deleting the batch on consume,
a survivor atomically claims its own row + any unclaimed siblings (stamping
consumed_by = itself) and accumulates exactly the rows it claimed. A second
survivor of the same batch finds its row already consumed by another job and
runs empty (no duplicate); a re-pulled survivor recognizes its own prior claim
and keeps its accumulated args; a never-batched job (CE/legacy) keeps its own
args. Non-accumulate debounce paths still hard-delete their batch rows.
- complete_debounced_job (EE companion) never completes a running predecessor,
so its in-flight run is not killed (no loss); the claim then prevents the
duplicate the guard would otherwise allow.
- monitor: GC sweep deletes consumed batch rows past a 1h grace.
Together with the running-survivor guard this makes debounce accumulation
exactly-once. Adds tests: batch_consumed_exactly_once, repull_keeps_accumulated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): exhaustive edge cases + tighten consumed-batch GC grace
Tighten the consumed debounce-batch GC grace 1h -> 10min: per-op cost of the
claim is unchanged (an indexed mark is as cheap as the old delete), so the only
cost of retaining consumed rows is table growth, which a shorter grace bounds
under high-throughput debounce (a survivor that could still reference a row is
pulled long before 10min; GC is not correctness-critical since a re-pull whose
row was swept falls back to its persisted args).
Adds edge-case tests: never-batched keeps own args (CE fallback), concurrent
claim partitions a batch disjointly (exactly-once under real concurrency),
three survivors -> first takes all / rest run empty, non-accumulate debounce
hard-deletes its batch rows (no leak), and the GC sweep deletes only
past-grace consumed rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): port the #9781 regression case, flow-node guard, full-path bench
- Port the regression from #9781
(test_post_preprocessing_debounce_into_running_survivor_loses_message):
post-preprocessing survivor accumulates + runs, a later same-key message must
start a new batch (survive) not be folded into the running survivor. Exercises
the full EE path via jobs_ee::maybe_debounce_post_preprocessing.
- Add the third EE entry point's guard:
test_flow_node_debounce_running_survivor_not_superseded (maybe_debounce_flow_node).
- Add an #[ignore] full-source throughput bench (bench_debounce_full_path) driving
the real maybe_debounce + maybe_apply_debouncing end-to-end.
All debounce tests exercise the real jobs_ee implementation (run with
--features private,enterprise); none stub it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): scalar-arg accumulation + GC-then-repull no-loss
Close two accumulation edge gaps (both run on --features private,enterprise,
exercising the real jobs_ee path):
- accumulate bare-scalar values (the T | T[] union fallback): each scalar is
wrapped and accumulated into the survivor's list.
- GC reclaiming a survivor's consumed batch row before a re-pull must not lose
data: the re-pull finds no row and keeps its already-persisted accumulated
args (had_row=false fallback), rather than running empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): real-worker end-to-end accumulation test
Drives the full real path on --features enterprise,deno_core,private: push 3
same-key debounced flow jobs (real push() -> maybe_debounce collapses the
batch), a real worker pulls the survivor (real pull() -> maybe_apply_debouncing
claim+accumulate) and executes the deno flow, then asserts the executed result
is the full accumulated set [1,2,3] and the two superseded messages are skipped.
Complements the in-process unit tests with a genuine worker-execution run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic claim+persist, GC only non-queued rows; reword comment
Address review findings:
- [P1] Claim and accumulated-args persist are now in one transaction. Before,
a crash between stamping batch rows consumed_by=self and the `UPDATE v2_job
SET args` could let a zombie re-pull see its own prior claim and keep only its
own args (dropping the siblings it had claimed). Wrapping claim + accumulate +
persist in a tx makes them commit together or roll back together (re-pull then
re-claims cleanly).
- [P1] GC of consumed batch rows now also requires the job to no longer be in
v2_job_queue. A consumed sibling can stay queued well past any time grace under
a concurrency limit / backlog; reclaiming its marker by age alone let its
eventual pull treat it as never-batched and re-run its item (a duplicate).
Keeping the row until the job leaves the queue preserves the "already consumed"
signal. Test extended with a still-queued consumed row that must survive GC.
- [P2] Drop "Customer" attribution from a test doc comment (AGENTS.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): emit accumulation log after committing the claim transaction
append_logs opened a second pool connection while the claim transaction (and its
batch row locks) were still held; under concurrent debounced pulls that risks
pool-exhaustion stalls/timeouts. Defer the log line until after tx.commit().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
This commit updates the EE repository reference after PR #631 was merged in windmill-ee-private.
Previous ee-repo-ref: 30d740e619fad219108ec4b4c6a9d67c1ab42d46
New ee-repo-ref: 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
Automated by sync-ee-ref workflow.
* fix(debounce): claim whole batch in one UPDATE (no deadlock); assert test setup
Both Codex (P1) and Claude (P2) flagged a deadlock: the claim used two writable
CTEs (claim_self then claim_rest), locking the self row before siblings, so two
survivors of the same batch pulled concurrently acquired row locks in opposite
order and PostgreSQL aborted one with deadlock_detected (a transient pull error
on exactly the two-survivors race this path handles).
Replace with a single `UPDATE ... WHERE debounce_batch = (...) AND consumed_at IS
NULL RETURNING id` that claims the whole batch: both transactions lock rows in
the same scan order, so one simply waits and re-evaluates under EvalPlanQual.
A `claimed_self` flag (EXISTS id = self in the claimed set) plus the `mine`
snapshot still distinguishes fresh-claim / consumed-by-other / own-re-pull.
Also assert add_survivor_to_batch_of actually inserts a row (rows_affected == 1)
so a mis-set-up test can't pass vacuously.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The Claude review workflow used a plain checkout, so the EE source (the *_ee.rs
files that live in windmill-ee-private and are symlinked/gitignored in this repo)
was absent — the reviewer could only see the CE surface and missed EE-only code
like windmill-queue/src/jobs_ee.rs. Mirror the EE-checkout the Codex/Pi review
workflows already do: read the PR head's backend/ee-repo-ref.txt via the API,
check out windmill-ee-private at that ref, and substitute the EE files in (copy).
Gated on WINDMILL_EE_PRIVATE_ACCESS being present.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Restoring a version from an app's Deployment History sets the editor value
directly (`onRestore`) without going through `loadApp`, so the fork base
pinned for the stale-draft check is never refreshed. The restored value
carries the `parent_version` that was baked in when that older version was
deployed, so the deploy-time guard (`compareVersions`) compares an outdated
base against the current head and falsely reports the editor is "not on
latest", surfacing a spurious override/diff confirmation on deploy.
Re-pin `parent_version` to the current head on restore, mirroring the
existing seed (loadApp) and after-deploy re-pin sites. Follow-up to #9768.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The apt-package trim in #9783 removed packages that transitively
provided libargon2.so.1. The PHP CLI binary copied from
php:8.3.30-cli-bookworm links against libargon2.so.1 (for argon2
password hashing), so PHP jobs fail at startup with:
/usr/bin/php: error while loading shared libraries: libargon2.so.1:
cannot open shared object file: No such file or directory
Explicitly install libargon2-1 so the dependency no longer relies on
an incidental transitive package.
Fixes WIN-2101
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The custom timeout configured in the script editor settings was only
honored for deployed script runs: it is persisted on the script row and
passed as custom_timeout when running by hash/path. Preview ("Test")
runs derive their timeout solely from the `timeout` query param of
/jobs/run/preview, which the editor never sent, so Test silently fell
back to the instance default.
Forward the editor's timeout setting through ScriptBuilder ->
ScriptEditor -> JobLoader.runPreview as the preview run's timeout query
param. The backend already clamps custom_timeout against the instance
max in resolve_job_timeout, so previews get the same ceiling as deployed
runs.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: self-host docs search for chat, mcp and cli; remove inkeep
Embed a vendored docs snapshot (llms.txt/llms-full.txt) in the backend and
serve ranking + page rendering from GET /api/docs/{search,page}. The AI chat,
the MCP searchDocs/readDocsPage tools, and 'wmill docs' all consume it, so docs
search works with no runtime egress and is no longer EE-gated. Removes the
inkeep proxy. EE companion deletes inkeep_ee.rs (ee-repo-ref bumped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: name read_docs_page param `url` instead of `path`
search_docs returns each hit's `Source` URL, so the read tool now takes a
`url` argument to match — the AI/MCP loop reads "search gives a Source URL,
read takes that url" rather than copying a `Source:` URL into a `path` slot.
A bare `/docs/...` path is still accepted and canonicalized before lookup.
Regenerated openapi-deref, the MCP endpoint tools, and the frontend client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: add scheduled workflow to refresh the vendored docs snapshot
The backend embeds docs_snapshot/*.gz at build time, so the in-product docs
corpus is otherwise only as fresh as the last manual fetch.sh run. This adds a
weekly (and manually dispatchable) job that re-runs fetch.sh, sanity-checks the
result against truncation/garbage, and opens a PR via the internal app when the
snapshot changed — so a human reviews the docs diff before it rides into the
next release build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: make docs tool-result strings caller-neutral
The search/page endpoints back three differently-named consumers (the AI chat
`read_docs_page` tool, the MCP `readDocsPage` tool, and the `wmill docs` CLI),
so the shared rendered text shouldn't name one of them. Refer to "the docs
page-reading tool" and its `url` argument instead, and add tests pinning the
caller-neutral follow-up guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: point ee-repo-ref at inkeep-removal companion rebased on EE main
The companion branch now carries only the inkeep_ee.rs deletion on top of EE
main (was based on the native-job-retry EE line, which polluted the EE PR diff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(docs): expose docs:read in token catalog; precompute lowercased corpus
Addresses two review nits on the self-hosted docs PR:
- docs:read was enforced (ScopeDomain::Docs) but missing from the token scope
catalog (token.rs ALL_SCOPES), so it couldn't be selected when creating a
standard scoped token in the UI — leaving scope-restricted CLI/MCP docs use
effectively ungrantable. Add a read-only "Documentation" group (no write
surface) and a test asserting it is exposed.
- search ran page.body.to_lowercase() on the whole corpus per query. Lowercase
body/title/description once at parse time (into the OnceLock corpus) and scan
the precomputed copies instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: update ee-repo-ref to 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
This commit updates the EE repository reference after PR #630 was merged in windmill-ee-private.
Previous ee-repo-ref: c7ec3a0c2fa38d4cb5e50bf0265eef4710de4860
New ee-repo-ref: 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: detect and guard against deploying stale drafts
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: extend stale-draft warning to low-code app drafts
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: discard stale draft on rebase instead of resetting to latest
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: animate AI chat thinking block open/close like tool calls
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: detect stale flow/app drafts by pinned version at load and deploy
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: reset version-staleness state on new drafts and after app deploy
Addresses review: new-draft route reuse left stale version/draftBaseVersion (false stale-draft modal on a fresh flow/app); app deploy left parent_version pinned to the superseded base (false 'not latest' on a follow-up deploy).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A corrupt local pip cache entry (e.g. wmill==1.739.0 missing s3_reader.py
after out-of-band file loss on a persistent/shared cache volume) was trusted
indefinitely: handle_python_reqs only checked the .valid.windmill marker on
the reuse fast path. verify_wheel_record already guarded the install and
S3-pull paths, but never ran again once the marker existed.
Re-verify the wheel RECORD on the first reuse of each cache entry per worker
process and repair (wipe + reinstall) on failure. A VERIFIED_VENVS in-memory
set makes every subsequent reuse skip the scan, so the warm-cache hot path
keeps paying only its original single stat.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: document delete_jobs auth contract and workspace-scope jobs_export purge
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`install_python` cleared the subprocess environment via `env_clear()` and
forwarded only a subset of variables, omitting `SSL_CERT_FILE` (from
`PY_INDEX_CERT`/`PIP_INDEX_CERT`) and `UV_NATIVE_TLS` (from `PY_NATIVE_CERT`).
This caused `invalid peer certificate: UnknownIssuer` errors when downloading
managed Python runtimes in environments with corporate/private CAs.
Forward both variables, mirroring the sibling `find_python` method and the
pip install path in `python_executor.rs`.
Fixes WIN-2100
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the last-iteration path of a parallel for-loop (nindex == len), the
DELETE FROM parallel_monitor_lock ran on the pool (db) while the
transaction tx (begun earlier) was still held. This dual-connection
pattern requires 2 simultaneous connections from the per-worker pool
(default max 5) and can trigger "pool timed out while waiting for an
open connection" under concurrent load.
Run the DELETE on the held transaction (&mut *tx) instead, matching the
fix PR #7861 applied to other queries in this file. The transaction is
committed shortly after, so including the DELETE in it is safe and
consistent with the non-last-iteration path.
Fixes WIN-2099
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(backend): native script retry without one-step-flow wrapping
Schedules and data pipelines that retry a single script previously wrapped
it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a
v2_job_status row, and UI projection complexity. This adds native retry on a
plain JobKind::Script job.
- RetrySettings: flatten Retry into a deduped retry_settings table, carried
via the existing runnable_settings_handle (lazy, off the hot path).
- push() materializes a bare-script-with-retry SingleStepFlow into a native
Script job (gated on min-version + no handlers/retry_if).
- add_completed_job re-pushes the next attempt on failure with backoff,
tracking the attempt counter in v2_job_queue.extras and the chain via
parent_job; schedule completion handlers fire only on the terminal attempt.
- frontend: ScriptRetryChain shows the attempt chain on the run page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(backend): native retry_if eval + per-occurrence schedule handlers
Extends native script retry to the two cases that previously stayed on the
one-step-flow path:
- retry_if: evaluated natively on the failure path via a feature-gated
windmill-jseval dep (quickjs) over the failure result + flow_input; push
materializes such policies natively only when quickjs is available.
- on_failure_times / on_recovery: apply_schedule_handlers now resolves each
past scheduled occurrence's terminal status across its native-retry chain
(root OR any parent_job=root child succeeded) and excludes the current
occurrence, so the counting is per-occurrence rather than per-attempt.
All scheduled-script retries now go native (schedule.rs gate removed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(backend): always materialize retry_if natively; unsupported without quickjs
retry_if is evaluated by the worker (which always has quickjs), not the
pusher, so gating materialization on the pusher's feature was wrong. The
flow path was never a real fallback either — the flow runtime needs quickjs
to evaluate retry_if too. retry_if now always goes native; on a worker
without quickjs it is unsupported and fails closed (no retry).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(backend): un-park asset-cascade (pipeline) retry
Native retry resolves the blocker that parked pipeline retry: a retried
subscriber is now a Script job (not a one-step flow / flow step), so it
stays eligible for asset dispatch and can trigger its own downstream on
recovery.
- scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset
edges (was dropped with a TODO warning).
- asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job,
so native-retry attempts dispatch on success while flow steps stay excluded.
- tests: retry-bearing subscriber now dispatches as a native Script carrying
the policy in runnable_settings_handle; native-retry attempt is eligible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): cap native retry interval, lazy result serialization, idempotent retry push
Hardening from a self-review of the native retry path:
- Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path
(evaluate_retry); the exponential formula could otherwise schedule up to
~18h vs the flow path's 6h.
- Serialize the failure result lazily (only when a retry_if policy needs it),
so the common failure no longer pays the serialization on the failure path.
- Push each retry with a deterministic id per (root, attempt). If a worker
dies between enqueueing the retry and finalizing the current attempt, the
reaper re-handles the attempt and lands here again — push rejects the
duplicate id, so the retry is enqueued exactly once (no double-retry).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): defer schedule handlers idempotently on retry-push replay (review P1)
Address local-review findings:
- P1: retry_pending was derived from the retry push *result*, so on a worker
crash + reaper replay the duplicate-id push returned Err → retry_pending
flipped to false → apply_schedule_handlers fired for the non-terminal
attempt (and the terminal attempt later fired them again). Pre-check whether
the deterministic retry id already exists and report it as pending without
re-pushing, so the handler-deferral invariant is crash-idempotent too.
- P2: refresh the stale 'wrap the script in a one-step flow' comment in the
asset-cascade retry push — it now materializes a native Script.
- Add RetrySettings <-> Retry round-trip unit tests (clamping edges).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(backend): native retry chain + per-occurrence status sqlx tests
Close the two integration-test gaps flagged in local review:
- chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry
through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts)
and asserts crash-replay idempotency (the P1 fix: a replayed completion reports
pending without double-enqueueing).
- per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence
terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but-
recovered occurrence counts as success, retries (parent_job set) are excluded
from occurrence counting, and the current occurrence is excluded.
- canceled_job_does_not_retry: cancellation wins over a pending retry.
Runtime sqlx API (no .sqlx cache entry needed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): exclude schedule handlers from the retry-attempt chain
The retry chain listed all script children of the root by parent_job, but
schedule completion handlers (on_failure/on_recovery/on_success) are also
script children — when the occurrence has no retries, the handler's parent is
the root itself, so a successful, never-retried job rendered a bogus
'Retries (1)' badge pointing at the handler. Filter children to re-runs of the
same script (matching script_hash); real retries keep the root's hash, handlers
run a different script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): surface schedule handlers on the run page
Extend the run-page chain component with schedule completion handlers:
- A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/
on_success runs that fired for that occurrence (found as children of the
terminal attempt, identified by their synthetic created_by).
- A handler's own run page now shows a 'Failure/Recovery/Success handler'
label with a link back to the run it handled and its schedule. on_recovery
and on_success share created_by, disambiguated by the recovery-only
error_started_at arg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare
An earlier `cargo sqlx prepare` on this branch ran before #8801's
folder_default_permissioned_as test merged in, so it pruned the 3 query caches
that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main.
* fix(backend): only cascade assets from native retry attempts, not handlers (review P1)
is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script
child became asset-eligible — including schedule/error/recovery handlers (Script
jobs with parent_job set and no flow_step_id). A handler that declares assets
would then trigger a cascade the old parent_job IS NULL guard prevented. Gate
parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as
its chain parent (handlers run a different script). Runtime query, no sqlx cache.
* fix(backend): cache the private-gated retry_setting asset-dispatch test query
The same prepare-without-private that dropped the folder_default caches also
pruned the cache for the retry_setting_dispatches_subscriber_as_native_script
test query (asset_trigger_dispatch.rs:721). Regenerated with --features private.
* fix(backend): exclude handler children from per-occurrence recovery (review)
A scheduled occurrence's on_failure/on_success handler runs as a successful
child (parent_job = occurrence), and the per-occurrence success EXISTS counted
ANY successful child — so a failed occurrence whose error handler succeeded was
marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in
the merge) and on_failure_times counting. EE query now scopes the EXISTS to
same-runnable children (only native retry attempts); regenerate sqlx cache + bump
ee-repo-ref. native_retry_test gains a handler-child regression case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(backend): scheduled-script retry is a native Script, not SingleStepFlow
test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted
the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a
native Script. Update both to assert kind='script' and that the retry policy is
carried via runnable_settings_handle.
* fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic)
Address cubic CI review:
- P1: the SingleStepFlow->native Script materialization dropped dedicated_worker,
so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve
it from the script row in push so the materialized Script keeps the dedicated tag.
- P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the
u32->i16 // retry count narrowing (scripts.rs) instead of wrapping.
* fix(backend): use a retry-specific signal, not runnable equality (codex review)
Address Codex CI review:
- P1: is_native_retry_attempt treated any same-runnable parented Script child as
a retry. WAC v2 inline children have that exact shape, so an inline child of an
asset producer would cascade. Use a retry-specific signal instead: the job
carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no
flow_innermost_root_job. Apply the same flow_innermost guard to the EE
per-occurrence EXISTS (WAC inline children must not count as a recovery).
- P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate
now resolves as 'retry pending' (re-check on the duplicate-id error) instead of
flipping retry_pending to false and firing handlers early.
- Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(backend): explicit native_retry_attempt marker, drop heuristics
Replace the per-site "is this a retry?" inference (parent_job + runnable match +
flow_innermost / retry_settings) with one explicit marker: a sparse
native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker
also carries the attempt counter (previously in v2_job_queue.extras), so it's the
single source of truth.
- asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker.
- EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost.
- maybe_enqueue: reads/writes the marker (persistent) instead of queue extras.
- Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap.
- Eliminates handler / WAC-inline-child misclassification by construction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex)
The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but
the periodic monitor.rs path deleted v2_job rows without it, orphaning markers.
Add the same WHERE job_id = ANY(...) sweep there.
* fix(backend): widen native_retry_attempt.attempt to integer (cubic)
The smallint column was cast to/from u32 and could wrap a retry chain longer than
i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32
attempt count, so no narrowing occurs on the maybe_enqueue read/write path.
* feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT
- Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi),
computed from the native_retry_attempt marker. The run-page chain now filters
retry attempts by is_retry instead of the script_hash heuristic, so WAC v2
inline children (same script, parent_job) no longer render as retries (codex).
- Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep
the plain catch-and-continue matching the other side-table deletes; the table is
created by a startup migration so it always exists when cleanup runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it
The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but
does not select is_retry, which errored with "no column found". Only the list
endpoint populates the marker; #[sqlx(default)] lets every other query omit the
column and default to None.
* feat(backend): select is_retry in single-job GET too for consistency
The list endpoint already exposes the marker; populate it on the single-job GET
(both completed and queued variants) as well so a run loaded directly reflects
its retry status. #[sqlx(default)] stays as a safety net for any other query.
* feat(backend): reap orphaned native_retry_attempt markers via periodic sweep
The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so
direct job deletions (workspace/job delete, schedule clearing) would leave marker
rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which
must then be remembered for every future path), reap orphans in the periodic
delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the
job). The table is sparse so the anti-join drives off it and probes v2_job by PK —
cheap. Retention still sweeps markers inline (keeps the table small so this stays
cheap); a transient orphan is harmless (nothing reads is_retry for a gone job).
* fix(frontend): include flow handlers in retry chain handler row (codex)
Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...),
whose handler job is a flow, not a script. The chain fetched children with
jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts
are still selected by is_retry and handlers by created_by, so both kinds surface.
* fix(backend): carry concurrency/debouncing settings into native retries
maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings
::default(), dropping the script/pipeline concurrency settings the failed job carried
in its runnable_settings_handle. A retry of a concurrency-limited script then inserted
no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and
pass them in the payload, which push forwards to the materialized retry. Adds a
regression test asserting the retry's handle resolves to the concurrency settings.
* fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex)
P1a (schedule.rs): the scheduled-retry materialization fetched the script's
concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings
::default() into the SingleStepFlow payload, so the root attempt's handle held only
the retry policy and the whole chain ran unbounded. Pass the fetched settings.
Regression test asserts the root handle resolves to retry + concurrency.
P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract
— it is pub only for the integration test; the sole production caller is the worker
completion path passing a DB-derived, already-authorized MiniCompletedJob.
* docs(backend): attach native-retry auth contract to the function itself (codex)
The doc block was merged with eval_retry_if's doc and bound to that function,
leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if
keeps its own doc; the native-retry + authorization contract now sits directly
above maybe_enqueue_native_script_retry.
* docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex)
- Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since
1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the
served spec couldn't see the field. Now current at 1.739.0.
- schedule.rs: a retry_if gate is evaluated at failure time and fails closed without
quickjs (no retry); it does not fall back to a flow path.
- windmill-types jobs.rs: is_retry is selected by both the list and single-job GET
endpoints (not list-only).
* docs(backend): fix remaining stale retry_if/quickjs comments (codex)
The retry_if block and the push materialization comments claimed push keeps
retry_if on a flow path / the worker always has quickjs. The code always
materializes native retry and the no-quickjs eval_retry_if path fails closed —
correct the comments to that constraint.
* docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex)
- Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated
and the job does not retry (no one-step-flow fallback).
- jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence
failure/recovery status across the retry chain, so the old 'restricted to
schedules whose handlers don't need per-occurrence counting' claim is dropped.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: bounded-cascade selective execution for pipelines (UI + CLI)
Run a prefix of a pipeline cascade: from a schedule/manual root, fan
downstream but stop at chosen end node(s) — the path-between set over the
asset-graph lineage DAG. Exposed as a canvas 'Run downstream up to…' pick
mode and a 'wmill pipeline run <folder> --to' CLI command. No backend or
parser changes; reads the existing graph, tags, and triggers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: surface bounded-run on the run caret, trigger-node kebab, and Test button
Move 'Run downstream up to…' from the runnable kebab onto the play-button
caret popover (Edit mode, next to Run / Run + trigger N downstream); add it
to the trigger-node kebab so schedule/data_upload entrypoints expose it on
the View page; and to the ScriptEditor Test split caret for the open script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address CI review on bounded-cascade (cubic)
- Port CLI engine test from Deno to bun:test under cli/test/ (won't run under bun test otherwise).
- closure() now excludes the start node on a cycle back to it (descendants/ancestors contract); regression tests both engines.
- CLI 'pipeline run --to' rejects unresolved/ambiguous end tokens instead of silently running a different subset.
- Sort a copy in the runSelection order test so the launch-order assertions aren't invalidated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: address standing review nits on bounded-cascade
Resolves the four recurring P1/P2 findings from the codex/pi/claude
reviews:
- UI gate (P1): the canvas/trigger-node "Run downstream up to…"
affordance was gated on the subscriber-only downstream map, so a valid
start whose only downstream is a pure reader had a non-empty bounded
set but no menu entry. Gate on the read-aware lineage downstream
(buildLineageDownstreamMap), matching the bounded engine.
- waitJob (CLI): a completed job without explicit success:true now
counts as a failure, mirroring the frontend waitJobTerminal — the
cascade only advances on a confirmed success.
- Comment fix (CLI): the unbounded `run` path uses the read-aware
lineage DAG (pure readers included); dropped the false "parity with
the canvas cascade" (subscriber-only) claim.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: expose bounded-run caret for pure-reader-only starts (codex P1)
The canvas wiring from the prior commit passed `onStartBoundedRun` from
the read-aware lineage map, but the leaf components still hid the popover
that holds the "Run downstream up to…" action behind a subscriber-only
gate:
- RunnableNode rendered the Run-button caret only when
`hasCascade = downstreamCount > 0` (subscriber-only). A valid start
whose only downstream is a pure reader got `onStartBoundedRun` but no
visible action. Now the caret opens when there's a cascade OR a
bounded-run start (`hasCaret`), and the "Run + trigger N downstream"
item is gated on `hasCascade` so it never reads "trigger 0".
- ScriptEditor's Test split button activated only when
`downstreamSubscribers > 0`, falling through to a plain Test button
(no caret) otherwise. Now it also activates when `onBoundedRun` is
set, with the "Test + trigger N" item gated on the count.
For a manual root (no trigger-node kebab fallback) with a pure-reader
downstream this was the only UI entry point, so it was previously
unreachable. Verified in-browser: a manual-root script writing an asset
read-only downstream now exposes "Run downstream up to…" on the
ScriptEditor Test caret with the cascade item hidden.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: gate ScriptEditor bounded-run on read-aware downstream; fix CLI asset-end warning (codex P2)
- Details-pane (ScriptEditor) bounded-run entry was gated only on
`validStartPaths`, broader than the canvas which also requires
read-aware downstream (`hasLineageDownstream`). An isolated start could
thus expose "Run downstream up to…" and enter pick mode with no
selectable end. Now gated on `lineageDownstreamPaths` (script paths with
a downstream in `buildLineageDownstreamMap`), matching the canvas.
- CLI dropped-end warning called `scriptPathOf(d)` unconditionally, which
slices `script:`-length chars off an asset id too — `datatable:main/raw`
printed as `le:main/raw`. Now prefix-checks like the JSON output.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: correct --from error to exclude only row-backed event triggers (codex P2)
The bounded-start validation message listed `kafka/webhook/…` as event
triggers that can't start a bounded run, but webhook/data_upload are
rowless and read as manual roots (valid starts). Only the row-backed
native kinds (kafka/mqtt/nats/postgres/sqs/gcp/email — EVENT_TRIGGER_KINDS)
are excluded; the message now names those.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: surface dropped ends in CLI JSON; disambiguate shared-trigger bounded start (codex P2)
- CLI `run --json` silenced the dropped-end warning, and the JSON payload
echoed the originally-resolved `--to` list with no reachable/dropped
split — a resolved-but-unreachable end looked like a clean plan that
silently runs only the start. JSON now includes `reachableEnds` and
`droppedEnds` (shared `idLabel` helper, asset-id safe).
- Trigger nodes dedupe per (kind, ref), so a schedule shared across
scripts collapses to one node, but `recordSourceTrigger` kept only the
first target path — the bounded-run action then rooted at an arbitrary
script (or hid when only that first script lacked downstream). Now all
target paths are tracked and the action is offered only when exactly one
is a valid start with downstream; multi-eligible nodes suppress it
rather than guess.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: don't run hidden drafts in View-mode bounded cascade (codex P1)
launchCascadeScript unconditionally preferred drafts.get(path) over the
deployed script. In View mode with drafts hidden (displayGraph is
deployed-only), a bounded run started from a trigger-node kebab would
execute preview jobs from hidden local draft content instead of the
deployed scripts the user is looking at.
Gate draft execution on `mode === 'edit' || includeDrafts` — the exact
condition under which displayGraph includes drafts — so execution always
matches the displayed graph. No-op for scripts without a draft; the
edit-mode "Run + trigger N downstream" cascade is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`get_variable_or_self`, `get_variable_or_self_as`, `get_secret_value_as_admin`
(and `transform_json_unchecked`'s `$var:` branch) in windmill-common always ran
the raw `variable.value` through `decrypt()`. With an external secret backend
(HashiCorp Vault / Azure Key Vault / AWS Secrets Manager) configured, that
column holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64
ciphertext, so base64 decoding failed with `Invalid byte 36, offset 0` (the
`$`). This broke GitHub App git sync (git_sync_ee.rs) and any other consumer of
these resolvers when an external backend is active.
Move backend resolution (`get_secret_backend`, `get_secret_value`,
`is_*_stored_value`, caching) into `windmill-common::secret_backend::resolver`
so the low-level variable resolvers can route external markers through the
configured backend's `get_secret()` instead of `decrypt()`. The windmill-store
and windmill-api `secret_backend_ext` modules now re-export these from
windmill-common (single source of truth / single backend cache) and keep only
their write-side helpers. No `_ee.rs` files change.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: allow hyphens in postgresql database name validation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: cover hyphen acceptance in validate_dbname
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): nested-loop "Test this step" resolves iter to innermost loop
In a loop-inside-a-loop, the inner step's "Test this step" tab prefilled
its arguments using the outermost ancestor as the parent module, so
flow_input.iter resolved to the parent loop's iteration value instead of
the inner loop's.
dfs(id, flow, true) returns [step, immediate parent, ..., root], so
modules[modules.length - 1] is the outermost ancestor. The prop picker
needs the immediate parent (modules[1]) so getFlowInput resolves iter at
the innermost loop's level. A single loop was unaffected because both
indices coincide; only depth >= 2 broke.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(frontend): nested-loop parent selection for test-step args
Pins that modules[1] from dfs(stepId, flow, true) is the immediate parent
for every step across all container types (for/while loops, branchone,
branchall, aiagent tools) and nesting depths, and that getStepPropPicker
then resolves flow_input.iter to the innermost enclosing loop.
Covers >400 step positions across 107 generated flow shapes, plus explicit
single/nested/while/branch iter-resolution cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(frontend): remove nested-loop parent selection test
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: data tests for ducklake pipeline materialization
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): data_test count badge on pipeline graph nodes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: surface annotation badges (incl. data_test) on deployed pipeline nodes
Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): keep materialize output edge when editing the producer in the pipeline graph
The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: run all data tests in one pass with a structured per-test result
Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): data-test pass/fail checklist in the job result
DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): move data-test badge onto the producer→asset edge with run status
The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): render custom data-test scripts as their own clickable graph nodes
A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g
BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): anchor edge badge on routed path + a11y text on test icons
Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests
Address review (cubic) findings:
- P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact.
- P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script.
- P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: fail loud if fewer data-test outcomes recovered than declared
Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: relationships data test same-lake reuse + schema-qualified target quoting
Address Codex/Pi review (two P1s in the relationships codegen):
- A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references.
- A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator.
Adds tests for both.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target
Address Codex review nits (both P2):
- resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch.
- AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: ducklake time-travel UX (snapshot history + AT VERSION reads)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: catalog-qualify ducklake time-travel FROM hints (lake. prefix)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: render ducklake snapshot_time (microseconds since epoch) correctly
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: merge ducklake History + Query into one master-detail tab
Snapshot list (left) selects the version previewed in the read-only grid (right); newest auto-selected. Copy-clause moved to the preview's SQL line.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scope ducklake snapshot history to the table's versions
Catalog-wide snapshots predate a table's creation; previewing AT a version before the table existed errored ("Table ... does not exist at version N"). The DUCKLAKE_SNAPSHOTS marker now takes the table and lists only snapshots from its first creation onward. Also: narrower snapshot-list pane on large screens (target a fixed width, not a fixed fraction).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: load ducklake preview columns at the pinned version + reset on asset switch
Addresses CI review (codex/pi P1, cubic P2):
- Historical previews loaded current-schema columns, so an AT(VERSION) read enumerating a column added in a later snapshot failed. Now DESCRIBE-loads the column set at the pinned version; the read is gated on columns matching the current version to avoid a stale-colDefs race on version switch.
- selectedVersion no longer sticks across assets: the panel is keyed on path (remounts per asset) and effectiveVersion falls back to newest when the pick isn't in the current list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: match History tab UI (master-detail, full-FROM copy) after merge
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: handle catalog-only ducklake asset paths (no table segment)
parseDbInputFromAssetSyntax threw on a catalog-only path like 'ducklake://main' (undefined.split('.')) — a real graph node (e.g. a consumer of the whole catalog). It now returns a table-less input instead of throwing, and DucklakeAssetPanel renders only the partition grid (no per-table history/time-travel) for table-less nodes. Adds parser unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: escape ducklake catalog name in client-built time-travel DESCRIBE
fetchDucklakeColumnsAtVersion interpolated the catalog name into an ATTACH string literal without escaping; double single-quotes (mirrors backend escape_sql_literal) so a quote-containing catalog name can't break out. Also fixed the v1.x docs checklist line to match the shipped full-FROM copy affordance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): add /clear session command to start a fresh conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): don't re-queue a built-in command flushed from the queue
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add two agent skills under .agents/skills (symlinked into .claude/skills):
- ai-chat: guidance for improving the Windmill AI chat / copilot, especially
global mode — benchmark before/after with ai_evals, optimize finalContextTokens
over cumulative, keep tool params and tool-result payloads minimal (no echoing
content the model already has), treat prompts/tool-descriptions as benchmarkable
surface.
- ai-evals: author and run black-box benchmark cases for the AI generation modes,
migrated from ai_evals/AGENTS.md and extended with run mechanics (workspace reuse,
reading the summary). ai_evals/AGENTS.md becomes a pointer stub to the skill.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): persist on-behalf-of user when redeploying raw apps
The raw-app deploy drawer reused AppEditorHeaderDeploy but never wired up
the `preserveOnBehalfOf` bindable nor forwarded `preserve_on_behalf_of` in
the createAppRaw/updateAppRaw request bodies. Without that flag, the shared
backend handler (create_app_internal/update_app_internal) resets the policy's
on_behalf_of to the deploying user on every deploy. So a publisher who set
"App executed on behalf of <other user>" would silently lose it on the next
deploy, unlike every other setting on the deploy page.
Mirror the classic (low-code) app header: declare `preserveOnBehalfOf`, bind
it to the deploy component, and send `preserve_on_behalf_of` on both create
and update.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): preserve on-behalf-of in the draft-deploy path
The draft-deploy path (deployDraft → AppService.createApp/updateApp for visual
apps, deployRawAppDraft → createAppRaw/updateAppRaw for raw apps) carries the
deployed app's policy forward but never sent preserve_on_behalf_of. So
deploying a draft via the "Review & deploy drafts" UI silently reset the
policy's on_behalf_of to the deploying user — the same backend reset behind the
deploy-drawer bug, on a surface that has no on-behalf-of selector to re-set it.
Send preserve_on_behalf_of whenever the carried policy has an on_behalf_of, for
both app types. The backend still gates actual preservation on
can_preserve_on_behalf_of, so a non-deployer cannot escalate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): preserve on-behalf-of in the AI-chat raw-app deploy
The global AI-chat deploy path (`deploy_workspace_item` → createAppRaw/
updateAppRaw in copilot/chat/global/core.ts) carried the recomputed policy
forward but omitted preserve_on_behalf_of, so deploying a raw app via chat
reset the policy's on_behalf_of to the deploying user — the last of the
deploy surfaces with this gap. Send the flag when the policy has an
on_behalf_of, mirroring the editor and draft-deploy paths; the backend still
gates preservation on can_preserve_on_behalf_of.
Add a regression test asserting the flag is forwarded when the deployed
policy carries an on_behalf_of.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): restore raw app 'open preview in separate window'
Re-adds the detached preview window dropped when preview hosting moved to
the host (ui-builder f52d8e5b). Live-synced preview + dark mode, and wires
the runnable bridge to the detached window so backend calls resolve there.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): repaint detached raw app preview after refresh
The detached preview tab is a blank app-preview.html shell fed by the
editor over postMessage. A one-shot opener load listener can't survive the
tab refreshing itself, so a manual reload left it blank. It now posts
'appPreviewReady' on every (re)load and the editor re-sends the build; the
orphaned window is also closed when the editor unmounts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): keep load-based feed for detached preview initial open
Relying solely on the appPreviewReady handshake left the detached window
blank on first open against app-preview.html artifacts that predate the
handshake (the pinned UI Builder tarball). Restore the one-shot load feed so
initial open works regardless of the served shell; the handshake still
covers manual refresh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): re-sync dark mode when refocusing detached preview
The focus-reuse path replayed the build but not the theme, so re-opening an
existing detached window after a dark-mode toggle kept the stale theme until
the next build. Extract a feedExternalPreview() helper (theme + build) used by
the open, focus-reuse, load and handshake paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(frontend): bump ui_builder artifact to 062d11c (preview handshake)
Pins the UI Builder artifact built from windmill-code-ui-builder#14, which
adds the appPreviewReady handshake, detached-preview favicon and title.
Activates refresh-repaint + favicon for the detached raw app preview.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(frontend): serve ui_builder static bundle in dev, proxy :4000 only as fallback
The postinstall downloads the pinned UI Builder artifact into static/ui_builder,
which SvelteKit already serves at /ui_builder. Skip the :4000 proxy when that
bundle is present so dev uses it directly (matching prod / the backend's
static-vs-:4000 fallback); no separate UI Builder dev server needed. Delete
static/ui_builder to develop the builder against :4000.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): harden detached preview origin + scope window name
Addresses cubic review on #9765:
- P1: only honor appPreviewReady from a same-origin sender and post the build
with targetOrigin=location.origin, so user app code that navigates the
detached window cross-origin can't trigger/receive a build (app source).
- P2: scope the detached window name per app path so two open editors don't
collide over one OS-level preview window.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): show context usage as a gauge with hover tooltip
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): consolidate model, thinking & params into one dropdown
Merge the model picker, reasoning-effort selector and prompt settings
into a single dropdown with a model list, a thinking-effort slider and a
hover-revealed Parameters submenu. The trigger shows the model and effort.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): polish model/thinking dropdown interactions
Register the model rows and thinking slider as melt menu items (roving
highlight + arrow-key navigation), keep the menu open on selection via a
new DropdownV2 closeOnItemClick prop, use melt's createSubmenu for the
Parameters flyout so it flips on screen edges, and use the brand accent
for the context-usage gauge and slider.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): stop popover drift and keep Thinking section when unsupported
Freeze the trigger width while the dropdown is open so the bottom-end
popover doesn't shift as the effort label resizes (released on close, so
no reserved padding). When a model has no reasoning support, show the
Thinking section disabled with a message instead of removing it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): restore reasoning slider drag inside the menu
The slider lives in a melt menu item, whose roving focus blurs the
focused element on pointermove and aborted the native thumb drag. Stop
the slider's pointer events from bubbling to the item so melt leaves it
alone; focus-based highlighting still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): move Parameters to the top of the model settings menu
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-chat): hide the @ context picker in global mode
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): only mark context gauge as a meter when the window is known
A meter is a 0–100% reading; with an unknown context window there is no max
to measure against, so role/aria-value* are dropped (previously valuenow fell
back to the raw token count against an implicit valuemax of 100).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(frontend): note closeOnItemClick is read at mount-time
Addresses a non-blocking review note on DropdownV2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(ai-chat): fix showContextPicker comment to match GLOBAL removal
Addresses Pi review P2: GLOBAL no longer offers the @ context picker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(ai-chat): clarify showContextPicker hides only the manual @ button
In GLOBAL, @-context is still invoked inline by typing @ in the input; only
the redundant picker button is hidden.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): stop flow step id generation from being poisoned by non-canonical keys
nextId computed the next step id from the max of charsToNumber over every
module id and flowState key. Only canonical auto-ids (a, b, ... aa, ab) have a
meaningful charsToNumber value, but flowState also holds copy ids ("z2"),
subflow result keys ("subflow:..."), reserved keys ("failure"/"preprocessor")
and user-renamed ids. The old `length >= 4` guard filtered long junk but let
short junk through, so e.g. duplicating step "z" (key "z2", charsToNumber 629)
made the next new step jump to "xg" and escalate from there.
nextId now only counts a key if it round-trips through numberToChars and is not
reserved, and the broken length cap is removed so large flows still get correct
ids.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): keep length cap in nextId to avoid regressing long renames
Address CI review: removing the length cap made all-lowercase renamed step
ids (e.g. "process", which round-trips through numberToChars) feed into the
max and poison id generation again — a regression versus the prior behavior,
since step ids can be renamed to ^[a-zA-Z][a-zA-Z0-9_]*$.
Restore the length>=4 skip and pair it with the round-trip canonical check,
so short non-canonical keys (copy ids "z2"/"c10", reserved/renamed short ids)
no longer poison the max while long renames stay out of the sequence. Update
the tests to reflect the actual coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reject symlink traversal in job-dir path validation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: cover dangling symlink in job-dir path validation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: close symlink-traversal bypass via in-bounds `..` in path check
Walk the normalized relative path instead of raw user components, so an
in-bounds `..` (e.g. `foo/../evil/payload`) can no longer drift the walk
past a planted symlink. Adds regression coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: scope AI sessions per workspace family with lifecycle reconcile
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: centralize session reconcile trigger + extract pure lifecycle decision
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf: remove unused workspace family index
* refactor: scope sessions by workspace root id, drop family_id column
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sessions): preserve user-archived sessions when archiving their workspace
archiveSessionsForWorkspace tagged every session archivedByWorkspace, including ones the user had already archived by hand, so a later workspace unarchive auto-restored them. Skip already-archived sessions so only workspace-archived ones are tagged, matching decideSessionLifecycle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: archived-session banner with unarchive, suppress workspace-gone banner while archived
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: re-root sub-fork sessions on reconcile when an ancestor is deleted
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: group AI sessions by workspace family with show-all-workspaces filter
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: revert unrelated AIProviderPicker cosmetic changes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: hide per-session unarchive when workspace is gone, show move/discard instead
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: GC attached files on lifecycle delete + reconcile on sidebar fork delete
Addresses Codex review: deleteSessionsForWorkspace/reconcile delete now GC linked files (deleteItemsForSession), matching deleteSession; sidebar deleteFork now reconciles so surviving child forks re-root off the deleted ancestor. Also de-flaked post-rehydrate reads in the IndexedDB tests via vi.waitFor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: don't strand user if post-delete reconcile throws; refresh stale warmSessions comment
Addresses auto-review P2s: wrap reconcileAfterWorkspaceChange in deleteFork so the parent switch + navigation always runs even on reconcile failure; correct the warmSessions comment which no longer holds under 'Show all workspaces'.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: don't fail/strand fork archive+delete when client session cleanup throws
Addresses cubic P1/P2 on forks/compare: the workspace archive/delete is authoritative; wrap the best-effort session cleanup + reconcile so a local IndexedDB failure neither falsely reports failure nor blocks navigation away from the gone fork. Mirrors the SidebarContent fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: drop drafting-history aside from reconcileAfterWorkspaceChange comment
Addresses auto-review P2: keep the refresh-before-reconcile invariant, drop the 'which they did inconsistently' narration per AGENTS.md (comments record constraints, not drafting history).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: clean up sessions on fork-id reuse + make all workspace-mutation cleanup best-effort
Addresses Codex P1s: (1) CreateWorkspaceInner 'permanently delete existing fork' (id-reuse) now drops local sessions for that id so they don't resurface on the recreated fork; (2) workspace_settings archive/delete and SidebarContent child-delete loop + main delete now treat post-mutation session cleanup as best-effort, so a local IndexedDB failure can't strand the user or abort remaining deletes (matching the compare-page fix).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: make fork-reuse session cleanup fire-and-forget (non-blocking)
Addresses cubic P2: don't await the best-effort cleanup so a slow IndexedDB op can't block the delete/reuse flow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: drop previous user's transient drafts on user change
Addresses Pi P1: hydrateSessions preserved transient (unsent) drafts across user changes, so user A's draft + its pending fork/workspace state bled into user B's list and got reused by createSession. onUserChange now drops transients when the email changes; reconcile (intra-user) still preserves them. Regression test added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): strip search_path=public from folder_labels migrations for non-public schema
The folder-labels migrations (20260610151334_folder_labels,
20260614075900_dedup_folder_labels) define `folder_labels(...)` with
`SET search_path = public` in their `CREATE FUNCTION` bodies. When Windmill
runs in a non-public schema (PG_SCHEMA), PostgreSQL validates the function
body against the `public` schema, where the `folder` table lacks the new
`labels` column, failing with `column "labels" does not exist`.
Add both migrations to OVERRIDDEN_MIGRATIONS, stripping the
`SET search_path = public` clause so the function inherits the current
search_path (which resolves the correct schema). Same regression and fix
pattern as PR #5400.
Fixes WIN-2093
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): pin folder_labels search_path FROM CURRENT instead of stripping it
Keep the SECURITY DEFINER injection hardening while resolving the correct
schema on non-public (PG_SCHEMA) installs: FROM CURRENT snapshots the
migration connection's search_path at function creation time (public on
normal installs, the custom schema otherwise) instead of dropping the pin
and inheriting the caller's search_path at call time.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): repair migration to re-pin folder_labels search_path on applied instances
Instances that already applied the folder-labels migrations with the hardcoded
SET search_path = public have a folder_labels function pinned to public. On a
non-public (PG_SCHEMA) schema that reads the wrong folder table at runtime; the
OVERRIDDEN_MIGRATIONS fix only helps instances that have not applied them yet.
Add a CREATE OR REPLACE ... SET search_path FROM CURRENT migration that re-pins
the function to the migration connection's schema. No-op on public installs
(re-pins to public) and idempotent on already-correct ones.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The global fork modal (and other modals) portal into `#content`, but that
element only existed in AiChatLayout's `!disableAi` branch. On the AI-session
route `disableAi` is true, so the `{:else}` branch rendered without `#content`,
and opening the fork modal there threw "No element found matching css selector:
#content". Give the else-branch container the same `id` so the portal target is
always present in this layout.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A customer's top-load query was the job_perms orphan sweep (cleanup_job_perms_orphaned:
6.9s mean, 41s max). The cost is discovery, not deletion (~2.1ms per row deleted): the
NOT EXISTS anti-join seq-scans the whole job_perms heap to find a few orphans, and that
scan tracks the heap's physical size. job_perms / job_result_stream_v2 get one row per job
and are drained only by these per-cycle sweeps, so they churn hard — but the bulk
vacuuming_tables() runs only ~hourly, so dead tuples bloat the heap between bulk vacuums.
Reclaim right after each sweep instead: VACUUM (SKIP_LOCKED) the swept table when it
deleted rows. Plain VACUUM (not FULL) takes only SHARE UPDATE EXCLUSIVE so concurrent job
creates/reads proceed; the visibility map skips unchanged pages so repeated runs are cheap;
SKIP_LOCKED means HA replicas don't pile up (one vacuums, the rest skip). Benchmarked ~7x:
a bloated 268MB job_perms heap swept in 35ms vs 5ms vacuumed. Chosen over an autovacuum
reloptions migration so the behavior is explicit and lives with the sweep it pairs with.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(audit): adaptive timestamp floor for S3 audit-log export (ee)
EE change in windmill-ee-private (src/ee.rs); this OSS commit carries the regenerated
sqlx cache for the new oldest-in-flight query and bumps ee-repo-ref.txt to the EE branch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to ed89574be9117cda5e2d7d9de02cb5db066e93e3
This commit updates the EE repository reference after PR #628 was merged in windmill-ee-private.
Previous ee-repo-ref: 8a7f645c0a194a284fe19dd20dbe79dd0733dfdb
New ee-repo-ref: ed89574be9117cda5e2d7d9de02cb5db066e93e3
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(apps): show raw-app fork diffs as per-file tree items
Raw-app diffs previously rendered as one big YAML diff of the whole
serialized app. This explodes a raw app into separate, independently
collapsible diff items — one per file, one per runnable, and an
app.yaml metadata item — that flow through the existing fork-diff
list, sidebar tree, search and count via composite paths
(<appPath>/<file>). Runnables render as script/flow rows (code shown
in a Content tab), and files get extension-specific icons reused from
the raw-app editor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove raw-app tree-diff plan doc from the branch
The implementation plan was an authoring aid, not product documentation; drop it so it doesn't ship in the PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: present raw app as an app-headed folder in the diff tree
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: narrow RawAppFileItem in diff viewer branch (fixes svelte-check)
DiffRow.kind is a plain string so the kind check didn't narrow the union; assert the synthetic item. Also size-guard on the larger side's line count instead of the doubled total, and document normalizeRawApp's per-field value-wrapper precedence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: single-line, lighter diff-tree rows for all item kinds
Add a singleLine mode to WorkspaceItemRow (summary ?? path on one line; DRY'd via a shared body snippet) and use it for every diff-tree leaf, so scripts/flows/triggers/resources/etc. match the raw-app header. Bump rows to py-1.5, force font-normal, and split colours: items in text-primary, folders in text-secondary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: extract pure diffTree model from WorkspaceDiffDrawer
Move tree construction + keyboard-nav traversal + the folder-keying convention out of the 775-line component into a pure, generic, tested module (buildDiffTree → root/order/parentKeyOf/firstChildKeyOf). Parent and first-child come from a child→parent map built during construction, not from re-splitting a path at the call site, so a node's tree position and its nav parent can't drift — the class of bug behind the ArrowLeft regression. Deletes the forkDiffNav half-seam (its bug lived in the untested caller). 12 new unit tests cover order/parent/first-child incl. the storage-key-vs-friendly-path case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(apps): keep raw-app metadata flag + dedup runnables across path collisions
Addresses two P2 review nits (Codex/claude): (1) rawAppDiffToItems marked metadata by matching path==='app.yaml', so when a real file is named app.yaml the reserved app.yaml~2 metadata item lost its flag/full-YAML toggle — now parseRawAppDiff tags the entry with isMetadata and the items read the flag; (2) runnable composite leaves weren't deduped against real files, so a real file at runnables/<name> could produce a duplicate leaf — now reserved (slash-normalized) like parseRawAppDiff. +2 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(apps): dedup /app.yaml metadata collision + disambiguate synthetic row keys
Two follow-up P2s from Pi/Codex re-review of the prior fix: (1) parseRawAppDiff's collision set used raw file keys, so a real file /app.yaml (leading slash, which joinAppPath strips) still collided with the synthetic app.yaml leaf — now slash-normalized via a shared stripLeadingSlash, +test. (2) synthetic raw-app items (runnables rendered as script/flow) could share kind+path identity with a real workspace script/flow at <appPath>/runnables/<name>, causing duplicate {#each} keys and broken nav — itemKey now prefixes synthetic items (rawapp:).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(apps): canonicalize raw-app file keys to dedup leading-slash collisions
Codex P2: a file keyed /App.tsx on one side and App.tsx on the other became two entries that joinAppPath collapsed to one composite path → duplicate row key. asFileMap now strips the leading slash so both sides resolve to one file. +test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(apps): lazy-mount per-file diff editors as they scroll into view
Exploding a raw app into N per-file rows mounted N Monaco DiffEditors at once (3 reviews flagged it). Each block's editor now mounts only when it scrolls within ~200px of the viewport (IntersectionObserver rooted on the scroll container), showing a light placeholder until then; mountedRows latches so it never unmounts on scroll-away. Verified: ~6 of 13 mount initially, the rest on scroll.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AI skills are only consumed by the GLOBAL chat mode's system prompt, and
global mode itself is dev-gated by isGlobalAiEnabled(). Gate the workspace
AI skills settings tab on the same flag so it isn't shown when the skills
can't be used, and add it to gate.ts's rip-out inventory.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(ai-chat): strip unclosed <summary> tag leaking into compaction summary
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-chat): strip analysis before matching summary to avoid scratchpad leak
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: forbid superadmin job tokens from global user and token management
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: extend superadmin job token guard to offboard and export routes
Apply forbid_superadmin_job_token to offboard_global_user and
export_global_users, the remaining global user-management routes that
were gated only by require_super_admin. Offboarding can delete a user
along with their tokens, password, invites and instance-group
membership, and export returns every user's password_hash, so both must
be unreachable by a superadmin job token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: render mermaid diagrams in chat code blocks
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: guard mermaid render against out-of-order async and transient streaming failures
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: only show mermaid diagram while it matches current source
Addresses Codex review: keeping the last good SVG through parse failures left a stale, mismatched diagram on screen when the source changed to something invalid. Tie the rendered SVG to the source that produced it and only display it while it still matches the current code, falling back to the raw source otherwise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(telemetry): disclose resource and infra usage stats
When minimal telemetry is disabled, the stats payload now includes resource
counts (workspaces, scripts per language, flows, workflows as code, low-code
and raw apps) and, on EE only, infrastructure info (container runtime,
database size, max connections, RDS detection).
Update the telemetry disclosure in instance settings accordingly: resource
counts are listed for both CE and EE; infra info is shown only on EE since it
is collected only there. Bump the EE ref and add the sqlx cache for the new
queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(telemetry): expand EE infra disclosure and add sysinfo dep
Disclose the expanded EE infrastructure telemetry (deployment mode, host
OS/arch/CPU/memory, filesystem space, Postgres version and connection counts,
object storage backend, sandboxing and retention settings) in instance
settings. Add sysinfo as a windmill-common dependency for host memory and
filesystem stats, bump the EE ref, and add the sqlx cache for the new queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(telemetry): focus EE infra disclosure on wrapping platform
Drop the single-server host details (OS, arch, CPU, memory, filesystem) and
tuning config from the EE infra disclosure, and revert the sysinfo dependency
they required. Reflect managed-database-provider detection in place of the RDS
flag. Bump the EE ref and update the sqlx cache for the revised queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(telemetry): drop deployment mode and worker count from disclosure
Remove deployment mode and worker count from the EE infra disclosure to match
the backend, and bump the EE ref. They reflect only the node sending telemetry,
not the deployment topology.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 6d3301507db50818f1683dac3941d3e0cf1152a7
This commit updates the EE repository reference after PR #627 was merged in windmill-ee-private.
Previous ee-repo-ref: d30e7d18d14992598a97356d0ed13f7d5d585115
New ee-repo-ref: 6d3301507db50818f1683dac3941d3e0cf1152a7
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* fix: allow non-super-admin object storage test, harden SSRF surface on cloud
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: validate effective object storage host to close region/bucket SSRF bypass
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: validate gcs_base_url/token_uri in GCS service account key to close SSRF bypass
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: match url scheme case-insensitively in object storage host validation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The expired-job retention loop re-scanned the same oldest rows on every batch. When
the oldest completed jobs are undeletable (children of a still-active root flow), the
ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20
batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch,
~180s/cleanup-cycle on a 1.5M-row prefix).
Carry a completed_at watermark (max deleted) across batches and re-apply it as
completed_at >= floor so each batch resumes past the already-processed prefix. Also skip
the v2_job join entirely when no old root flow is active (the common case), since nothing
is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms.
The watermark only ever skips rows the current run already deleted, was protecting, or
skip-locked — all deferred to the next run, identical to the unbounded scan's row set
(verified: union of batched deletes == single delete, 0 diff). Mirrored in
windmill-api-settings log_cleanup.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): debounce external code→Monaco sync in Editor
Make the external `code` prop → Monaco sync always-on and 500ms
debounced, replacing the opt-in `syncExternalCode` prop. Removes the
prop from the two inline rawscript call sites in FlowModuleComponent.
Includes temporary debug scaffolding (A→B executeEdits button and
console logs) for diagnosing successive-edit behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(frontend): share alignCodeWithEditor + bump debounce to 800ms
Extract the full-range executeEdits sync into alignCodeWithEditor() and
reuse it from both setCode and the debounced external-code effect. Bump
the external-sync debounce 500ms -> 800ms. ScriptEditor now calls
editor.setCode when syncing external code in.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* nits
* nits
* Fix AI not seeing latest code
* remvoe debug button
* nit types
* nits
* Nits
* Check timeoutModel is undefined
* fix(frontend): suppress editor echo in external code sync to prevent typing clobber
* fix(frontend): cancel pending keystroke debounce in setCode to prevent clobber
* fix(frontend): preserve pending external code write in updateCode
* Revert "fix(frontend): preserve pending external code write in updateCode"
This reverts commit 731d877730.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Diego Imbert <diego@windmill.dev>
Closing the Instance settings drawer cleared the underlying script
editor. On unmount, SuperadminSettingsInner.removeHash() stripped the
`#superadmin-settings` hash with a SvelteKit `goto()`, and that
navigation re-fired the script editor page's path-reactive `$effect`,
reloading the script and wiping unsaved editor content.
Use `replaceState` to drop the hash without a navigation (matching the
existing RunForm.svelte pattern), guarded against router-teardown throws.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reject pipeline `# tag` annotation false-positives on regular comments
`parse_pipeline_annotations` treats any comment line starting with
`# tag <text>` as a worker-tag annotation. In Python scripts, ordinary
English comments beginning with "# tag ..." were misinterpreted: values
over 50 chars failed the `script.tag` INSERT (varchar(50)), and shorter
ones silently overrode the script's worker tag.
Worker tags are single-word identifiers (e.g. `heavy`, `gpu`), so reject
any candidate that contains whitespace or exceeds 50 characters. Mirror
the same validation in the TS parity parser and add regression tests on
both sides.
Fixes WIN-2090
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: restrict pipeline annotation scan to the leading comment header
The root cause of the `# tag` false-positive is broader than the `tag`
keyword: `parse_pipeline_annotations` scanned every comment line in the
whole file, so any body comment matching an annotation grammar
(`on`, `freshness`, `tag`, `retry`, ...) was misinterpreted. The `tag`
case was the most visible because an over-length value crashed the
`script.tag` INSERT (varchar(50)).
Windmill's other comment-directive parsers (BashAnnotations::sandbox_image,
ssh_target) already scan only the leading comment header and stop at the
first line of real code. Align parse_pipeline_annotations (and its TS
mirror) with that convention: skip blank lines, break on the first
non-comment line. This eliminates body-comment false-positives for every
annotation, not just `tag`.
The `tag` whitespace/length guard from the previous commit is kept as
defense for prose that sits in the header itself.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-23 12:35:37 +00:00
676 changed files with 50687 additions and 9161 deletions
description:Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window.
---
## Always benchmark before and after
No context or behavior change ships without an `ai_evals` A/B on the affected mode.
Add or adjust cases for exactly what you changed — see the `ai-evals` skill for
authoring and the full run reference.
Run the affected mode **before** your change and **after**, same model(s), same cases.
## Measure the window first, and cumulative second
Optimize **`finalContextTokens`** (window occupancy — what drives overflow and
compaction), then cumulative prompt tokens.
## Context discipline
The dominant fixed cost is per-iteration overhead: the system prompt **plus every
tool schema** is re-sent on every loop iteration. So:
- **Every tool and every parameter is a permanent tax.** Justify each one and measure
it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead
params rather than leaving them in the schema.
- **Tool results return the minimum.** Never echo content the model already has. The
canonical mistake: a write tool that returns the whole edited artifact right after
the model authored it — return `{ success, message }` instead. When you touch a
*shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check
this invariant for **all** the write tools routing through it — the echo has
regressed before via a shared refactor.
## Prompts and tool descriptions are part of the surface
The system prompt and tool descriptions steer behavior as much as the tools
themselves, and are benchmarkable the same way. A description that advertises
truncation makes the model self-limit; the path-conventions block changes where
drafts land. Treat prompt/description edits as real changes and A/B them — a
pure-prompt change is a legitimate, measurable improvement.
description:Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes.
---
# AI evals — authoring and running benchmark cases
`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes:
`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production
prompts, tools, and guidance in this checkout. Each attempt runs the real production
path, deterministic validation, then LLM judging.
The goal is to test current production guidance with realistic user requests — **not**
to pin one exact implementation shape.
## Running benchmarks
```bash
cd ai_evals
bun install # first time; frontend modes also need `cd frontend && bun install`
bun run cli -- models # list model aliases
bun run cli -- cases global # list cases for a mode
bun run cli -- run global global-test1-script-create --model sonnet
```
Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill
backend's `/api/w/<ws>/ai/proxy`, so you need **any** reachable backend:
@@ -24,6 +24,7 @@ Open-source platform for internal tools, workflows, API integrations, background
## Dev Environment
- **Backend**: `cargo run` from `backend/` (API at http://localhost:8000)
- **DuckDB local jobs**: before running DuckDB scripts locally, build the FFI shared library with `cd backend/windmill-duckdb-ffi-internal && ./build_dev.sh`. Re-run it after clean builds or when `backend/target/debug/libwindmill_duckdb_ffi_internal.*` is missing.
- **Frontend**: `REMOTE=http://localhost:8000 npm run dev` from `frontend/` (port 3000+)
* **ai:** add Azure AI Foundry as a native AI provider ([#9879](https://github.com/windmill-labs/windmill/issues/9879)) ([d9b080f](https://github.com/windmill-labs/windmill/commit/d9b080f57fa0be144cefa773d39742c45b40f043))
* **frontend:** group compare & deploy items by folder ([#9880](https://github.com/windmill-labs/windmill/issues/9880)) ([7b04820](https://github.com/windmill-labs/windmill/commit/7b04820f8ef8c7f02f79dd4239a877f667d23e6a))
* **frontend:** pipelines index page and sql editor hint ([#9881](https://github.com/windmill-labs/windmill/issues/9881)) ([20351a6](https://github.com/windmill-labs/windmill/commit/20351a6b4c262184c5f815eeb5de007ab1eaf4a0))
* **pipeline:** backfill a range of partitions from the asset drawer ([#9885](https://github.com/windmill-labs/windmill/issues/9885)) ([53bbb92](https://github.com/windmill-labs/windmill/commit/53bbb92953178eb6d0017818ef870f3cb2399dfd))
* **s3:** replace CE 50MB upload cap with 10GiB workspace storage quota ([#9874](https://github.com/windmill-labs/windmill/issues/9874)) ([af01e90](https://github.com/windmill-labs/windmill/commit/af01e90b5c65d1b1cfacf4433f8cff7effe73768))
* support workspace forks on cloud using parent workspace limits ([#9864](https://github.com/windmill-labs/windmill/issues/9864)) ([7c7d747](https://github.com/windmill-labs/windmill/commit/7c7d7474cc86a4052272032f281cc4d7a85db37b))
* **frontend:** home New submenus fall back below, hugging the right edge ([#9894](https://github.com/windmill-labs/windmill/issues/9894)) ([186ac49](https://github.com/windmill-labs/windmill/commit/186ac4933b79aed57fce23ebcf3b525fcfd1c474))
* **frontend:** show inline workspace name editor on general settings (Fixes GIT-911) ([#9892](https://github.com/windmill-labs/windmill/issues/9892)) ([a49c087](https://github.com/windmill-labs/windmill/commit/a49c0871d7ab2aaf78a7713b8a786ead937434da))
* **frontend:** stack cron field and cron builder button on narrow screens ([#9871](https://github.com/windmill-labs/windmill/issues/9871)) ([7989795](https://github.com/windmill-labs/windmill/commit/79897950e7646b00d92a28a009174d91c705b251))
* invalidate bun bundle cache on transitive relative-import changes ([#9891](https://github.com/windmill-labs/windmill/issues/9891)) ([d15033c](https://github.com/windmill-labs/windmill/commit/d15033cde6a474b548ebbaf18ff02223fc21f701))
* make SMTP username and password optional in frontend validation ([#9895](https://github.com/windmill-labs/windmill/issues/9895)) ([37bb574](https://github.com/windmill-labs/windmill/commit/37bb57474e8336823bb31527f2a708ef41cd39c4))
* **parsers:** infer py s3 assets from S3Object constructor and dict forms ([#9877](https://github.com/windmill-labs/windmill/issues/9877)) ([659642e](https://github.com/windmill-labs/windmill/commit/659642e4889361f86e8addb038cda62fc3471006))
* stale AI chat context picker after workspace item changes ([#9893](https://github.com/windmill-labs/windmill/issues/9893)) ([5af91a6](https://github.com/windmill-labs/windmill/commit/5af91a677cad88faccba702e3556fc4fb7b6e640))
* **frontend:** add zoom and download to Mermaid graphs ([#9859](https://github.com/windmill-labs/windmill/issues/9859)) ([289017b](https://github.com/windmill-labs/windmill/commit/289017bcb28c049c8258b2ffd7da0ec3e6ef120b))
* use derived username instead of email for non-member superadmins ([#9857](https://github.com/windmill-labs/windmill/issues/9857)) ([76a9523](https://github.com/windmill-labs/windmill/commit/76a95230095ca3f43c9dc9eecde0e9de6520242f))
* **folders:** allow dots and at-signs in folder owner validation ([#9856](https://github.com/windmill-labs/windmill/issues/9856)) ([383c705](https://github.com/windmill-labs/windmill/commit/383c70523bf81c5c07784a4379ef6b1c93ff86e5))
* **forks:** require admin of both sides for the compare visibility guard ([#9869](https://github.com/windmill-labs/windmill/issues/9869)) ([7363d2c](https://github.com/windmill-labs/windmill/commit/7363d2c217cb04391f03b2f9958be70a9d0b5325))
* **forks:** reset diff tally on trigger delete + guard compare visibility for admins ([#9866](https://github.com/windmill-labs/windmill/issues/9866)) ([6a6f129](https://github.com/windmill-labs/windmill/commit/6a6f12960e29c314d11ad541519c71412f42567b))
* **jobs:** give flow dynselect a path and its worker tag, like scripts ([#9867](https://github.com/windmill-labs/windmill/issues/9867)) ([1a9debb](https://github.com/windmill-labs/windmill/commit/1a9debb689f756f38db085d1360c8fe7691ada48))
* **offboarding:** make global reassignment per-workspace and optional ([#9863](https://github.com/windmill-labs/windmill/issues/9863)) ([3586164](https://github.com/windmill-labs/windmill/commit/35861641f807a02b5c205608fb592e20ee7cad7f))
* add copy-to-clipboard button to rendered Mermaid diagrams in AI chat ([#9838](https://github.com/windmill-labs/windmill/issues/9838)) ([a27e814](https://github.com/windmill-labs/windmill/commit/a27e814a03c615259381eaf684aa90d56569b0af))
* add dev workspaces paired with a lockable prod workspace ([#9793](https://github.com/windmill-labs/windmill/issues/9793)) ([b4b0c6a](https://github.com/windmill-labs/windmill/commit/b4b0c6a93e52152251fadefe319773faf42549b2))
* **ansible:** support repo-provided ansible.cfg in delegate_to_git_repo ([#9851](https://github.com/windmill-labs/windmill/issues/9851)) ([68bf0da](https://github.com/windmill-labs/windmill/commit/68bf0daf5815307cda6ce23214dd5159b6aa33b4))
* **licensing:** enforce offline license seat cap ([#9845](https://github.com/windmill-labs/windmill/issues/9845)) ([83f3d7f](https://github.com/windmill-labs/windmill/commit/83f3d7f910b331c09f60cc9ff556728afa3dec07))
* **object-store:** make GCS service account key optional for Workload Identity ([#9842](https://github.com/windmill-labs/windmill/issues/9842)) ([83ed011](https://github.com/windmill-labs/windmill/commit/83ed011e264f20ffa66a7bf933f2fe3615cf6b67))
* **pipeline:** local development for data pipelines (CLI --local + pipeline dev preview) ([#9840](https://github.com/windmill-labs/windmill/issues/9840)) ([74f579e](https://github.com/windmill-labs/windmill/commit/74f579e6d9ef08e74460f904a4c22ed9d6a3b5b0))
* **pipelines:** add managed SCD2 history materialize strategy ([#9850](https://github.com/windmill-labs/windmill/issues/9850)) ([5a66127](https://github.com/windmill-labs/windmill/commit/5a661279a3690e2393b9b16996f5d1a5a509259c))
* grant dispatch_event table to windmill roles ([#9852](https://github.com/windmill-labs/windmill/issues/9852)) ([f05b50d](https://github.com/windmill-labs/windmill/commit/f05b50d29ac2fdbb808a97057fb92c8e425b4a2f))
* grant workspace_diff, materialized_partition, debounce_stale_data to windmill roles ([#9853](https://github.com/windmill-labs/windmill/issues/9853)) ([293647d](https://github.com/windmill-labs/windmill/commit/293647de4c13cb8468cbd81ff1924cba90e164b4))
* honor verify-ca/verify-full sslmode for postgres connections ([#9835](https://github.com/windmill-labs/windmill/issues/9835)) ([bf6be96](https://github.com/windmill-labs/windmill/commit/bf6be967fa8c74e1299cf63f813c1cfa34b97f3e))
* **s3_proxy:** preserve URL-encoding on Hive-partition proxy writes ([#9848](https://github.com/windmill-labs/windmill/issues/9848)) ([6b79bdd](https://github.com/windmill-labs/windmill/commit/6b79bddd42fe55f891c17cb71a7e36ee31337bac))
* validate workspace name length (max 50 chars) on create and fork ([#9854](https://github.com/windmill-labs/windmill/issues/9854)) ([b52972d](https://github.com/windmill-labs/windmill/commit/b52972d0de89004e98d18241d238ca028e4eecba))
* **home:** redesign create-new popover and home header ([#9827](https://github.com/windmill-labs/windmill/issues/9827)) ([2493eaf](https://github.com/windmill-labs/windmill/commit/2493eaf031f30072637a297398674e761f039005))
* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e))
* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d))
* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62))
* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e))
* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69))
* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048))
* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5))
* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57))
### Bug Fixes
* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98))
* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971))
* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32))
* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b))
* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a))
* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b))
* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279))
* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1))
* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89))
### Bug Fixes
* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25))
* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe))
* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b))
* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f))
* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9))
* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8))
* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d))
* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443))
* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b))
* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f))
* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319))
* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7))
### Performance Improvements
* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de))
* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43))
* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209))
* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381))
* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d))
* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548))
* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38))
* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2))
* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c))
### Bug Fixes
* **ai-chat:** strip unclosed <summary> tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d))
* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d))
* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43))
* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23))
* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5))
* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb))
* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37))
* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d))
* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc))
* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b))
### Bug Fixes
* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a))
* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e))
* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9))
* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7))
* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3))
* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6))
### Performance Improvements
* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10))
This folder contains black-box benchmark cases for:
Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`,
`script`, `cli`, `global`).
-`flow`
-`app`
-`script`
-`cli`
-`global`
**Authoring and running cases is documented in the `ai-evals` skill** — load it
before adding/changing a case or running a benchmark. Claude Code reads
`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read
`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in
Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`.
The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape.
For AI chat / copilot changes that these evals measure, see the `ai-chat` skill.
## Core rules
1. Write prompts like a real user request.
2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details.
3. Keep deterministic validation narrow and hard.
4. Put semantic expectations in `judgeChecklist`.
5. Use `expected` fixtures only when exact structure really matters.
## Prompt writing
Prompts should sound like something a user would naturally ask.
Good:
- "Create a flow that routes support requests based on customer tier."
- "Add a reset button that sets the counter back to 0."
- "Create a flow that reuses the existing greeting script instead of duplicating the logic."
Bad:
- "Use `branchone` with 3 branches and a default branch."
- "Create a `rawscript` step with this exact topology."
- "This is a benchmark harness."
Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow.
## Flow-specific rules
This is the main principle you asked for:
- flow prompts should read like requests from a user who does not know the product internals
- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs
That means:
- creation cases should describe the business behavior and expected result
- modification cases may mention existing step names, because the user can see the current flow
- only mention special Windmill constructs when the case is explicitly about those constructs
Examples:
- acceptable creation prompt:
"Create a purchase approval flow that pauses for approval and asks the approver for a comment."
- avoid:
"Create a suspend step with one required event and a resume form."
For flow cases, do not fail a case just because the model chose a different valid topology.
## App-specific rules
App prompts should focus on user-visible behavior:
- what the UI should let the user do
- what should persist
- what backend behavior is needed
Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app.
## CLI-specific rules
CLI prompts can be more explicit about paths and file names because real CLI users often do specify them.
Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction.
When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior.
## Global-specific rules
Global prompts should exercise workspace-level drafting behavior:
- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant
- writing AI drafts rather than saving or deploying by default
- producing coherent multi-artifact changes when the request crosses artifact boundaries
Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them.
Datatable cases should set `skipJudge: true` and validate through tool-use
(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions
(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`,
`['update', 'insert into']`). Two reasons the judge is unreliable here:
-`list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql`
produce no drafts, and the global judge only sees the drafts artifact — it
scores a no-draft conversational answer as empty (same as the
`askUserQuestion` cases).
- Even a case that *does* produce a draft (a script reading the data table via
`wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK
reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the
- builds a data pipeline node as a script (not a flow)
- marks the script as a pipeline member with the pipeline annotation in the script's comment syntax (`-- pipeline` for a DuckDB/SQL node, not `// pipeline`)
- declares a schedule trigger and writes its output to a managed DuckLake table
- leaves the result as an AI draft and does not deploy or save it
- id:global-test-pipeline-two-node-chain
prompt:|-
Build a small data pipeline in the `f/evals/global` folder: one step that
ingests orders into a DuckLake table, and a second step that reads that table
and writes a daily order-count rollup table. Wire the second step to run off
the first step's output. Keep everything as drafts — don't deploy.
"query":"UPDATE materialized_asset_schema\n SET snapshot_id = $5, job_id = $6, captured_at = now()\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n AND version = $4",
"query":"\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ",
"query":"\n WITH completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n CASE WHEN q.running\n THEN $3::text::jsonb\n ELSE $4::text::jsonb\n END,\n s.flow_status,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n WHERE q.id = $1\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id = $1\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n VALUES ($5, $1, $2)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ",
"query":"SELECT consumer_path AS \"consumer_path!\", macro_name AS \"macro_name!\"\n FROM macro_usage\n WHERE workspace_id = $1\n AND ($2::text IS NULL OR consumer_path LIKE $2)",
"query":"\n SELECT\n kind AS \"kind!: AssetKind\",\n path AS \"path!\"\n FROM asset\n WHERE workspace_id = $1\n AND usage_kind = 'script'\n AND usage_path = $2\n AND usage_access_type IN ('w', 'rw')\n ",
"query":"SELECT version, columns AS \"columns: Json<Vec<SchemaColumn>>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1",
"query":"\n WITH RECURSIVE chain AS (\n SELECT id, parent_workspace_id, 0 AS depth\n FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.parent_workspace_id, chain.depth + 1\n FROM workspace w\n JOIN chain ON w.id = chain.parent_workspace_id\n WHERE chain.depth < 20\n )\n SELECT COALESCE(MAX(depth), 0)::bigint AS \"depth!\" FROM chain\n ",
"query":"INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id, is_dev_workspace)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END, $5\n FROM workspace WHERE id = $3",
"query":"SELECT version, columns AS \"columns: Json<Vec<SchemaColumn>>\",\n snapshot_id, job_id, captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC",
"query":"\n WITH RECURSIVE tree AS (\n SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT COALESCE(MAX(depth) FILTER (WHERE NOT deleted), 0)::bigint AS \"height!\" FROM tree\n ",
"query":"SELECT path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork FROM workspace_diff\n WHERE source_workspace_id = $1 AND fork_workspace_id = $2",
"query":"SELECT path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork FROM workspace_diff\n WHERE source_workspace_id = $1 AND fork_workspace_id = $2\n AND NOT EXISTS (\n SELECT 1 FROM ws_specific ws\n WHERE ws.path = workspace_diff.path\n AND ws.item_kind = workspace_diff.kind\n AND ws.workspace_id IN (workspace_diff.source_workspace_id, workspace_diff.fork_workspace_id)\n )",
"query":"INSERT INTO ws_specific (workspace_id, item_kind, path)\n SELECT $1::varchar, 'resource', $2::varchar\n WHERE EXISTS (SELECT 1 FROM resource WHERE workspace_id = $1::varchar AND path = $2::varchar)\n ON CONFLICT DO NOTHING",
"query":"DELETE FROM macro_usage WHERE workspace_id = $1 AND (consumer_path = $2 OR macro_name IN (SELECT name FROM macro_definition WHERE workspace_id = $1 AND provider_path = $2))",
"query":"SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND upload_id <> $2 AND created_at > now() - ($3::text)::interval)::bigint as \"other_reserved!\"",
"query":"SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n COALESCE(\n (SELECT MIN(computed_at) FROM workspace_storage_usage WHERE workspace_id = $1) < now() - interval '10 minutes',\n true) as \"stale!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at > now() - ($2::text)::interval)::bigint as \"reserved!\"",
"query":"INSERT INTO usr\n (workspace_id, email, username, is_admin)\n SELECT $1, email, username, is_admin FROM usr\n WHERE workspace_id = $3 AND email = $2\n ",
"query":"INSERT INTO usr\n (workspace_id, email, username, is_admin)\n SELECT $1, email, username, is_admin FROM usr\n WHERE workspace_id = $3 AND email = $2\n ON CONFLICT DO NOTHING\n ",
"query":"\n WITH RECURSIVE tree AS (\n SELECT id, deleted, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, w.deleted, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT COUNT(DISTINCT id) AS \"count!\" FROM tree WHERE id != $1 AND NOT deleted\n ",
"query":"DELETE FROM flow_conversation_message m\n USING flow_conversation c\n WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)",
"query":"WITH potential AS (\n SELECT email, operator FROM usr WHERE is_service_account IS false\n UNION\n SELECT email, operator FROM workspace_invite\n ),\n per_user AS (\n SELECT email, bool_and(operator) AS only_operator FROM potential GROUP BY email\n )\n SELECT\n COUNT(*) FILTER (WHERE NOT only_operator) AS \"authors!\",\n COUNT(*) FILTER (WHERE only_operator) AS \"operators!\"\n FROM per_user\n WHERE email NOT IN (SELECT email FROM password WHERE disabled IS true)",
"query":"\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND (content LIKE '%' || $2 || '%' OR path = ANY($3))\n ORDER BY path, created_at DESC\n ",
"query":"SELECT COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n ORDER BY d.email NULLS LAST",
"query":"SELECT COALESCE(u.username, p.username, CASE WHEN p.email IS NOT NULL THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n LEFT JOIN password p\n ON p.email = d.email\n AND p.super_admin = true\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n ORDER BY d.email NULLS LAST",
"query":"SELECT bool_and(operator) FROM (\n SELECT operator FROM usr WHERE email = $1\n UNION ALL\n SELECT operator FROM workspace_invite WHERE email = $1\n ) t",
"query":"SELECT bool_and(operator) FROM (\n SELECT operator FROM usr WHERE email = $1 AND is_service_account IS false\n UNION ALL\n SELECT operator FROM workspace_invite WHERE email = $1\n ) t",
"query":"SELECT\n (SELECT COALESCE(SUM(bytes), 0) FROM workspace_storage_usage WHERE workspace_id = $1)::bigint as \"committed!\",\n COALESCE(\n (SELECT MIN(computed_at) FROM workspace_storage_usage WHERE workspace_id = $1) < now() - interval '10 minutes',\n true) as \"stale!\",\n (SELECT COALESCE(SUM(GREATEST(inflight_bytes - target_existing_size, 0)), 0)\n FROM workspace_multipart_inflight\n WHERE workspace_id = $1 AND created_at > now() - ($2::text)::interval\n AND ($3::text IS NULL OR upload_id <> $3))::bigint as \"reserved!\"",
"query":"SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\",\n result AS \"result: Json<Box<RawValue>>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5",
"query":"SELECT workspace.id, workspace.name, usr.username, workspace_settings.color, workspace.parent_workspace_id,\n CASE WHEN usr.operator THEN workspace_settings.operator_settings ELSE NULL END as operator_settings,\n usr.disabled\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n JOIN workspace_settings ON workspace_settings.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false",
"query":"SELECT workspace.id, workspace.name, usr.username, workspace_settings.color, workspace.parent_workspace_id,\n workspace.is_dev_workspace,\n CASE WHEN usr.operator THEN workspace_settings.operator_settings ELSE NULL END as operator_settings,\n usr.disabled\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n JOIN workspace_settings ON workspace_settings.workspace_id = workspace.id\n WHERE usr.email = $1 AND workspace.deleted = false",
"query":"SELECT name AS \"name!\", provider_path AS \"provider_path!\",\n params AS \"params!\", is_table_macro AS \"is_table_macro!\"\n FROM macro_definition\n WHERE workspace_id = $1\n ORDER BY provider_path, name",
"query":"\n WITH RECURSIVE tree AS (\n SELECT id, 0 AS depth FROM workspace WHERE id = $1\n UNION ALL\n SELECT w.id, tree.depth + 1 FROM workspace w\n JOIN tree ON w.parent_workspace_id = tree.id\n WHERE tree.depth < 20\n )\n SELECT id AS \"id!\" FROM tree WHERE id != $1\n ",
"query":"SELECT runnable_path AS \"runnable_path!\", kind::text AS \"kind!\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'asset'\n ORDER BY runnable_path",
"query":"DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script' AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)",
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.