- CE: the auto-pull and fork-PR toggles are disabled with an EE badge, and
new sync repos only default them on when licensed (basic git sync is
available on CE since #8493, but auto-pull is EE and the backend rejects it)
- The pull modal passes clone_ref for wm-fork- forks (wm-fork/<tracked>/<id>)
so a manual pull fetches the fork branch instead of the tracked branch head
- PR-on-deploy skips no-op pushes: when the push script reports pushed=false
(e.g. the deploy was caused by an auto-pull), the completion hook no longer
ensures a PR, so closed PRs aren't recreated by the sync loop
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
- The PR diff completion hook maintains one managed comment on the PR
(Cloudflare deploy-preview style: workspace, status, commit, collapsible
change list), upserted per synchronize via a hidden marker. The check run
stays for required-check gating.
- A settings difference in the diff summary is worded by cause: the PR
changes wmill.yaml, vs pre-existing drift between the repo's wmill.yaml
and the workspace, vs undetermined (neutral wording).
- Deploy-status check titles name the target workspace ("Deployed 2
change(s) to staging"), since GitHub shows a head commit's checks on any
PR containing it and a bare "Deployed" read as if the PR had deployed.
- Token-based repos see a hint pointing at the open-pr-on-commit /
open-pr-on-fork-commit workflows where the app-only PR toggles would be;
an API-set toggle on a non-app repo now logs a warning naming the
fallback; the design doc lists app-only features and their degradation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
A fork of a dev workspace now roots its wm-fork/** branch on the dev's
environment-label branch (the content it diverged from) and its PR merges
back into that branch: the backend passes parent_dev_workspace_label with
the deploy (parent row joined in both enqueue paths), the CLI gains
--parent-dev-workspace-label and checks it before the wm-fork- prefix
fallback when rooting a fork-of-a-fork branch, and the PR completion hook
uses it as the PR base.
Fork sync routing covers the whole live descendant chain of the
webhook/poller workspace (recursive, depth-capped) instead of direct
children only, and fork_open_prs is resolved at the root ancestor — only
the root can hold auto-pull config, so grandchild forks sync through it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
A dev workspace's git branch is its environment label verbatim (dev/
staging, default dev) — a first-class env branch like the documented
push-on-merge-staging layout — instead of the wm-fork/** form. The label
rides the deploy job args (backend → hub script → CLI
--dev-workspace-label), the PR completion hook derives the same head, the
webhook/poller route label branches into the matching dev-workspace child
(poller lists them alongside wm-fork/* via extra ls-remote refs / per-label
API lookups), and manual pulls from the UI pass clone_ref accordingly. The
CLI refuses to deploy when the label branch equals the checked-out tracked
branch, which would otherwise commit fork content straight to it.
Because the branch is keyed on the label, the label is now immutable after
creation: set at create/attach only, the set_dev_workspace_label endpoint
is removed and the settings tab shows it read-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
Fork sync (push-on-merge-to-forks parity): a parent-level
auto_pull.sync_forks toggle routes changes on each fork's wm-fork/** branch
into that fork workspace, via the parent's existing webhook and one extra
fork-heads listing per poll tick (git ls-remote pattern for token repos,
git/matching-refs for app-backed). Fork state is a server-written
status-only auto_pull blob on the fork's own repo entry; the fork's card
shows a read-only "managed in the parent workspace" line with its branch
and last pull status. Dev workspaces (prefix-less ids) use the same branch
parsing (unit-tested in windmill-common).
PR-on-deploy: opening PRs for Windmill-pushed branches moves into the
deploy pipeline, per repo toggle (promotion_open_prs on the promotion
repo; parent-level fork_open_prs for fork deploys). The push job carries a
marker and the job-completion hook derives the pushed branch (helper
unit-tested against the CLI formula) and opens the PR outbound, so it
works without inbound webhooks; the webhook-side wm_deploy PR arm is
removed. The documented open-pr-* GitHub Actions remain valid alternatives
(PR creation is idempotent).
Fork guards: promotion mode, enabled auto-pull, and fork_open_prs are
rejected on fork workspaces (they are parent-managed; a fork's deploys
always target its wm-fork/** branch) and the promotion card is hidden in a
fork's settings. Enabling auto-pull now also requires EE, and the
post-commit webhook reconcile persists the normalized delivery mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
The post-commit webhook reconcile in edit_git_sync_repository and
edit_git_sync_config wrote the whole pre-reconcile git_sync snapshot back
after the main save committed. A concurrent git-sync edit or poller status
write that landed in the gap could then be dropped by the stale snapshot.
Re-read the current row and merge only the reconciled webhook id/secret/error
for the repos the reconcile actually changed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
Both write endpoints (edit_git_sync_repository, edit_git_sync_config)
persisted caller-supplied auto_pull.webhook_id / webhook_secret /
webhook_error / last_synced_sha / last_pull_status when adding a repo or
newly enabling auto-pull, letting a client inject a webhook id/secret or
fake sync state. Strip those server-owned fields from the request up front;
existing repos re-derive them from the DB (carried over), new ones start
clean and the server (re)creates the webhook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
The poller scanned workspace_settings directly, so an archived (soft-deleted)
or renamed-away workspace — whose settings row persists — kept polling and
could enqueue a pull into a dead workspace. Join workspace and require
NOT deleted. The EE webhook receiver gets the same filter (ee ref bumped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
- Export (P1): strip the server-owned auto_pull state (webhook secret/id/error
+ synced sha + last pull status) from git_sync before it is written into an
export's settings.json for both settings formats. The HMAC webhook secret
must never leave the server (matching the GET-settings redaction), and a
re-imported workspace must not inherit another install's hook/sync state.
- Docs: the webhook receiver is a single per-workspace endpoint
/api/w/{workspace}/github_app/webhook (host-aware for managed + self-managed);
update the stale push_webhook/{id} and instance-global /api/github_app/webhook
references.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
- edit_git_sync_config: also delete a repo's old webhook when the save drops
the repo OR clears its auto_pull. Webhook fields are only preserved onto a
Some auto_pull, so a save that present-but-clears a repo would otherwise
orphan its hook.
- GitSyncRepositoryCard: isFork now uses parent_workspace_id OR the wm-fork-
prefix (was AND), matching the backend/CLI rule, so prefix-less dev
workspaces are detected as forks and don't show the parent fork-PR toggle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
edit_git_sync_config preserved server-owned webhook fields but never
created or deleted the managed GitHub webhook, so enabling auto-pull
through the whole-config endpoint only polled, and disabling or removing
a repo left an orphan hook still delivering. Mirror the per-repository
endpoint: after the commit is durable, reconcile every saved repo's
webhook (sync_repo_webhook) and delete the hooks of repos the save
removed, including the clear-whole-config case. Addresses the Codex nit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
Removes the "Keep forks in sync with the tracked branch" toggle and its
fan-out. Pulling the tracked branch straight into every fork was the
inconsistent piece; the consistent model is per-fork branch sync (each
fork tracks its own wm-fork/** branch), which is a separate follow-up.
fork_open_prs is kept. Also tightens the fork toggle-section spacing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
migrate() and fix_flow_versioning_migration re-acquired a second connection from the pool while already holding one (the migrator's checked-out, advisory-locked connection). That deadlocks any backend limited to one connection at a time — connection-constrained managed Postgres, PgBouncer transaction pooling, or an embedded single-connection dev database. Route those housekeeping queries onto the already-held connection via a new CustomMigrator::connection() accessor. Fewer connections during migration and, for fix_flow_versioning, the existence check and write now run on the same advisory-locked connection. Default multi-connection behavior is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: add cosmetic dev/staging label for dev workspaces
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: prefill dev fork name and use a link to switch its label
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: reword the dev/staging label link copy
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: preview the dev/staging label as a badge in the switch link
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: show the dev/staging badge in the session diff drawer header
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): auto-derive cascade trigger edges from ducklake/s3 reads
Within a `// pipeline`, a read of a ducklake table or s3 object now
auto-wires its cascade trigger edge straight from the FROM clause, so
`// on <asset>` is only needed for edges inference can't see (dynamic SQL)
or to carry per-edge opts. Two opt-outs: `// mute <asset>` suppresses a
single derived edge (a lookup / SCD input read every run but not cascaded
on), and `// mute all` opts the script out of derivation entirely (back to
explicit-`// on`-only). Explicit `// on` still wins the dedup.
Scoped to ducklake + s3 reads; resource/datatable/volume stay explicit.
Read-write (RW) and write inputs are excluded so a self-referential
merge can't loop-trigger itself; ambiguous (None) access is skipped.
- parser: `mute` / `mute_all` in PipelineAnnotations (Rust + TS mirror)
- deploy: derive_pipeline_asset_trigger_refs → script_trigger rows
- frontend: resolveGraph mirrors derivation for the live edit-mode canvas
- tests: shared parity corpus + derive-helper units + resolveGraph overlays
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): mark auto-derived cascade edges with a persisted derived flag + "auto" badge
Persist script_trigger.derived (deploy: true for ducklake/s3-read derivation,
false for explicit // on) and return it from the asset-graph endpoint so the
canvas renders a Sparkles "auto" badge on auto-wired edges — the inference is
now visible on both the deployed graph and the live edit canvas, not just
implied. Dispatch (fetch_subscribers) ignores the flag, so a derived edge fires
identically to an explicit // on. Also copy derived in the workspace-clone
trigger copy, and backfill muteAssets/muteAll into two empty PipelineAnnotations
literals the base commit left stale (check:fast).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): derive cascade edge from effective (alt-fallback) asset access
derive_pipeline_asset_trigger_refs gated on the raw parser access_type, but the
persisted asset.usage_access_type and the frontend canvas both use
access_type.or(alt_access_type). An ambiguous parse with a manual read override
was persisted/drawn as a read yet derived no edge, so the auto edge silently
vanished on deploy. Gate on the effective access type for parity.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): badge muted reads instead of auto-derived edges
Auto-derivation is the default now, so badging every derived cascade edge is
noise. Drop the "auto" badge and the persisted `script_trigger.derived` flag
(migration + insert param + graph field + clone copy) that only powered it, and
instead badge the exception: a ducklake/s3 asset a script reads but does NOT
cascade — `// mute <asset>` / `// mute all`. `computeMutedReadKeys` marks a
read-only ('r') supported read with no cascade trigger and no self-write; the
canvas renders a bell-off "muted" badge on that read edge.
Also fixes two review parity nits:
- TS `// on` parser now strips trailing `key=value` opts (e.g. `debounce=60s`)
like the Rust `split_trailing_kv_opts`, so the ref dedups against inference.
- A `// materialize` producer reading its own target is upgraded to `rw`
(deploy) / excluded via the materialize write refs (canvas), so it neither
self-cascades nor shows as a muted read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): drop redundant // on for auto-derived reads; gate muted badge to pipeline scripts
- Templates no longer scaffold `// on <asset>` for a ducklake/s3 input the body
reads — the read auto-wires the cascade now that derivation is the default.
Kept for datatable/resource (not auto-derived) and native triggers. The
discoverability hint now mentions `// mute` (the newly relevant annotation).
- computeMutedReadKeys only badges reads by `// pipeline` scripts. A plain
script or flow reading a ducklake/s3 asset never had an auto trigger to
suppress, so it must render as ordinary lineage, not "muted" (Codex review).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): only drop template // on when the body actually reads the input
The redundant-`// on` removal assumed the generated body reads the ducklake/s3
input, but postgres/bash/generic bodies (and `data_upload`, which reads the
picker file) ignore `input` — dropping `// on` there left the asset-created
script with no cascade at all. Gate the drop on READS_INPUT_LANGS
(bun/deno/python/duckdb) so non-reading templates keep the explicit trigger.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(auth): add runtime NO_AUTH mode for authentication bypass
Adds a runtime `NO_AUTH` env flag that makes every request resolve as the
`admin@windmill.dev` superadmin with no login required, so self-hosted
deployments can front Windmill with their own authenticating gateway
without building a dedicated `oss` (compile-time `no_auth`) binary.
- `NO_AUTH` is honored in any build but is force-disabled when
`CLOUD_HOSTED` is set, so the managed cloud always enforces real auth.
- The existing compile-time `no_auth` feature keeps its always-on behavior
(`cfg!(feature = "no_auth") || *NO_AUTH`), so `oss` builds are unchanged.
- `Tokened` now yields a synthetic token in no-auth mode so handlers that
require it (e.g. global_whoami, called by the frontend on load) resolve.
- A loud startup banner warns when the mode is on; `HIDE_NO_AUTH_BANNER`
silences it once the operator has deliberately deployed behind a gateway.
Fixes WIN-2131
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(auth): dismissable NO_AUTH warning banner via global setting
Replaces the HIDE_NO_AUTH_BANNER env flag with a UI warning banner that
can be permanently dismissed for all users from within the running
instance (not exposed in instance settings).
- New `no_auth_banner_dismissed` global setting, only ever written by
dismissing the banner itself.
- `GET /api/settings/no_auth_banner` returns whether to show the banner
(true only when NO_AUTH is active and it hasn't been dismissed).
- NoAuthBanner.svelte renders a top-of-app warning in NO_AUTH mode; its
dismiss button opens a confirmation modal, then writes the global
setting via the existing setGlobal endpoint so it stays hidden for
everyone.
- The server still logs the startup NO_AUTH warning unconditionally.
Fixes WIN-2131
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(auth): resolve NO_AUTH in AuthCache so all_runnables works
Codex/Pi review flagged that `/api/users/all_runnables` still failed in
NO_AUTH mode: `get_all_runnables` extracts `Tokened` and re-validates the
request token per workspace via `AuthCache::get_authed`, which rejected the
fabricated `"no_auth"` token (no matching DB row) with a 400.
Short-circuit `AuthCache::get_opt_job_authed` (the resolver behind
`get_authed`) to the admin superadmin in no-auth mode, so any direct cache
caller resolves without a real token. Single-source the mode check and the
synthetic identity via `is_no_auth()` / `no_auth_admin_authed()` and reuse
them across the extractor, resolver, and login paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(auth): drop the NO_AUTH dismissable UI banner
The in-app banner added a GET /api/settings/no_auth_banner request to every
instance load for little benefit. The startup log warning already surfaces
that auth is bypassed to operators, so drop the banner, its endpoint, and the
no_auth_banner_dismissed global setting entirely.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codex nits:
- edit_git_sync_config saved the client config verbatim, so the webhook_secret
redacted from the GET response would be dropped (breaking delivery). Preserve
server-owned auto-pull state (webhook id/secret, synced sha, last status) per
repo from the existing settings, matching edit_git_sync_repository.
- addSyncRepository (the visible add path) didn't set the auto_pull default, so
new sync repos added from the UI came up with auto-deploy off. Match
addRepository's default (webhook + polling fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
Codex nit: edit_git_sync_repository ran sync_repo_webhook before the transaction
committed. The rollback only covered created hooks, but sync_repo_webhook also
deletes a hook on disable/switch-to-polling — a commit failure then left the DB
with a webhook_id whose hook was already gone (and it wouldn't be recreated).
Save + commit first, then reconcile the webhook against the durable config and
persist any hook id/secret change (best-effort). Bump EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
Codex re-review nits:
- delete_git_sync_repository deleted the webhook before the settings transaction
committed; a failed save would then leave the repo pointing at a hook that no
longer exists (sync_repo_webhook treats a set webhook_id as live and won't
recreate it). Capture the hook id, commit the DB removal, then delete the hook.
- Reword a fork-copy comment to drop drafting-history wording per AGENTS.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
- Deleting a git-sync repository now tears down its managed GitHub webhook
(deletion bypassed the sync_repo_webhook lifecycle, orphaning the hook so
GitHub kept delivering to the instance).
- Worker completion hook rolls back the optimistic auto-pull sha on job failure
(OSS side of the EE change) + caches the new marker query. Bump EE ref.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PP5gBSPfo1YtkL1sWVAjJm
* wip: partial work before earlyoom-recovery relaunch
* fix(pipelines): scaffold the strftime {partition} filter idiom (frontend-only)
The DuckDB materialize scaffold and the AI pipeline prompt now teach the
grain-agnostic `WHERE strftime(<ts_col>, '<fmt>') = {partition}` filter instead
of the naive `= TIMESTAMP {partition}` cast. `{partition}` substitutes to the
partition IDENTITY string (`2026-07-05T23`, `2026-W27`, `2026-07`), which is not
a valid DuckDB TIMESTAMP literal for any non-daily grain — so the naive form
raises a `Conversion Error` for hourly/weekly/monthly (only daily parses).
Adds a frontend unit test asserting the hourly scaffold emits the strftime
idiom (`%Y-%m-%dT%H`) for every grain and never scaffolds the naive TIMESTAMP
cast as executable SQL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): scope strftime partition idiom to time grains
Review nit: `dynamic` partitioning's identity is a caller-supplied key, not a
timestamp, so `strftime` doesn't apply. Scope the scaffold + AI prompt claim to
time grains and add a `dynamic` example that filters on the user's own key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): wm_partition macro for grain-agnostic partition filters
The materialize runtime now injects a `wm_partition(ts)` temp macro as the first
setup statement of a time-partitioned script, so filtering the source to the
active slice is one grain-agnostic line — `WHERE wm_partition(<ts_col>) =
{partition}` — instead of a hand-written `strftime` format the author must keep
in lockstep with the resolver, or the `= TIMESTAMP {partition}` cast that only
parses for daily and Conversion-Errors for hourly/weekly/monthly.
The macro's format comes from `PartitionKind::default_time_format` in
windmill-parser, the same source the EE resolver reads to stamp the `{partition}`
identity, so the two can't drift. `dynamic` partitions get no macro (their
identity is a caller-supplied key → `WHERE <key_col> = {partition}`).
Replaces the earlier 9-line strftime comment block in the scaffold with the
single macro line; AI pipeline prompt and design doc updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): verify wm_partition strftime parity vs chrono through real DuckDB
Runs the bundled DuckDB engine in-memory and asserts strftime renders every
grain format (daily/hourly/weekly `%G-W%V`/monthly) byte-for-byte identically to
chrono — the engine the resolver uses to stamp the `{partition}` identity —
across ISO-week year boundaries (2027-01-01 → 2026-W53 etc.). Also proves the
injected `wm_partition` macro buckets the whole slice and that the naive
`TIMESTAMP '<weekly|monthly identity>'` cast Conversion-Errors.
Closes the one cross-engine assumption the pure-Rust/frontend tests couldn't
reach (flagged by CI review for weekly ISO-week rendering).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 0de2412ff0734b11e12ba378c9bcc373ff9ae800
This commit updates the EE repository reference after PR #649 was merged in windmill-ee-private.
Previous ee-repo-ref: ad6c6685689d7741058e7d2c9ecbe95d982e6268
New ee-repo-ref: 0de2412ff0734b11e12ba378c9bcc373ff9ae800
Automated by sync-ee-ref workflow.
* fix(pipelines): classify CREATE TEMP MACRO as a DuckDB prepare-path setup statement
The FFI prepare/diagnostics pass only EXECUTES statements recognized by
is_setup_statement (ATTACH/USE/INSTALL/…); everything else is merely prepared.
`CREATE [OR REPLACE] TEMP MACRO` wasn't recognized, so on a `-- prepare` run of a
partitioned materialize the injected `wm_partition` macro was never created on
the connection, and the later generated `CREATE TABLE … SELECT … WHERE
wm_partition(...)` failed to bind ("function does not exist"). The same latent
gap affected the workspace-macro splicer, which injects TEMP MACRO blocks too.
Classify CREATE [OR REPLACE] TEMP|TEMPORARY MACRO as setup so it's executed
before dependent blocks and excluded from the PrepareQueryResult count
(persistent CREATE MACRO stays a user statement). Adds a prepare-path test that
fails without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
The `duckdb` bundled feature compiles the whole DuckDB C++ library from
source (~2min), which dominates the FFI crate's build. A fresh git
worktree had an empty target dir and paid that cost every time.
build_dev.sh now builds into a per-user cache shared across worktrees,
keyed by Cargo.lock + build.rs so distinct DuckDB versions don't collide.
Uncommitted changes to the crate source fall back to an isolated
per-worktree ./target so active FFI development neither disturbs nor is
disturbed by the shared cache. Add a crate .gitignore for /target, and
note the shared cache in the AGENTS.md / backend CLAUDE.md build steps.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: resolve extensionless bun relative imports on windows loader
* fix: resolve local module probe against importer dir not job root
* fix: keep node_modules-internal relative imports out of windmill resolver on windows
* fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect
The SDK object forms — TS `writeS3File({s3:"exports/x"})` and Python
`write_s3_file(S3Object(s3="exports/x"))` — resolve to the URI `s3:///exports/x`
(empty default storage), whose parsed asset path was `/exports/x` (leading
slash). DuckDB `read_csv('s3://exports/x')` and the `// on s3://exports/x`
trigger form yielded the bare `exports/x`. The same object thus produced two
asset identities, so a DuckDB consumer never connected to a TS/Python producer
in the pipeline graph.
`parse_asset_syntax` (shared by the native backend parsers and the wasm parser
that drives `frontend/src/lib/infer.ts` and the CLI `localGraph`) now strips a
single leading slash from S3 paths, so `s3:///key`, `s3://storage/key`, DuckDB
`s3://…`, and `// on` all canonicalize to one key. Both deploy-time inference
and editor/CLI inference agree, and the producer's write edge and the
consumer's read/trigger edge share a node.
Only one leading slash is stripped, so `s3:///` triple-slash default-storage
keys collapse to the bare key while Hive-partition keys
(`s3://bucket/y=2024/f.parquet`) and explicit-storage `s3://storage/key` paths
are untouched. Non-S3 asset kinds (res://, ducklake://, …) keep their paths
verbatim.
Note: existing deployed pipelines that recorded `/key` paths need a redeploy to
pick up the canonical `key`; the fix is forward-consistent for anything parsed
after this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): mark S3 asset-path normalization (item 6) resolved
The open-issues list still flagged the SDK-form leading-slash vs bare-URI
no-slash mismatch as "Still open", contradicting the fix in this PR. Mark it
resolved to match the updated Language-coverage prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs+test(pipelines): disclose S3 explicit-storage vs default-storage-nested-key aliasing
Collapsing to one canonical key means `s3://storage/key` (explicit storage) and
`s3:///storage/key` (default-storage nested key) now alias to the same node
`storage/key`, though they name different objects. Low-probability (needs a
storage config named to match a default-storage prefix) and inherent to a
best-effort lineage graph that doesn't split the first segment as a storage
name, but previously undisclosed. Document the tradeoff and pin the intended
aliasing with a test so it's intentional, not a latent surprise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): mirror S3 leading-slash strip in frontend live-preview parser
The pipeline graph live preview parses `// on` annotations client-side via the
hand-written `parsePipelineAnnotations.ts` (a TS mirror of the Rust annotation
scanner), NOT the wasm parser. Its `parseAssetSyntax` still returned the raw
suffix, so `// on s3:///exports/x` yielded `/exports/x` while the deploy-time
and wasm parsers now canonicalize to `exports/x`. `resolveGraph` synthesizes
trigger edges from that path, so the browser preview could still render
disconnected `/exports/x` and `exports/x` nodes for the exact triple-slash case
this PR fixes at deploy time.
Mirror the S3-only single-leading-slash strip in the TS parser and extend the
shared parity fixture corpus (run by both the Rust and TS parity suites) with
the triple-slash trigger case, so Rust/TS drift on this is now caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): seed slashless S3 template asset paths to match canonical key
`autoOutputAsset` seeded new S3 template outputs with a leading slash
(`/pipelines/…`), which the old parser required to match `s3:///key` writes.
This PR made `parse_asset_syntax` strip that slash, so the seeded draft asset
(stored as `outputAssets`, used by `resolveGraph` for inactive-draft node
identity) no longer matched the body-inferred identity `pipelines/…` — the live
preview could render a duplicate `/pipelines/…` node and a phantom post-deploy
drift warning.
Seed the canonical slashless key instead, and switch the DuckDB body's S3 URIs
from `s3://${path}` to `s3:///${path}` so the generated runtime URI stays the
triple-slash default-storage form byte-for-byte (the SDK sites already build
`s3:///` + bare key). Add a pure-logic parity test asserting, for every
language and S3 output kind, that the seeded asset path is slashless and that
every S3 URI the generated body emits is triple-slash and canonicalizes back to
that seeded path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): canonicalize S3 keys in CLI + frontend bounded-cascade resolvers
Two more hand-written S3-URI sites returned the raw suffix, so `s3:///exports/x`
stayed `/exports/x` while native/wasm parsers now canonicalize to `exports/x`:
- `cli/src/commands/pipeline/localGraph.ts` — the no-wasm fallback `// on`
scanner (go/bash/ruby). A fallback consumer's `// on s3:///x` would not
connect to a wasm-inferred `x` producer in `wmill pipeline show/run --local`.
- `boundedCascade.ts` `assetUriToNodeId` (duplicated in the CLI and the frontend
AssetGraph engines, kept in sync) — `--to s3:///exports/x` / a cascade bound
token would not resolve against the canonical graph node `s3object:exports/x`.
`resolveToken` delegates here, so it is covered too.
Mirror the S3-only single-leading-slash strip in all three, and add `s3:///`
tests to the CLI local-graph fallback suite and both bounded-cascade suites
(explicit-storage and Hive-partition keys asserted untouched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): phrase S3 template test comment as a current invariant
Describe the slashless-seed requirement as the invariant it is, not as change
history, per the AGENTS.md "describe the code as it is" rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): strip all leading slashes from S3 keys so trigger refs round-trip
`parse_asset_syntax` stripped only one leading slash, so `S3Object(s3="/x")` —
which resolves to the quad-slash URI `s3:////x` — parsed to path `/x`. But
`trigger_spec_to_row` rebuilds a stored trigger ref as `s3://<path>` =
`s3:///x`, which `parse_asset_trigger_ref` then parses back to `x`. The
producer recorded `/x` while its consumer trigger resolved to `x` → a broken
edge. The same asymmetry affects every `s3://`+path reconstruction site
(backend refs, frontend `assetUri`, page refs) whenever a path starts with `/`.
Strip ALL leading slashes so a canonical S3 path never starts with `/`; naive
`prefix + path` reconstruction then round-trips everywhere. Applied uniformly
across all six S3-URI sites (Rust `parse_asset_syntax`, the TS live-preview
parser, template `s3Key`, and the frontend+CLI `assetUriToNodeId` and CLI
fallback scanner). The pathological leading-slash key collapses to the bare key
— acceptable for a best-effort lineage graph that never split storage anyway.
Tests: a windmill-common round-trip test (parse → trigger_spec_to_row →
parse_asset_trigger_ref) over every URI form incl. the quad-slash case; a
`s3:////x` shared parity fixture (Rust + TS); and quad-slash assertions in the
Rust parser test and both bounded-cascade suites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): align S3 template parity helper with strip-all canonicalization
The template seed/body parity test's `canonicalS3Key` helper (and its comment)
still stripped a single leading slash, so it no longer mirrored the parser it
claims to pin. Strip all leading slashes to match `parse_asset_syntax` and the
frontend/CLI mirrors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`ATTACH 'datatable://main'` (or any datatable schema/executor path) failed
with a bare "datatable main not found", giving the user no way forward — the
datatable substrate has no auto-provisioning like a DuckLake catalog, so the
fix is always to create one in workspace settings, but nothing said so.
`get_datatable_resource_from_db_unchecked` now returns a NotFound error that
lists the workspace's configured data tables (to catch typos) and points at
the "Data tables" settings tab, noting `main` is the default name used by
`datatable://main`. The message bubbles up wherever the resolver is called
(pipeline ATTACH, schema fetch, postgres executor, agent HTTP endpoint).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipelines): link SCD2 <dim>_current view to its producer across all graph surfaces
An SCD2 producer (`// materialize … history`) creates the base table AND a
`<dim>_current` view at runtime. The deploy path already registered both writes,
but the CLI `--local` graph and the frontend live-editor graph only emitted the
base write, so a consumer reading only `<dim>_current` orphaned there. Centralize
the companion derivation in `MaterializeSpec::write_targets` /
`scd2_current_target` (+ TS `scd2CurrentTargetPath` mirror), emit the `_current`
write in every surface, and mark the companion node `derived_from` the base so the
canvas renders it as a derived "current view" instead of an unrelated table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): keep scd2 _current write edge when editing a saved producer
Addresses Codex CI review (P1): opening a deployed scd2 materialize producer for
editing dropped its persisted `<dim>_current` write edge. `liveRefKeys` (the set
of asset keys a saved-script edit preserves against stale-filtering) only added
the base materialize target, so the companion `_current` write was judged stale
and filtered — orphaning consumers of only the view mid-edit. Add
`scd2CurrentTargetPath(m)` to `liveRefKeys` too; covered by a new saved-edit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): order data_test relationships refs before the tested script in a cascade
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): key custom-test reads by (usage_kind, path) to avoid same-path flow collisions
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Custom `// data_test <path>` scripts must be a single SELECT reading the
freshly-materialized target via the internal `_wm_target.<table>` alias —
neither was documented or scaffolded. Make the codegen errors name the exact
violation (multi-statement, non-SELECT, wrong alias, empty) and append a
copyable `SELECT * FROM _wm_target.<table> WHERE <condition>` example. Add a
DuckDB-only 'Data test' pipeline output kind that scaffolds that starter body.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two correctness/validation improvements to managed materialization:
1. A keyed `merge` (`key=<col>`) is delete-by-key + insert-all and does NOT
deduplicate its source, so two incoming rows sharing a key both landed
under that key — silently breaking the one-row-per-key contract. Codegen
now emits an in-transaction guard (same `error(...)` shape as the schema
-drift guard) that fails the run when the SELECT returns more than one row
for a non-NULL key, naming the key. Authors deduplicate in the SELECT or
switch to `append`. NULL keys are exempt, matching the delete's `IN (...)`
scope.
2. The two SCD2 misconfigurations that were only caught at run time — `history`
without `key=`, and `history` + `// partitioned` — now fail fast at deploy
via a shared `MaterializeSpec::validate`, called from `create_script_internal`.
The DuckDB executor keeps the same check as a safety net for preview/test
runs that never deploy (shared message, no drift).
Adds unit tests for the merge guard codegen and for `validate` (all four
cases), and updates docs/ducklake-materialization.md and docs/pipelines-vs-dbt.md.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(object-storage): remove 20-file bucket-browser cap in CE
The Community Edition build rejected the object-storage `list_stored_files`
endpoint with an error once a workspace bucket held more than 20 objects,
making the bucket browser unusable on larger buckets. The listing already
collects up to `max_keys` objects, so the hard cap was purely a gate.
Drops the CE listing cap (in the EE-symlinked `job_helpers_ee.rs`, tracked
in the companion windmill-ee-private PR) and removes the now-inaccurate
sentence from the workspace object-storage settings banner. The 10 GiB
total-storage write quota remains as the intentional CE limit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 4bd7b73dcef1b77dc2866bc58e0c86962559495c
This commit updates the EE repository reference after PR #648 was merged in windmill-ee-private.
Previous ee-repo-ref: aa14d0724216030948c2f575bcc19c0e6e0476a7
New ee-repo-ref: 4bd7b73dcef1b77dc2866bc58e0c86962559495c
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(duckdb): render FFI errors with real newlines and no stray quoting
The DuckDB FFI returns errors as ERROR <json-encoded-message>, so the
executor was surfacing the serde_json-escaped form (wrapping quotes,
literal \\n). Multi-line errors like the write-audit-publish data-test
breakdown were unreadable. Decode the JSON string back to the raw message
at both FFI error sites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: tighten decode_ffi_error comment to the invariant
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): on_schema_change write guardrails + data_test deploy validation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to fa7ac11c1e0ab39e84a0c18973ba427a240933ca
This commit updates the EE repository reference after PR #647 was merged in windmill-ee-private.
Previous ee-repo-ref: bd23b2a904cb2e6554c7ff209ff8adb9d91775d1
New ee-repo-ref: fa7ac11c1e0ab39e84a0c18973ba427a240933ca
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): per-lake isolated/shared choice at fork creation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): chain-aware defer discovery + per-location fork namespace registry
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): cleanup deletes fork data from the registered storage identity
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): collapse fork data-path segment to one component (slash-safe ids)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): run fork ducklake cleanup inline in delete_workspace
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter
- extract require_prod_admin_for_dev_workspace, used by both delete_workspace
and drop_forked_ducklake_namespaces so the gates cannot drift
- key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in
cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL
re-registers its namespaces
- filter listed object locations to the segment boundary before deletion
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated
parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its
parent with an empty ancestor chain while its cloned config still points
at the shared lake. Key the isolation gate on the wm-fork- prefix as well
as the chain (mirroring workspace_is_fork): orphaned forks get the write
redirect, registration and cleanup with zero ancestors (no defer), and
keep their 'fork' graph chips.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace
Chain position alone classified the last ancestor as a root, but an
orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the
chain the same way while its data lives in its fork namespace — its
descendants' defer views bound the dead root's lake instead. Key the
root-vs-fork decision on the wm-fork- prefix too, matching the
resolution gate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions
- fork creation strips cloned fork_behavior stamps before applying the
request's shared_ducklakes list: sharing is a per-creation choice, a
fork of a shared fork defaults back to isolated
- fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the
durable cleanup ledger and outlive the workspace when physical cleanup
fails post-commit; fork creation retries leftover rows for the reused
id and refuses to create while a metadata schema still cannot be
dropped (data-file leftovers alone are inert once the schema is gone
and are swept by the next successful same-prefix cleanup)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources
- ledger rows gain a schema_dropped phase flag: set when the schema drop
succeeded but data cleanup failed, so later retries skip the schema
phase and need no catalog credentials at all; registration resets it
on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the
schema
- retry-path $res: resolution falls back to the workspace being forked
(the deleted fork's resources were clones of a parent's); live paths
(delete_workspace prepare, drop endpoint) pass no fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph
A failed materialization must not disguise a physically existing fork
table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the
table, so reads hit fork data while the graph claims parent defer.
- record_mat upsert preserves the last committed snapshot_id on failure
- defer discovery and graph chips treat fork rows with a committed
snapshot as fork-owned even when status is failed
- inspect_fork_catalog also lists live fork tables (same round trip) and
the defer list is filtered against them — covers rows recorded before
this fix and tables created by raw SQL
- drop stale FK-cascade wording in the design doc and sidebar comment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip
The workspace-settings ducklake editor had no fork awareness: no
reminder of each lake's isolated/shared choice and no warning about
what edits mean in a fork. It also rebuilt each lake explicitly on
save, silently dropping fork_behavior — any settings save in a shared
fork flipped the lake back to isolated.
- fork detection mirrors the backend gate (parent link or wm-fork- prefix)
- info banner explaining isolated vs shared semantics in a fork
- per-lake chip (emerald 'isolated' / amber 'shared with parent') with
tooltips, matching the pipeline graph chip colors
- fork_behavior added to DucklakeSettingsType and preserved through
convertDucklakeSettingsToBackend
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): capture violating-row samples for data tests
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): byte-accurate sample cap and leaf-level payload sanitize
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump ee-repo-ref to WAP guard probe adaptation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: WAP failures are counts-only — samples exist only on commit-then-test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: qualify where sample row data appears — job result and failed-job log line
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: error handlers receive the full result incl. samples, like any failed job
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 80d309edebb899e36a3bdcdf4ea73c4db070534d
This commit updates the EE repository reference after PR #646 was merged in windmill-ee-private.
Previous ee-repo-ref: 16e916bf11f26381920560b55771fce693e668c6
New ee-repo-ref: 80d309edebb899e36a3bdcdf4ea73c4db070534d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(pipelines): ingestion (EL) templates + docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): review nits — draft collision guard, template-mode selection reset, invariant test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): lead the insert menu with ingestion templates
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(pipelines): ingestion story as docs-only — drop editor template UI
The insert-menu template section mixed two selection grammars in one popover and confused more than it helped. The three E2E-verified example pipelines now live verbatim in docs/pipeline-ingestion.md; the Python bare-string S3 key fix in pipelineTemplates.ts stays.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(sdk): bare string S3 keys in py/ts clients + asset parsers
A plain string passed where an S3Object is expected is now a bare key in the default storage — previously the py client silently degraded it to s3="" (auto-generated key) and both asset parsers canonicalized it without the leading slash, splitting lineage. parseS3Object moves to s3Types.ts so it is unit-testable without the generated services. The pipeline template fix from the earlier commit is superseded (bare strings are the supported spelling again); docs examples flipped to bare keys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(sdk): enforce s3:// URIs for string S3Object params
Bare strings now raise/throw with a hint pointing at the s3:///<key> spelling instead of being treated as keys (previous commit) or silently degrading to an empty key (original behavior). One string spelling everywhere: SDK calls, // on annotations, and DuckDB SQL all use s3:///<key>. TS regains the s3://-template-literal type; the asset parsers record no asset for a bare string (the call can only error); templates emit the URI form.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(pipelines): move ingestion (EL) guide to windmilldocs, keep design constraints
User-facing how-to (engine choice, cursor recipes, schema drift, worked examples) moves to windmilldocs core_concepts/63_pipelines (windmilldocs#1462); the repo keeps only the design constraints future feature work must not break, as a section of ducklake-materialization.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: regenerate system prompts after parse_s3_object docstring change
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(sdk): reject empty-key s3 URIs; align asset parsers with the runtime rule
Addresses CI review: s3:/// and s3://bucket/ now raise (an empty key would fall back to the auto-generated-key path the strict contract exists to prevent); the asset parsers' string branch applies the same valid-URI-with-non-empty-key rule so no R/W edge is recorded for a call that can only error (the generic URI-literal scan still records ambiguous access-None assets, by design); comments rephrased as current constraints per AGENTS.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(ducklake): scheduled lake maintenance (snapshot expiry, compaction, orphan cleanup)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ducklake): review fixes — starts_with not LIKE, CE license-lapse escape
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(ducklake): auth-contract docs + _unchecked rename per codex review
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(ducklake): move maintenance payload construction into EE module
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ducklake): fall through to script resolution for non-managed reserved-prefix schedules
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(ducklake): document accepted pre-existing-schedule limitation on the reserved prefix
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ducklake): CE save-off clears the managed schedule row and queued occurrence
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 2fab310d4f50ed7c34857d69c9b854f4491bf217
This commit updates the EE repository reference after PR #645 was merged in windmill-ee-private.
Previous ee-repo-ref: fff1fd830a36beba732486f05941ec243cf6b640
New ee-repo-ref: 2fab310d4f50ed7c34857d69c9b854f4491bf217
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(pipeline): write-audit-publish for materialization data tests (EE)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: EE worktree E0583 troubleshooting + duckdb feature check row
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: clarify EE symlink example (absolute target, EE repo layout)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipeline): move bootstrap DDL inside guarded WAP transaction
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(pipeline): move WAP guard SQL builder into EE, OSS keeps placement only
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump ee-repo-ref to EE branch rebased on EE main
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: reword test comment as current invariant per AGENTS.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: bump ee-repo-ref (EE module doc update)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(pipeline): OSS emits typed materialize plan, EE owns WAP transform
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test: make rewrite assertion build-aware; refresh oss module doc
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c
This commit updates the EE repository reference after PR #644 was merged in windmill-ee-private.
Previous ee-repo-ref: 63cabae75329429f647e01083936d70f8197dc9e
New ee-repo-ref: 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(ai-agent): support reasoning effort in AI agent workflow steps
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): round-trip native Anthropic thinking blocks and fix DeepSeek/Mistral reasoning
Address review: native Anthropic now captures the signed thinking block during streaming and replays it before tool_use across iterations (prevents a 400 on multi-turn tool use). DeepSeek 'off' sends thinking:{type:disabled} instead of the rejected reasoning_effort:none, and Mistral drops temperature when reasoning is on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-agent): move reasoning effort into the provider/model selector
Store reasoning_effort on ProviderConfig (next to the model) instead of a separate flow arg, and render the selector inside AIProviderPicker under the model dropdown. Add an explicit 'off' option on models that disable reasoning by omission (e.g. Claude), so reasoning can always be turned off from the UI.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-agent): use DropdownV2 for reasoning effort, matching copilot chat
Replace the Select combobox with the same DropdownV2 action-menu the copilot chat reasoning selector uses. Each option carries an action instead of a bound value, so click selection is unambiguous and there is no typeahead/sentinel-value mismatch on the off/default entries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(ai-agent): regenerate system prompts for ProviderConfig.reasoning_effort
Refresh system_prompts/auto-generated and cli skills.gen after adding reasoning_effort to the OpenFlow ProviderConfig schema (check-freshness).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): clear stale reasoning effort on model change; dedup bedrock reasoning folding
Address cubic review: (P1) the reasoning picker now clears the stored effort when the newly selected model doesn't accept it (e.g. carrying 'xhigh' from Opus onto a model that tops out at 'high'), not only when the model can't reason at all. (P3) the proxy's accumulate_reasoning_delta now delegates to the shared bedrock_stream_event_to_reasoning_delta so worker and proxy folding can't drift.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-agent): stream reasoning summary and show a thinking affordance in flow chat
Add StreamingEvent::ReasoningTokenDelta, emitted from every worker reasoning path (Anthropic native thinking deltas, Bedrock, Gemini thought parts, OpenAI-compatible reasoning_content, OpenAI Responses reasoning_summary_text with summary:auto). The flow chat parses it and renders a collapsible 'Thinking' affordance on the assistant message (thinking tokens bill regardless of display, so surfacing the summary is billing-neutral).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): persist streamed reasoning onto the finished chat message
Reasoning isn't stored server-side, so the completion re-poll (which swaps temp messages for the persisted ones) was dropping the streamed thinking summary. Carry it onto the final assistant message so the 'Thought process' affordance survives the run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(ai-agent): match flow-chat thinking box to the copilot chat reasoning UI
Replace the plain <details> thinking affordance with the same bordered, collapsible reasoning box the copilot chat uses (chevron + Brain/spinner + 'Thinking' header, markdown body, expand-while-streaming/collapse-on-answer).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): attribute streamed reasoning per turn by content; drop duplicated comment
Address review: the completion-poll carry-over now matches each temp assistant turn's thinking summary to its persisted message by content, so a multi-turn response (reasoning -> tool call -> final answer) no longer misattributes an earlier turn's thinking to the final answer or drops intermediate turns. Also removes a leftover duplicated comment block above the AIReasoningEffortPicker effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): address review round 2 (carry-over edges, off-token validity, aria, test)
cubic round 2: (1) reasoning carry-over now consumes temp turns in order verifying content, so identical/empty-content multi-turn responses attribute thinking correctly and reasoning-only turns aren't dropped; (2) the picker's stale-value check only accepts the off token when the model can actually disable reasoning; (3) add aria-expanded to the Thinking toggle; (4) add a test for the failed tool_result path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): add bottom margin below the flow-chat thinking box
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): don't request OpenAI reasoning summary, matching the copilot chat
OpenAI gates reasoning summaries behind org verification, so requesting summary: auto would 400 for unverified orgs. The copilot chat requests effort only and never asks for a summary; align the worker with it (reasoning: { effort }) and drop the now-unreachable summary-delta parsing. OpenAI reasoning no longer streams a summary in flow chat (consistent with the copilot); Anthropic/Bedrock/Gemini/DeepSeek reasoning display is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): scope reasoning carry-over to newly persisted messages
cubic round 3: matching by content over the full history could attach a new turn's reasoning to an older message with identical text. Restrict eligible targets to the messages just fetched for this response (via afterSeq), so historical turns are never touched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): carry reasoning newest-first instead of gating on the final poll
cubic round 4: gating the carry-over on the final poll's filteredResponse dropped reasoning for messages already fetched by an earlier streaming poll (their id is excluded by afterSeq). Walk persisted newest-first and consume the newest matching pending summary, stopping once summaries run out. This response's turns are always at the end, so they claim their own reasoning (P1) before older history is reached (P2), regardless of which poll persisted them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(ai-agent): drop flow-chat reasoning display, keep backend + effort picker
The chat-side thinking box relied on non-deterministically matching streamed (ephemeral) reasoning back onto persisted messages, which kept spawning edge cases. Remove the flow-chat display entirely (ChatMessage box, FlowChatManager carry-over/threading, parseStreamDeltas reasoning) and keep the sound backend: per-provider reasoning-effort requests, thinking-block round-trips for tool calls, and ReasoningTokenDelta streaming. A display can be built on top later, deterministically (e.g. once the stream carries the persisted message id).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ai-agent): include reasoning_effort in default-config compare; document reasoning_token_delta
Codex/Pi nits: isSameAsStoredConfig now compares reasoning_effort so the 'use as personal default' toggle reflects effort-only changes; openflow streaming-events doc lists the reasoning_token_delta event (regenerated auto prompts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): passive asset freshness tracking on the graph
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore(pipelines): drop dead freshness-enforcement stub, document query ordering
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(pipelines): freshness watchdog (EE) — auto re-run stale producers
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666
This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private.
Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459
New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(pipelines): record upstream snapshot ids on cascade-dispatched jobs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: batch upstream-snapshot lookup and memoize per subscriber
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
hugocasaandClaude Fable 5
Ruben Fiszelandrubenfiszel
windmill-internal-app[bot]andhugocasa