mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
ruben/win-2151-wdyt-of-httpsgithubcomwindmill
13798 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
f2a18beca4 |
fix(security): remove git from Deno sandbox allow-run (GHSA-gj6h-vw66-mr8f)
The `// sandbox` annotation restricts Deno to `--allow-run=git,/usr/bin/chromium`. git can be coerced into spawning `/bin/sh` via hook configs such as `git -c core.fsmonitor=<cmd> status`, and that subprocess is spawned by git — not Deno — so it is invisible to Deno's permission model. This let any user with script-execution permission run arbitrary OS commands as root inside the worker, fully defeating the sandbox. The advisory's alternative (injecting `-c core.fsmonitor=false -c core.hooksPath=/dev/null`) does not apply here: the user's own script invokes git directly via `Deno.Command`, so Windmill cannot inject hardening flags into that call. Removing git from the allowlist is the only complete fix. git was originally allowed for git-sync-adjacent use, which no longer needs it. Verified with the advisory PoC: git invocation now returns `Requires run access to "git"` and the sandbox escape is closed. chromium (puppeteer) support is preserved. Fixes WIN-2151 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
8dd5e48a68 |
chore(main): release 1.751.0 (#9965)
* chore(main): release 1.751.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>v1.751.0 |
||
 Guilhem
|
f3da86512a |
theme-aware prose palette for markdown in dark mode (#9971)
* fix(frontend): theme-aware prose palette for markdown in dark mode * chore(frontend): add markdown example to kitchen_sink showcase |
||
|
Ruben Fiszel
|
97d14d979f |
bun bootstrap housekeeping on the migrator's held connection (#9970)
migrate() and fix_flow_versioning_migration re-acquired a second connection from the pool while already holding one (the migrator's checked-out, advisory-locked connection). That deadlocks any backend limited to one connection at a time — connection-constrained managed Postgres, PgBouncer transaction pooling, or an embedded single-connection dev database. Route those housekeeping queries onto the already-held connection via a new CustomMigrator::connection() accessor. Fewer connections during migration and, for fix_flow_versioning, the existence check and write now run on the same advisory-locked connection. Default multi-connection behavior is unchanged. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Guilhem
|
2e14302e4a |
feat(frontend): custom skills — detail modal, batch manage, shared validation (#9847)
* feat(frontend): simplify custom skills workspace settings UI Collapse the "Custom skills" AI settings section into a single block. When no skills exist, show two side-by-side zones: a drag-and-drop folder dropzone (reusing FileInput) and a paste textarea whose add button appears only once content is entered. When skills exist, an "+ Add skills" dropdown offers "Import a folder of skills" (native picker) and "Paste a skill" (modal), above the skills list. Folder ingestion is shared by both the picker and the dropzone via processFolderFiles. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): custom skills detail modal + shared zod validation Rework the Custom skills settings: header Add-skills dropdown, per-row ellipsis menu (edit/delete), a Show more detail modal with a view/edit toggle (rendered markdown in read mode), accent Save gated by dirty detection and inline validation, and a folder-import conflict modal with per-skill overwrite toggles. Extract skill parsing/validation into a shared Zod-backed aiSkills module used by both the modal and the importer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): use Button for Show more; surface empty-body validation Address review: swap the raw <button> Show-more affordance for the design-system Button (per frontend component standards), and render the Save/inline-error block whenever editing an existing skill so clearing the body surfaces "body is required" instead of hiding both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(frontend): unit-test aiSkills; use themed border token Add aiSkills.test.ts covering parseSkillMd (BOM, CRLF, malformed YAML), validateSkill (code-point vs byte limits, name pattern), parseAndValidateSkill (nameOverride precedence) and buildSkillMd round-trip. Replace the hardcoded gray borders with the themed border-border-light token. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): cap custom skills list height and scroll Constrain the skills list to max-h-96 with overflow-y-auto so a large number of skills scrolls within the section instead of pushing the page. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): manage-mode batch delete for custom skills Add a "Manage skills" button (shown only with more than one skill) that enters a multi-select mode: a checkbox per row plus a sticky select-all (tri-state) header, and a batch Delete gated on the selection with a confirmation. Manage mode auto-exits when the list drops to one skill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): destructive delete, subtle manage button, Esc exits manage mode Batch Delete uses the destructive accent variant, Manage skills uses the subtle variant, and Escape leaves manage mode (mirroring Done) unless a modal or menu is open. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): add neutral 'info' type to ConfirmationModal; use for skill import ConfirmationModal only had 'danger' and 'reload' semantics, so a constructive confirmation like importing skills defaulted to danger (red warning + destructive button). Add a neutral 'info' type (blue Info icon, non-destructive accent confirm) and use it for the Import skills modal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 hugocasa
|
fd8e64d11f |
feat: add cosmetic dev/staging label for dev workspaces (#9959)
* feat: add cosmetic dev/staging label for dev workspaces Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: prefill dev fork name and use a link to switch its label Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: reword the dev/staging label link copy Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: preview the dev/staging label as a badge in the switch link Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: show the dev/staging badge in the session diff drawer header Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Guilhem
|
6587019d26 |
fix: critical alerts modal mute toggles no longer close popover or fail to save (#9969)
* fix: mute toggles in critical alerts modal no longer close popover or fail to save Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: mark popover content root as dropdown-portal so padding clicks don't close modal Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: derive no-channels warning from mute state so it survives modal reopen Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Guilhem
|
45946d1185 |
fix(assets): responsive layout for small screens (#9961)
* fix(assets): handle card/header overflow on small screens Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(assets): keep filter row label on one line with min spacing from refresh Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(assets): wrap card header actions below title instead of collapsing docs to icon Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(assets): widen card basis to 340px so cards wrap sooner and header stays one row Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Guilhem
|
9821596251 | fix(frontend): theme-aware code block background in prose markdown (#9968) | ||
|
Ruben Fiszel
|
3dcd3949a1 |
feat(pipelines): auto-derive cascade edges from ducklake/s3 reads (+ muted-read badge) (#9963)
* feat(pipelines): auto-derive cascade trigger edges from ducklake/s3 reads Within a `// pipeline`, a read of a ducklake table or s3 object now auto-wires its cascade trigger edge straight from the FROM clause, so `// on <asset>` is only needed for edges inference can't see (dynamic SQL) or to carry per-edge opts. Two opt-outs: `// mute <asset>` suppresses a single derived edge (a lookup / SCD input read every run but not cascaded on), and `// mute all` opts the script out of derivation entirely (back to explicit-`// on`-only). Explicit `// on` still wins the dedup. Scoped to ducklake + s3 reads; resource/datatable/volume stay explicit. Read-write (RW) and write inputs are excluded so a self-referential merge can't loop-trigger itself; ambiguous (None) access is skipped. - parser: `mute` / `mute_all` in PipelineAnnotations (Rust + TS mirror) - deploy: derive_pipeline_asset_trigger_refs → script_trigger rows - frontend: resolveGraph mirrors derivation for the live edit-mode canvas - tests: shared parity corpus + derive-helper units + resolveGraph overlays Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): mark auto-derived cascade edges with a persisted derived flag + "auto" badge Persist script_trigger.derived (deploy: true for ducklake/s3-read derivation, false for explicit // on) and return it from the asset-graph endpoint so the canvas renders a Sparkles "auto" badge on auto-wired edges — the inference is now visible on both the deployed graph and the live edit canvas, not just implied. Dispatch (fetch_subscribers) ignores the flag, so a derived edge fires identically to an explicit // on. Also copy derived in the workspace-clone trigger copy, and backfill muteAssets/muteAll into two empty PipelineAnnotations literals the base commit left stale (check:fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): derive cascade edge from effective (alt-fallback) asset access derive_pipeline_asset_trigger_refs gated on the raw parser access_type, but the persisted asset.usage_access_type and the frontend canvas both use access_type.or(alt_access_type). An ambiguous parse with a manual read override was persisted/drawn as a read yet derived no edge, so the auto edge silently vanished on deploy. Gate on the effective access type for parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): badge muted reads instead of auto-derived edges Auto-derivation is the default now, so badging every derived cascade edge is noise. Drop the "auto" badge and the persisted `script_trigger.derived` flag (migration + insert param + graph field + clone copy) that only powered it, and instead badge the exception: a ducklake/s3 asset a script reads but does NOT cascade — `// mute <asset>` / `// mute all`. `computeMutedReadKeys` marks a read-only ('r') supported read with no cascade trigger and no self-write; the canvas renders a bell-off "muted" badge on that read edge. Also fixes two review parity nits: - TS `// on` parser now strips trailing `key=value` opts (e.g. `debounce=60s`) like the Rust `split_trailing_kv_opts`, so the ref dedups against inference. - A `// materialize` producer reading its own target is upgraded to `rw` (deploy) / excluded via the materialize write refs (canvas), so it neither self-cascades nor shows as a muted read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): drop redundant // on for auto-derived reads; gate muted badge to pipeline scripts - Templates no longer scaffold `// on <asset>` for a ducklake/s3 input the body reads — the read auto-wires the cascade now that derivation is the default. Kept for datatable/resource (not auto-derived) and native triggers. The discoverability hint now mentions `// mute` (the newly relevant annotation). - computeMutedReadKeys only badges reads by `// pipeline` scripts. A plain script or flow reading a ducklake/s3 asset never had an auto trigger to suppress, so it must render as ordinary lineage, not "muted" (Codex review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): only drop template // on when the body actually reads the input The redundant-`// on` removal assumed the generated body reads the ducklake/s3 input, but postgres/bash/generic bodies (and `data_upload`, which reads the picker file) ignore `input` — dropping `// on` there left the asset-created script with no cascade at all. Gate the drop on READS_INPUT_LANGS (bun/deno/python/duckdb) so non-reading templates keep the explicit trigger. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
91e1b087a2 |
feat(auth): add runtime NO_AUTH mode for authentication bypass (#9962)
* feat(auth): add runtime NO_AUTH mode for authentication bypass Adds a runtime `NO_AUTH` env flag that makes every request resolve as the `admin@windmill.dev` superadmin with no login required, so self-hosted deployments can front Windmill with their own authenticating gateway without building a dedicated `oss` (compile-time `no_auth`) binary. - `NO_AUTH` is honored in any build but is force-disabled when `CLOUD_HOSTED` is set, so the managed cloud always enforces real auth. - The existing compile-time `no_auth` feature keeps its always-on behavior (`cfg!(feature = "no_auth") || *NO_AUTH`), so `oss` builds are unchanged. - `Tokened` now yields a synthetic token in no-auth mode so handlers that require it (e.g. global_whoami, called by the frontend on load) resolve. - A loud startup banner warns when the mode is on; `HIDE_NO_AUTH_BANNER` silences it once the operator has deliberately deployed behind a gateway. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(auth): dismissable NO_AUTH warning banner via global setting Replaces the HIDE_NO_AUTH_BANNER env flag with a UI warning banner that can be permanently dismissed for all users from within the running instance (not exposed in instance settings). - New `no_auth_banner_dismissed` global setting, only ever written by dismissing the banner itself. - `GET /api/settings/no_auth_banner` returns whether to show the banner (true only when NO_AUTH is active and it hasn't been dismissed). - NoAuthBanner.svelte renders a top-of-app warning in NO_AUTH mode; its dismiss button opens a confirmation modal, then writes the global setting via the existing setGlobal endpoint so it stays hidden for everyone. - The server still logs the startup NO_AUTH warning unconditionally. Fixes WIN-2131 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(auth): resolve NO_AUTH in AuthCache so all_runnables works Codex/Pi review flagged that `/api/users/all_runnables` still failed in NO_AUTH mode: `get_all_runnables` extracts `Tokened` and re-validates the request token per workspace via `AuthCache::get_authed`, which rejected the fabricated `"no_auth"` token (no matching DB row) with a 400. Short-circuit `AuthCache::get_opt_job_authed` (the resolver behind `get_authed`) to the admin superadmin in no-auth mode, so any direct cache caller resolves without a real token. Single-source the mode check and the synthetic identity via `is_no_auth()` / `no_auth_admin_authed()` and reuse them across the extractor, resolver, and login paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * revert(auth): drop the NO_AUTH dismissable UI banner The in-app banner added a GET /api/settings/no_auth_banner request to every instance load for little benefit. The startup log warning already surfaces that auth is bypassed to operators, so drop the banner, its endpoint, and the no_auth_banner_dismissed global setting entirely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
cc2f638de6 |
fix(ai): centralize Anthropic Messages API routing across completion paths (#9960)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
dc6b99775b |
fix(cli): quote non-identifier property names in resource-type namespace (#9964)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5fe7e1f3e8 |
chore(main): release 1.750.0 (#9952)
* chore(main): release 1.750.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>v1.750.0 |
||
|
Guilhem
|
056ebdb035 |
fix: read chat drafts via own-draft route so drawer-kind drafts deploy (#9913)
* fix: read chat drafts via own-draft route so drawer-kind drafts deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover trigger and resource chat-draft read/deploy regressions Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover non-secret variable chat-draft read/deploy regression Completes the drawer-kind matrix from the review notes on #9913: schedule, trigger, and resource already had full write→read→deploy regressions; this adds the variable one (non-secret — the secret flow deploys through the ephemeral in-memory value and is pinned by the existing ephemeral tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ai_evals): mock getOwnDraft so eval draft hydration stays in-memory The frontend eval adapter intercepts DraftService for benchmark workspaces, but only updateDraft/getDraftForUser/listDrafts. Global eval output collection hydrates draft values through getGlobalDraft, which reads via getOwnDraft — so draft-producing global cases fell through to the real generated client instead of the in-memory benchmark store. Adds a getBenchmarkOwnDraft helper (null on miss, mirroring the 200/null route semantics), wires it into the adapter mock, and pins it in mockBackendDrafts.test.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
Ruben Fiszel
|
98013483c8 |
restore auto-review & command gating for private org members (#9958)
* fix(ci): gate auto-review on non-fork PR not author_association (skips private members) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): authorize private org members for command workflows via app-token gate Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
ea19cc9dc4 |
fix(ai): test key routes Azure Foundry Claude models via Anthropic Messages API (#9956)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
43044c2e28 |
feat(pipelines): wm_partition macro for grain-agnostic partition filters (#9950)
* wip: partial work before earlyoom-recovery relaunch
* fix(pipelines): scaffold the strftime {partition} filter idiom (frontend-only)
The DuckDB materialize scaffold and the AI pipeline prompt now teach the
grain-agnostic `WHERE strftime(<ts_col>, '<fmt>') = {partition}` filter instead
of the naive `= TIMESTAMP {partition}` cast. `{partition}` substitutes to the
partition IDENTITY string (`2026-07-05T23`, `2026-W27`, `2026-07`), which is not
a valid DuckDB TIMESTAMP literal for any non-daily grain — so the naive form
raises a `Conversion Error` for hourly/weekly/monthly (only daily parses).
Adds a frontend unit test asserting the hourly scaffold emits the strftime
idiom (`%Y-%m-%dT%H`) for every grain and never scaffolds the naive TIMESTAMP
cast as executable SQL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): scope strftime partition idiom to time grains
Review nit: `dynamic` partitioning's identity is a caller-supplied key, not a
timestamp, so `strftime` doesn't apply. Scope the scaffold + AI prompt claim to
time grains and add a `dynamic` example that filters on the user's own key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): wm_partition macro for grain-agnostic partition filters
The materialize runtime now injects a `wm_partition(ts)` temp macro as the first
setup statement of a time-partitioned script, so filtering the source to the
active slice is one grain-agnostic line — `WHERE wm_partition(<ts_col>) =
{partition}` — instead of a hand-written `strftime` format the author must keep
in lockstep with the resolver, or the `= TIMESTAMP {partition}` cast that only
parses for daily and Conversion-Errors for hourly/weekly/monthly.
The macro's format comes from `PartitionKind::default_time_format` in
windmill-parser, the same source the EE resolver reads to stamp the `{partition}`
identity, so the two can't drift. `dynamic` partitions get no macro (their
identity is a caller-supplied key → `WHERE <key_col> = {partition}`).
Replaces the earlier 9-line strftime comment block in the scaffold with the
single macro line; AI pipeline prompt and design doc updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): verify wm_partition strftime parity vs chrono through real DuckDB
Runs the bundled DuckDB engine in-memory and asserts strftime renders every
grain format (daily/hourly/weekly `%G-W%V`/monthly) byte-for-byte identically to
chrono — the engine the resolver uses to stamp the `{partition}` identity —
across ISO-week year boundaries (2027-01-01 → 2026-W53 etc.). Also proves the
injected `wm_partition` macro buckets the whole slice and that the naive
`TIMESTAMP '<weekly|monthly identity>'` cast Conversion-Errors.
Closes the one cross-engine assumption the pure-Rust/frontend tests couldn't
reach (flagged by CI review for weekly ISO-week rendering).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 0de2412ff0734b11e12ba378c9bcc373ff9ae800
This commit updates the EE repository reference after PR #649 was merged in windmill-ee-private.
Previous ee-repo-ref: ad6c6685689d7741058e7d2c9ecbe95d982e6268
New ee-repo-ref: 0de2412ff0734b11e12ba378c9bcc373ff9ae800
Automated by sync-ee-ref workflow.
* fix(pipelines): classify CREATE TEMP MACRO as a DuckDB prepare-path setup statement
The FFI prepare/diagnostics pass only EXECUTES statements recognized by
is_setup_statement (ATTACH/USE/INSTALL/…); everything else is merely prepared.
`CREATE [OR REPLACE] TEMP MACRO` wasn't recognized, so on a `-- prepare` run of a
partitioned materialize the injected `wm_partition` macro was never created on
the connection, and the later generated `CREATE TABLE … SELECT … WHERE
wm_partition(...)` failed to bind ("function does not exist"). The same latent
gap affected the workspace-macro splicer, which injects TEMP MACRO blocks too.
Classify CREATE [OR REPLACE] TEMP|TEMPORARY MACRO as setup so it's executed
before dependent blocks and excluded from the PrepareQueryResult count
(persistent CREATE MACRO stays a user statement). Adds a prepare-path test that
fails without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben Fiszel
|
a1c5b7aa3e |
feat(pipelines): require data uploads before running a pipeline (#9953)
* feat(pipelines): require data uploads before running a pipeline Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): require every S3Object filled for data-upload readiness Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipelines): Run pipeline defaults schedule-triggered scripts to their schedule args Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: correct why only schedules default their args in Run pipeline Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): close cascade double-start race and gate data-upload readiness on full-schema validity Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: expose ScriptEditor validity via callback, not banned bindable-with-default Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
4b1f2207b8 |
ci: replace expiring-PAT org membership gate with author_association (#9957)
* ci for broken links + fix broken links * ci: replace expiring-PAT org membership gate with author_association The shared check-org-membership.yml reusable workflow authenticated to the GitHub API with the ORG_ACCESS_TOKEN PAT to confirm org membership. That PAT expired ~1 year after issuance, so the API could no longer see private org members and check-membership emitted is_member=false — silently skipping every auto-review, command-triggered review, /ai, /plan, and git-command job while still reporting success. Gate on the event payload's author_association (OWNER/MEMBER/COLLABORATOR) instead, which comes from the built-in GITHUB_TOKEN and never expires. The trusted internal bot and existing draft/fork/command guards are preserved; the workflow_call paths stay open as trusted upstream. Deletes the now-unused reusable workflow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 windmill-internal-app[bot]
|
aaeb9524b3 |
chore: refresh vendored docs snapshot (#9955)
Co-authored-by: hugocasa <15649739+hugocasa@users.noreply.github.com> |
||
|
Guilhem
|
a6c0b3756b |
feat: chat-scoped session changes bar + unified diff drawer (#9762)
* feat(frontend): chat-scoped unified session changes bar Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(frontend): drop diff-baseline toggle, show natural per-row diffs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): carry Draft marker to expanded raw-app file rows Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): show raw-app Draft badge once at tree root, not per file Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(frontend): reuse shared DraftBadge in session diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): show draft-author avatars in session diff badge, icon-only in sidebar Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): keep badge pill around avatar in icon-only DraftBadge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): small draft marker = indigo pen + avatar; correct itemKind label Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): drop package-lock churn from merge (match origin/main) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): dedup diff-button count for legacy fork sessions; test mask helper Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): collapsible session diff panel + per-row open-diff action Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): shared sessionDeployModel for review & deploy (S1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): model-driven session review drawer, deploy inert (S2) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): wire session deploy + on-behalf/conflict gating (S3) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): behind banner + Update fork + deployment request (S4) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): session changes dock opens drawer by filter (S5) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): session review UI polish (badge, dock, In parent, tree width) - draft rows show only the avatar DraftBadge, not a duplicate state pill - drop redundant dock Review button (same as "N to review") - rename Done -> In parent with a "deployed in parent workspace" tooltip - widen the file tree Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): existence-gate In parent rows; Badge filters; badge hover - drop discarded mask-only items from the In-parent segment (existence check) - use the Badge component for the drawer filter segments and the changes dock - soften the blue Badge hover (blue-50 base was jumping to blue-200) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): polish session diff drawer (layout, badges, actions) - remove empty fork-banner gap; uniform sidebar tree padding - full-bleed diff list: drop card borders/side padding, separators between items - clamp tree x-overflow; right-align tree badges (min-w-0 on the row button) - brand-compliant selected filter badges; smaller draft badge - hide per-row open-diff button when the panel is open - rename "Delete draft" to "Discard draft" (destructive); remove header Review button - larger sm deploy/discard action buttons; remove per-item diff-content collapse Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): session Edits dock — deploy gating + change-op tracking Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): session bar per-status badges; drop change-op tracking Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): unwrap raw apps into per-file tree in session diff sidebar Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): dot-parcours pipeline (badge-derived, melt tooltip) + discard confirm Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): behind-only session item reads as deployed, not bare Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(frontend): even sidebar tree margins; gutter-aware right padding Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): pass chat id as from_session; wire deploying flag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): session diff drawer scroll-to-flush, ordering, spacer, deploy gating Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): preserve chat mask on compact; guard stale existence checks; clear poll timers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): refresh bar after drawer deploys; conflict hint over chip; plain conflict badge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(frontend): diff drawer card layout with flash ring and aligned insets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): hide stale deployed chip once row status badge reads deployed Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(frontend): session dock to two states; drop parent deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): staged deploy animation in session edits drawer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): stale-draft warning in session edits drawer Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): keep chat mask honest on deploy and discard Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): map trigger_email deploy kind; serialize mask persists Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): reset mask on new chat; close review-flagged races Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(frontend): rename session drawer title to Edited during session Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): keep mask persist queue alive after a failed save Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): sync session chatId on chat rotation; gate deploy on canWrite Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): keep compare handoff for deletion-only session edits Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): hold deploy success beat across re-keyed rows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
234a7254d5 |
chore: share DuckDB FFI build cache across worktrees (#9954)
The `duckdb` bundled feature compiles the whole DuckDB C++ library from source (~2min), which dominates the FFI crate's build. A fresh git worktree had an empty target dir and paid that cost every time. build_dev.sh now builds into a per-user cache shared across worktrees, keyed by Cargo.lock + build.rs so distinct DuckDB versions don't collide. Uncommitted changes to the crate source fall back to an isolated per-worktree ./target so active FFI development neither disturbs nor is disturbed by the shared cache. Add a crate .gitignore for /target, and note the shared cache in the AGENTS.md / backend CLAUDE.md build steps. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
bf9662172a |
fix: resolve extensionless bun relative imports on windows loader (#9949)
* fix: resolve extensionless bun relative imports on windows loader * fix: resolve local module probe against importer dir not job root * fix: keep node_modules-internal relative imports out of windmill resolver on windows |
||
|
Ruben Fiszel
|
ad6f23d6bf |
fix(cli): HD-1 test_edges + HD-2 scd2 _current write in --local pipeline graph (#9947)
* fix(cli): emit HD-1 test_edges + HD-2 scd2 _current write in --local pipeline graph Close the remaining local-vs-deployed graph parity gaps in `wmill pipeline show <folder> --local` so it matches the deployed graph (backend `asset_graph`, windmill-api-assets): - HD-1 `test_edges`: synthesize ordering-only producer → tested-script edges from parsed `// data_test` annotations. A `relationships` test references its `to_path` asset; a custom `// data_test <script>` resolves best-effort against that script's parsed reads. Each referenced asset is resolved to its in-pipeline producer via the write edges; self-edges and producer-less (external) assets are dropped — mirroring the backend set semantics. Routed through the asset node in boundedCascade's lineage DAG (asset → tested script) so a cold/bounded cascade orders the referenced dimension first, matching the frontend. - HD-2 scd2 `<dim>_current` companion write: a managed `// materialize … history` (scd2 && !manual) also produces a `<dim>_current` view. Register it as a second write edge and mark the asset `derived_from` its base dimension, so a consumer reading only the view links back to the producer instead of orphaning. Gated exactly like the backend `MaterializeSpec::write_targets` / `scd2_current_target`. The pinned `windmill-parser-wasm-asset` (1.740.0) predates the `scd2` materialize flag, so `buildLocalPipelineGraph` takes an injectable parser and the HD-2 test injects one that re-adds `scd2` for a `history` materialize — exercising the already-shipped companion-write branch until a wasm carrying `scd2` is republished (cf. #9926). Extends cli/test/pipeline_local_graph_unit.test.ts with HD-1 (relationships, no-producer, self-test, custom) and HD-2 coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(cli): pin windmill-parser-wasm-asset 1.749.0, drop HD-2 test parser seam Now that windmill-parser-wasm-asset 1.749.0 (which serializes the `scd2` materialize flag) is published, bump the CLI pin and retire the temporary injection seam: - Remove the `infer?` parameter from `buildLocalPipelineGraph`; it always uses the wasm-backed `inferScriptAssets` again. - The HD-2 `<dim>_current` companion-write test drives the real wasm directly (drops the `inferWithScd2` wrapper that re-added `scd2` against the pinned 1.740.0 build). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): pin windmill-parser-wasm-asset 1.749.0 to match CLI Restore the CLI↔frontend lockstep on the asset parser wasm broken by the previous commit: every other windmill-parser-wasm-* package is pinned to the same version in both cli/package.json and frontend/package.json, so keep the asset parser aligned too. The frontend derives materialize/scd2 from its own TS annotation parser (`parsePipelineAnnotations`), so this bump only affects body asset inference in the live graph — moving it in step with the CLI `--local` graph and the deployed backend parser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6eabb96ae7 |
fix(pipelines): make node & pipeline-level run affordances always visible (#9948)
* fix(pipelines): make node & pipeline-level run affordances always visible Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): don't leave node Run disabled by stale form validity The always-visible node Run button read `isValid` directly, but that flag is only meaningful while PipelineRunForm is mounted to set it. On a same-path re-resolve (the component is keyed on script.path) from an input-carrying script to an input-less one, the form unmounts leaving `isValid=false`, which wrongly kept the empty-args Run button disabled. Gate validity through a `runValid` derived that is true whenever no form is rendered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): run with {} when node has no form, not stale args The always-visible node Run button dispatched $state.snapshot(args) unconditionally. `args` persists across a same-path re-resolve (component keyed on script.path), so a script that once had inputs/partition args and is re-resolved as input-less would run with the stale hidden args instead of {}. Send {} whenever no form is rendered, matching the no-form run intent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f9840cd477 |
add missing amber-950 shade so dark pipeline callouts render (#9951)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
891b32195a |
chore(main): release 1.749.0 (#9938)
* chore(main): release 1.749.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>v1.749.0 |
||
|
Ruben Fiszel
|
e3f43033ca |
fix(cli): macro-library parity in --local pipeline graph + read-only run --dry-run (#9942)
* fix(cli): surface macro libraries in --local pipeline graph + make run --dry-run read-only
* fix(cli): resolve workspace-wide macro libraries in --local graph (shared libs outside the pipeline folder)
* fix(cli): macro-lib consumers + //-prefix parity in --local pipeline graph
Address Codex review P1s: (1) macro libraries that consume another library's
macros now produce lib->lib edges (any folder DuckDB script is a consumer, not
just // pipeline members) so an upstream provider node no longer disappears;
(2) parseMacroAnnotations accepts //, --, and # prefixes like the backend, so a
.duckdb.sql library headed with // macros is detected locally. Both edge
endpoints are forced into the node set. Verified byte-for-byte against deployed.
* fix(cli): exclude non-pipeline macro-consumer nodes from --local run selection
Address Codex P1: buildMacroEdges surfaces macro-consumer nodes (a DuckDB script
calling a macro but not marked // pipeline) for lineage display. Those have no
local file, so pipeline run --local must not treat them as manual roots — a
dry-run listed them and a real run failed resolving local content. Exclude any
--local graph node absent from localScripts (the previewable set) from starts and
selection, alongside the existing macro-library exclusion.
* fix(cli): reject display-only macro consumers in explicit --from (post-merge with #9945)
The mid-DAG --from feature (#9945, now on main) admits any autorun-able script
via validFromStarts/fromEligible, which was filtered only by macroLibPaths. A
non-// pipeline macro-consumer helper (a --local display node) therefore passed
--from eligibility and produced an empty plan. Filter fromEligible by the broader
notRunnablePaths too, and reject such a --from with a clear message instead of a
silent empty plan.
* chore(cli): remove NUL edge-key separator + refresh stale macro comments
Address Codex P2 nits: (1) the macro edge map packed (lib, consumer) into a
string with a literal NUL separator, which made localGraph.ts read as a binary
file to grep/rg — replace with a nested lib->consumer Map (no separator); (2)
comments claiming macro nodes/edges are 'deployed graph only' contradicted this
PR's local derivation — describe the code as it is.
* fix(cli): tag unused // pipeline + // macros libraries so --local run excludes them
Address Codex P1: the deployed builder sets 'macros' on any node whose path
provides macros (edge or not), so a // pipeline + // macros script with no
consumers is still recognized as definition-only. Local enrichment only tagged
edge providers, leaving an unused pipeline macro library as a bare runnable that
pipeline run --local would schedule as a manual root. Also tag any library whose
path is already a runnable; unused non-pipeline libraries stay suppressed.
* fix(pipelines): `// macros` takes precedence over `// pipeline` (a library is never a member)
A macro library is definition-only — its macros are injected into consumers and
running it is a no-op — so marking it `// pipeline` is meaningless and only
produced a confusing state (an unused pipeline macro library appearing as a
manual root). Make `// macros` win: parse_pipeline_annotations forces in_pipeline
false when macros is set. Mirrored in all three parsers that must agree — the Rust
canonical parser (drives deploy membership), the frontend TS parser (live graph),
and the CLI local graph (pinned wasm still reports in_pipeline, so precedence is
applied when skipping members). Shared parity fixture + unit tests on each side.
* docs(cli): trim narrative comment blocks to non-obvious constraints
Address Codex P2: duckdbMacros.ts opened with a ~19-line narrative block whose
parity rationale belongs in the PR description; reduce to the two real constraints
(keep in lockstep with duckdb_macros.rs; dynamic-SQL calls need // use). Per the
AGENTS.md comment policy.
* fix(cli): model macro libraries as pipeline members, matching the deployed graph
Reverts the parser-precedence approach (
|
||
|
Ruben Fiszel
|
b13113964a |
fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect (#9939)
* fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect
The SDK object forms — TS `writeS3File({s3:"exports/x"})` and Python
`write_s3_file(S3Object(s3="exports/x"))` — resolve to the URI `s3:///exports/x`
(empty default storage), whose parsed asset path was `/exports/x` (leading
slash). DuckDB `read_csv('s3://exports/x')` and the `// on s3://exports/x`
trigger form yielded the bare `exports/x`. The same object thus produced two
asset identities, so a DuckDB consumer never connected to a TS/Python producer
in the pipeline graph.
`parse_asset_syntax` (shared by the native backend parsers and the wasm parser
that drives `frontend/src/lib/infer.ts` and the CLI `localGraph`) now strips a
single leading slash from S3 paths, so `s3:///key`, `s3://storage/key`, DuckDB
`s3://…`, and `// on` all canonicalize to one key. Both deploy-time inference
and editor/CLI inference agree, and the producer's write edge and the
consumer's read/trigger edge share a node.
Only one leading slash is stripped, so `s3:///` triple-slash default-storage
keys collapse to the bare key while Hive-partition keys
(`s3://bucket/y=2024/f.parquet`) and explicit-storage `s3://storage/key` paths
are untouched. Non-S3 asset kinds (res://, ducklake://, …) keep their paths
verbatim.
Note: existing deployed pipelines that recorded `/key` paths need a redeploy to
pick up the canonical `key`; the fix is forward-consistent for anything parsed
after this change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): mark S3 asset-path normalization (item 6) resolved
The open-issues list still flagged the SDK-form leading-slash vs bare-URI
no-slash mismatch as "Still open", contradicting the fix in this PR. Mark it
resolved to match the updated Language-coverage prose.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs+test(pipelines): disclose S3 explicit-storage vs default-storage-nested-key aliasing
Collapsing to one canonical key means `s3://storage/key` (explicit storage) and
`s3:///storage/key` (default-storage nested key) now alias to the same node
`storage/key`, though they name different objects. Low-probability (needs a
storage config named to match a default-storage prefix) and inherent to a
best-effort lineage graph that doesn't split the first segment as a storage
name, but previously undisclosed. Document the tradeoff and pin the intended
aliasing with a test so it's intentional, not a latent surprise.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): mirror S3 leading-slash strip in frontend live-preview parser
The pipeline graph live preview parses `// on` annotations client-side via the
hand-written `parsePipelineAnnotations.ts` (a TS mirror of the Rust annotation
scanner), NOT the wasm parser. Its `parseAssetSyntax` still returned the raw
suffix, so `// on s3:///exports/x` yielded `/exports/x` while the deploy-time
and wasm parsers now canonicalize to `exports/x`. `resolveGraph` synthesizes
trigger edges from that path, so the browser preview could still render
disconnected `/exports/x` and `exports/x` nodes for the exact triple-slash case
this PR fixes at deploy time.
Mirror the S3-only single-leading-slash strip in the TS parser and extend the
shared parity fixture corpus (run by both the Rust and TS parity suites) with
the triple-slash trigger case, so Rust/TS drift on this is now caught.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): seed slashless S3 template asset paths to match canonical key
`autoOutputAsset` seeded new S3 template outputs with a leading slash
(`/pipelines/…`), which the old parser required to match `s3:///key` writes.
This PR made `parse_asset_syntax` strip that slash, so the seeded draft asset
(stored as `outputAssets`, used by `resolveGraph` for inactive-draft node
identity) no longer matched the body-inferred identity `pipelines/…` — the live
preview could render a duplicate `/pipelines/…` node and a phantom post-deploy
drift warning.
Seed the canonical slashless key instead, and switch the DuckDB body's S3 URIs
from `s3://${path}` to `s3:///${path}` so the generated runtime URI stays the
triple-slash default-storage form byte-for-byte (the SDK sites already build
`s3:///` + bare key). Add a pure-logic parity test asserting, for every
language and S3 output kind, that the seeded asset path is slashless and that
every S3 URI the generated body emits is triple-slash and canonicalizes back to
that seeded path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): canonicalize S3 keys in CLI + frontend bounded-cascade resolvers
Two more hand-written S3-URI sites returned the raw suffix, so `s3:///exports/x`
stayed `/exports/x` while native/wasm parsers now canonicalize to `exports/x`:
- `cli/src/commands/pipeline/localGraph.ts` — the no-wasm fallback `// on`
scanner (go/bash/ruby). A fallback consumer's `// on s3:///x` would not
connect to a wasm-inferred `x` producer in `wmill pipeline show/run --local`.
- `boundedCascade.ts` `assetUriToNodeId` (duplicated in the CLI and the frontend
AssetGraph engines, kept in sync) — `--to s3:///exports/x` / a cascade bound
token would not resolve against the canonical graph node `s3object:exports/x`.
`resolveToken` delegates here, so it is covered too.
Mirror the S3-only single-leading-slash strip in all three, and add `s3:///`
tests to the CLI local-graph fallback suite and both bounded-cascade suites
(explicit-storage and Hive-partition keys asserted untouched).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(pipelines): phrase S3 template test comment as a current invariant
Describe the slashless-seed requirement as the invariant it is, not as change
history, per the AGENTS.md "describe the code as it is" rule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): strip all leading slashes from S3 keys so trigger refs round-trip
`parse_asset_syntax` stripped only one leading slash, so `S3Object(s3="/x")` —
which resolves to the quad-slash URI `s3:////x` — parsed to path `/x`. But
`trigger_spec_to_row` rebuilds a stored trigger ref as `s3://<path>` =
`s3:///x`, which `parse_asset_trigger_ref` then parses back to `x`. The
producer recorded `/x` while its consumer trigger resolved to `x` → a broken
edge. The same asymmetry affects every `s3://`+path reconstruction site
(backend refs, frontend `assetUri`, page refs) whenever a path starts with `/`.
Strip ALL leading slashes so a canonical S3 path never starts with `/`; naive
`prefix + path` reconstruction then round-trips everywhere. Applied uniformly
across all six S3-URI sites (Rust `parse_asset_syntax`, the TS live-preview
parser, template `s3Key`, and the frontend+CLI `assetUriToNodeId` and CLI
fallback scanner). The pathological leading-slash key collapses to the bare key
— acceptable for a best-effort lineage graph that never split storage anyway.
Tests: a windmill-common round-trip test (parse → trigger_spec_to_row →
parse_asset_trigger_ref) over every URI form incl. the quad-slash case; a
`s3:////x` shared parity fixture (Rust + TS); and quad-slash assertions in the
Rust parser test and both bounded-cascade suites.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): align S3 template parity helper with strip-all canonicalization
The template seed/body parity test's `canonicalS3Key` helper (and its comment)
still stripped a single leading slash, so it no longer mirrored the parser it
claims to pin. Strip all leading slashes to match `parse_asset_syntax` and the
frontend/CLI mirrors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
2d3a773441 |
feat(pipelines): mid-DAG selective execution (dbt model+) for pipeline runs (#9945)
* feat(pipelines): mid-DAG selective execution (dbt `model+`) for pipeline runs
Relax the root-only constraint on bounded-cascade starts so `--from` can name
any node in a pipeline — not just a schedule/manual root. A mid-DAG start runs
that node plus its transitive downstream and never re-runs upstream, giving
dbt's most common gesture (`dbt run --select model+`) a direct form:
wmill pipeline run f/orders --from fct_orders_daily
Previously this errored with "Starts must be schedule-triggered or manual
roots". The bounded-run engine already computed downstream/path-between sets
generically; only the eligibility gate was root-only.
- Shared engine (`boundedCascade.ts`, CLI + frontend mirror): add
`validFromStarts` — every autorun-able script (roots AND mid-DAG asset
subscribers / pure readers), excluding only event/input-only handlers
(kafka/mqtt/…/webhook/data_upload) that can't run with empty args.
- CLI: `--from` accepts any `validFromStarts` node; asset `--from` and
non-autorun handlers still rejected (the latter runnable via `--upload`). An
explicit mid-DAG start is protected from the barrier cut. Help text + regenerated
system_prompts describe the new surface.
- Frontend graph UI parity: any node with downstream now offers "Run + downstream…"
(was roots-only). With no end picked the bounded-run bar runs the full downstream
closure (`model+`); picking end(s) still bounds the path-between set.
- Unit tests for the new selection semantics in both engines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipelines): address CI review — scheduled-root --from regression + pick-mode barrier parity
Codex review findings on #9945:
- P1: explicit `--from` rejected a scheduled root that also carries a secondary
non-autorun trigger (e.g. `// on schedule` + `// on data_upload`), even though
it stays a valid IMPLICIT start. `validFromStarts` excluded anything in
`nonAutorunTriggerScripts`; now it unions in `validStarts` (which lets the
schedule identity win over the secondary trigger), so a scheduled root is
`--from`-eligible in both CLI and the graph UI. Regression tests added in both
engines.
- P2: bounded-pick mode built `eligible` (pickable end bounds) from raw
`descendants`, so an event handler — or a node only reachable through one —
could be clicked as an end yet be silently dropped from the barrier-cut run.
`eligible` is now the barrier-cut closure, so those nodes are dimmed and
non-pickable. The highlighted `bounded` ring now also reflects the actual
(barrier-cut) run set, including the no-ends "Run + downstream" case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipelines): frontend barrier set must exclude all valid roots, not just the picked start
Codex review follow-up: the frontend `boundReachable` barrier set only protected
the picked start (`id !== boundPickStart`), while the CLI protects every valid
root (`!starts.has(id)`). So a scheduled root that also carries an event trigger,
reached downstream from another start, was wrongly treated as a barrier — the UI
dimmed/skipped it and its downstream, diverging from the CLI run set.
Exclude `validStarts` from the barrier set too (a scheduled/manual root runs on
its own identity even with a secondary event trigger). Regression test asserts a
scheduled-event root and its downstream stay reachable from an upstream start,
and that the naive (start-only) barrier set would have dropped them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(pipelines): frontend must exclude webhook/data_upload as mid-DAG autorun starts
Codex review follow-up: the frontend `validFromStarts` only excluded
`EVENT_TRIGGER_KINDS`, so a mid-DAG `webhook`/`data_upload` subscriber was added
by the new eligibility loop — the UI would offer "Run + downstream" and launch it
with empty args (no uploaded S3Object / webhook body). The CLI mirror already
excludes these input-only kinds.
Add a frontend `NON_AUTORUN_TRIGGER_KINDS` (event kinds + webhook + data_upload),
mirroring the CLI, and use it in both `validFromStarts` (exclude such mid-DAG
handlers from starts) and `nonAutorunTriggerScripts` (cut them as barriers).
When the marker is visible (editor overlay / draft) these are now handled
exactly as the CLI does; the deployed-graph blind spot (no webhook/data_upload
rows) remains the documented pre-existing `validStarts` limitation.
Regression test: a `data_upload`/`webhook` mid-DAG subscriber is not an eligible
start and is barrier-cut (with its exclusive downstream) when running from an
upstream root.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
475b072987 |
feat(pipelines): partition run-arg picker + first-run setup signpost (#9943)
* feat(pipelines): partition run-arg picker + first-run setup signpost Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): honor partition start= anchor in picker default and per-input upstream hint Addresses CI review (Codex): - P1: never seed/offer a pre-start bucket — the worker takes an explicit partition arg verbatim, so seeding today's bucket before start= would materialize early. defaultBucket now clamps to the start bucket and drops pre-start recent-missing chips; a hint explains the start anchor. - P2: upstream-missing hint checks each partitioned ducklake input separately instead of unioning, so a fan-in where one input has the bucket no longer masks another that lacks it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): reseed partition picker on header change and fail safe on bad tz/start Addresses second CI review (Codex): - P1: run form is now keyed on the parsed partition spec as well as the schema, so editing the // partitioned header (same schema) remounts the picker and reseeds/re-strips instead of keeping a stale bucket that re-bypasses the start anchor. - P1: malformed metadata is fail-safe in parity with the backend — invalid tz= no longer throws in Intl (falls back to UTC for display), invalid start= (e.g. 2026-02-31) is rejected via round-trip check, and neither auto-seeds an explicit partition (which would bypass the worker's own tz/start validation). A warning hint points at the header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6ae8dd37b1 |
fix(pipelines): pipeline-level run control, tables label, data-test rollback + fork badges (#9944)
* fix(pipelines): pipeline-level run control, tables label, data-test rollback + fork badges - Add always-visible "Run pipeline" header control (edit mode) that runs every script in dependency order via the bounded-cascade engine, so a run no longer requires hovering a node's play button. - Header summary counts ducklake/datatable assets as "tables" (and s3object as "files") instead of the raw kind, collapsing shared nouns. - Surface a data-test outcome badge on guarded asset nodes: EE shows a rolled-back (previous version left live) state, CE shows published-despite- failure — driven by the producer's last run state and the edition. - Make the fork data-environment marker a prominent labeled chip (⑂ fork / ↗ parent) instead of a bare icon. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): address CI review — scope Run pipeline to members, anchor guard badge, spin loader - Run pipeline now filters to `in_pipeline` script runnables, so it never launches dependency-only endpoints the graph shows for context (macro libraries, custom data-test scripts, out-of-folder producers). - Data-test guard badge only attaches to the producer's declared `// materialize` target, so a multi-output producer no longer badges its other ducklake writes. - Spin the Loader2 icon in the "Run pipeline" button while a run is in progress (startIcon classes), matching every other loading affordance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): data-test badge copy speaks to write policy, not failure cause producerFailed is a generic job-failure signal, so the failed-state tooltip no longer claims the run "failed its data tests" (it could be a runtime/worker error). It now states the edition's behavior on any failed materialize: EE rolls back (previous version left live), CE may leave a failing write live. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): Run pipeline keeps independent branches running after a failure runSelection used a single global fail-fast flag, so once any node failed it refused to schedule *any* newly-ready node — a failure in one branch could strand an unrelated healthy branch as 'skipped' depending on job timing. Now a failure poisons only its transitive descendants; independent branches finish. Add regression tests: independent-branch-survives-failure and join-node-skipped -when-one-upstream-fails. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
a582e04bf4 | fix: rebuild windows bun loader main.ts filter from forward-slash cdir (#9946) | ||
|
Ruben Fiszel
|
b40b504513 | docs: document feature set + gotchas for running DuckLake pipelines from source (#9940) | ||
|
Ruben Fiszel
|
55451db009 |
fix(datatable): self-teaching error for unresolved datatable:// references (#9941)
`ATTACH 'datatable://main'` (or any datatable schema/executor path) failed with a bare "datatable main not found", giving the user no way forward — the datatable substrate has no auto-provisioning like a DuckLake catalog, so the fix is always to create one in workspace settings, but nothing said so. `get_datatable_resource_from_db_unchecked` now returns a NotFound error that lists the workspace's configured data tables (to catch typos) and points at the "Data tables" settings tab, noting `main` is the default name used by `datatable://main`. The message bubbles up wherever the resolver is called (pipeline ATTACH, schema fetch, postgres executor, agent HTTP endpoint). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
574d3ac9ff |
fix(pipelines): link SCD2 <dim>_current view to its producer across all graph surfaces (#9933)
* fix(pipelines): link SCD2 <dim>_current view to its producer across all graph surfaces An SCD2 producer (`// materialize … history`) creates the base table AND a `<dim>_current` view at runtime. The deploy path already registered both writes, but the CLI `--local` graph and the frontend live-editor graph only emitted the base write, so a consumer reading only `<dim>_current` orphaned there. Centralize the companion derivation in `MaterializeSpec::write_targets` / `scd2_current_target` (+ TS `scd2CurrentTargetPath` mirror), emit the `_current` write in every surface, and mark the companion node `derived_from` the base so the canvas renders it as a derived "current view" instead of an unrelated table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): keep scd2 _current write edge when editing a saved producer Addresses Codex CI review (P1): opening a deployed scd2 materialize producer for editing dropped its persisted `<dim>_current` write edge. `liveRefKeys` (the set of asset keys a saved-script edit preserves against stale-filtering) only added the base materialize target, so the companion `_current` write was judged stale and filtered — orphaning consumers of only the view mid-edit. Add `scd2CurrentTargetPath(m)` to `liveRefKeys` too; covered by a new saved-edit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
46be39dfb7 |
fix(pipelines): order data_test relationships refs before the tested script in a cascade (#9934)
* fix(pipelines): order data_test relationships refs before the tested script in a cascade Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): key custom-test reads by (usage_kind, path) to avoid same-path flow collisions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
0ad174fa49 |
feat(pipelines): self-teaching custom data_test errors + scaffold (#9937)
Custom `// data_test <path>` scripts must be a single SELECT reading the freshly-materialized target via the internal `_wm_target.<table>` alias — neither was documented or scaffolded. Make the codegen errors name the exact violation (multi-statement, non-SELECT, wrong alias, empty) and append a copyable `SELECT * FROM _wm_target.<table> WHERE <condition>` example. Add a DuckDB-only 'Data test' pipeline output kind that scaffolds that starter body. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
52ce805f61 |
fix(pipelines): dedup guard for keyed merge + deploy-time SCD2 validation (#9936)
Two correctness/validation improvements to managed materialization: 1. A keyed `merge` (`key=<col>`) is delete-by-key + insert-all and does NOT deduplicate its source, so two incoming rows sharing a key both landed under that key — silently breaking the one-row-per-key contract. Codegen now emits an in-transaction guard (same `error(...)` shape as the schema -drift guard) that fails the run when the SELECT returns more than one row for a non-NULL key, naming the key. Authors deduplicate in the SELECT or switch to `append`. NULL keys are exempt, matching the delete's `IN (...)` scope. 2. The two SCD2 misconfigurations that were only caught at run time — `history` without `key=`, and `history` + `// partitioned` — now fail fast at deploy via a shared `MaterializeSpec::validate`, called from `create_script_internal`. The DuckDB executor keeps the same check as a safety net for preview/test runs that never deploy (shared message, no drift). Adds unit tests for the merge guard codegen and for `validate` (all four cases), and updates docs/ducklake-materialization.md and docs/pipelines-vs-dbt.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
22452ce540 |
fix(object-storage): remove 20-file bucket-browser listing cap in CE (#9935)
* fix(object-storage): remove 20-file bucket-browser cap in CE The Community Edition build rejected the object-storage `list_stored_files` endpoint with an error once a workspace bucket held more than 20 objects, making the bucket browser unusable on larger buckets. The listing already collects up to `max_keys` objects, so the hard cap was purely a gate. Drops the CE listing cap (in the EE-symlinked `job_helpers_ee.rs`, tracked in the companion windmill-ee-private PR) and removes the now-inaccurate sentence from the workspace object-storage settings banner. The 10 GiB total-storage write quota remains as the intentional CE limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 4bd7b73dcef1b77dc2866bc58e0c86962559495c This commit updates the EE repository reference after PR #648 was merged in windmill-ee-private. Previous ee-repo-ref: aa14d0724216030948c2f575bcc19c0e6e0476a7 New ee-repo-ref: 4bd7b73dcef1b77dc2866bc58e0c86962559495c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
799b9e3b7c |
chore(main): release 1.748.0 (#9914)
* chore(main): release 1.748.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>v1.748.0 |
||
|
Ruben Fiszel
|
182b10b2ad |
render FFI errors with real newlines and no stray quoting (#9932)
* fix(duckdb): render FFI errors with real newlines and no stray quoting The DuckDB FFI returns errors as ERROR <json-encoded-message>, so the executor was surfacing the serde_json-escaped form (wrapping quotes, literal \\n). Multi-line errors like the write-audit-publish data-test breakdown were unreadable. Decode the JSON string back to the raw message at both FFI error sites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tighten decode_ffi_error comment to the invariant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5769b6036c |
fix(pipelines): activity-axis label clarity + select failed node on cascade failure (#9931)
* fix(pipelines): clarify activity-window axis label + select failed node on cascade failure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): clear active draft so failed-node focus is not masked PipelineGraphEditor gives an open draft (activeDraftPath) priority over selection via openScriptPath, so the cascade-failure focus set only selection and stayed masked while a draft pane was open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
377c02ec47 |
feat(pipelines): on_schema_change write guardrails + data_test deploy validation (#9930)
* feat(pipelines): on_schema_change write guardrails + data_test deploy validation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to fa7ac11c1e0ab39e84a0c18973ba427a240933ca This commit updates the EE repository reference after PR #647 was merged in windmill-ee-private. Previous ee-repo-ref: bd23b2a904cb2e6554c7ff209ff8adb9d91775d1 New ee-repo-ref: fa7ac11c1e0ab39e84a0c18973ba427a240933ca Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
28a6b086c8 |
fix(cli): pipeline + workspace UX batch (init/bind stub, run errors, macro libs, lock-job report, upgrade errors) (#9929)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
744a7597ed |
fix(cli): publish all windmill-parser-wasm-* deps so local pipeline graph keeps write edges (#9926)
* fix(cli): publish all windmill-parser-wasm-* deps so local pipeline graph keeps write edges Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: trim explanatory comment blocks to core constraints --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
39eb9de1bc |
feat(pipelines): fork data environments for ducklake materialization (dev data) (#9915)
* feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): per-lake isolated/shared choice at fork creation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): chain-aware defer discovery + per-location fork namespace registry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): cleanup deletes fork data from the registered storage identity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): collapse fork data-path segment to one component (slash-safe ids) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): run fork ducklake cleanup inline in delete_workspace Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter - extract require_prod_admin_for_dev_workspace, used by both delete_workspace and drop_forked_ducklake_namespaces so the gates cannot drift - key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL re-registers its namespaces - filter listed object locations to the segment boundary before deletion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its parent with an empty ancestor chain while its cloned config still points at the shared lake. Key the isolation gate on the wm-fork- prefix as well as the chain (mirroring workspace_is_fork): orphaned forks get the write redirect, registration and cleanup with zero ancestors (no defer), and keep their 'fork' graph chips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace Chain position alone classified the last ancestor as a root, but an orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the chain the same way while its data lives in its fork namespace — its descendants' defer views bound the dead root's lake instead. Key the root-vs-fork decision on the wm-fork- prefix too, matching the resolution gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions - fork creation strips cloned fork_behavior stamps before applying the request's shared_ducklakes list: sharing is a per-creation choice, a fork of a shared fork defaults back to isolated - fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the durable cleanup ledger and outlive the workspace when physical cleanup fails post-commit; fork creation retries leftover rows for the reused id and refuses to create while a metadata schema still cannot be dropped (data-file leftovers alone are inert once the schema is gone and are swept by the next successful same-prefix cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources - ledger rows gain a schema_dropped phase flag: set when the schema drop succeeded but data cleanup failed, so later retries skip the schema phase and need no catalog credentials at all; registration resets it on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the schema - retry-path $res: resolution falls back to the workspace being forked (the deleted fork's resources were clones of a parent's); live paths (delete_workspace prepare, drop endpoint) pass no fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph A failed materialization must not disguise a physically existing fork table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the table, so reads hit fork data while the graph claims parent defer. - record_mat upsert preserves the last committed snapshot_id on failure - defer discovery and graph chips treat fork rows with a committed snapshot as fork-owned even when status is failed - inspect_fork_catalog also lists live fork tables (same round trip) and the defer list is filtered against them — covers rows recorded before this fix and tables created by raw SQL - drop stale FK-cascade wording in the design doc and sidebar comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip The workspace-settings ducklake editor had no fork awareness: no reminder of each lake's isolated/shared choice and no warning about what edits mean in a fork. It also rebuilt each lake explicitly on save, silently dropping fork_behavior — any settings save in a shared fork flipped the lake back to isolated. - fork detection mirrors the backend gate (parent link or wm-fork- prefix) - info banner explaining isolated vs shared semantics in a fork - per-lake chip (emerald 'isolated' / amber 'shared with parent') with tooltips, matching the pipeline graph chip colors - fork_behavior added to DucklakeSettingsType and preserved through convertDucklakeSettingsToBackend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d4b4374de8 |
feat(pipelines): capture violating-row samples for data tests (#9919)
* feat(pipelines): capture violating-row samples for data tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): byte-accurate sample cap and leaf-level payload sanitize Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref to WAP guard probe adaptation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: WAP failures are counts-only — samples exist only on commit-then-test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: qualify where sample row data appears — job result and failed-job log line Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: error handlers receive the full result incl. samples, like any failed job Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 80d309edebb899e36a3bdcdf4ea73c4db070534d This commit updates the EE repository reference after PR #646 was merged in windmill-ee-private. Previous ee-repo-ref: 16e916bf11f26381920560b55771fce693e668c6 New ee-repo-ref: 80d309edebb899e36a3bdcdf4ea73c4db070534d Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
5ad2de91a2 |
feat(sdk): enforce s3:// URIs for string S3 params + ingestion (EL) docs (#9912)
* feat(pipelines): ingestion (EL) templates + docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review nits — draft collision guard, template-mode selection reset, invariant test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lead the insert menu with ingestion templates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipelines): ingestion story as docs-only — drop editor template UI The insert-menu template section mixed two selection grammars in one popover and confused more than it helped. The three E2E-verified example pipelines now live verbatim in docs/pipeline-ingestion.md; the Python bare-string S3 key fix in pipelineTemplates.ts stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sdk): bare string S3 keys in py/ts clients + asset parsers A plain string passed where an S3Object is expected is now a bare key in the default storage — previously the py client silently degraded it to s3="" (auto-generated key) and both asset parsers canonicalized it without the leading slash, splitting lineage. parseS3Object moves to s3Types.ts so it is unit-testable without the generated services. The pipeline template fix from the earlier commit is superseded (bare strings are the supported spelling again); docs examples flipped to bare keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(sdk): enforce s3:// URIs for string S3Object params Bare strings now raise/throw with a hint pointing at the s3:///<key> spelling instead of being treated as keys (previous commit) or silently degrading to an empty key (original behavior). One string spelling everywhere: SDK calls, // on annotations, and DuckDB SQL all use s3:///<key>. TS regains the s3://-template-literal type; the asset parsers record no asset for a bare string (the call can only error); templates emit the URI form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(pipelines): move ingestion (EL) guide to windmilldocs, keep design constraints User-facing how-to (engine choice, cursor recipes, schema drift, worked examples) moves to windmilldocs core_concepts/63_pipelines (windmilldocs#1462); the repo keeps only the design constraints future feature work must not break, as a section of ducklake-materialization.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: regenerate system prompts after parse_s3_object docstring change Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): reject empty-key s3 URIs; align asset parsers with the runtime rule Addresses CI review: s3:/// and s3://bucket/ now raise (an empty key would fall back to the auto-generated-key path the strict contract exists to prevent); the asset parsers' string branch applies the same valid-URI-with-non-empty-key rule so no R/W edge is recorded for a call that can only error (the generic URI-literal scan still records ambiguous access-None assets, by design); comments rephrased as current constraints per AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |