mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-19 16:02:14 +00:00
sql-s3-streaming-issue
1823 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ruben Fiszel
|
77222a24ec | fix: relabel materialization timeout and document per-phase budget | ||
|
Ruben Fiszel
|
c9d6c59ed8 | fix: keep job ping alive while materializing (s3object) SQL args | ||
|
Ruben Fiszel
|
17872018cc |
feat(nsjail): make python/ansible rlimit_as configurable per worker (GIT-921) (#10138)
nsjail caps a jailed job's virtual address space at rlimit_as (4096 MiB for python3 and ansible). JIT runtimes (Bun/JavaScriptCore, the JVM) reserve large virtual ranges up front, so a subprocess spawned from a jailed Python/Ansible job can crash against this cap even when its physical memory use is modest (e.g. the Bun-compiled claude CLI hitting JSC/pthread allocation failures). Most other language protos already run with disable_rl: true (unlimited); python3 and ansible are the outliers with an explicit rlimit_as. This exposes that cap via a per-language env var (NSJAIL_PY_RLIMIT_AS_MB, NSJAIL_ANSIBLE_RLIMIT_AS_MB) so operators can raise or lift it on a dedicated worker pool without a source patch/rebuild and without weakening the mount/PID/user-namespace isolation that provides the real security boundary. Only the address-space limit changes; cpu/fsize/nofile rlimits are untouched. Value is in MiB, or unlimited/none/inf/0 to uncap (rlimit_as_type: INF). Unset keeps the historical 4096 default. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6407d9ff5c |
fix(bash): normalize CRLF line endings before running scripts (#10131)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
27ead8d084 |
fix(ai): disable redirects on worker AI provider client (GHSA-5q4v) (#10122)
* fix(ai): disable redirects on worker AI provider client (GHSA-5q4v) The worker AI request path issued provider requests with the shared HTTP_CLIENT, which follows up to 10 redirects without revalidating each hop. SSRF validation on the provider base_url is single-shot, so a public base_url could 3xx the worker into a private/internal host (e.g. cloud metadata), bypassing the private-endpoint protection. The API proxy was already hardened in #9370; the worker path was missed. Add a dedicated AI_HTTP_CLIENT with redirects disabled and use it for the user-controlled provider endpoint, mirroring the API proxy client. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai): add ALLOW_AI_BASE_URL_REDIRECTS debug escape hatch Off by default (redirects stay disabled). When set, restores redirect following on the AI HTTP client for debugging non-standard/self-hosted gateways, with a startup warning that it weakens SSRF protection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ai): correct redirect comment for the escape hatch override Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(ai): condense redirect invariant comments per review Anchor the SSRF rationale to ALLOW_AI_BASE_URL_REDIRECTS (the knob that would break it) and shorten the AI_HTTP_CLIENT and call-site comments to avoid restating it at multiple sites (AGENTS.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 hugocasa
|
7ebfad382a |
feat(ai-agent): give tools a real description instead of the tool name (#10083)
* feat(ai-agent): use a real tool description instead of the tool name Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): render tool-name error full width and hoist it above the description Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): make tool description field hug its content so a single line is vertically centered Add an optional minHeight param to the autosize action (default unchanged at 30px) and pass minHeight 0 for the tool description so an empty/one-line field no longer reserves the 30px floor and leaves dead space below the text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ai-agent): regenerate OpenFlow-derived prompts, CLI guidance, and copilot zod schema for tool description Fixes the check-freshness CI failure (system_prompts + skills.gen.ts) and makes the flow copilot's openFlow.json / openFlowZod.gen.ts aware of the new AgentTool.description field so AI-authored tools can set it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ab38e1418e |
fix: keep agent-worker server job-completed processor alive & self-healing (#10033)
* fix: keep agent-worker server job-completed processor alive on init-script failure The agent-worker API server's background job-completed processors relay completions on behalf of many remote agent workers. The processor loop exited (dropping its receiver) on an init-script failure, but on the server that failed init script belongs to a remote worker, not the server. Once enough processors exited, the shared completion channel disconnected and every /send_result POST returned 500, stranding completions and creating zombie-job restart loops. Add an is_agent_server flag so server relay processors don't self-terminate on init-script failure. Pins the EE companion change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump EE ref for send_result wait-for-processor change Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: agent-worker server survives a failed init script End-to-end regression for the agent-worker-server processor bug: an agent worker runs a failing init script, POSTs the failed init-script completion to /send_result, and the test asserts the server's background job-completed processor stays alive (a subsequent job completes and no bg-processor critical alert is raised). Fails if the is_agent_server guard is removed (the processor breaks, the supervisor raises a critical alert). Requires --features enterprise,license,private,agent_worker_server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: replace heavyweight init-script e2e with focused unit tests The panic/respawn/alert and 503 timeout paths are now covered by fast, deterministic unit tests in windmill-api-agent-workers (supervise_processor, classify_send). Drop the enterprise-only, global-config-mutating e2e in favor of those. Bump EE ref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to bc45d9275d4307132927dc8ad3e82049b1aed463 This commit updates the EE repository reference after PR #653 was merged in windmill-ee-private. Previous ee-repo-ref: 2aca03f28bb37e938ae548b81f1620b2e00dc0f7 New ee-repo-ref: bc45d9275d4307132927dc8ad3e82049b1aed463 Automated by sync-ee-ref workflow. * chore: bump EE ref for bg-processor alert rate-limiting Picks up windmill-ee-private#654: exponential backoff + rate-limited critical alerts in supervise_processor, addressing the code-review nit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to d48c0e01e8601a372353c032dd237ddb6fa3bbad This commit updates the EE repository reference after PR #654 was merged in windmill-ee-private. Previous ee-repo-ref: f89eeb6e333614850ef650e7df78e3c2335f107c New ee-repo-ref: d48c0e01e8601a372353c032dd237ddb6fa3bbad Automated by sync-ee-ref workflow. * chore: bump EE ref for graceful-shutdown-during-backoff fix Picks up windmill-ee-private#655: supervise_processor re-checks shutdown before respawn and selects on the shutdown broadcast during backoff, so a crash-loop backoff can't hang graceful shutdown. Addresses the Codex P1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 9bc5dfb9ce73a2d9b981a1de86eea6aa26688b79 This commit updates the EE repository reference after PR #655 was merged in windmill-ee-private. Previous ee-repo-ref: 8dc3b3d9ec8f9c28b227d36c2a1327b4b2017665 New ee-repo-ref: 9bc5dfb9ce73a2d9b981a1de86eea6aa26688b79 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
c029d6dcde |
fix(security): drop --allow-run from Deno sandbox (GHSA-gj6h-vw66-mr8f) (#10039)
The `// sandbox` annotation (and global nsjail sandboxing) restricted Deno to `--allow-run=git,/usr/bin/chromium`. Both binaries can be coerced into spawning `/bin/sh`, escaping Deno's permission model: - git via hook configs, e.g. `git -c core.fsmonitor='/bin/sh -c <cmd>' status` - chromium via subprocess-launcher flags, e.g. `--renderer-cmd-prefix` / `--gpu-launcher`, pointed at a launcher the script writes into `./` Because the subprocess is spawned by git/chromium — not Deno — it is invisible to Deno's permission checks, giving any user with script-execution permission arbitrary OS command execution (root, in the default worker container). Critically, the Deno runtime is the ONE language never wrapped in nsjail (there is no run.deno.config.proto; every other language has one). So for deno the Deno permission model is the *entire* sandbox — there is no OS-level containment to fall back on, and handing it any subprocess-spawning binary is an unconditional escape regardless of the nsjail setting. Fix: emit no `--allow-run` in the restricted path, denying all subprocess execution. The advisory's alternative (inject `-c core.fsmonitor=false ...`) doesn't apply — the user controls the git/chromium argv, so any injected hardening is overridden. Admins who accept the risk (e.g. puppeteer) can still re-add specific binaries via `DENO_FLAGS`. Verified with both PoCs on a running worker: git and chromium invocations now return `Requires run access to "<bin>"`; the sandbox escapes are closed. Fixes WIN-2151 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
689b20a470 |
fix(security): remove git from Deno sandbox allow-run (GHSA-gj6h-vw66-mr8f) (#10038)
The `// sandbox` annotation restricts Deno to `--allow-run=git,/usr/bin/chromium`. git can be coerced into spawning `/bin/sh` via hook configs such as `git -c core.fsmonitor=<cmd> status`, and that subprocess is spawned by git — not Deno — so it is invisible to Deno's permission model. This let any user with script-execution permission run arbitrary OS commands as root inside the worker, fully defeating the sandbox. The advisory's alternative (injecting `-c core.fsmonitor=false -c core.hooksPath=/dev/null`) does not apply here: the user's own script invokes git directly via `Deno.Command`, so Windmill cannot inject hardening flags into that call. Removing git from the allowlist is the only complete fix. git was originally allowed for git-sync-adjacent use, which no longer needs it. Verified with the advisory PoC: git invocation now returns `Requires run access to "git"` and the sandbox escape is closed. chromium (puppeteer) support is preserved. Fixes WIN-2151 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
87f8d46aaf | fix(ai-agent): align agent_actions_success with agent_actions for mcp and websearch (#9983) | ||
|
Ruben Fiszel
|
43044c2e28 |
feat(pipelines): wm_partition macro for grain-agnostic partition filters (#9950)
* wip: partial work before earlyoom-recovery relaunch
* fix(pipelines): scaffold the strftime {partition} filter idiom (frontend-only)
The DuckDB materialize scaffold and the AI pipeline prompt now teach the
grain-agnostic `WHERE strftime(<ts_col>, '<fmt>') = {partition}` filter instead
of the naive `= TIMESTAMP {partition}` cast. `{partition}` substitutes to the
partition IDENTITY string (`2026-07-05T23`, `2026-W27`, `2026-07`), which is not
a valid DuckDB TIMESTAMP literal for any non-daily grain — so the naive form
raises a `Conversion Error` for hourly/weekly/monthly (only daily parses).
Adds a frontend unit test asserting the hourly scaffold emits the strftime
idiom (`%Y-%m-%dT%H`) for every grain and never scaffolds the naive TIMESTAMP
cast as executable SQL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(pipelines): scope strftime partition idiom to time grains
Review nit: `dynamic` partitioning's identity is a caller-supplied key, not a
timestamp, so `strftime` doesn't apply. Scope the scaffold + AI prompt claim to
time grains and add a `dynamic` example that filters on the user's own key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(pipelines): wm_partition macro for grain-agnostic partition filters
The materialize runtime now injects a `wm_partition(ts)` temp macro as the first
setup statement of a time-partitioned script, so filtering the source to the
active slice is one grain-agnostic line — `WHERE wm_partition(<ts_col>) =
{partition}` — instead of a hand-written `strftime` format the author must keep
in lockstep with the resolver, or the `= TIMESTAMP {partition}` cast that only
parses for daily and Conversion-Errors for hourly/weekly/monthly.
The macro's format comes from `PartitionKind::default_time_format` in
windmill-parser, the same source the EE resolver reads to stamp the `{partition}`
identity, so the two can't drift. `dynamic` partitions get no macro (their
identity is a caller-supplied key → `WHERE <key_col> = {partition}`).
Replaces the earlier 9-line strftime comment block in the scaffold with the
single macro line; AI pipeline prompt and design doc updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(pipelines): verify wm_partition strftime parity vs chrono through real DuckDB
Runs the bundled DuckDB engine in-memory and asserts strftime renders every
grain format (daily/hourly/weekly `%G-W%V`/monthly) byte-for-byte identically to
chrono — the engine the resolver uses to stamp the `{partition}` identity —
across ISO-week year boundaries (2027-01-01 → 2026-W53 etc.). Also proves the
injected `wm_partition` macro buckets the whole slice and that the naive
`TIMESTAMP '<weekly|monthly identity>'` cast Conversion-Errors.
Closes the one cross-engine assumption the pure-Rust/frontend tests couldn't
reach (flagged by CI review for weekly ISO-week rendering).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 0de2412ff0734b11e12ba378c9bcc373ff9ae800
This commit updates the EE repository reference after PR #649 was merged in windmill-ee-private.
Previous ee-repo-ref: ad6c6685689d7741058e7d2c9ecbe95d982e6268
New ee-repo-ref: 0de2412ff0734b11e12ba378c9bcc373ff9ae800
Automated by sync-ee-ref workflow.
* fix(pipelines): classify CREATE TEMP MACRO as a DuckDB prepare-path setup statement
The FFI prepare/diagnostics pass only EXECUTES statements recognized by
is_setup_statement (ATTACH/USE/INSTALL/…); everything else is merely prepared.
`CREATE [OR REPLACE] TEMP MACRO` wasn't recognized, so on a `-- prepare` run of a
partitioned materialize the injected `wm_partition` macro was never created on
the connection, and the later generated `CREATE TABLE … SELECT … WHERE
wm_partition(...)` failed to bind ("function does not exist"). The same latent
gap affected the workspace-macro splicer, which injects TEMP MACRO blocks too.
Classify CREATE [OR REPLACE] TEMP|TEMPORARY MACRO as setup so it's executed
before dependent blocks and excluded from the PrepareQueryResult count
(persistent CREATE MACRO stays a user statement). Adds a prepare-path test that
fails without the fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben Fiszel
|
bf9662172a |
fix: resolve extensionless bun relative imports on windows loader (#9949)
* fix: resolve extensionless bun relative imports on windows loader * fix: resolve local module probe against importer dir not job root * fix: keep node_modules-internal relative imports out of windmill resolver on windows |
||
|
Ruben Fiszel
|
a582e04bf4 | fix: rebuild windows bun loader main.ts filter from forward-slash cdir (#9946) | ||
|
Ruben Fiszel
|
52ce805f61 |
fix(pipelines): dedup guard for keyed merge + deploy-time SCD2 validation (#9936)
Two correctness/validation improvements to managed materialization: 1. A keyed `merge` (`key=<col>`) is delete-by-key + insert-all and does NOT deduplicate its source, so two incoming rows sharing a key both landed under that key — silently breaking the one-row-per-key contract. Codegen now emits an in-transaction guard (same `error(...)` shape as the schema -drift guard) that fails the run when the SELECT returns more than one row for a non-NULL key, naming the key. Authors deduplicate in the SELECT or switch to `append`. NULL keys are exempt, matching the delete's `IN (...)` scope. 2. The two SCD2 misconfigurations that were only caught at run time — `history` without `key=`, and `history` + `// partitioned` — now fail fast at deploy via a shared `MaterializeSpec::validate`, called from `create_script_internal`. The DuckDB executor keeps the same check as a safety net for preview/test runs that never deploy (shared message, no drift). Adds unit tests for the merge guard codegen and for `validate` (all four cases), and updates docs/ducklake-materialization.md and docs/pipelines-vs-dbt.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
182b10b2ad |
render FFI errors with real newlines and no stray quoting (#9932)
* fix(duckdb): render FFI errors with real newlines and no stray quoting The DuckDB FFI returns errors as ERROR <json-encoded-message>, so the executor was surfacing the serde_json-escaped form (wrapping quotes, literal \\n). Multi-line errors like the write-audit-publish data-test breakdown were unreadable. Decode the JSON string back to the raw message at both FFI error sites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: tighten decode_ffi_error comment to the invariant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
377c02ec47 |
feat(pipelines): on_schema_change write guardrails + data_test deploy validation (#9930)
* feat(pipelines): on_schema_change write guardrails + data_test deploy validation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to fa7ac11c1e0ab39e84a0c18973ba427a240933ca This commit updates the EE repository reference after PR #647 was merged in windmill-ee-private. Previous ee-repo-ref: bd23b2a904cb2e6554c7ff209ff8adb9d91775d1 New ee-repo-ref: fa7ac11c1e0ab39e84a0c18973ba427a240933ca Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
39eb9de1bc |
feat(pipelines): fork data environments for ducklake materialization (dev data) (#9915)
* feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): per-lake isolated/shared choice at fork creation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): chain-aware defer discovery + per-location fork namespace registry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): cleanup deletes fork data from the registered storage identity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): collapse fork data-path segment to one component (slash-safe ids) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): run fork ducklake cleanup inline in delete_workspace Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter - extract require_prod_admin_for_dev_workspace, used by both delete_workspace and drop_forked_ducklake_namespaces so the gates cannot drift - key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL re-registers its namespaces - filter listed object locations to the segment boundary before deletion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its parent with an empty ancestor chain while its cloned config still points at the shared lake. Key the isolation gate on the wm-fork- prefix as well as the chain (mirroring workspace_is_fork): orphaned forks get the write redirect, registration and cleanup with zero ancestors (no defer), and keep their 'fork' graph chips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace Chain position alone classified the last ancestor as a root, but an orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the chain the same way while its data lives in its fork namespace — its descendants' defer views bound the dead root's lake instead. Key the root-vs-fork decision on the wm-fork- prefix too, matching the resolution gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions - fork creation strips cloned fork_behavior stamps before applying the request's shared_ducklakes list: sharing is a per-creation choice, a fork of a shared fork defaults back to isolated - fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the durable cleanup ledger and outlive the workspace when physical cleanup fails post-commit; fork creation retries leftover rows for the reused id and refuses to create while a metadata schema still cannot be dropped (data-file leftovers alone are inert once the schema is gone and are swept by the next successful same-prefix cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources - ledger rows gain a schema_dropped phase flag: set when the schema drop succeeded but data cleanup failed, so later retries skip the schema phase and need no catalog credentials at all; registration resets it on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the schema - retry-path $res: resolution falls back to the workspace being forked (the deleted fork's resources were clones of a parent's); live paths (delete_workspace prepare, drop endpoint) pass no fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph A failed materialization must not disguise a physically existing fork table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the table, so reads hit fork data while the graph claims parent defer. - record_mat upsert preserves the last committed snapshot_id on failure - defer discovery and graph chips treat fork rows with a committed snapshot as fork-owned even when status is failed - inspect_fork_catalog also lists live fork tables (same round trip) and the defer list is filtered against them — covers rows recorded before this fix and tables created by raw SQL - drop stale FK-cascade wording in the design doc and sidebar comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip The workspace-settings ducklake editor had no fork awareness: no reminder of each lake's isolated/shared choice and no warning about what edits mean in a fork. It also rebuilt each lake explicitly on save, silently dropping fork_behavior — any settings save in a shared fork flipped the lake back to isolated. - fork detection mirrors the backend gate (parent link or wm-fork- prefix) - info banner explaining isolated vs shared semantics in a fork - per-lake chip (emerald 'isolated' / amber 'shared with parent') with tooltips, matching the pipeline graph chip colors - fork_behavior added to DucklakeSettingsType and preserved through convertDucklakeSettingsToBackend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
d4b4374de8 |
feat(pipelines): capture violating-row samples for data tests (#9919)
* feat(pipelines): capture violating-row samples for data tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): byte-accurate sample cap and leaf-level payload sanitize Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref to WAP guard probe adaptation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: WAP failures are counts-only — samples exist only on commit-then-test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: qualify where sample row data appears — job result and failed-job log line Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: error handlers receive the full result incl. samples, like any failed job Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 80d309edebb899e36a3bdcdf4ea73c4db070534d This commit updates the EE repository reference after PR #646 was merged in windmill-ee-private. Previous ee-repo-ref: 16e916bf11f26381920560b55771fce693e668c6 New ee-repo-ref: 80d309edebb899e36a3bdcdf4ea73c4db070534d Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
dce247c6d2 |
feat(pipeline): write-audit-publish for materialization data tests (#9911)
* feat(pipeline): write-audit-publish for materialization data tests (EE) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: EE worktree E0583 troubleshooting + duckdb feature check row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: clarify EE symlink example (absolute target, EE repo layout) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipeline): move bootstrap DDL inside guarded WAP transaction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipeline): move WAP guard SQL builder into EE, OSS keeps placement only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref to EE branch rebased on EE main Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style: reword test comment as current invariant per AGENTS.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump ee-repo-ref (EE module doc update) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipeline): OSS emits typed materialize plan, EE owns WAP transform Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: make rewrite assertion build-aware; refresh oss module doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c This commit updates the EE repository reference after PR #644 was merged in windmill-ee-private. Previous ee-repo-ref: 63cabae75329429f647e01083936d70f8197dc9e New ee-repo-ref: 7be0bad1a6d6b5c3a107c0a2cd4bf003c36ec34c Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasa
|
a368d49bd8 |
feat(ai-agent): support reasoning effort in AI agent workflow steps (#9886)
* feat(ai-agent): support reasoning effort in AI agent workflow steps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): round-trip native Anthropic thinking blocks and fix DeepSeek/Mistral reasoning Address review: native Anthropic now captures the signed thinking block during streaming and replays it before tool_use across iterations (prevents a 400 on multi-turn tool use). DeepSeek 'off' sends thinking:{type:disabled} instead of the rejected reasoning_effort:none, and Mistral drops temperature when reasoning is on. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-agent): move reasoning effort into the provider/model selector Store reasoning_effort on ProviderConfig (next to the model) instead of a separate flow arg, and render the selector inside AIProviderPicker under the model dropdown. Add an explicit 'off' option on models that disable reasoning by omission (e.g. Claude), so reasoning can always be turned off from the UI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-agent): use DropdownV2 for reasoning effort, matching copilot chat Replace the Select combobox with the same DropdownV2 action-menu the copilot chat reasoning selector uses. Each option carries an action instead of a bound value, so click selection is unambiguous and there is no typeahead/sentinel-value mismatch on the off/default entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ai-agent): regenerate system prompts for ProviderConfig.reasoning_effort Refresh system_prompts/auto-generated and cli skills.gen after adding reasoning_effort to the OpenFlow ProviderConfig schema (check-freshness). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): clear stale reasoning effort on model change; dedup bedrock reasoning folding Address cubic review: (P1) the reasoning picker now clears the stored effort when the newly selected model doesn't accept it (e.g. carrying 'xhigh' from Opus onto a model that tops out at 'high'), not only when the model can't reason at all. (P3) the proxy's accumulate_reasoning_delta now delegates to the shared bedrock_stream_event_to_reasoning_delta so worker and proxy folding can't drift. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-agent): stream reasoning summary and show a thinking affordance in flow chat Add StreamingEvent::ReasoningTokenDelta, emitted from every worker reasoning path (Anthropic native thinking deltas, Bedrock, Gemini thought parts, OpenAI-compatible reasoning_content, OpenAI Responses reasoning_summary_text with summary:auto). The flow chat parses it and renders a collapsible 'Thinking' affordance on the assistant message (thinking tokens bill regardless of display, so surfacing the summary is billing-neutral). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): persist streamed reasoning onto the finished chat message Reasoning isn't stored server-side, so the completion re-poll (which swaps temp messages for the persisted ones) was dropping the streamed thinking summary. Carry it onto the final assistant message so the 'Thought process' affordance survives the run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai-agent): match flow-chat thinking box to the copilot chat reasoning UI Replace the plain <details> thinking affordance with the same bordered, collapsible reasoning box the copilot chat uses (chevron + Brain/spinner + 'Thinking' header, markdown body, expand-while-streaming/collapse-on-answer). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): attribute streamed reasoning per turn by content; drop duplicated comment Address review: the completion-poll carry-over now matches each temp assistant turn's thinking summary to its persisted message by content, so a multi-turn response (reasoning -> tool call -> final answer) no longer misattributes an earlier turn's thinking to the final answer or drops intermediate turns. Also removes a leftover duplicated comment block above the AIReasoningEffortPicker effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): address review round 2 (carry-over edges, off-token validity, aria, test) cubic round 2: (1) reasoning carry-over now consumes temp turns in order verifying content, so identical/empty-content multi-turn responses attribute thinking correctly and reasoning-only turns aren't dropped; (2) the picker's stale-value check only accepts the off token when the model can actually disable reasoning; (3) add aria-expanded to the Thinking toggle; (4) add a test for the failed tool_result path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): add bottom margin below the flow-chat thinking box Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): don't request OpenAI reasoning summary, matching the copilot chat OpenAI gates reasoning summaries behind org verification, so requesting summary: auto would 400 for unverified orgs. The copilot chat requests effort only and never asks for a summary; align the worker with it (reasoning: { effort }) and drop the now-unreachable summary-delta parsing. OpenAI reasoning no longer streams a summary in flow chat (consistent with the copilot); Anthropic/Bedrock/Gemini/DeepSeek reasoning display is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): scope reasoning carry-over to newly persisted messages cubic round 3: matching by content over the full history could attach a new turn's reasoning to an older message with identical text. Restrict eligible targets to the messages just fetched for this response (via afterSeq), so historical turns are never touched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): carry reasoning newest-first instead of gating on the final poll cubic round 4: gating the carry-over on the final poll's filteredResponse dropped reasoning for messages already fetched by an earlier streaming poll (their id is excluded by afterSeq). Walk persisted newest-first and consume the newest matching pending summary, stopping once summaries run out. This response's turns are always at the end, so they claim their own reasoning (P1) before older history is reached (P2), regardless of which poll persisted them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ai-agent): drop flow-chat reasoning display, keep backend + effort picker The chat-side thinking box relied on non-deterministically matching streamed (ephemeral) reasoning back onto persisted messages, which kept spawning edge cases. Remove the flow-chat display entirely (ChatMessage box, FlowChatManager carry-over/threading, parseStreamDeltas reasoning) and keep the sound backend: per-provider reasoning-effort requests, thinking-block round-trips for tool calls, and ReasoningTokenDelta streaming. A display can be built on top later, deterministically (e.g. once the stream carries the persisted message id). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-agent): include reasoning_effort in default-config compare; document reasoning_token_delta Codex/Pi nits: isSameAsStoredConfig now compares reasoning_effort so the 'use as personal default' toggle reflects effort-only changes; openflow streaming-events doc lists the reasoning_token_delta event (regenerated auto prompts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
d600c7ecfe |
fix(ai): route Azure Foundry Claude models via Anthropic Messages API (#9908)
* fix(ai): route Azure Foundry Claude models via Anthropic Messages API Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai): keep explicit Azure OpenAI deployment base URLs intact build_azure_openai_url only appends /openai/v1 for a bare resource root; any base with an explicit path (e.g. .../openai/deployments/<id>) is preserved. Adds a regression test and a unit test for usesAnthropicMessagesApi. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(ai): enable Claude extended thinking on Azure Foundry Route azure_foundry+Claude through the Anthropic reasoning branch (adaptive thinking + output_config.effort) instead of the gpt/o gate, and recognize claude-sonnet-5. Live-verified: sonnet-5 and opus-4-8 on Foundry accept the low/medium/high/xhigh/max ladder and render summarized thinking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
84141add1d |
feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890)
* feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud |
||
|
Ruben Fiszel
|
d15033cde6 |
fix: invalidate bun bundle cache on transitive relative-import changes (#9891)
* fix: invalidate bun bundle cache on transitive relative-import changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: do not memoize transient fetch errors in bundle-key import cache Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: use regular comment on lazy_static block (deny unused_doc_comments) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: align bundle-key import version selection with loader content endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
b883adbc00 |
fix(duckdb): auto-declare partition arg for // partitioned scripts (#9878)
* fix(duckdb): auto-declare the partition arg for // partitioned scripts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(cli): pipeline run --arg to pass plain run args to cascade scripts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
5a661279a3 |
feat(pipelines): add managed SCD2 history materialize strategy (#9850)
* feat(pipelines): add managed SCD2 history materialize strategy `// materialize ducklake://... key=<col> history [track=...]` (alias: `scd2`) upgrades the keyed merge to SCD type 2: diff the current snapshot against live rows, close changed versions (valid_to/is_current) and open new ones in one transaction, keeping full history. Adds a consumer-convenience <dim>_current view; effective-dated joins via native ASOF JOIN >= valid_from. Managed, so // data_test and schema capture work (unlike manual mode). Non-partitioned v1, soft-delete on absence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pipelines): document scd2 track= spacing, reserved _current suffix, schema-freeze Addresses non-blocking CI-review nits on the new SCD2 public surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): null-safe scd2 key matching + create _current view inside txn Addresses CI review: (1) Codex P1 — NULL natural keys were flagged as changed but silently dropped because `key IN (...)` never matches NULL; close/open now match with `IS NOT DISTINCT FROM` via correlated EXISTS. (2) cubic P2 — the `<dim>_current` view was created after COMMIT and CREATE VIEW advances the DuckLake snapshot, so the summary recorded the view's snapshot instead of the data write; the view is now created inside the write transaction. Validated both against a real DuckLake (NULL key materialized; one snapshot per run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): create scd2 _current view with IF NOT EXISTS to keep no-change runs no-op Addresses CI review (Codex P2): CREATE OR REPLACE VIEW advances the DuckLake snapshot every run, so an unchanged rerun still minted/recorded a snapshot. The view definition is static, so IF NOT EXISTS creates it once (folded into the first data-write snapshot) and is a true no-op thereafter — verified an unchanged rerun keeps max(snapshot_id) constant. Also softens the reserved-name collision: IF NOT EXISTS skips silently instead of erroring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): add scd2 deletes=close (hard-delete-close) Opt-in `deletes=close` closes the current version of a key that disappears from the snapshot (dbt's hard_deletes=close); default stays soft-delete. Codegen adds a vanished-key temp set (current keys EXCEPT snapshot keys) + a second null-safe close UPDATE with no reopen; a reappearing key opens a fresh version (validity gap = correct SCD2). Wired through both parsers with parity fixtures/tests, worker derivation, unit + codegen tests, and docs. Verified end-to-end against a real DuckLake incl. delete-close + reactivation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): align materialize deploy precedence warning with runtime (scd2>append>merge) The deploy-time conflict warning only knew append>key, so warned 'append wins' while the runtime (duckdb_executor) runs SCD2 (history wins). Warn for history+append (history wins, append ignored) before the append+key case, mirroring the runtime strategy precedence. (Pi review P2.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): register scd2 _current view as a produced asset for cascade dispatch The docs present the companion <dim>_current view as a subscribable produced asset (// on ducklake://.../<dim>_current), but deploy registered only the base table as a write asset, so a subscriber on the view would never be dispatched (the cascade fans out from deploy-time asset rows). Register <dim>_current as a produced write asset when scd2 so those subscribers fire. (Codex review P1.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): don't register _current asset for manual+history (no view created) Manual mode short-circuits before the scd2 codegen, so no <dim>_current view is created; gate the produced-asset registration on !manual so a contradictory // materialize manual ... history doesn't leave a false write edge dispatching subscribers on a nonexistent view. (Codex review P2.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
68bf0daf58 |
feat(ansible): support repo-provided ansible.cfg in delegate_to_git_repo (#9851)
* feat(ansible): support repo-provided ansible.cfg in delegate_to_git_repo Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): accept colon delimiter and collections_paths alias in cfg parser Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
bf6be967fa |
fix: honor verify-ca/verify-full sslmode for postgres connections (#9835)
* fix: enforce tls verification for postgres verify-ca/verify-full sslmode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: make PG_ACCEPT_INVALID_CERTS value-based and keep cache key well-formed Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: grandfather existing postgres resources via per-resource trust_cert flag Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope pg trust_cert migration to resource data, drop schema patch Hub resource type sync (windmill cache-rt + startup SYNC_CACHED_RT) only touches the admins workspace and is opt-in, so the schema is left to the hub; the migration just grandfathers existing resource values so the upgrade is non-breaking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: replace pg trust_cert with verify-*-scoped accept_invalid_certs, drop migration Per-resource accept_invalid_certs (default false for new resources) replaces the trust_cert flag and grandfather migration. It only applies to verify-ca/verify-full; unset falls back to legacy behavior (verify only when a root cert is present) so existing and git-synced resources are not broken on upgrade. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: warn in job logs when a verify-* postgres resource skips cert verification Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ade74b297f |
feat: capture managed-materialize output schema as asset metadata (#2a) (#9812)
* feat: capture managed-materialize output schema as asset metadata (#2a) After a managed `// materialize` run, capture the producer's output schema via a DESCRIBE folded into the existing one-row summary read (no extra round-trip) and persist it in a new versioned `materialized_asset_schema` sidecar table. This is the producer-side capture that pipeline parity gap #2b (save-time consumer-ref contract enforcement) will read back. - materialized_asset_schema sidecar (asset-level grain), versioned: a new version row is inserted only when the captured column set changes. - output_schema column added to the materialize summary codegen. - worker extracts + records the schema on a successful materialize. - /assets/asset_schemas read endpoint exposing the evolution history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on schema capture (partition col, order, status gate) - exclude the synthetic `_wm_partition` column from the captured schema for partitioned assets, so the recorded contract is the producer's logical output, not Windmill's storage detail (claude/cubic P1). - make the captured column list explicitly ordered (`row_number()` over the DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder columns and spuriously bump the schema version (cubic P2). - gate the API `record_materialization` schema upsert on a `Materialized` status, so a failed/running write (or a client attaching a schema to one) can't advance the schema history (cubic P2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Codex review (manual-mode schema gate + auth contract docs) - gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a `// materialize manual` run — whose result is the user's own query output — can't persist a caller-shaped `output_schema` into materialized_asset_schema (Codex P2). Verified e2e: a manual run returning a fabricated `output_schema:[{injected,EVIL}]` records the partition but writes no schema version, while the managed path still captures normally. - document the authorization contract on the new public `record_asset_schema` and `list_asset_schemas` helpers: they perform no access control (mirroring the materialized_partition siblings) and require callers to pass a workspace-authorized executor (Codex P1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): schema-history tab on the ducklake asset node (#2a) Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema versions persisted by the materialize run. Master-detail (mirrors the History tab): the version list (newest first, newest auto-selected) shows column count + snapshot + capture time; selecting a version renders its column/type table. Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling PartitionStatusGrid convention (these materialization endpoints are not in the generated client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: schema tab is strategy-aware (history vs fixed schema) Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema table, so their schema is pinned at first materialize and the "history" framing is degenerate (always one version). - backend: surface the managed `materialize_strategy` (`replace`/`append`/ `merge`) on the asset-graph runnable node, alongside the existing `partition_kind` (same parse-from-annotation path). - frontend: the pipeline page derives `schemaCanEvolve` for the selected asset from its write-producer (`replace` && not partitioned) and threads it to the Schema tab. Evolvable → master-detail version history; fixed → a single current-schema table with a short "schema is fixed" note. Unknown defaults to evolvable so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: schemaCanEvolve fails open on unknown producer strategy Previously a producer present but missing `materialize_strategy` (e.g. a draft-overlay runnable, synthesized without the field) fell through to canEvolve=false, hiding captured history behind the fixed-schema view — contradicting the "unknown defaults to evolvable" intent. Now the fixed view shows only when *every* producer is a known insert-style write (append/merge, or partitioned replace); any producer with unknown (missing) strategy is treated as evolvable, so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
40110bc715 |
fix: skipped suspend step no longer parks the flow forever (#9821)
* fix: skipped suspend step no longer parks the flow forever A flow step that declares a `suspend` (approval) but is skipped via `skip_if` was leaving the flow stuck waiting for a resume that would never arrive. Suspend gates the *next* step: before pushing step N, `needs_resume` checks whether step N-1 declared a non-zero `suspend` and finished as `Success`. A step skipped via `skip_if` is also recorded as `FlowStatusModule::Success` (with `skipped: true`), so `needs_resume` treated a skipped approval gate as a real one and parked the flow waiting for an event that nothing ever sends — until the suspend timeout (up to 24h). The skip is most visible when the skipped suspend step is followed by a branch/subflow: the flow appears stuck on the *following* predicate node with a generic resume button, while none of the branch/subflow steps ran. Fix: honor the `skipped` flag in `needs_resume` and do not gate the next step on a suspend that was skipped. Adds regression test `skipped_suspend_step_does_not_block_next_step` (times out without the fix, completes with it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: reword regression test comment as a current invariant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
0dbd9c1231 |
perf: eliminate dual-connection DB pool contention across worker, queue, and api (#9798)
* perf: eliminate dual-connection DB pool contention across worker, queue, and api Reuse the held transaction (or move pool reads before begin()) instead of checking out a second pool connection while a tx is open, extending the fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus several server-pool API handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pass owned pool to get_email_from_permissioned_as in http trigger handler The generified signature takes impl PgExecutor; the http trigger handler passed &db where db is already &DB, yielding &&Pool which does not impl PgExecutor (only surfaced under the full feature set in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint Addresses review of the dual-connection sweep: - worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The caller swallows the error and keeps using tx, so a DB read failure must not leave the outer transaction aborted (it would fail the later commit). Matches the previous pool-read semantics. - Revert reads that were moved onto an RLS (user_db) transaction back to the non-RLS pool, since RLS row-visibility/role context can change results: push_scheduled_job (email/tag/settings lookups; reachable with a user_db tx from api-schedule/api-flows), push_inner native-retry dedicated_worker routing (RLS isolation variants), resources.rs app-namespace folder auto-create (non-admins must not be blocked), and the script archive/delete UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read Adds a worker-crate failpoints feature and a data-driven hook: when the stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still completes (iteration marked failed) — which only holds if the savepoint keeps the outer status-update transaction committable. Without the savepoint the abort would poison the outer tx and the job would never complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
b28f974e50 |
fix: opt out of Deno minimum-dependency-age for private npm registries (#9802)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
11d83ab1ec |
fix(python): serialize concurrent installs into shared wheel cache dir (#9787)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6c71c33470 |
fix(python): re-verify wheel RECORD on local cache reuse (once per worker) (#9775)
A corrupt local pip cache entry (e.g. wmill==1.739.0 missing s3_reader.py after out-of-band file loss on a persistent/shared cache volume) was trusted indefinitely: handle_python_reqs only checked the .valid.windmill marker on the reuse fast path. verify_wheel_record already guarded the install and S3-pull paths, but never ran again once the marker existed. Re-verify the wheel RECORD on the first reuse of each cache entry per worker process and repair (wipe + reinstall) on failure. A VERIFIED_VENVS in-memory set makes every subsequent reuse skip the scan, so the warm-cache hot path keeps paying only its original single stat. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
962758c02d |
fix: pass SSL cert env vars to uv python install (#9790)
`install_python` cleared the subprocess environment via `env_clear()` and forwarded only a subset of variables, omitting `SSL_CERT_FILE` (from `PY_INDEX_CERT`/`PIP_INDEX_CERT`) and `UV_NATIVE_TLS` (from `PY_NATIVE_CERT`). This caused `invalid peer certificate: UnknownIssuer` errors when downloading managed Python runtimes in environments with corporate/private CAs. Forward both variables, mirroring the sibling `find_python` method and the pip install path in `python_executor.rs`. Fixes WIN-2100 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
754cae956a |
fix: use transaction for parallel_monitor_lock DELETE in last-iteration path (#9789)
In the last-iteration path of a parallel for-loop (nindex == len), the DELETE FROM parallel_monitor_lock ran on the pool (db) while the transaction tx (begun earlier) was still held. This dual-connection pattern requires 2 simultaneous connections from the per-worker pool (default max 5) and can trigger "pool timed out while waiting for an open connection" under concurrent load. Run the DELETE on the held transaction (&mut *tx) instead, matching the fix PR #7861 applied to other queries in this file. The transaction is committed shortly after, so including the DELETE in it is safe and consistent with the non-last-iteration path. Fixes WIN-2099 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
f6998ec54c |
feat: data tests for ducklake pipeline materialization (#9708)
* feat: data tests for ducklake pipeline materialization Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data_test count badge on pipeline graph nodes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: surface annotation badges (incl. data_test) on deployed pipeline nodes Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): keep materialize output edge when editing the producer in the pipeline graph The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: run all data tests in one pass with a structured per-test result Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data-test pass/fail checklist in the job result DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): move data-test badge onto the producer→asset edge with run status The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): render custom data-test scripts as their own clickable graph nodes A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): anchor edge badge on routed path + a11y text on test icons Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests Address review (cubic) findings: - P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact. - P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script. - P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail loud if fewer data-test outcomes recovered than declared Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: relationships data test same-lake reuse + schema-qualified target quoting Address Codex/Pi review (two P1s in the relationships codegen): - A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references. - A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator. Adds tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target Address Codex review nits (both P2): - resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch. - AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
88fca6a8c1 |
fix: enforce containment of python module dir for preview jobs (#9704)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
fa3596885b |
fix: allow SQL args in managed // materialize scripts (#9733)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasa
|
e403f92d7e |
fix: enforce job_dir containment when writing module files (#9703)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
ace7b68a28 |
fix: sanitize git credentials from ansible executor errors and logs (#9697)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
3ebf24359d | feat: ducklake materialization for data pipelines (#9689) | ||
|
hugocasa
|
c1f31c0e47 |
fix(backend): validate ansible vault_id entries before config generation (#9681)
* fix(backend): validate ansible vault_id entries before config generation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): cover ansible.cfg write-boundary vault_id validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 Pyra
|
ab3bc97cd9 |
feat(python, windows): enable S3 to cache wheels (#5199)
* add .minio-data to .gitignore
* feat(python): make S3 cache arch specific
Current schema:
S3-Bucket
├── python_311
│ ├── wheel==1.0
│ └── wheel2==1.0
└── python_312
├── wheel==1.0
└── wheel2==1.0
New schema:
S3-Bucket
├── linux_aarch64
│ └── ...
└── linux_x86_64
├── python_311
│ ├── wheel==1.0
│ └── wheel2==1.0
└── python_312
├── wheel==1.0
└── wheel2==1.0
* remove .minio-data from .gitignore
* remove unneeded tracing::error
* feat(python, windows): enable S3 to cache wheels
* fix(python, windows): drop residual unix cfg gates on S3 cache items
Post-merge, PIPTAR_UPLOAD_CHANNEL and its call sites were already
cross-platform, but the types/functions they reference (PiptarUploadTask,
handle_piptar_uploads, pull_from_tar, OBJECT_STORE_SETTINGS) remained
unix-gated, breaking the Windows build.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
cafb473494 |
fix(python): split PIP_TRUSTED_HOST by whitespace to support multiple hosts (#9675)
* fix(python): split PIP_TRUSTED_HOST by whitespace for multiple hosts When PIP_TRUSTED_HOST contains multiple space-separated hostnames, the whole string was passed as a single --trusted-host argument (--trusted-host "host1 host2") rather than one flag per host. This matches pip's documented PIP_TRUSTED_HOST convention by emitting a separate --trusted-host for each host, mirroring the existing pip_extra_index_url handling. Fixes WIN-2077 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(python): use shell word-splitting for nsjail trusted-host args Address review feedback: the previous tr/sed pipeline only split on single spaces, diverging from the Rust split_whitespace() paths. Repeated or leading/trailing spaces produced empty --trusted-host flags and tab-separated hosts were not split. Use an unquoted for-loop over $TRUSTED_HOST so the shell's own IFS word-splitting handles arbitrary whitespace and skips empty fields, matching the non-nsjail paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
7e4df02bd6 |
fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures (#9662)
* fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures When a worker is OOM-killed mid-step, the zombie job handler fails the step via handle_job_error with unrecoverable=true. update_flow_status_after_job_completion had `false if unrecoverable => false`, which silently completed the flow with the error and skipped the flow's failure module (error handler). It would also have pinned the failure module to the dead worker via same_worker. Unrecoverable failures now route to the failure module instead of being retried or silently dropped, and the error-handler step is pushed as a regular queued job that any live worker can pick up. Fixes WIN-2070 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: skip retry on unrecoverable flow failures, add retry-skip regression test Address review: the failure-module-on-unrecoverable change must also bypass the per-step retry policy in push_next_flow_job, otherwise an OOM/zombie-killed step with a retry config would be retried instead of routing to the error handler. Gate the retry evaluation on !unrecoverable and add a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: add sqlx offline cache for new flow-step zombie test query Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: route unrecoverable continue_on_error step failures to the error handler Addresses Codex/Pi review (P1): with continue_on_error on the failed step, the step counter was advanced before the unrecoverable decision branch, so push_next_flow_job pushed the next normal step instead of the failure module — hiding the worker death and letting the flow complete successfully. - Do not advance the step counter (inc) for an unrecoverable continue_on_error failure. - Let the Failure arm in push_next_flow_job route to the failure step even on a continue_on_error module when unrecoverable. - Add a regression test (a[continue_on_error] -> b + failure_module): asserts the failure module runs and step b does not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
7155a0bb96 |
feat: Data Pipelines alpha (#9193)
* feat: add workspace asset graph view Workspace-wide canvas of assets and their producer/consumer scripts, reachable from the assets page. Left-to-right layered layout via d3-dag sugiyama, rendered with @xyflow/svelte (same stack as the flow editor). GET /w/:ws/assets/graph returns deduped nodes + edges. Follow-ups: filters (kind/folder/search), node detail drawer, inline script edit from a clicked node. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * all * all * update * all * all * all * feat(pipeline): output-kind picker and per-(lang, output) templates Add a third stage to PipelineInsertMenu that asks what kind of asset the new script will produce (datatable / ducklake / s3 parquet / s3 object / none). The picked kind drives a real wmill SDK skeleton — typed datatable inserts, ducklake CREATE+INSERT, s3 parquet COPY, etc. — with the upstream asset auto-wired as the input source when added from an asset node. Reorder languages to bun → duckdb → python → sql so data-shaped languages surface first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * chore(main): release 1.693.4 (#8994) * chore(main): release 1.693.4 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: ansible delegate_to_git_repo install_requirements, dynamic fields, --limit (#8997) * feat: ansible delegate_to_git_repo install_requirements, dynamic fields, --limit Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: include .yaml variants in collections/roles requirements lookup Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): only preserve case for raw-app runnableIds, not app/flow summaries (#9000) * fix(cli): only preserve case for raw-app runnableIds, not app/flow summaries PR #8940 stopped lowercasing in sanitizeForFilesystem to fix #8939, where a raw-app runnableId like CamelCaseTSRunnable produced a CamelCase YAML metadata file but a lowercased code file, making them desync and register as duplicate runnables on push. That fix overshot. sanitizeForFilesystem is also reached by newPathAssigner, which serves normal apps and flows where the input is the script's human summary ("Get Users Data") rather than an identifier. There the on-disk filename is the only artifact — there's no companion YAML to keep in sync — so lowercasing was the right behavior. Removing it changed both the on-disk filename and the !inline reference in app.yaml / flow.yaml from get_users_data.inline_script.ts to Get_Users_Data.inline_script.ts on the next pull, surfacing as unwanted case churn for users updating to 1.693.x. Add a preserveCase option to sanitizeForFilesystem (default false → lowercase). newRawAppPathAssigner opts in; newPathAssigner stays on the default. Update unit tests accordingly and add an end-to-end raw-app round-trip in raw_app_sync.test.ts that pushes a CamelCase backend runnable, pulls it back, and asserts both YAML and code file preserve case with no lowercase orphan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(cli): use readdir for exact-case orphan check on Windows The CamelCase round-trip test used fileExists("camelcasetsrunnable.ts") to assert no lowercase orphan was produced, which false-positives on Windows since the filesystem is case-insensitive and resolves the lookup to the existing CamelCaseTSRunnable.ts. Switch to readdir + toContain so the exact on-disk casing is compared identically on Linux and Windows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): wmill-lock.yaml auto-fill + --rehash-only + path-prefix dedup (#8978) * fix(cli): canonical lockfile hashes + lock upgrade migration to v3 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): use __app_hash subpath in rehash missing-entry check Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): run sync pull lockfile auto-fill regardless of changes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: regenerate system prompts for new lock and rehash-only commands Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on lock upgrade Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): drop v3 marker; always run fallback; fail-fast on unknown lockfile version Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): drop yaml-round-trip legacy hash variant; recover via --rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): include legacy hash in script push staleness warning check Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * revert(cli): drop canonical hash formula; keep raw-bytes hashing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * perf(cli): reuse change-tracker map for sync pull lockfile auto-fill Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): pin lockfile hash + yaml format and cover regression cases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): byte-stable snapshot tests for flow.yaml format Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(cli): add app and script-metadata yaml snapshot fixtures Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address claude review on rehash-only Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): factorize script-path to remote-path derivation Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address claude + cubic review (dry-run mutation, rehash short-circuit) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(cli): make rehash a subcommand and factorize fs walks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): normalize line endings in yaml snapshot tests for windows ci Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(cli): address review feedback on rehash + auto-fill - Flat-layout scripts now clearGlobalLock before rehash write so legacy ./-prefixed duplicates get cleaned up (matches flow/app behavior). - Add MalformedLockfileError; sync pull auto-fill re-throws it alongside UnknownLockVersionError instead of silently warning + continuing. - Document the legacy step-removal false-negative in isFlowDirectlyStale / isAppDirectlyStale and the categorizeLocalFiles ignore-filter invariant. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * fix: use otel.status_message for OTLP Status.message on failed jobs (#8995) tracing-opentelemetry only recognizes otel.status_code and otel.status_message as fields that map to the OTLP Status proto. The previously-used otel.status_description fell through to the generic attribute recorder, leaving Status.message unset and preventing OTLP consumers from filtering spans on error status. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: route email trigger path through standard info channel (#8996) * docs(skill): document email triggers and S3 attachments Add an "Email triggers" section to the triggers skill covering the local-part config, the parsed_email/raw_email/email_extra_args payload, the URL-style extras convention, where to find trigger_path (only with a preprocessor, at event.trigger_path), and — most importantly — that binary attachments are uploaded to the workspace S3 bucket and surface as `{ s3: "windmill_emails/<job_id>/attachments/<filename>" }`. Scripts must use wmill.loadS3File / wmill.load_s3_file to read them. Also pulls EmailTrigger into the schema mappings so a real `email_trigger.schema.yaml` is generated, and adds Email/Azure to the trigger kinds list in the CLI agent guidance. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref for email trigger path fix Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 26184ab7a4aadfc529dcedf038aa08d36c7ad381 This commit updates the EE repository reference after PR #553 was merged in windmill-ee-private. Previous ee-repo-ref: 318a46897a605dc9be3817901f35ba5a99a0a525 New ee-repo-ref: 26184ab7a4aadfc529dcedf038aa08d36c7ad381 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * update git sync version to 1.693.5 * fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999) * fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(pg): wrap encoder errors with arg context, add fallback test Followups on #8999 review: - Wrap rust-postgres "error serializing parameter N" failures with the arg name, JSON value kind, and asserted Postgres type plus a hint about an explicit cast — so users see actionable context instead of an opaque WrongType. - Drift-prevention meta-test: assert otyp_to_pg_type and convert_val agree on the Type for every recognised arg_t when the JSON value matches its natural Rust kind. Catches future drift if either side changes. - Integration test for the prepare + query_raw fallback path: confirms unrecognised arg_t (custom enum) is routed through prepare and the server-resolved type appears in the failure surface — flips into a test failure if a regression accidentally routes unrecognised types through query_typed_raw. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add otyp_inferred flag + regex-based placeholder renumbering Two follow-ups from the review of #8999: 1. **Issue #1 (Number/Bool + explicit text decl in WHERE)** Add `Arg::otyp_inferred: bool` to the parser. The PG SQL parser sets it `true` only at the "no info → fall back to text" site (bare `$N`, no inline cast, no `-- $N (TYPE)` decl). All other arg sources keep it `false`. In `convert_val` this flag distinguishes: - explicit text-like target (`-- $1 (text)` or `$1::text`) — coerce `Bool`/`Number` → `Box<String>` so `WHERE text_col = $1` works (`text = text` operator). Pre-#8988 behaviour, restored. - parser-default text (bare `$N`) — bind the value's natural Rust type so the regression case (`Value::Bool` against a real `bool` column via `CAST AS bool`) keeps working. `Arg` is in `windmill-parser`; the new field has `#[serde(default)]` so persisted signatures stay backward-compatible. 2. **Issue #4 ($5/$50 substring rewrite collision)** Replace the per-index `String::replace` chain (which turned `$50` into `$10` when oidx=5 was processed first) with a single regex pass. `\d+` is greedy, so `$5` and `$50` match as distinct units; indices outside the mapping are left intact. 3. Tests: - parser: `test_parse_pgsql_otyp_inferred_flag` covers bare/inline- cast/decl/mixed shapes. - executor unit: `convert_val_bool_against_every_arg_t` and `convert_val_*_number_*` split each text-like target into explicit vs inferred expectations. - executor unit: `renumber_sparse_placeholders_no_collision`. - integration: `test_postgresql_arg_type_combinations` adds 4 cases covering decl(text)+Number/Bool in WHERE, bare $1+Bool, and sparse positional args ($5/$50). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg+sdk): enum support, extended String arms, position-aware $N rewrite, SDK quality Backend: 1. **`AnyTextValue` ToSql/FromSql wrapper**: vanilla `tokio_postgres`'s `ToSql for String` / `FromSql for String` reject `Kind::Enum` and `Kind::Domain` even though the wire format is plain UTF-8. The wrapper accepts those kinds in both directions. End result: explicit `$1::my_enum` / `CAST($1 AS my_enum)` casts now round-trip without the ugly `CAST($1::text AS my_enum)` workaround, AND `SELECT enum_col` results come back as JSON strings instead of erroring at the FromSql layer. 2. **#10 — Value::String → numeric/real/double/oid/bool**. Without these arms, a string-encoded value (`"3.14"`, `"true"`) for a non-text / non-temporal arg_t fell through to `Box<String> + TEXT`, which then failed at the server (no implicit cast text→numeric in expression context). Now strings are parsed into the matching native type with clear error messages on parse failure. 3. **Position-aware `$N` rewrite**: replaces the regex-based renumbering (which fixed the `$5/$50` substring collision but still walked through string literals and comments, mangling `'price: $5'` etc.) with a walk over `parse_pg_statement_arg_positions` — the same string/comment/dollar-quote-aware tokenizer used for index discovery. Adds `parse_pg_statement_arg_positions` to the parser's public API. SDK: 4. **BigInt support**: `JSON.stringify(BigInt)` throws. The SDK now stringifies bigints before serialisation; the executor accepts numeric strings into BIGINT arg slots via the existing `Value::String → INT8` parsing arm. SDK-side `inferSqlType` is split so `BigInt` always resolves to `BIGINT` (was reaching `Number.isInteger(BigInt)` which returns false → wrong default). 5. **Homogeneous array auto-tag**: `${[1,2,3]}` against an `int[]` column now emits `$1::BIGINT[]` instead of `$1::JSON`. Detection covers primitive types only (number / bigint / string / boolean); mixed or nested arrays still fall back to JSON. Mixed int/float widens to `DOUBLE PRECISION[]`. 6. **`.query()` positional bug**: previously the `.query()` method abused the template-tag builder, which appended `$N::TYPE` after the user's literal SQL string instead of binding by position (`SELECT $1, $2` became `SELECT $1, $2$1::BIGINT`). Now `.query()` builds the executor-shaped content directly: a `-- $N argN (TYPE)` declaration block followed by the user's SQL verbatim. Tests: - Parser: `test_parse_pg_statement_arg_positions_skips_strings_and_comments` asserts string literals, comments, and dollar-quoted blocks don't produce positions (so renumbering doesn't mangle them). - Executor unit: `renumber_sparse_placeholders_no_collision_no_string_mangling` uses the new position-aware path and includes string-literal + comment + `$$…$$` cases. Existing convert_val tests grow to cover new String→numeric/real/double/oid/bool arms. - Integration: `test_postgresql_arg_type_combinations` adds 13 cases (enum round-trip both directions, string→numeric/real/double/bool/oid, string-literal `$N` non-mangling). The prepare-fallback test now asserts SUCCESS (not failure) for enum encoding via AnyTextValue. - SDK: new `typescript-client/tests/sqlUtils.test.ts` (42 tests) exhaustively covering inferSqlType primitives + arrays, parseTypeAnnotation, datatable() template tag (with all the new shapes — BigInt, homogeneous arrays, RawSql, schema preamble), datatable().query() positional, and ducklake() shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): replace DISCARD ALL with curated reset (preserves typeinfo cache) Found while exhaustively probing custom-type DX: every cached-connection reuse was running `DISCARD ALL`, whose included `DEALLOCATE ALL` deallocates *all* prepared statements server-side — including the typeinfo statements that tokio_postgres caches per-Client to resolve custom enum / domain Oids. tokio_postgres still held `Statement` objects whose names the server had forgotten, so the next custom-type query failed with intermittent "prepared statement \"sN\" does not exist" errors. The failure was easy to reproduce: any sequence that forced typeinfo lookup for two different custom-type kinds on the same cached connection (e.g. enum followed by domain) would hit it. Replace `DISCARD ALL` with a curated reset that explicitly targets the state we actually care about, *without* touching prepared statements: RESET ALL — GUC parameters (search_path, application _name, statement_timeout, …) RESET SESSION AUTHORIZATION — undoes both `SET SESSION AUTHORIZATION` and `SET ROLE` (RESET ALL does NOT — these aren't GUC parameters, so without this an elevated role from a previous job would silently leak) UNLISTEN * — drops LISTEN registrations CLOSE ALL — closes open cursors Trade-off: temp tables, advisory locks (session-scoped), and user-created PREPARE statements may persist across cached-connection reuse — rare in datatable / PG-script workloads. tokio_postgres's typeinfo cache survives intact, so custom enum / domain queries are fast on subsequent reuse. Tests: - `test_postgresql_custom_types_on_cached_connection` — runs 10× alternating enum + domain queries on a cached connection. Pre-fix this failed with `prepared statement "sN" does not exist` after the first reuse; post-fix passes. - `test_postgresql_set_role_does_not_leak_across_cached_connection` — switches `SET ROLE` and `SET SESSION AUTHORIZATION` to a non-postgres role, then runs a follow-up job and asserts current_user/session_user are restored. Specifically catches the case where someone might switch back to `RESET ALL` alone (which doesn't cover SET ROLE / SESSION AUTHORIZATION) and silently introduce a permission-leak vector. - All existing session-isolation tests (`test_postgresql_cached_connection_resets_session`, `test_postgresql_single_worker_session_isolation`, `test_postgresql_100_jobs_cached`) continue to pass. Found via end-to-end probing of datatable / PG-script DX, not previously covered: the existing isolation tests only did `SET ROLE postgres`, the connecting user, so the leak was invisible. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): address PR #8999 review (cubic + claude) cubic (P1, real bug): - `convert_vec_val` for `timetz` array asserted `Type::TIMETZ_ARRAY`, but chrono `NaiveTime` only encodes for TIME (same caveat as the scalar arm). Switch to `Type::TIME_ARRAY`; rely on PG's implicit `time→timetz` assignment cast at the column site. Add an explicit unit test. claude (#1, silent failure → explicit error): - `Bool` + explicit `(char)` / `(character)` decl previously silently bound BOOL, hoping the server would cast at the use site — but PG has no implicit `bool→char` and the resulting error ("operator does not exist: bool = char") was opaque. Now error at bind time with an actionable hint to use `bool` decl or pass the value as a "t"/"f" string. claude (#2, asymmetry doc): - Object/Array still coerce to text on `matches!(typ, Typ::Str(_))` (covers both explicit AND inferred-default text), unlike Bool/Number which key on `explicit_text_target`. The asymmetry is intentional (no implicit `jsonb → text` cast in expression context vs PG having implicit `bool/int → text` casts) — added a body comment so future maintainers don't try to "align" them. claude (#3, perf): - `parse_pg_statement_arg_indices` and `parse_pg_statement_arg_positions` walked the SQL tokenizer twice. Fold into a single pass that derives the index set from the position list. claude (#4, fmt drift): - `cargo fmt` over the parser crates I touched with perl scripts in the earlier commit (windmill-parser-{sql,bash,ts,go,php,java,csharp,nu,py, rust,graphql,yaml,r}). Net cosmetic. claude (#5, parseTypeAnnotation): - One-line caveat in the SDK's `parseTypeAnnotation` that the returned string is presence-only (e.g. `${x}::DOUBLE PRECISION` returns `"DOUBLE"`, `CAST(${x} AS int)` returns `"int)"` — neither matches a real PG type, but the only consumer just checks `!== undefined`). While here — discovered + fixed independently while exhaustively probing DX: - **Replace `DISCARD ALL` with curated reset** (`RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;`). DISCARD's `DEALLOCATE ALL` killed tokio_postgres' typeinfo cache, producing intermittent `prepared statement "sN" does not exist` errors on custom-type queries after cached-conn reuse. New regression tests: `test_postgresql_custom_types_on_cached_connection` and `test_postgresql_set_role_does_not_leak_across_cached_connection` (the latter catches the case where someone might switch back to `RESET ALL` alone and silently introduce a permission-leak vector — RESET ALL doesn't cover SET ROLE / SET SESSION AUTHORIZATION). - **ISO-8601 timestamp results** (`pg_cell_to_json_value`). Pre-fix `TIMESTAMP` was rendered with a space separator ("2024-01-15 10:30:00") and `TIMESTAMPTZ` with " UTC" suffix ("2024-01-15 10:30:00 UTC") — neither parseable by `date-fns parseISO`, JavaScript `new Date()` is lenient enough to handle them but several frontend `App*Input.svelte` components use parseISO and fail silently. Switched to ISO-8601 with `T` separator and `+00:00` offset; arg-parsing path still accepts the legacy " UTC" suffix for back-compat. Test coverage: - 17/17 unit (`pg_executor::tests`) - 9/9 integration (`backend/tests/worker.rs`, `test_postgresql_*`) - 27/27 parser (`windmill-parser-sql`) - 42/42 SDK (`typescript-client/tests/sqlUtils.test.ts`) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): bounded one-shot warning on numeric precision loss + ISO-8601 + NaN handling Found while probing PG-script DX with millions of numeric cells: 1. **Numeric precision-loss warning**: `numeric` results are still serialised as JSON Number (back-compat — switching to JSON String would silently break user code doing arithmetic on results), but we now detect `Decimal -> f64 -> Decimal` round-trip failure and emit a single job-log warning recommending a `::text` cast in the SQL. Bounded by `NUMERIC_PRECISION_CHECK_BUDGET = 256` cells per query (one atomic load + one fetch_sub on the hot path; first lossy value short-circuits to a single load thereafter). Worst-case overhead on a 1M-cell numeric-heavy query: ~25µs of checks + 5ns × N atomic loads (vs. ~100ms unbounded). 2. **ISO-8601 timestamps**: `pg_cell_to_json_value` previously returned `"2024-01-15 10:30:00"` (TIMESTAMP) and `"2024-01-15 10:30:00 UTC"` (TIMESTAMPTZ) — neither parseable by date-fns `parseISO`, which is what the apps `App*Input.svelte` components use, so timestamp values silently failed to round-trip into date pickers. Switch to ISO-8601 (`T` separator + `+00:00` offset) on the result side; arg-parser continues to accept the legacy `" UTC"`-suffixed format for back-compat. 3. **Float NaN / Infinity results**: `Number::from_f64` returns None for NaN / ±Inf, which `pg_cell_to_json_value` was raising as "invalid json-float" — failing the *entire* query if any cell held one of these special values. Now serialise them as JSON strings ("NaN", "Infinity", "-Infinity") and let the rest of the row come through. Arg-side: `s.parse::<f64>()` already accepts the same strings. Tests: - `decimal_fits_f64_losslessly_predicate` — covers fits / doesn't-fit cases for the precision-loss predicate. - `precision_check_budget_caps_per_query_overhead` — locks in the budget cap and the loss-flag short-circuit. - All 9 PG integration tests + 17 unit tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): add pg_advisory_unlock_all to reset; warn on missing args; honor decl defaults While probing PG-script DX further found three more frictions: 1. **Advisory lock leak** (cubic P2): switching from `DISCARD ALL` to `RESET ALL; RESET SESSION AUTHORIZATION; UNLISTEN *; CLOSE ALL;` meant session-scoped advisory locks (`pg_advisory_lock`) leaked across cached-connection reuse. Add `SELECT pg_advisory_unlock_all()` to the chain — `DISCARD ALL` covered this implicitly via `DISCARD PLANS / DEALLOCATE / pg_advisory_unlock_all` and we lost it in the switch. 2. **Missing-arg silent NULL**: an arg declared in the SQL (e.g. `-- $1 amount (numeric)`) but not provided in the args object was bound as NULL with no error / warning. Misspelling the key in the args object silently produced a row of NULLs — a notorious DX debugging trap. Now: collect the names of declared-but-missing args during dispatch and emit a single one-shot warning to the job logs at end-of-query naming each one. Bound NULL is preserved for back-compat. 3. **Declaration defaults ignored**: `-- $1 a (int) = 5` carries `arg.default = Some(Number(5))`, but the dispatch fell straight to NULL when the arg was missing. Now: respect the default — user-supplied value > declaration default > NULL. Also fixes the warning logic above (only warn for args that *don't* have a default). Tests: existing 19 unit + 9 integration pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(pg): multi-word PG types with [] suffix lost the array-ness; array arms accept stringified values Two more frictions found while probing SDK end-to-end against a real datatable resource: 1. **Multi-word array types lose the [] suffix in the parser**. `transform_types_with_spaces` recognises aliases for "double precision", "character varying", "timestamp with time zone", etc. but its return type was `&'a str` — only the bare alias, never with a trailing `[]`. The `RE_CODE_PGSQL` regex's `\w+` captures stop at the first space, so the regex's own `(?:\[\])?` array-suffix branch sees only `"double"` (not `"double precision[]"`); the `[]` was silently lost. Result: `$1::double precision[]` (which the SDK now emits for homogeneous float arrays via the new auto-tag) routed through `Value::Array → Type::JSONB` and the server failed with "cannot cast type jsonb to double precision[]". Fix: switch `transform_types_with_spaces` to return `Cow<'a, str>` and re-check the trailing bytes after a multi-word match. If they start with `[]`, return `format!("{alias}[]")` — Owned. Single-word types and the no-match path keep returning Borrowed slices, so no allocation in the hot path. 2. **Array arms in `convert_vec_val` rejected stringified values for numeric / int* / bool / oid / real / double**. The scalar `convert_val` already parses strings into the matching native type for these arg_ts, but the array variant only accepted JSON-native counterparts. Sending `["1.5", "2.5", "3.5"]` against `$1::numeric[]` (e.g. via `unnest` for bulk loading, or `JSON.stringify(BigInt[])` round-trip) failed with "Mixed types in array". Now the array arms mirror the scalar ones — `as_<native>().or_else(|| as_str().and_then(parse))` — so both shapes round-trip cleanly. Tests: 19 unit + 9 integration pass; existing parser tests cover the multi-word array forms (the regex-cap behaviour didn't break for single-word types, and Cow plumbing is transparent to all callers). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(parsers): add otyp_inferred field to Arg literals in tests + 3 missed src files CI failures: the perl-driven sweep that added `otyp_inferred: false` to every `Arg { ... }` literal when I introduced the field in the parser schema covered `src/lib.rs` files but missed: - parsers/windmill-parser-bash/src/lib.rs (mass-edited but a later format pass un-applied a few sites) - parsers/windmill-parser-go/src/lib.rs (same) - parsers/windmill-parser-graphql/src/lib.rs (same) - parsers/windmill-parser-nu/tests/tests.rs (test file — not swept the first time) - parsers/windmill-parser-ts/tests/tests.rs (test file — same) Also tightened the regex to handle `oidx: None` without the trailing comma (some test files had the field as the last initialiser line). `cargo build --features <CI feature combo> --workspace --all-targets` is clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sdk): Date → TIMESTAMPTZ; NaN / ±Infinity → string Two more frictions found while running the actual SDK end-to-end against a live datatable resource: 1. **JS `Date`** fell into the typeof "object" branch and was tagged `::JSON`. It worked accidentally for `${date}::timestamptz` via PG's `json → text → timestamptz` implicit cast chain, but `${date}` against a `timestamptz` column without a user-supplied cast bound the value as a JSON string and the comparison `timestamptz = json` failed. Now: `inferSqlType` recognises `Date` and tags `::TIMESTAMPTZ`; `serializeArgValue` emits `Date.toISOString()` so the executor's `Value::String → TIMESTAMPTZ` arm parses it cleanly. 2. **JS `NaN` / `±Infinity`** silently became NULL. `JSON.stringify(NaN)` returns `"null"` per the JS spec, so the value reached the executor as JSON null — the SDK's `::DOUBLE PRECISION` tag then bound a NULL double. Fix: detect non-finite numbers in `serializeArgValue` and stringify them as `"NaN" / "Infinity" / "-Infinity"`. The executor's `Value::String → FLOAT8` arm (`f64::from_str`) accepts these literals directly, and the result-side already renders the values as JSON strings (matching round-trip). SDK unit tests grow from 42 → 44 passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(pg): integration coverage for multi-word arrays + stringified array elements Locks in the two array fixes from the previous commit (`fix(pg): multi-word PG types with [] suffix lost the array-ness`) with end-to-end cases in `test_postgresql_arg_type_combinations`: - `double precision[]`, `character varying[]`, `timestamp without time zone[]` — verifies the parser keeps the `[]` suffix after multi-word alias resolution. - `numeric[]` / `int[]` / `bool[]` from stringified primitives — verifies the array arms of `convert_vec_val` apply the same string-coercion the scalar arms do. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: fix indentation drift on otyp_inferred lines cargo fmt cleanup of leftover indentation where the perl-driven sweep that introduced the otyp_inferred field landed at the wrong column. No behaviour change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * feat: support assigning a worker tag to app inline scripts (#9002) * feat: support assigning a worker tag to app/raw-app inline scripts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: omit empty tag field from inline script raw_code payload Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * style: shrink tag popover width --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> * feat(pipeline): 2-col picker, draft path edit, save-all + leave guard Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * all * update * fix(cli): forward HEADERS env var on every backend fetch call (#9075) Several `fetch()` callers in the CLI bypassed `OpenAPI.HEADERS` and skipped the `HEADERS` env var, causing requests to fail behind auth gateways like Cloudflare Access (same shape as #6421): - `pushScript()` `/scripts/create` and `/scripts/create_snapshot` — regressed in #8936 when the call switched from `wmill.createScript()` (SDK) to a raw `fetch` for the `skip_if_noop` query param. - Script preview `/jobs/run/preview_bundle`. - App dev `/jobs_u/getupdate_sse` SSE stream. - `wmill docs` `/api/inkeep`. All four now spread `getHeaders()` and call `detectAuthGatewayChallenge()` so a Cloudflare/SSO challenge surfaces a clear error instead of an opaque JSON parse failure. Adds `test/headers_env_var.test.ts`: spins up an auth-gateway proxy that 403s requests missing `CF-Access-Client-Id` / `CF-Access-Client-Secret` and otherwise reverse-proxies to the test backend, then runs `wmill sync push` of a fresh script through the proxy. Negative case (no `HEADERS` env) verifies the proxy actually gates; positive case asserts every request including `/scripts/create` reaches the backend with the headers attached. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): add --parallel flag to generate-metadata (#9074) * feat(cli): add --parallel flag to generate-metadata * fix(cli): validate --parallel input and harden flush ordering * perf(flows): skip flow_env DB+transform work when no resolution is needed (#9078) * fix(cli-tests): stabilize flow lock-gen race + Windows path (#9080) * fix(cli-tests): stabilize flow lock-gen race + Windows path Three CLI test failures on the latest main, all flaky on CI: 1. `Mixed Case Paths: pull and push flow with capitalized folder` and `Integration: Mixed scripts and flows with nonDottedPaths are idempotent`: flow create/update queues an async FlowDependencies job that fills inline-script lockfiles and rewrites flow.value. The tests pulled/pushed before the worker finished, so dry-run idempotency saw phantom `*.inline_script.lock` adds and `flow.yaml` edits. Added a `waitForFlowDependencyJob` helper that polls `/flows/get` for the latest `dependency_job` and `/jobs_u/completed/get` until it lands, and called it after each API/CLI flow write in both tests. 2. `HEADERS env var is forwarded on every CLI fetch` (Windows-only, added in #9075): the new test built the CLI entrypoint via `new URL("..", import.meta.url).pathname`, which yields `/C:/...` on Windows and `Bun.spawn` rejected before reaching the proxy, leaving `rejectedRequests.length` at 0. Switched to `fileURLToPath` + `node:path.join` to match `cargo_backend.ts`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli-tests): use /flows/deployment_status to actually wait for dep job CI reviewers (Claude, Codex) flagged the prior `waitForFlowDependencyJob` as a no-op: it read `flow.dependency_job` from `/api/w/{ws}/flows/get`, but `Flow` / `FlowWithStarred` (backend/windmill-types/src/flows.rs:20-60) do not include that field. The helper exited on the first iteration without polling. Switch to `/api/w/{ws}/flows/deployment_status/p/{path}`, which returns `{ lock_error_logs, job_id }`. `job_id` is the FlowDependencies UUID written into `deployment_metadata` in the same tx as the dep-job push (backend/windmill-api-flows/src/flows.rs:660-672 and :1275-1292), so by the time the create/update API call returns, the response carries the latest dep-job UUID. Then poll `/jobs_u/completed/get/{job_id}` as before. Local runtime for `mixed_case_paths.test.ts` jumps from ~9s to ~32s, confirming the helper now actually waits instead of returning immediately. The 404 short-circuit in `sync_pull_push.test.ts` still works — `get_deployment_status` returns 404 when the flow is absent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(flows): cache resolved flow_env per flow execution (#9079) * perf(flows): cache resolved flow_env per flow execution * perf(flows): tighten flow_env cache cap to 1024 and clarify memory note * perf(flows): don't cache transient flow_env resolution failures * chore(main): release 1.698.0 (#9076) * chore(main): release 1.698.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: reject root-rooted paths in ansible playbook validator on windows (#9081) * fix(native-triggers): serialize Google channel renewal across replicas (#9060) * fix(native-triggers): serialize Google channel renewal across replicas `sync_all_triggers` runs every 5 minutes on every windmill-app replica with no leader election. Multiple replicas were each rotating the webhook token, creating a new Google watch channel, and racing the trigger UPDATE — leaving the loser's new token (in `token`) and channel (in Google) orphaned. Cloud was accumulating ~5 leaked tokens/week without the silent best-effort `delete_token_by_hash` ever logging a warning. Wrap each per-trigger renewal in a transaction and acquire the row with `SELECT … FOR UPDATE SKIP LOCKED`. Contending replicas skip the row instead of duplicating the work. The lock spans `rotate_webhook_token` → Google API call → `update_native_trigger_service_config` and is only released on commit. Re-checks `should_renew_channel` after acquiring the lock so a replica that committed seconds earlier doesn't trigger a duplicate renewal. The pattern matches existing batch-cleanup paths in `monitor.rs` (job-retention sweep) and other `FOR UPDATE SKIP LOCKED` call sites. Also logs at `debug!` when `delete_token_by_hash` finds no matching row, so future investigations can distinguish "deleted" from "not found" without changing the `Ok(false)` contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Address claude review: - #5: per-skip log info -> debug (expected outcome under SKIP LOCKED) - #2: warn moved out of delete_token_by_hash to the call site that knows the expected state (try_renew_channel_locked); other callers are race-prone and shouldn't warn - #3: NULL service_config now warns (anomalous case) - #4: post-Google-API DB-update + commit failures log distinctly so the channel-orphan case is grep-able Plus: add 14d expiry to Google webhook tokens via ServiceName::webhook_token_expiration, mint fresh ephemeral-webhook-{service}-{rd5} labels at create + rotate so the existing 'ephemeral-' filter excludes them from user-token email/critical-alert paths (no filter changes in 3 places). Orphans now self-clean via the existing expiry sweep in monitor.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Address second-round review: - Claude #1 (P2): username_override_from_label now strips the 'ephemeral-' prefix for ephemeral-webhook-* labels, so created_by stays webhook-{service}-{rd5} instead of changing to label-ephemeral-webhook-... (preserves audit/job-list filter compatibility) - Codex (P2): updated renew_channel doc — labels are no longer copied; rotate mints fresh ephemeral-webhook-google-{rd5} with 14d expiration - Claude #3 (optional): test_rotate_webhook_token now asserts the rotated Google token has an ephemeral-webhook-google-* label and a populated expiration Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! fix(native-triggers): serialize Google channel renewal across replicas fixup! fix(native-triggers): serialize Google channel renewal across replicas Reconsider the previous fixup: stripping the 'ephemeral-' prefix made created_by no longer match token.label exactly, defeating the linking purpose. Just allowlist 'ephemeral-webhook-' alongside the other recognized webhook/email/ws prefixes — created_by becomes ephemeral-webhook-google-XXXXX, matching token.label exactly. The 'ephemeral-' substring also informs operators that this is a system-managed auto-expiring token vs a user-managed webhook trigger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): bump svelte version in `wmill app new` template (#9084) * fix(cli): bump svelte version in `wmill app new` template The svelte5 template pinned `svelte` to `5.45.2`, but the Svelte compiler bundled in `wmill app dev` emits `$.delegated('click', ...)` calls. The `delegated` export was added later, so 5.45.2 doesn't have it — esbuild warns `Import "delegated" will always be undefined`, replaces the call with `void 0`, and the page crashes at first event-handler bind (white screen). Bump to `^5.55.5` so the compiler and runtime stay in sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): bump svelte version in raw_apps UI template Mirror the CLI fix: the UI's `Add raw app` flow scaffolds a package.json with `svelte: "5.45.2"`. That works today only because the bundled rolldown worker also pins 5.45.2 — when the worker is upgraded past 5.51.1, the compiler will emit `$.delegated()` and the runtime won't have it, producing the same white-page crash that hit the CLI. 5.55.5 still exports `event` (used by the current bundled compiler), so this is forward-compatible: it works with the 5.45.2 compiler now and won't break when the worker is upgraded. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(flows): gate flow_env resolve on expr text and share cache with handle_flow (#9085) * feat: parse windmill_failure field to tag run as failure (#9073) * feat: parse windmill_failure field in job result to tag run as failure * feat: preserve top-level fields when windmill_failure tags a run as failure * fix: address review findings on windmill_manual_failure * refactor: rename windmill_manual_failure to wm_failure and add wm_* aliases * fix: prefer injected ManualFailure error over sibling name/message in OTel * fix: hide _ENTRYPOINT_OVERRIDE jobs from script/flow history panel (#9088) * fix(flows): populate error handler input args from failure picker (#9087) * fix(flows): populate error handler input args from failure picker * style(flows): fix indentation in failure-step branch * fix(python): verify wheel RECORD on cache pull/install, finalize piptar (#9090) The Python per-package dependency cache could persist an incomplete wheel extraction with `.valid.windmill` set, then propagate that broken artifact to every worker through the object store. Customer hit this on argon2-cffi==25.1.0 (missing argon2/_utils.py), and previously on botocore/httpx (truncated tars). Symptom is a runtime ImportError that looks like a missing dependency declaration rather than a Windmill bug. Three changes that together stop the propagation: 1. After `pull_from_tar`, parse the wheel's `<dist-info>/RECORD` and confirm every listed path exists on disk before writing `.valid.windmill`. On failure, wipe the directory and fall through to a fresh local install — the next install also self-heals the broken object-store entry by pushing a fresh tar. 2. After `uv pip install` succeeds, run the same RECORD check before queuing the piptar upload or writing `.valid.windmill`. A bad install never becomes the source of a broken tar in the object store. 3. Finalize the tar (`drop(tar.into_inner()?)`) before reading its bytes for upload, so we never push an unfinalized archive (no end-of-archive marker) to the object store. Verified with a 60-package end-to-end integration test (first-fill → clear-local-cache → re-pull-from-objectstore → corrupt-objectstore-tar → detect-and-self-heal). All 27 packages on the live test pulled cleanly, and the deliberately corrupted argon2-cffi tar was caught with the exact expected log line ("wheel RECORD lists files missing on disk: argon2/_utils.py") and replaced with a fresh tar. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(main): release 1.699.0 (#9082) * chore(main): release 1.699.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat(cli): auto-infer args for `wmill app push` (#9091) Run `wmill app push` from inside an app folder (e.g. `f/foo/my_app.app/`) with no args. The local path defaults to CWD, and the remote path is derived from CWD relative to `wmill.yaml`, with `.app`/`.raw_app`/ `__app`/`__raw_app` suffixes stripped. Either, both, or neither positional argument can be passed. Also resolves `file_path` against the user's original CWD before `resolveWorkspace` may chdir to the wmill.yaml root, so a relative `file_path` argument is interpreted from where the user invoked the command (previously it could resolve against the wrong directory). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * all * fix(pipeline): live-update graph for annotations and body assets * fix(pipeline): persist draft body edits across node switches * fix(pipeline): persist live writes per draft to keep output node fresh after switch * feat(pipeline): animate graph edges only while a runnable is executing * feat(pipeline): add run button on script nodes + recomputing hint on preview * feat(pipeline): compact preview layout, two-way Test/Run sync * fix(pipeline): test button cross-browser placement (no overflow trick) * style(log-viewer): replace took/mem-peak labels with timer/cpu icons * style(log-viewer): hyphenate Auto-scroll label and prevent wrapping * style(log-viewer): lowercase auto-scroll label, force vertical scrollbar * style(log-viewer): force horizontal scrollbar instead of vertical * fix(log-viewer): scope overflow-x to top bar so pre doesn't drive panel width * fix(pipeline): overlay live body-asset writes for persisted scripts too * fix(pipeline): persist inferred body assets at save so edges survive page reload * fix(pipeline): snapshot live draft writes at persist time so they survive reload * fix(pipeline): keep inferred body writes on the canvas across selection changes * fix(pipeline): untrack inferredWrites cache mutation to break effect loop * fix(pipeline): refetch asset graph after persisted-script save * feat(pipeline): optional AI prompt when creating a pipeline script * all * all * test: cover asset-trigger dispatch end-to-end through worker * feat(pipeline): split-button Test with optional downstream cascade * feat(pipeline): cascade option on graph Run + match button heights * style(pipeline): match caret bg/text to Test button's accent-secondary * feat(pipeline): split Run pill on graph node exposes cascade option * feat: live run activity + status badges in pipeline asset graph - folder-scoped queue poll lights up the downstream asset-trigger cascade (not just the launched script); zero requests at rest, catch-up for fast hops, auto-disarm when idle - per-runnable node badge: last-run status + session run count - animate unsaved/live-parsed edges (was unconditionally suppressed) - background-pane click no longer clears selection - run-bridge guarded so node selection/save no longer triggers a test Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: live activity log, optimistic badges, node-avoiding graph edges - collapsible folder activity log (PipelineEventLog): live job feed, polls only while open/active, slow idle cadence, capped + pruned - composable: observe mode + events list + run-count anchored to graph-open time (pre-existing history excluded) - optimistic node badge: launched script shows running instantly via the zero-latency activeRunnable hint, keeps the polled run count - activity pane height capped (min(18rem,40vh)) then scrolls - route asset-graph edges through sugiyama-computed waypoints so they go around nodes instead of under them; bezier fallback for adjacent-layer / draft-overlay edges Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: prefetch all folder script assets so graph is stable on load On pipeline load, eagerly infer body assets for every persisted folder script and seed the existing inferredWritesByPath overlay, instead of only filling it when a node is selected. Scripts whose persisted asset rows are missing (e.g. object-form writeS3File) now have their edges from first paint, so clicking a node no longer re-layouts the graph. One-shot per (workspace, base-graph) load, untracked map reads, generation-cancelled, pool-capped fetches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: guard no-op poll re-layout; dedupe write-asset extraction - skip reactive ids/states/events reassignment when unchanged, so an idle poll tick no longer re-runs the full sugiyama layout every 3-6s - bound countedJobIds (rebuilt from eventsById in lockstep with prune) - extract shared extractWrites() helper, replacing 4 copy-pasted write-asset filter/map blocks in the pipeline page - compute activeRunnable node-id once, reuse for the active-edge set and the optimistic badge (flattened ternary); trim narrating docs Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: live read-lineage overlay for inferred body assets Renaming e.g. duckdb read_parquet('s3://...') / loadS3File now updates the asset->reader edge live instead of only after Save re-derives the persisted asset rows. - extractReads() (+ shared refsByAccess) mirroring extractWrites - inferredReadsByPath sticky cache, filled by handleAssetsChange and the load prefetch alongside writes - replace the write-only overlay loop with one overlayLineage(map, access) helper invoked for both 'w' and 'r' (net DRY) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: detect S3 assets passed as SDK object arg in ts parser Mirrors merged PR #9181 so feat/asset-graph-view is self-contained (local origin/main is stale and lacks it). Object/{ s3, storage } form of writeS3File/loadS3File is now detected, not only the bare s3:// string literal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate wasm Cargo.lock + frontend package-lock Lockfile churn from local wasm-pack (asset target) + npm operations during the asset-graph work. No source/dependency-intent change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: revert to bezier graph edges; add parsing-assets hint The sugiyama-waypoint routing looked worse than the original; revert AssetGraphEdge/assetGraphLayout to the pre-routing bezier logic (same as the flow editor's BaseEdge) and drop the now-unused route plumbing from the canvas. Add a small 'Parsing assets…' hint shown while the load-time prefetch sweep is still inferring folder scripts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: extract pure resolveGraph merge + unit tests Move the ~230-line graphWithDraft precedence/merge (base < session- inferred < draft-seeded < open-script-live, +read/write/annotation overlays, +dedup) out of the 1648-line route into a pure, testable resolveGraph() module; the route's graphWithDraft is now a thin $derived. Behaviour extracted verbatim. 10 unit tests cover the precedence matrix. Phase 1 of the state/render split. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: graph controls top-right, lift minimap, hide Save when unchanged Controls -> top-right horizontal, no lock toggle; MiniMap !mb-10 so it clears the activity bar; hide the per-script Save button when the script is already at its latest save point (drafts still show Create). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: scope runtime-asset prune by id to spare static lineage rows prune_runtime_assets deleted by (workspace_id, path, kind) tuple, so trimming surplus usage_kind='job' rows for an s3 path also wiped the static usage_kind='script'/'flow' producer rows for the same path — silently breaking the asset-trigger cascade (fetch_producer_writes found no writes; downstream never dispatched; required band-aid re-syncs). Delete the surplus job rows by id instead; the inner query is already scoped to usage_kind='job'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't re-pulse already-running jobs after they finish The catch-up pulse re-added a completed job to the active set if its start was within the (lagging) lookback window — even one we'd already animated the whole time it ran — keeping its edges lit ~a poll interval past completion (~5s after a 3.5s test). Track job ids seen in-flight and skip the pulse for them; it still fires for hops whose whole lifetime fell between two polls. Bound the set in lockstep with eventsById; cleared on dispose. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't catch-up-pulse the runnable launched from the graph If the poll never sampled a launched run's in-flight window, the catch-up pulse re-flashed its edges one tick after it correctly stopped (the page already animated it zero-latency via activeRunnable). arm(launchedId) records the launched runnable id; catch-up skips it. Cascade hops (other ids) still pulse. launchedIds cleared on stop. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: nudge graph controls left to clear panel toggle Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: partition value resolver + asset-cascade propagation windmill-common/partition: pure resolver — time kinds (tz/format/start anchor) + dynamic $.a.b JSONPath; 9 unit tests. asset_dispatch: read the producer's resolved partition and thread it into every cascaded subscriber's args + trigger.partition, so a chain resolves once at the top. No migration (cascade needs no spec lookup). Stage 1+3 of pipeline partition runtime; run-start resolution is Stage 2. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: show args form in compact pipeline preview when script has inputs AssetGraphDetailsPane keeps the compact (hideArgs) preview but, via a new previewPanel.argsAboveLogs flag, renders a compact SchemaForm between the floating Test button and the logs/result panel when the script declares inputs (e.g. a partitioned script needing a `partition` arg). The preview pane also grows ~18pts so the args form doesn't shrink logs/result. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: parser join-mode (`// trigger all`) + script_trigger.join_all Stage A: JoinMode{Any(default),All} + `// trigger any|all` directive in parse_pipeline_annotations; TriggerSpec::is_partition_bearing() (path contains {partition}); join_mode threaded through all 4 asset-parser crates (ts/py/sql/yaml). Stage B: reversible migration adds script_trigger.join_all; insert_script_trigger writes it; deploy path sets it from the parsed annotation. No reader yet (AND-join dispatch is the next stage) so runtime behaviour is unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: resolve pipeline partition at job execution time Stage C: in handle_code_execution_job, once the script content is loaded, parse the // partitioned annotation (free here) and resolve the concrete partition once — schedule fire-time (scheduled_for anchor, not wall-clock) for time kinds, triggering payload for dynamic. The value is injected into the in-memory args the body sees (via a shadowed job clone) and persisted back to v2_job.args so dispatch_asset_triggers propagates the same value down the cascade. Already-set (explicit/backfill/cascade) partitions are never re-resolved (run identity immutable); unresolvable partitioned runs fail with a clear error. Integration test exercises the full worker loop + cascade propagation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: AND-join barrier for partitioned pipeline subscribers Stage D: a // trigger all subscriber no longer fires on any input. New join_pending_inputs slot table keyed (workspace, subscriber, partition); fetch_subscribers now returns join_all and the dispatch loop records each partition-bearing input arrival, pushing the subscriber once only when every partition-bearing input it declares is present for that partition. Per-partition slots, cleared on fire (re-accumulate, no double-fire), skew-immune (unlike debounce). Case-3 guard: an unpartitioned producer or a reference (non-{partition}) input never fires a partitioned join. Integration test covers wait/fire/isolation/no-double-fire. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: opt-in // debounce for asset-cascade subscribers (parser + schema) Stage E1+E2. Parser: script-level // debounce <dur> + per-// on debounce=<dur> override (edge wins, else script default, else none = fan-out, unchanged); TriggerSpec::Asset carries the per-edge override; split_trailing_kv_opts separates the ref from trailing key=val opts. Schema/deploy: reversible migration adds script_trigger.debounce_s; parse_duration_secs (bare int or <n>s|m|h|d, fail-safe on garbage) resolves the effective per-edge window at deploy and writes it per row. No reader yet (dispatch wiring is E3) so runtime is unchanged. New unit tests for the parser directive and duration parsing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: apply opt-in debounce to asset-cascade subscriber dispatch Stage E3. fetch_subscribers now also returns debounce_s; push_subscriber builds real DebouncingSettings (delay + a (subscriber, partition) key, so distinct partitions never collapse and latest-in-window falls out) instead of ::default() when the edge opted in. Default stays no-debounce (fan-out — the prior deliberate behaviour, now overridable rather than reversed). Wiring test asserts the dispatched job carries the configured window/key and an undebounced edge carries none. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix: atomic AND-join gate + preserve resolved partition; drop scratch artifacts Addresses local-review findings before PR: - P1: record_and_check_join_slot was a non-atomic check-then-act on a pooled connection; concurrent completion of a subscriber's last two partition-bearing inputs on different workers could double-dispatch. Now one transaction guarded by a tx-scoped advisory lock keyed on (workspace, subscriber, partition) so the gate fires exactly once. - P2: the preprocessed-args overwrite in result_processor replaced args wholesale, dropping a partition resolved by resolve_partition_for_job; the UPDATE now preserves an existing persisted partition key. - P2: gate resolve_partition_for_job on a cheap code.contains check so non-pipeline script jobs skip the annotation scan on the hot path. - P2: remove 40 scratch screenshot PNGs, a flicker-debug script and a local scheduler lock accidentally committed; gitignore the lock. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: AND-join fires once under concurrent upstream completion Regression for the check-then-act race fixed by the advisory-locked transactional gate: releases N producer dispatches simultaneously via a barrier and asserts the AND subscriber is pushed exactly once and the slot is cleared. The invariant holds for the correct gate regardless of interleaving; a non-atomic regression fails it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: fuller partitioned join + multi-hop pipeline coverage Exercises a complex pipeline combining options end to end: two partitioned producers fanning into a // trigger all join, then a multi-hop downstream chain. Asserts the resolved partition propagates unchanged at every hop, chain depth increments per hop, the AND barrier fires exactly once, and a second partition opens an independent slot with no cross-partition bleed across the whole graph. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * refactor: simplify pipeline code per review (dedup, single-parse, constant) - ParseAssetsOutput::new() collapses the 6-line annotation copy-paste across the 4 asset-parser crates to one call site. - asset_dispatch: parse the cascade trigger object once and pass it to the depth/partition readers instead of deserializing it twice; add a TRIGGER_ARG constant for the previously stringly-typed key (3 sites). - scripts deploy: drop a redundant debounce_default clone. No behavior change; 29 parser + 6 dispatch integration tests green. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: reap abandoned AND-join slots after a TTL (default 60d, per-slot) join_pending_inputs slots are normally cleared when the join fires; partial slots whose inputs never all arrive (upstream removed/renamed, one-off dynamic partition key, permanent skew) would otherwise leak. windmill_queue::asset_dispatch::reap_stale_join_slots, called from the monitor's delete_expired_items loop, deletes a (workspace, subscriber, partition) slot only when its MOST RECENT row is older than JOIN_SLOT_TTL_SECS (60d) — per-slot, never per-row, so a legitimately slow join is not corrupted mid-accumulation. Conservative default; per-join configurable TTL via the annotation is a planned follow-up. Test covers stale-reaped / fresh-kept / mixed-slot-kept. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * update * feat: path-less native trigger markers + missing-trigger placeholder * feat: pipeline // tag and // retry annotations + dispatch_event log * fix: derive test-pane min from split-axis dimension (height in bottom layout) * feat: show last run logs/result when a script node is selected * fix: backfill asset rows from script.assets for pre-feature scripts * feat: job-id link + dispatch popover above script log/result * style: drop 'dispatched' label, keep just the check icon * fix: drop tag picker from pipeline script editor (set via // tag annotation) * Nicer UI * refactor: move google ai proxy handling to windmill-ai (#9260) * refactor: add ai proxy execution mode * refactor: move google ai proxy handling * refactor: share google ai request building * fix: early return should consider failure_module result (#9241) * fix(flows): flag noLogs jobs and lazily resolve them in log panel (#9099) * fix(flows): flag noLogs jobs and lazily resolve them in log panel * fix appending to flag * fix: preserve WM_LOGS_SKIPPED sentinel on SSE/replay completion pickMoreCompleteLogs resolved both sentinel and undefined to '', so the SSE completion event (whose job field is fetched .without_logs()) would clobber the sentinel placed by flagSkippedLogs. The module log panel then saw '' instead of the sentinel, defeating the lazy-resolve path. Also wire onLogsResolved on the OutputPickerInner inline LogViewer so a lazy resolve writes back to flowStateStore.previewLogs, matching ModulePreviewResultViewer and avoiding repeated fetches on remount. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(main): release 1.705.0 (#9229) * chore(main): release 1.705.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * chore: add playwright mcp for frontend verification (#9269) * feat: CLI datatable serve / psql (#9267) * feat(cli): add datatable list and run commands * feat(cli): render datatable query results as a table * feat(cli): serve datatables as a postgres-wire endpoint * feat(cli): add 'datatable psql' to launch psql against the proxy * feat(cli): route datatable serve by client-supplied database name * override database list + password option * fix: support extended queries in datatable serve * fix: correct cloud size threshold log and parse CLI descriptions with parens/trailing comma * refactor: extract raw_output envelope encoding into pg_raw_output module --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> * oom_adj nit * feat: add UV_PYTHON_INSTALL_MIRROR env and instance setting (#9271) * feat: add UV_PYTHON_INSTALL_MIRROR env and instance setting Allows operators to point `uv python install` at a private mirror of the python-build-standalone releases. Configurable via the `UV_PYTHON_INSTALL_MIRROR` env var or the `uv_python_install_mirror` instance setting, with the env var as the boot fallback and the instance setting taking precedence at reload. Fixes WIN-1966 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: hoist uv_python_install_mirror binding above sandboxing branch The non-sandboxed uv pip install branch referenced a binding that was only declared inside the sandboxed branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: neutral placeholder for uv_python_install_mirror The previous placeholder was the default public URL the setting is meant to redirect away from. A neutral example mirror URL is clearer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(indexer): tell admins when ingress routes search to wrong pod (#9274) * [ee] fix(indexer): tell admins when ingress routes search to wrong pod When the IndexReader is absent on the pod handling a search request but another pod is actively holding the indexer lock, the EE handler now returns a tailored error pointing at the ingress/load-balancer configuration instead of the generic "indexer not running" message. The indexer status endpoint reads the DB lock so it reports "running" from any pod, but search endpoints need the in-memory IndexReader that only exists on the lock holder. In multi-replica deployments this looks like the indexer is healthy but every search 404s. Companion: windmill-labs/windmill-ee-private#TBD Fixes WIN-1968. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 This commit updates the EE repository reference after PR #586 was merged in windmill-ee-private. Previous ee-repo-ref: 7dd43d1850813071cc18ba49ba090583e7321f4b New ee-repo-ref: eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * feat(cli): add `wmill init prompts` and custom override slot (#9266) * feat(cli): add `wmill init prompts` and custom override slot Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): replace init prompts with refresh prompts + AGENTS.md/AGENTS.cli.md split Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): dedupe claude skills via @-includes and add prompts freshness check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): drop migration-choice flags from `refresh prompts` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cli): add 'Running and previewing local changes' section to AGENTS.cli.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): write full skill content to .claude/, drop @-include wrapper Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): reconcile CLAUDE.md the same way as AGENTS.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): address PR review nits — argv parsing, lazy import, comment detection, error propagation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add yolo mode for ai chat tools (#9258) * feat: add yolo mode for ai chat tools * nit * fix: align chat footer controls * feat: add ai chat autonomy modes * feat: add autonomy mode dropdown * fix: highlight yolo autonomy icon * fix: auto accept flow edits * fix: hide unsupported autonomy modes * fix: handle auto-accept flow editor races * fix(debugger): add non-root user support to Dockerfile (#9277) Mirrors the main Windmill Dockerfile pattern: creates a windmill user (UID/GID 1000) and makes cache/work directories world-writable so the image runs cleanly under Kubernetes securityContext.runAsNonRoot or runAsUser: 1000 without permission errors on Bun, pip, or windmill cache writes. Fixes WIN-1969 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path (#9276) * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path The AI proxy handler accepts an X-Resource-Path header to override the configured workspace AI provider. When supplied, the handler loaded the resource value from the resource table using the root DB pool with no resources:read scope check, so any authenticated workspace user could point X-Resource-Path at a restricted AI resource (e.g. one in a folder they cannot read) and the proxy would use that resource's provider credentials for the outbound AI request. For user-supplied resource paths, now require resources:read:{path} scope and fetch the resource through user_db.begin(&authed) so RLS enforces the same folder/group boundary as the resource API. The RLS- scoped $var: resolution stays in place as defense in depth. The admin-configured workspace/instance ai_config path is unchanged. Fixes WIN-1971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai): regression test for X-Resource-Path RLS enforcement Cover all four cases: - non-admin pointing X-Resource-Path at a restricted resource is rejected - non-admin pointing it at a resource they own still works - admin can point it at any resource - workspace-configured proxy flow (no X-Resource-Path) is unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add userdraft listing primitives (#9268) * feat: add userdraft listing primitives * fix: cancel stale userdraft discard writes * docs: remove global ai userdraft plan * feat(nsjail): optional disk-backed /tmp via instance setting (#9272) * feat(nsjail): optional disk-backed /tmp via instance setting * test(nsjail): unit-test tmp mount resolver and narrow visibility * refactor(nsjail): switch tmp backing to select + conditional UI * ui(nsjail): make tmpfs the visible default in /tmp backing select * fix(nsjail): refuse preexisting jail_tmp to block symlink escape * fix(nsjail): allow jail_tmp reuse on sequential nsjail calls Codex flagged that python/ruby/rust executors invoke nsjail twice per job_dir (install then run). The previous resolver treated any preexisting jail_tmp as hostile and silently fell back to tmpfs on the second call, so disk-backed mode never reached the main script run for those langs. Use symlink_metadata().is_dir() to distinguish a real directory left by an earlier call in the same job_dir (safe to reuse) from a symlink or other entity (still refused, as the codebase-tar escape requires). Also loosen the frontend visibility predicate: only hide nsjail settings when job_isolation is explicitly 'none' or 'unshare', so deployments that enable nsjail via DISABLE_NSJAIL=false with no DB setting can still see the controls. * chore(main): release 1.706.0 (#9270) * chore(main): release 1.706.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix(nsjail): gate unix-symlink test behind cfg(unix) for Windows build (#9280) The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls std::os::unix::fs::symlink directly, which doesn't exist on Windows targets. Without a cfg gate, `cargo check --tests` fails on Windows with E0433. Other symlink call sites in this crate (php_executor, bun_executor, rust_executor, etc.) already follow this pattern. Fixes WIN-1972 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reduce slim image vulnerability surface (#9279) * Reduce slim image vulnerability surface * chore(docker): drop apt-get upgrade -y from slim images apt-get upgrade hurts build reproducibility (same Dockerfile + same commit at different times produces divergent images) and trips hadolint DL3005. The freshness it buys is dominated by simply rebuilding against the periodically-refreshed debian:bookworm-slim base image. The --no-install-recommends and apt-list cleanup wins are kept. --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> * fix(git-sync): bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) (#9282) * fix(git-sync): revert LATEST_GIT_SYNC_SCRIPT_PATH to hub/28230 to restore GPG-signed deploys (WIN-1974) hub/28231 (PR #9230) is the "thin" script that hands the actual `git commit` to the CLI's hidden `sync git-deploy`. The hub script still does the GPG setup (import key into a fresh GNUPGHOME, dummy `gpg -bsau` to warm the agent passphrase cache, then `git config user.signingkey` + `commit.gpgsign` locally), but the commit no longer runs in the same `git_push` flow — it runs minutes later inside the CLI after workspace API resolution, zip pull, file extraction, and lockfile autofill. By the time the spawned `git commit` asks gpg-agent for the cached passphrase, the cache state is no longer reliable (or the spawned `gpg` ends up talking to a fresh agent), so signing fails non-interactively with `gpg failed to sign the data`. hub/28230 is hub/28217's in-script logic rebuilt with windmill-cli@1.703.3: the GPG setup and the in-script `sh_run("git commit ...")` happen back-to-back in `git_push`, so the cache is always fresh. It preserves wm_deploy / fork branch behavior, the EE deployment-callback `main()` signature is unchanged, and the only min-version check in EE (`is_script_meets_min_version(28103)`) is comfortably below 28230 — so this revert is safe. Forward fix (separate PR): publish a new thin script that, alongside the existing GPG setup, writes a `gpg.program` wrapper using `--pinentry-mode loopback --passphrase-file` so signing is independent of the agent's cache state. Re-bump past 28231 then. Fixes WIN-1974 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): check in source-of-truth for the next hub script (gpg.program wrapper) This is the script that will be published to hub.windmill.dev once verified on a customer GPG-signed deploy. It replaces hub/28231's agent-cache pre-warm (`gpg -bsau` with --passphrase) with a stateless gpg.program wrapper + chmod-600 passphrase file. Every git-invoked gpg call goes through the wrapper, which always uses --pinentry-mode loopback (and --passphrase-file when a passphrase exists). Signing no longer depends on gpg-agent having a cached passphrase by the time the CLI's `git commit` runs — which closes WIN-1974. Not wired in yet: LATEST_GIT_SYNC_SCRIPT_PATH stays on hub/28230 until this script is uploaded and the new hub id is known. This file is checked in so the diff is reviewable, future bumps have a source of truth, and a CLI regression test can `cat` it for fixture parity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): skip format/pattern validation for $var/$res/$jsonvar references in ArgInput A resource field with a `pattern` constraint (e.g. the gpg_key.private_key field, whose pattern enforces a `-----BEGIN PGP PRIVATE KEY BLOCK-----` prefix) rejects values like `$var:u/me/gpg-private-key` with an "invalid format" error in the resource editor — even though `$var:`/`$res:`/`$jsonvar:` are placeholders the backend resolves at runtime, not the actual string that needs to match the regex. Bail out of all format/pattern checks (email, ipv4, ipv6, uuid, custom pattern) when the value is one of these references. Required/numeric bounds/array checks still apply since they're shape-level, not regex. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(git-sync): bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28234 (gpg.program-wrapper fix) hub/28234 is the forward fix for WIN-1974: replaces hub/28231's agent-cache pre-warm (which became stale by the time the CLI's `git commit` ran) with a stateless `gpg.program` wrapper that uses `--pinentry-mode loopback` (and `--passphrase-file` when a passphrase exists) on every gpg invocation. Bundled CLI is windmill-cli@1.705.0. Verified via reproducer at /tmp/git-sync-diff/test-gpg-fix.sh: deliberately killing gpg-agent between GPG setup and `git commit` reproduces the customer's `gpg failed to sign the data` error verbatim under the old flow, and the wrapper signs through it. Holds for passphrase-protected keys, split-subkey [C]+[S] layouts, and unprotected keys. Drops the local source-of-truth copy (`hub-scripts/`) — hub is canonical now that 28234 is published. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): drop verbose comment above LATEST_GIT_SYNC_SCRIPT_PATH The git history (this PR) carries the why; the constant name + value carry the what. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): wmill sync git-deploy stops committing; caller owns commit+push (#9284) Single contract for the deployment-callback path: the CLI does branch checkout + pull, the caller (hub script in production, test in test) does git add + commit + push. This restores the WIN-1974 invariant — GPG setup and `git commit` run back-to-back in the same process, so the agent's pre-warmed passphrase cache is still warm at sign time — without needing a `--skip-commit` flag for the hub case and a default "also-commit" for everything else. Same behavior in every call site. Changes: - sync.ts: drop the gitSyncDeployPush call from pull()'s deploy path (both the onlyCreateBranch fast-return and the post-pull commit). `gitSyncDeployPush` stays exported for any caller that wants the same commit/push semantics — just not invoked by the CLI subcommand. - gitsync_promotion.test.ts: e2e test now does its own git add + commit + push after `wmill sync git-deploy`, mirroring what the hub script does in production. Same regression coverage (wm_deploy branch created in Case A, main untouched; main updated in Case B, no new wm_deploy). CLI typecheck unchanged (two pre-existing TarAsZip errors at lines 2578/3307, present before this PR). All 743 unit tests still pass. The accompanying hub script (option-C — CLI for branch+pull, script for commit+push) lives at /tmp/git-sync-diff/sync-script-to-git-repo-windmill.option-C.ts. Once published, a follow-up bumps LATEST_GIT_SYNC_SCRIPT_PATH to its id. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bump git sync to 28236 * fix: fork compare visibility for non-admins and stale-token superadmins (#9283) * fix: use fork-scoped authed for fork visibility in compare_workspaces * test: add EE end-to-end repro for fork rename visibility * chore: restore concurrency_locks sqlx cache lost in cleanup * test: add regression for stale-superadmin-token fork visibility bug * chore: update sqlx cache for new test queries * chore(main): release 1.706.1 (#9281) * chore(main): release 1.706.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: add wmill job rerun subcommand (#9275) * feat: add wmill job rerun subcommand * feat: add wmill job restart subcommand for flow restart-at-step * chore(system_prompts): point plugin skills sync at plugins/windmill/ (#9287) * chore(system_prompts): point plugin skills sync at plugins/windmill/ The plugin checkout's plugin folder is being renamed from `plugins/windmill-code-plugin/` to `plugins/windmill/` to shorten the slash-command namespace and align with the matching Cursor plugin layout. Paired with windmill-labs/windmill-claude-plugin#8. That PR must merge first so the next sync run finds the new folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(system_prompts): update plugin-dir example to plugins/windmill Co-authored-by: centdix <centdix@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> * fix(cli): wmill sync pull updates wmill-lock.yaml for raw apps (#9289) * fix: flow recording teardown crash + rename package to @windmill-labs/components (#9288) * fix: guard against null recording during FlowRecordingReplay teardown Navigating away from a flow recording inside a workspace file-tree view threw `TypeError: Cannot read properties of null (reading 'flow')` from FlowGraphViewer once during the teardown tick. Svelte 5 compiles child component props as live getters that close over `$$props.recording.flow`. When `recording` flips to null on the parent's navigation, an outer `{#if !recording?.flow}` doesn't stop those getters from firing one more time as derived effects re-evaluate before the unmount lands — so the getter dereferences null and throws. Fix at the two layers where the deref actually happens: - FlowRecordingReplay: use `recording?.flow` at the binding sites (FlowViewer + graph-snippet FlowGraphViewer) so the compiler emits an optional-chained getter, and guard the snippet branch with `{:else if recording?.flow}` so it doesn't mount when there's nothing to show. - FlowGraphViewer: finish the optional chaining the rest of the file already used everywhere else (`flow?.value?.skip_expr`, `flow?.value?.cache_ttl`, `flow?.schema`). When the upstream binding returns undefined during teardown, the graph degrades to an empty frame instead of crashing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rename package to @windmill-labs/components - frontend/package.json: rename `windmill-components` → `@windmill-labs/components` - frontend/publish.sh: drop the in-place sed rename dance; the checked-in name now matches what's published, so `npm run package && npm publish` is enough - frontend/package-lock.json, system_prompts/auto-generated/prompts.d.ts: regenerated by `npm run package` under the new name Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * default script name * save logic * Keyboard nav * finish keynav * nits * CI fix * nit stop propagation * Merge branch 'main' into feat/asset-graph-view * commit * update * fix: cropped save button on small screens * progress * managed scheduled removed * all * progress * feat: add data upload pipeline trigger with auto S3 picker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: avoid pane editor remount flicker when deploying a pipeline draft Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: show only the edited script's I/O in the asset graph, not the saved version's Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: derive script asset rows server-side at deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: shared fixture corpus keeps annotation parsers in parity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: dev-run draft pipeline chains, live badges, deploy drift warning Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: ungate cascade producers, squash pipeline migrations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop committed cli-sync fixtures and stray screenshots Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: show skip-asset-dispatch flag as badge instead of args row Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: pipeline view mode default with activity feed, drafts overlay chip Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: treat DROP TABLE as table-level write in sql asset parser Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: wmill datatable create + actionable sql extension error Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: ephemeral data-pipelines demo sync repo zip for handoff Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: wmill pipeline list/show renders the asset DAG in the terminal Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * nits * nits * nits * nits * fix: defer draft persist-back past the batch so discard sticks first click Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: band-reserving tidy-tree asset graph layout with join breakpoints Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: route skip-layer and long graph edges around occupied columns Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: seed s3 template outputs with canonical leading-slash paths Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * all * feat: bundle data-pipeline drafts into the DB-backed user draft system Pipeline drafts were browser-only (localStorage `pipeline-<folder>`), so they didn't sync across devices, weren't server-visible, and never showed in the drafts list. Store them instead as one per-user `draft` row of a new `data_pipeline` kind, keyed at the folder (`f/<folder>/data_pipeline`), holding the same `{ drafts, activeDraftPath }` bundle. Stage 1 — backend kind: add `data_pipeline` to DRAFT_KIND (migration) and `UserDraftItemKind` (deployed_table=None, private). The list/update handlers and folder-path access check already cover a backing-table-less kind. Stage 2 — sync: add `GET /drafts/get_own/{kind}/{path}` so an editor with no deployed-overlay GET can load its own draft. The pipeline page now hydrates from the DB on mount (one-time localStorage import for in-flight drafts) and persists via UserDraftDbSyncer (debounce + optimistic-concurrency), keeping a localStorage crash mirror. Stage 3 — surface: the drafts review page renders the bundle as a "pipeline" row that opens `/pipeline/<folder>` (open-only; excluded from bulk deploy). Verified end-to-end in-browser: DB-seeded draft hydrates to "Edit (1)", edits persist back, and the row shows with Open pipeline / Discard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: pipeline Activity panel grouping, run↔graph highlight, deploy-conflict handling Activity panel (view mode): - Group cascade runs by the connected component of the asset-dispatch graph (new GET /jobs/asset_dispatch_edges over the dispatch_event table, incl. join_pending inputs), headed by the earliest originating run + its trigger, with a "+N" chip for joins fed by multiple triggers. - Success/failure count histogram with drag-to-filter brushing, an always-on time axis + per-bar tooltips, a Reset, and Last hour/24h/48h/7/30/90d ranges. - Node run-count/status badges now derive from the same merged historic+live events the panel shows (previously session-only). Run ↔ graph highlight: - Hovering a run row (or a group header → the whole cascade) rings the node(s), animates their incident edges, and borders the adjacent assets in the edge hue (blue write / gray read); expanding a run pins a soft-blue ring. - Switching edit→view re-surfaces the Activity feed. Deploy: - Live-content autosave for the open pipeline draft + an autosave indicator. - Re-saving a script now chains off the hash just created instead of a stale parent_hash (fixes the "lineage must be linear" error on a second save), and a genuine concurrent deploy opens a keep-mine / view-latest conflict modal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pipeline editor badge requires asset-parse, not just main-function parse A pipeline script's asset lineage is load-bearing — a deploy that can't parse assets silently records no edges. The editor "parsable" dot only reflected inferArgs (the main function), so a body the asset parser rejects (e.g. a trailing `/////` in DuckDB) still showed green and deployed with empty lineage. ScriptEditor gains `requireValidAssets` (set by the pipeline pane); when on, the EditorBar badge is green only if BOTH the main function and inferAssets parse, with the tooltip distinguishing "Main function not parsable" / "Assets not parsable" / "Parsable". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: route asset-graph edges around nodes that sit in their path Edges could draw straight through an unrelated node (a join fan-out or long cross-component edge), making it ambiguous whether that node shared the input. AssetGraphEdge only saw its own endpoints, so it could only detour the near-vertical same-column skip case. The canvas now (once per layout, O(edges × nodes) — no per-frame cost) samples each edge's straight run against every non-incident node center and, on a crossing, passes a clear gutter lane to the edge via `data.detourX`; AssetGraphEdge routes the rounded-orthogonal detour through it. Verified: 0 edge↔node box crossings on the orders pipeline. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: deploy pipeline drafts with freshly-inferred assets, not a stale snapshot "Save all" spread `...draft.script` into createScript, which carries a `assets` snapshot that isn't refreshed when the body is edited. So a renamed/removed output (e.g. an old `CREATE TABLE exciting_en32z9` later changed to `exciting_880909`) was re-deployed as a phantom write edge and lingered as an orphan asset on the graph — shown with no producer, and shifting position on click as the graph re-derived. saveDraft now re-runs inferAssets on the current body and passes the result as `assets`, overriding the snapshot — mirroring the per-pane save. The backend clears+reinserts from the sent set, so a re-deploy drops the stale rows. Verified: deploying with the fresh asset set removes the orphan from the graph. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: collect upstream reads from CTAS and CREATE VIEW in SQL asset parser `CREATE TABLE x AS SELECT … FROM y` (and `CREATE VIEW`) recorded only the write to x — the source read of y was silently dropped. Table-level reads are gathered in the `Statement::Query` arm via handle_table_with_joins; the generic table-factor visitor only picks up read-functions and string literals, not plain `FROM <table>` references. The AS-query of a CTAS isn't a `Statement::Query`, so its FROM tables were never walked. On the pipeline canvas this meant a `datatable://…` upstream consumed by a CTAS step showed no read node/edge — the step looked like it produced its output from nothing. Factor the Query arm's read collection into handle_query_reads and call it from the CreateTable (when it has an AS-query) and CreateView arms, balancing the cte_name_stack push in post_visit_statement. Updated the drop_then_create test (which had pinned the old drop-the-read behavior) and added CTAS + CREATE VIEW read coverage. Verified against the rebuilt asset wasm: the live editor now infers the read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * update * updates * refactor: dedup asset-graph code, squash migrations, drop artifacts Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: gate asset dispatch on a cached per-workspace producer set Cache the producer-path→writes map per workspace and invalidate it from the asset-clear paths via the notify_event polling system, so a top-level script/preview completion that isn't an asset producer costs an in-memory lookup instead of a per-completion query. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: remove dead unquote fn that failed backend check under -D warnings Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: green the frontend check (pin published wasm-asset, fix type errors) Pin windmill-parser-wasm-asset to the published 1.728.1 (was a file: link to a gitignored, CI-unbuilt pkg-asset). Exclude test files from svelte-check (the parity test reads a backend fixture via node:fs, which the browser app tsconfig has no @types/node for; vitest still runs them). Fix pre-existing branch type errors: drop the unsupported 2nd getScriptByPath arg, cast script.schema to Schema for inferArgs, coerce has_preprocessor to a definite boolean, and wrap the cancelJob handler so it isn't possibly-undefined. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: move pipeline partition resolution to ee-private (free-CE) Partition resolution becomes a private module (partition_ee in windmill-ee-private, hidden from the public repo) with an OSS no-op fallback (partition_oss); call sites resolve via the aliased windmill_common::partition. Not enterprise-gated — free to run in CE. Bumps ee-repo-ref to the ee branch carrying partition_ee. Verified building in default, private, and private,enterprise (offline). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: move asset-cascade join/debounce/retry to ee-private (free-CE) Join barrier, debounce, and retry become the private windmill_queue::cascade module (cascade_ee in windmill-ee-private); OSS gets cascade_oss no-op fallbacks (plain OR fan-out). Core cascade stays public. Bumps ee-repo-ref. Verified default/private/private,enterprise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: skeleton enterprise pipeline freshness + backfill (TODO, ee-private) Gated windmill_common::pipeline_advanced (private; pipeline_advanced_ee) with OSS fallback; entry points return a clear not-implemented error. Deploy surfaces a TODO when a script declares // freshness. Bumps ee-repo-ref. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: repair asset_trigger_dispatch test after cascade carve-out + cache its queries Stage-2 moved reap_stale_join_slots to windmill_queue::cascade; update the integration test's import. Also commit the test's sqlx query cache (was never prepared with --tests, so SQLX_OFFLINE cargo test failed pre-existing). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: invalidate producer-cache in asset dispatch tests (mirror deploy) The tests seed asset rows directly and run no notify poller, so the per-workspace producer cache went stale across tests → 0 dispatched. Clear it at the seed point, as a deploy would via notify_event. All 8 asset_trigger_dispatch tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to ba677ea142011462ad4dfe77e8375a6dd274cdef This commit updates the EE repository reference after PR #619 was merged in windmill-ee-private. Previous ee-repo-ref: 925c350cff55d3ea738d9e2e4098d9ce4bdda418 New ee-repo-ref: ba677ea142011462ad4dfe77e8375a6dd274cdef Automated by sync-ee-ref workflow. * test: disable producer cache in asset dispatch tests (isolated-DB safe) The .remove(WS) approach still raced: #[sqlx::test] gives each test its own DB but they share one workspace id, so the WS-keyed process-global cache clobbered across DBs under concurrent threads. Add an ASSET_PRODUCER_CACHE_DISABLED test hook and set it in the tests so every dispatch reads its own DB. 8/8 pass at --test-threads=10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: replace asset-cascade depth cap with cycle detection The hardcoded MAX_CHAIN_DEPTH=5 truncated legitimate deep pipelines (silently — the check returned before event logging). Replace it with per-edge cycle detection: carry the producer lineage in trigger.chain and skip only a subscriber already in the chain, recording a visible cycle_detected dispatch_event. Acyclic pipelines of any depth now cascade fully; a high MAX_CHAIN_LEN backstop guards against runaway. Tests + UI label updated; 8/8 pass at --test-threads=10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: update dispatch_event reason examples (depth_cap → cycle_detected) Comment-only; the migration is idempotent and already in the potentially_stale self-heal list, so the checksum change re-applies cleanly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: park cascade retry (P1 dead-end) + clear stale script_triggers on rename Two deploy-path fixes: - Retry is parked: a retried subscriber is wrapped in a SingleStepFlow, whose run is a flow step and ineligible for asset dispatch, so it would silently dead-end the cascade (P1). Stop persisting retry to script_trigger and warn at deploy; TODO(pipeline-retry) to re-enable once dispatch handles flow-wrapped producers. (Dispatch plumbing kept + still tested via direct seeding.) - Rename leaves stale script_trigger rows: clear was keyed on ns.path only, so old-path '// on' edges lingered and could trigger a script later recreated at that path. Also clear the old path on rename (assets already handled via the parent-hash clear). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> Co-authored-by: hugocasa <hugo@casademont.ch> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Arnaud <31803803+Araden14@users.noreply.github.com> Co-authored-by: Diego Imbert <diego@windmill.dev> Co-authored-by: centdix <40307056+centdix@users.noreply.github.com> Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Aldrin Jenson <aldrinjenson@gmail.com> Co-authored-by: centdix <centdix@users.noreply.github.com> |
||
|
Ruben Fiszel
|
aff0a4ec18 |
fix(security): sanitize dependency names & connection strings against command/SQL injection (#9590)
* fix(security): sanitize dependency names & connection strings against command/SQL injection Follow-up to the PowerShell module-name injection fix (#9587, CWE-78): the same audit surfaced the identical "secondary identifier interpolated into an interpreter/SQL command without escaping" pattern in a few other executors. - R executor (the real twin, HIGH): package name/version parsed from a user-supplied renv.lock were interpolated raw into an `Rscript -e "...renv::install(\"{pkg}@{version}\"...)"` eval string. A double-quote in the name/version broke out → arbitrary R (unsandboxed under DISABLE_NSJAIL / non-Linux). Now validated in parse_renv_lock (charset) and escaped at the sink as defense-in-depth (also escapes the lib path, which holds backslashes on Windows). - DuckDB ATTACH (MED): the connection string built from resource fields (host/db/user/password) is embedded in a single-quoted DuckDB literal; escape quotes so a field value can't break out of the ATTACH statement. - PgDatabase::to_uri: URL-encode host and dbname (user/password already were), so '@'/'/'/'?'/'&' can't reshape the parsed URI (feeds live PG connect and DuckDB ATTACH). - DuckDB CREATE SECRET (FFI): wrap the interpolated S3 key/secret/endpoint in the existing sql_single_quote() helper, consistent with the resource-limits setup right above it. - PowerShell: also escape the configured private repo URL/PAT in the install template (same sink as the module names; the escape landed after #9587 was squash-merged so it was not in the merged change). Adds unit tests for the R validation/escaping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(security): escape ducklake/snowflake/bigquery identifiers; preserve IPv6 host - to_uri: don't percent-encode bracketed IPv6 literal hosts ([::1]) — encoding their brackets/colons would stop them parsing as a host (review fix). - duckdb ducklake ATTACH: the catalog conn string, storage and data_path are embedded in single-quoted DuckDB literals; escape quotes so a resource field can't break out (the ducklake path bypassed the ATTACH escape added earlier). - snowflake: validate account_identifier (it forms the request hostname). - bigquery: validate project_id (it forms a request URL path segment). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
6acce7a887 |
fix(powershell): sanitize module names to prevent command injection (CWE-78) (#9587)
Module names parsed from user-supplied lock content (parse_modules_json)
and script imports (parse_script_imports) were interpolated directly into
PowerShell single-quoted string literals when building the module install
command, with no escaping or validation. A module name containing a single
quote could break out of the literal and execute arbitrary PowerShell.
Two-layer fix:
- Validate module names against ^[a-zA-Z0-9._-]+$. parse_modules_json
rejects invalid names with an error; parse_script_imports skips them
with a warning.
- Defense-in-depth: escape single quotes (' -> '') on both name and
version before interpolating into the install hashtable literals,
mirroring val_to_pwsh_param.
Fixes WIN-2049
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben Fiszel
|
317a8629d1 |
fix(ai): enforce resource authz when loading MCP tools in agent worker (#9571)
* fix(ai): enforce resource authz when loading MCP tools in agent worker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: describe MCP authz test invariants instead of drafting history Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |