56 Commits

Include renames

Author	SHA1	Message	Date
3 people	6186a0645d	feat: edit variables, resources and triggers in their own session tab (#11206) ... * feat: edit variables, resources and triggers in their own session tab Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: take a trigger's flow kind from its config and name the written trigger Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: one operating-workspace context for editors acting on a session's workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: reload page item tabs only in sessions acting on the tool's workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve open in workspace for every preview tab kind in one place Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: guard every component under a session editor against the navigation workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: show a page item tab whose item is gone instead of mounting its editor Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: follow a saved page item in the tab's own workspace and restore in-frame rows Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: close a page item tab from its editor header Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: judge trigger permissions by the user acting in the session workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: judge every session-editor permission by the user acting in that workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep session permissions reactive to the acting role and restore tabs as last seen Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: default Path's owner controls to the user acting in its workspace Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-09-19 23:29:14 +02:00
Ruben FiszelandClaude Opus 5	6860521b4e	refresh a dbt column trace with its graph, and stop calling whole ones cut (#11015) ... * fix: refresh a dbt column trace with its graph, and stop calling whole ones cut Follow-up to #11014, addressing two findings from the review round that landed after it merged. **A deliberate graph refresh now re-asks for the lineage.** The dedup key held only the workspace, the pin and the seed relations, all of which a redeploy leaves alone — so the pipeline page's Refresh refetched the graph and left the trace as it was, pairing the new version's SQL and columns with the previous one's edges. The key now carries which fetch of the graph is on screen, taken from `graphRes.current`'s identity: it moves on a Refresh, a deploy and a folder switch, and on nothing else, so an editor keystroke still cannot make the pane re-ask. **`truncated` is set only with evidence.** `pending` was read as proof the component had been cut, but it only says a relation's owners have not been asked about yet — and those owners are usually the project already in hand. A project holding more than the expansion budget across unrelated families therefore reported a small, complete component as truncated. The owners query now runs before the budget and round stops, so a trace is called cut only when a project this caller may read is left unread, or when the walk itself was cut. Two smaller things from the same round: a failed lineage request says so instead of rendering the empty trace a project without the analysis pass renders — the two were indistinguishable, and a Refresh now retries it — and `asset_paths` is capped as well as refused when empty. `MAX_HELD_EDGES` is renamed `EXPANSION_EDGE_BUDGET`: it never bounded what its name claimed, since the seeds' own projects are read whole whatever their size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NY4kuFy2jAnGEzaCc1CseL * fix: show a failed column-lineage request beside a partial trace, not only instead of one Review-round findings on this PR. The failure line only rendered when the trace had no nodes at all. A ducklake selection whose producers declare column lineage has nodes from the graph the canvas already carries, so a failed dbt request left a trace that rendered and was missing a half — which is the reading the line exists to prevent. It now renders beside a drawn trace as well, and says the trace may be incomplete rather than that nothing loaded. The dbt branch of the details pane also opened on `selectionColumnLoading` but not on the failed state, so a relation with neither SQL nor a column schema fell through to "no inline preview" and the line never rendered at all. Dropped "Refresh to try again": the dbt editor has no Refresh for this, and its recovery is a re-parse or reselecting. The comment on the error handler says both paths again rather than only the one the pipeline page uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NY4kuFy2jAnGEzaCc1CseL * fix: clear a column-lineage failure when the next request goes out Round-2 nits, both reviewers on the same state. `failed` was cleared only when an answer landed, so a retry kept saying the trace may be incomplete while it was being fetched, and a new selection inherited the previous one's failure until its own answer arrived. It is cleared as the request goes out instead. Also documents the bounds on `asset_path` in the two routes that take it: the 1000-relation cap and the at-least-one rule were both enforced and neither was written down, so a caller met them as a 400 with no way to have known. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NY4kuFy2jAnGEzaCc1CseL --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-08 11:41:05 +02:00
Ruben FiszelandClaude Opus 5	33f9828c3e	feat: draw a dbt column trace, across projects and the pipeline boundary (#11014) ... Serves the edges `dbt_column_edge` has been storing. `assets/column_lineage` answers the connected component a set of relations' columns sit in, and the details pane draws it beside the model's SQL — in the dbt editor, on the pipeline page, and for a run through `jobs/dbt_column_lineage/{id}`. The unpinned component crosses projects. A relation one project produces is another's source, so resolving owners once — for the relations asked about — stops the trace at the first boundary. Owners are resolved to a fixpoint instead, and the caller's gate is re-applied to every project the expansion discovers: reaching a relation says nothing about who may read the project on the far side of it. A pinned answer needs none of it, by version or by job: the pin says which stored graph is on screen, and another project's live graph is not part of it. One request per selection, whatever it reaches: the endpoint takes every relation at once and answers their union, so nothing is held between selections and there is no staleness, retry bookkeeping or per-click dedup to balance. The answer is bounded. A synthetic 3000-model project whose models share a column has 58k direct edges and returns 7.3MB, which no column diagram can draw; the walk is breadth-first from the asked-for relations and stops at 5000 edges, so what survives is the part nearest the selection, and `truncated` says the trace was cut rather than ended. Also adds the columns section the pipeline page's asset pane was missing, so `column_schema` is visible there and not only in the dbt editor. Claude-Session: https://claude.ai/code/session_01NY4kuFy2jAnGEzaCc1CseL Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-08 08:29:36 +00:00
Ruben FiszelandClaude Opus 5	0139467b01	feat: ingest dbt column lineage and real column schemas from the engine's parquet index (#10977) ... * feat: column-level lineage for dbt from the engine's parquet index `manifest.json` carries no column-to-column edges, which is why decision 14 recorded column lineage as unavailable. The edges live in a different artifact: `dbt compile --static-analysis strict --write-index` writes `target/index/`, whose `dbt.column_lineage.parquet` holds them and whose `dbt.node_columns.parquet` holds every column of every node, typed and ordered rather than only the ones an author documented. Strict analysis rejects SQL the default accepts, so this is a separate compile with its own `--target-path`, opt-in per project via `column_lineage: true`, and best-effort throughout: a project it cannot analyze keeps exactly the graph it had, with the engine's own diagnostics in the job log. Storage mirrors `dbt_edge`: `dbt_column_edge` keyed by (path, version, job) with the same composite FK to `script` and the same sweeps. The typed column list lands in `dbt_node.column_schema`, beside `columns` rather than merged into it, so `columns` stays what the author declared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: address review findings on the dbt column-lineage pass - The workspace fork copied every other dbt sidecar table and not this one, so a fork lost its column lineage silently and could not recover it: the cloned digest covers the column edges, so a dynamic run in the fork matched it and stored nothing. - The parquet was collected whole before the edge cap applied, which is exactly the input the cap exists for — a project whose `scan` lineage is quadratic in its widest model could take the worker process down. Decoded a row at a time with the bound enforced during the decode. - The pass swallowed every error from the runner, including the job poller's cancellation and deadline, so a run that blew its timeout inside an optional annotation could still publish a graph and report success. `run_captured` now carries the exit status in its value, so only a failed COMPILE is downgraded, and the pass may spend at most half the remaining wall clock so it cannot starve the build that follows it. - `scan` edges are stored but no longer served: they are most of a project's lineage, nothing renders them, and the graph endpoint is polled by the run page. They are also the first thing the storage cap gives up now, rather than evicting the direct edges the trace draws. - `column_schema` and the column edges take the same gate as the model's SQL. A column-level view is the shape of what the author wrote, one level finer than the `ref()` graph, which is ungated only because it draws relations the caller already sees. - `graph_digest` hashes the new section only when it has edges, so a project that never asked for the pass keeps the digest it has instead of re-snapshotting on every dynamic run until it is redeployed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: the editor buffer's column lineage, and three bounds that were wrong Round-2 review found four defects, all of them introduced by the round-1 fixes. - The `script_visible` gate on the column edges was copied from the node query without its `script_hash IS NULL` arm. `= NULL` is never true, so every version-less row was filtered out and an editor buffer's parse rendered its typed columns and none of their lineage — the one place the feature is meant to be used. Pinned by an assertion in `dbt_pinned_graph.rs`, which is where this class of bug already had a home. - The phase budget was handed to the poller, whose expiry is an `Err` indistinguishable from a cancellation or the job's own deadline, so a slow but valid analysis aborted the build it exists to annotate. The runner gets the full deadline again — those two must still fail the job — and the budget is a race around the whole pass, where expiring is this budget and nothing else. - The decode cap counted parquet ROWS, so `scan` and out-of-graph rows could spend it before a single drawn edge was read. It now counts what is kept, takes direct kinds in a first pass, and is handed the graph's own nodes so the budget cannot go on rows that could never be stored. - Hashing the new digest section conditionally did not preserve old digests, because an absent `column_schema` still serialized as `null` inside the nodes. It is skipped when absent instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: split the lineage pass by error contract, and read it in one query Round 3's findings were all consequences of round 1 and 2's fixes, clustered in the same two files, so this reshapes those two seams rather than patching again. The worker pass was one function being three things at once — a subprocess runner with job-lifecycle error semantics, a bounded decoder, and a best-effort degrader — which is why each fix to one perturbed another. It is now `compile_index`, which owns the JOB's semantics (only a cancellation or the job's deadline can `Err`; a non-zero exit, the output ceiling and the phase budget are outcomes), and `read_index`, which owns the ARTIFACT's and knows nothing about the job. The budget wraps the compile alone, so a decode can no longer outlive the timeout that reported the build would get the rest. The output ceiling likewise becomes a value rather than a job error, for the caller that can carry on without the tail of a compile's stdout. The column edges were read by a fourth hand-written copy of the `live`/`chosen` CTEs and the version/editor-buffer join conditions, and copying them is what dropped the `script_hash IS NULL` arm and hid every buffer parse's lineage. Both kinds of edge now come from ONE statement over a `UNION ALL`'d edge source, so those conditions exist once. The union is at the source rather than a join because column lineage can name a node pair `dbt_edge` has no row for: a model reading `{{ this }}` gets edges from itself to itself, and `parent_map` has no self-loop. The cap on the column half now sits after the scope filter, the visibility check and the graph joins — the scope moved into SQL via the existing `ScopePathFilter` — so a row the caller may not read can no longer spend it and leave an allowed project's trace short. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: serve dbt column lineage from its own endpoint The column edges rode on the folder-wide asset graph, which a run page polls, while the trace is drawn for one selected relation. That needed a cap, and a cap has to be applied after every filter that can drop a row. Keyed to the asset there is no cap: `assets/column_lineage` answers for one relation, and the caller's `scripts:read` scope and the project's visibility are decided once, for the script that owns it. Pinning to a run's snapshot or the editor's parse of its buffer costs the job-read gate, so that form is `jobs/dbt_column_lineage/{id}` — the same shape `jobs/dbt_graph/{id}` has. The worker's decode now bounds work and memory separately, and a compile stopped by the output ceiling reports as truncated rather than complete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve the owning dbt version the way the graph does The unpinned arm picked the newest live version at the path without narrowing to dbt, so a path since redeployed in another language answered with no lineage while the graph beside it still drew that project's stale nodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate pinned column lineage on reading the project, and answer the component Four things round 5 found, three of them in code this branch rewrote: - The pinned arm resolved the version from the job and stopped there, so a share-link viewer entitled to a run got the project's column names and edges while the graph beside it still redacted `raw_code` and `column_schema`. Resolving WHICH version answers is not deciding whether the caller may read it; the version-less editor buffer keeps its exemption, having no `script` row to ask. - The answer was the whole owning project's edges. The canvas lays out the connected component of the selected relation's columns, so the rest was unrenderable weight; a recursive walk over both directions returns exactly what is drawn, and the project key travels with it so a `unique_id` two projects share cannot walk from one graph into the other. - The decode had no exit but the 4M-row backstop once its buckets were full, spending wall clock the build below does not get. - An unreadable index was reported as a missing one, sending the reader to look at their engine rather than at the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stitch the two column graphs, and walk the component in Rust Round 6's two findings, both regressions this branch introduced: - The decode returned `Continue` on the edge that FILLED the direct-edge budget, so a `scan`-only tail after it decoded to the 4M-row backstop with nowhere to put anything. The read now ends on that edge. - Seam 3 made the pipeline page choose between the dbt graph and the producer one. They share node ids — `// column total <- dbt://wh/analytics/orders.amount` mints the same `(dbt, path, column)` node dbt's own lineage does — so choosing ended a trace at the boundary in both directions. They are merged again, and a ducklake selection asks about the dbt relation its producers name so the chain continues past it. The dbt editor gets the same merge. Also: the component is walked in Rust rather than by a recursive CTE. A CTE has no index, so the recursive term rescanned the doubled edge set once per level — 1243ms against 59ms for the query alone on a 3000-model project, 11.7M rows in the plan. Same answers, same tests; end to end 1.48s to 0.73s there and 1.60s to 0.26s on a 1000-deep chain. The client stops re-asking for a component it already holds, which is most clicks within one project. The four doc sites that described a whole-project answer are rewritten around what it now is, rather than edited where they disagreed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: expand every dbt boundary a selection reaches, and only skip what was asked Round 7's findings, all in the frontend seam this branch added: - A ducklake selection seeded the dbt fetch from the FIRST boundary relation it found, so a table derived from two unconnected dbt relations expanded one and left the other a leaf — the same "stops at the boundary" symptom the round-6 fix removed, one hop further along. Every distinct boundary is fetched now and the components merged. - The component cache skipped a relation merely PRESENT in the graph in hand. A relation two projects describe has an owner row in each, and a component fetched for one carries it as an endpoint without the other's half, so that skipped the request that would have resolved the second owner. Only a relation actually asked about under this pin is skipped. - A comment still called the producer graph gated to ducklake selections after it was widened to dbt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: land dbt column lineage as storage and ingest only The API surface that draws a column trace moves to a follow-up PR, on `dbt-column-lineage-surface`. It kept generating findings — a client cache whose premise was wrong for a two-owner relation, then staleness and a lost retry from tightening it, and a seed walk that stopped at the first boundary — and the fix for the last of them is a transitive owner expansion, which has to re-apply the caller's gate to every newly discovered project. That is the same shape as the leak four reviewers caught in the pinned arm, and it wants its own review rather than being the fourth fix at the end of this one. What lands here stands on its own: the analysis pass, `dbt_column_edge`, `dbt_node.column_schema`, the engine gating and the error-contract split — plus the one user-visible half, the typed and ordered column list, which rides the asset graph the details pane already fetches and replaces a panel that could only show the columns an author had documented. Also fixes a real bug in the pass, found in review: it compiled without the build's `--full-refresh`. `is_incremental()` branches on that flag, so an incremental model reading `{{ this }}` compiles its self-join — and any `ref()` inside that branch — only when the flag is absent, and the pass was storing lineage for SQL a full-refresh run never executed. The flag now comes from one place shared with the build, and a run that overrides it gets its own graph rather than standing as the version's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say why direct kinds get the budget without naming a view The bucketing comments explained the priority by what a trace draws, which is a forward reference now that the surface moved out. The reason stands on its own: `copy`/`mod` say the value travelled, `scan` says the column was read to produce the row and so reaches every output column of its model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: round-9 findings on the descoped PR - The `full_refresh` helper was inserted between `selection_is_overridden` and its doc comment, so thirteen lines about `select`/`exclude` echoes documented the wrong function and the one they were written for had none. Moved below it. - The parse path ran the analysis compile and the parquet decode BEFORE the guard that returns when there is no warehouse identity, paying for both and dropping the result. Moved after it. - Three sites still described a `/column_lineage` endpoint this branch no longer has, and two user-facing strings promised a column trace it no longer renders: the panel's hint and the descriptor template now say what the flag actually buys, which is the typed column schema. - Dropped test scaffolding the removed suite left behind: a `raw_orders` node and `dbt_edge` whose only assertion re-tested pre-existing graph behaviour, and a second editor-buffer node nothing asserts on. Documented rather than fixed: an incremental model has two shapes, and which one the index holds depends on whether the target existed when the pass ran. `is_incremental()` is false with no target as well as under `--full-refresh`, and dbt has no mode that emits both — so a version's graph describes the compile that produced it, and only a re-ingesting run describes its own run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep lineage_kind in the edge key, and one answer for --full-refresh - Both unique indexes omitted `lineage_kind`, so a column that is projected AND used as a predicate for the same output column — an ordinary shape — had its `copy` and `scan` edges collapse under `ON CONFLICT DO NOTHING`, while the digest counted both. The kind is part of the fact, so it is part of the key. Edited in the migration rather than added as a second one: it has not landed. - `full_refresh` was shared between the build and the analysis pass without the `command != "test"` condition that sat at the build's call site, so the two disagreed for exactly the runs that build nothing. The condition moved inside the function, which is the point of sharing it, and the command is threaded to the pass. - The "what a trace draws" rewrite missed the copy in `dbt_manifest.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop the unreachable full_refresh threading, test the uniqueness key `DBT_COMMANDS` is `["build", "retry", "show", "parse"]` and `default_command` returns `build` in every arm, so `command == "test"` cannot happen — the guard the last commit moved into `full_refresh` was already inert where it came from. Threading the command through five signatures to preserve it bought nothing, so it is gone; the build and the pass call one function of the descriptor and the invocation, which is what the sharing was for. The uniqueness-key fix now has a test: a column projected AND used as a predicate for the same output column stores both its `copy` and its `scan` row. Verified against the old key, where it returns 1 instead of 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: restore the dbt test --full-refresh guard I removed on a wrong premise The previous commit removed it after reading `DBT_COMMANDS` and concluding `"test"` was unreachable. That is only true of the command a CALLER can name: `run_dbt` is invoked with `"test"` directly for the `after_all` test phase, so an `after_all` project with `full_refresh: true` reached it — and dbt rejects `--full-refresh` on `test`, failing the phase. Both reviewers caught it. The guard is back inside the shared function, where the build and the pass get one answer, and its doc now records why reading the allowlist alone is misleading. The test covering the `test` case is restored with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: notice a job that ended during the decode, and name truncation as the cause - The parquet decode runs on a blocking thread with no poller watching it, so a cancellation or an expired deadline during it was invisible: `dbt_dep` went on to publish the graph and the job returned success. The job's state is checked once the decode returns, before the caller publishes anything, and an ended job `Err`s — which this module may always do for the job's own semantics. - A compile stopped by the output ceiling could leave no artifact, and the log then blamed the engine's capability, sending the reader to check their adapter rather than the ceiling. Truncation now names itself in the missing and unreadable branches too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read cancellation from the DB after the decode, not from a poller's field `ctx.canceled_by` is only ever written by a poller, and no poller runs during the blocking decode — which is the exact window the check was added for. So the guard caught only a cancellation already observed before it, and the comment beside it claimed more than it did. It now queries `v2_job_queue` directly, the same probe `worker_lockfiles` uses before it overwrites a flow. A failed probe answers "still running": this decides whether to discard work already done, so an unreachable database must not be the reason a healthy deploy loses its graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: reuse job_is_canceled rather than a second copy of it The probe added last round was `job_is_canceled` from the same file, retyped — same query, same `Connection::Http` behaviour. Reused instead. Its doc said a non-database connection was "a failed probe", which reads as an error path. It is not: it is the agent worker, and on one there is no database to ask, so only the deadline answers and a cancel issued during the decode is not observable. The retry path avoids that by refusing to run on an agent worker at all — which an optional annotation has no business doing — so the gap is recorded at both ends instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: close the agent-worker cancellation gap instead of documenting it The previous commit said a cancel issued during the decode is not observable on an agent worker. It is: `ping_job_status` returns `canceled_by` over both connection kinds, and is how the poller itself notices one there. So the check asks through the ping rather than querying `v2_job_queue` directly, and holds on an agent worker, where a direct query reaches no database at all. `job_is_canceled` goes back to private and its doc to what it said before — the retry that calls it still refuses to run on an agent worker for its own reasons. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decode the index under the job poller instead of checking after it Two findings with one cause: the decode was the only phase of this pass with no subprocess behind it, so nothing heartbeated while it ran. A large index left the worker silent for as long as it took, which the zombie sweep reads as a dead job and restarts — and the cancellation check bolted on afterwards could only ever report what had already happened, while dropping the ping's `already_completed`, so a force-cancelled deploy still published its graph. Running it under `run_future_with_polling_update_job_poller` answers all of it: the poller pings throughout, and ends the phase with an `Err` on cancellation, `AlreadyCompleted` or the phase timeout. The bespoke probe is gone with it. Verified on a live deploy: 32 edges and 4 typed schemas ingested through the polled decode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop a cancelled decode, and say what the read phase can now do Putting the decode under the poller heartbeats it and ends the phase when the job does, but dropping a `JoinHandle` detaches a blocking task rather than cancelling it — so a cancelled job left a thread decoding up to four million rows for a job that was over. The row loop reads an abandonment flag that a drop guard on the awaiting future sets, so the decode stops at its next row. That same change made the read phase able to `Err`, and three places still said it could not — decision 14 in as many words. The distinction that holds is narrower: nothing the ARTIFACT does or fails to do can fail a job, so absent, unreadable and partial are all values; the JOB can still end the phase the read runs in. Stated that way in the module doc, the `Artifact` doc, `MAX_INDEX_ROWS` and the decision. Verified on a live deploy: 32 edges and 4 typed schemas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: share AbortOnDrop, and stop citing a hazard that is now handled `Abandon` was `ansible_executor`'s `AbortOnDrop` retyped — same struct, same reason, same `spawn_blocking` shape. Moved to `common` and used from both. The paragraph explaining why the phase budget wraps the compile alone gave as its reason "a decode still running on a blocking thread", which is exactly what the abandonment flag now prevents. The reason that survives is the one that was always the point: the budget exists to leave the build its share of the clock, and only the compile can spend that share unboundedly. The decode's end is the job's, through the poller it runs under. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: put both doc comments back on the items they describe Moving AbortOnDrop orphaned a doc at each end: it landed between `raw_to_string`'s doc and `raw_to_string`, and the doc of the struct it replaced stayed behind to prefix `fetch_repo_archive`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the binding the row loop actually reads `Abandoned` was neither the type nor the binding; the flag is `abandoned`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-08 09:58:38 +02:00
Ruben FiszelandClaude Opus 5	c6e0302d7c	feat: let // materialize declare a dbt:// warehouse-relation write (#10978) ... * feat: let `// materialize` declare a `dbt://` warehouse-relation write `// materialize manual dbt://<warehouse>/<schema>/<name>` lets an ingestion script in any language declare that it writes a warehouse relation, so it and the dbt model reading that relation land on one asset node instead of two disconnected pictures. `manual` is the only mode a warehouse target has — nothing generates warehouse DDL — and the non-`manual` spelling is refused rather than silently degraded. The `<warehouse>` segment is resolved against the workspace's configured warehouses, like a descriptor's `profile.warehouse`. The run records the same `materialized_partition` row a DuckLake target does, from the generic job path rather than an executor: the DuckLake write engine is DuckDB's, this declaration is anyone's. With a non-dbt producer now possible, the blanket deploy-time refusal of `# on dbt://<relation>` narrows to the shape that still cannot fire — every writer of the relation being a dbt script, since a dbt run does not dispatch. "Nothing produces it yet" stays accepted, as for every other asset kind, so deploy order does not matter. A dbt script may not subscribe at all: its graph ingest clears its own `dbt://` trigger rows. The one ordering the deploy cannot catch — a subscription accepted before any producer, then claimed by a dbt project — is named in that project's deploy log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rw1WrKeRRzyYHjfkuB83ek * fix: address review — preview stamping, stale producer set, public doc Three findings from the local review round: - Record the warehouse write only for a DEPLOYED script job. The annotation is a deploy-time contract (`manual`, three segments, a configured warehouse) checked where write access to the path is also required; honouring it in a preview, hub or inline-flow body let `jobs:run` alone restamp any relation's last writer from a script that never touched it. - Exclude the deploying script's own rows from the producer set. Read committed, they describe the version being replaced, so a script dropping its `// materialize` while adding a subscription counted itself as the producer that would wake it and committed a dormant edge. It could not be that producer anyway — the dispatcher skips self-loops. - `AssetKind::Dbt`'s doc no longer claims dbt is the exclusive producer of a warehouse relation, on both the types and the parser enum. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: review round 1 — dbt-script materialize, set-form rule, doc - Refuse `// materialize` on a dbt script, the producer half of the rule the trigger loop already applies to `// on`: the graph ingest republishes that path's asset rows wholesale, so a declared write is wiped by the deploy that accepted it while its runs keep stamping the relation. - `dormant_dbt_subscriptions` now spells the same predicate its singular sibling does: the producer set has to be non-empty (nothing produces it yet is deploy order, not a dormant edge) and excludes the subscriber's own path (a script never wakes itself). Both divergences are pinned by tests. - The docs no longer claim the dbt deploy log covers a native producer that drops its `// materialize`; it does not, and nothing else reports that case. - An integration test over the deploy contract, since only a real deploy proves the handler feeds `sole_dbt_producer` the canonical key `asset.path` holds — the spelling that has to agree across the materialize target, the `// on` ref and the refusal that joins them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: qualify the any-language claim, and pin the dbt-script refusal `AssetKind::Dbt`'s contract (both enums), the two runtime guides and the deploy comment said a script of any language may declare a `dbt://` write, which the dbt-script refusal added last round contradicts. They now say "any language but dbt's own", with the reason: a project's writes are read from its manifest. The deploy-contract integration test covers that refusal for both annotations. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: teach the pipeline AI guidance the warehouse-relation target The pipeline prompt (both sources, plus the regenerated bundle) told the model `// materialize` is DuckDB-only and rejected on any other target, which now steers users away from the very thing this PR adds. It distinguishes the managed DuckLake write, still DuckDB-only, from the warehouse-relation declaration any language but dbt's own may make. `dbt_manifest.rs`'s module doc carried the same "the only thing that creates one" overclaim the other four sites lost last commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: draw an explicit dbt:// subscription on the canvas The editor suppressed every `// on dbt://…` overlay, which was right while the deploy refused all of them. It now refuses only a relation dbt alone builds, so the suppression hid the author's own annotation for exactly the case this PR adds — a subscription woken by a native `// materialize manual dbt://…` producer. The deploy stays the gate. Also the two stale claims round 4 named: the live pipeline prompt dropped the dbt-script exception the base prompt carries, and the doc's e2e requirements still said every `dbt://` subscription is refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse `// data_test` beside a `dbt://` materialize target `// data_test` checks are verifier probes the DuckDB executor splices around a managed write. A warehouse relation is written by the script itself, in any language, so nothing would run them — and unlike the DuckLake `manual` case, which at least fails loudly in that executor, a declarer in another language deployed green with its data-quality assertions silently skipped. Covered in the deploy-contract test and documented beside the annotation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exclude a renamed producer from the sole-dbt producer set The producer set already excluded the deploying script's own path, because its committed rows describe the version being replaced. Under a rename the write sits at the OLD path — still committed, and removed by the same uncommitted transaction — so a producer renamed while it drops its `// materialize` and adds `// on dbt://…` still counted as the producer that would wake it, and committed a dormant edge. The deploy-contract test covers it: without the exclusion the rename deploys 201 instead of being refused. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: take the rename test's parent hash from the create response `format!("{:x}", …)` over the stored i64 drops leading zeros, while `ScriptHash`'s deserializer hex-decodes and demands 8 bytes — so a hash below 2^60 would 422 the request instead of reaching the refusal it asserts on, on roughly one in sixteen spellings of that script body. The create response already carries the zero-padded form, as the rest of the suite uses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the concurrent-ingest interleaving honestly `sole_dbt_producer`'s doc claimed the concurrent-deploy race only ever resolves toward refusing. It does when the uncommitted producer is native; when it is the dbt ingest, the check sees an empty producer set and accepts, and if that ingest then commits and runs its warning query before the subscriber's trigger row lands, neither side reports the dormant edge. Not serialized: the two would have to share a per-relation lock, and the ingest takes `script … FOR UPDATE` before its own advisory lock, so a deploy holding relation locks first inverts that order into a cross-subsystem deadlock — a worse failure than the cosmetic edge. Recorded beside the other orphaning the deploy cannot catch, with the bound both share: the next deploy of that project warns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse a `dbt://` subscription that is not a whole relation `# on dbt://main/analytics` deployed and persisted a trigger row. Every producer spells `<warehouse>/<schema>/<name>` — the manifest ingest derives it from `relation_name`, a `// materialize` target is checked against it — so a partial one is an edge nothing can ever wake, which is what the dbt-only refusal exists to prevent. The shape now has one definition (`is_full_relation_path`) that both halves of the deploy ask, rather than a segment count spelled twice: a subscription and a write that disagreed would refuse and accept the same string. Also rewrites the canvas test's comment as a current constraint per AGENTS.md. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hold both halves of the deploy to one `dbt://` relation validator A subscription checked the relation's shape but not its warehouse, so `# on dbt://<unconfigured>/<schema>/<name>` deployed and persisted a trigger row for something no producer can ever write: the write side refuses that exact string, and a dbt project's `profile.warehouse` resolves against the same config, so no later deploy fixes it and the dormant-edge warning cannot report it either. The shape rule and the warehouse rule now live in one `validate_dbt_relation` that both halves call, rather than being spelled per site — the previous two rounds each closed one half of one rule, which is the drift that invites. Also moves the parser test out from between a comment and the test it documents, and names both refusals in the doc's list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop the subscription-only clause from the shared refusal message "so nothing can produce it" reads backwards on the `// materialize` side, which is the producer. The remaining sentence says what is wrong on both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound a `dbt://` relation by the asset-path column in the shared validator `asset.path` is VARCHAR(255) and the manifest ingest drops a relation that outgrows it rather than failing the whole graph, so past the column no producer row can exist on either side. `script_trigger.trigger_ref` is unbounded text, so an overlong subscription deployed and stayed dormant for good; an overlong write reached Postgres and failed the deploy on a `value too long` instead of a message. Both now refuse in the validator the two halves share, against the ingest's own constant. The integration case computes the ref from that constant so it cannot drift back under the bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report a warehouse-lookup failure as the failure it is, and correct the boundary `dbt_warehouse_exists` fails three ways — no such warehouse, the query itself, and a setting with no `resource_path` — and all three became a 400 blaming the user's warehouse name. A pool timeout mid-deploy told a retrying sync that a transient server error was a permanent client one. Only `NotFound` is the annotation's fault now. The known-boundary paragraph claimed a flow-runner run still cascades. It does not: it is routed by `flow_step_id`, which `is_eligible_kind` rejects, as `asset_trigger_dispatch.rs` pins. Recording and cascading are decided separately, so the paragraph now names all three routes rather than merging two of them — and the row it omitted, an ordinary flow step, which records and never cascades. E2E item 7 said "deployable" where the rule is "wakeable": with only the dbt project reading the relation the producer set is empty, which deploys fine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct two rationales the last commit got wrong `Error::SqlErr` already maps to 400 in this codebase, so the query case's status was never the thing at stake. What the `NotFound` match earns is that a query failure and a malformed setting stop being described as an unconfigured warehouse name, and that the malformed-setting `InternalErr` reaches its own 500 instead of being flattened. And a flow step is two shapes, not one: a step running a deployed script is a `Script` job that records and never cascades, while a step with an inline body is `FlowScript`, which the recording guard excludes along with previews. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: warn about dormant subscriptions from the run that publishes ownership too A run whose static descriptor finds its profile moved re-ingests the version's graph and republishes path ownership, exactly as a deploy does — so it can be what leaves a subscription accepted while the relation had no producer with dbt as its only one. That path discarded `persist_ingest`'s result and emitted no warning, which also made the doc's enumeration of unreported orphanings wrong. Both ownership-publishing points warn now. An agent worker still cannot: it reaches these tables only through the API and its ingest publishes without reading back, which the doc now says. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: an agent run publishes no ownership, and the warning has two callers The agent-worker sentence called it an exception that publishes ownership without warning. It publishes none: `Connection::Http` forces per-run models, and `publishes_ownership()` is the negation of that, so an agent stores a job-pinned snapshot and leaves workspace ownership with the deployed graph — it cannot orphan a subscription at all. `warn_dormant_subscribers`' own doc still named the deploy log as the only place the warning shows, one commit after it gained its second caller. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: stop the managed-write rule from contradicting the dbt:// target The sentence after the warehouse-relation paragraph says `// materialize` means the runtime writes the table for you and the body is a bare SELECT. That is the managed DuckLake rule, written before a `dbt://` target existed, and unqualified it tells the model the opposite of what the paragraph above it just said — a model following the more prominent one emits a SELECT for a warehouse relation, which deploys and then writes nothing. Both prompt sources now scope it, and both name the `// data_test` refusal beside a `dbt://` target, which the badge list advertised without the caveat. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-07 17:38:07 +02:00
Ruben FiszelandClaude Fable 5.1	fdd3b36423	feat: workspace setting to hide the AI assistant, agent steps unaffected (#10941) ... * feat: workspace setting to hide the AI assistant, agent steps unaffected Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: load workspace AI config on cold /sessions load and say hidden, not disabled Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: follow workspace switches on /sessions gate and drop deprecated button size Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: key the /sessions hidden-assistant gate on the acting workspace's own config Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: tag the /sessions hidden-assistant verdict with its workspace and drop superseded reads Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: overlay the /sessions hidden-assistant gate so warm sessions survive workspace switches Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: hide the pipeline insert menu AI prompt and refuse chat turns where the assistant is hidden Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: shrink the home Build with AI / CLI / Hub line to a flush hint row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL * fix: frame the workspace toggle as hide AI sessions at the bottom of the AI settings Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011eNweUugVqerex6MLxjbeL --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>	2026-09-02 17:21:10 +02:00
Ruben Fiszel	d85050f505	feat: upgrade bun to 1.4.0 and demote deno in the language picker (#10784) ... * chore: upgrade bun to 1.4.0 in dockerfiles and CI pins * chore: move deno last in the language picker and relabel it Deno * chore: move deno last in the pipeline language picker too * chore: pin debugger image to bun 1.4.0 and trim the deno picker comment * chore: state the deno picker constraint without referencing the old order * fix: stamp bun lockfiles back to v1 while the fleet predates bun 1.4 * fix: ask bun for a v1 lockfile instead of rewriting one, and refuse an escalated lock * chore: warn instead of silently storing a lockfile with no readable version	2026-08-21 01:16:59 +02:00
3 people	d0089758e6	feat: open a session edit in the preview panel from the edits list (#10486) ... * feat: open a session edit in the preview panel from the edits list * refactor: move the deploy-kind preview mapping next to its siblings * feat: make preview the primary action in the session edits list Clicking a row in the Edits popover now opens the item in the session preview panel; kinds the panel cannot host fall back to their diff. Each row gains an explicit Diff button for that item, and a pinned footer row opens Review & deploy for the whole set. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: fold the open-and-flash rule into SessionPreviewTabs The rule for when a preview open should flash the tab was written out at three call sites, one of them with a looser condition. Move it onto the tab owner as openAndPulse so the three agree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: preview data-pipeline edits from the edits list A pipeline bundle is stored at `f/<folder>/data_pipeline` while its editor is the folder's pipeline view, so the deploy-kind mapping has to route on the folder. Without it the one remaining previewable kind fell through to the drawer. Also pin the open-and-flash rule with tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: state the pipeline bundle path layout once `f/<folder>/data_pipeline` was parsed independently in the compare page, the home list and the session preview mapping — two of them disagreeing on whether the trailing segment is checked — and built by hand in the editor. Route all four through $lib/pipelinePaths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: route the pipeline page through the shared bundle path The route built the bundle path by hand and passed the draft kind as a literal, the two halves of the editor disagreeing on where the layout is stated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: name a pipeline edit by its folder in the edits list The bundle path is an implementation detail and the row's click lands on the folder's editor, so showing `f/<folder>/data_pipeline` named something the click doesn't open. Also derive the bundle regex from the draft-kind const it has to agree with. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(cli): drop unrelated package-lock.json change The lockfile diff was an incidental regeneration from an older manifest (it downgraded the locked svelte range below what cli/package.json requires) and had nothing to do with this PR. Reset to main's version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-08-06 17:37:03 +02:00
Ruben Fiszel	562ef438e5	signpost the dbt migration path on the pipelines page (#10506) ... * feat: signpost the dbt migration path on the pipelines page * fix: frame the dbt signpost as a separate runtime, not a pipeline * fix: claim only graph visibility for dbt models and gate the signpost on operators * fix: reword the dbt signpost title	2026-08-04 12:26:17 +00:00
Ruben Fiszel	4db432f004	fix: normalize the pipeline folder so AI node paths aren't double-prefixed (#10479) ... * fix: normalize the pipeline folder so ai node paths aren't double-prefixed * refactor: move the pipeline folder normalizer to its own util	2026-08-03 19:32:20 +02:00
3 people	baefa1345b	feat: give dbt its own editor with an explicitly refreshed model graph (#10448) ... * feat: give dbt its own editor with an explicitly refreshed model graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: bump ee ref for the agent-worker dbt editor graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope editor graph retention by principal, carry parse context, honor nlang Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: bump ee ref Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the dbt editor's model graph and log panel mounted across tabs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the dbt_edge to dbt_node joins on an index-usable equality Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: poll a parse until the job ends, resolve the project key, correct the docs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: surface a slow parse's job, bound poll failures, drop banned bindable defaults Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide the dbt Generated UI content, not only its tab Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: honor disabled Triggers in the dbt tab fallback, record permissioned_as Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one dbt pane with the run drawn on the models, and a full-height script graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: move the dbt build arguments behind the Build button Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: trim the dbt editor toolbar and stop the graph asserting a cause it lacks Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: mark dbt as alpha in the language picker and announce it once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: trim the dbt alpha notice Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: give a selected dbt model the whole detail section, with a close that deselects Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: close the dbt detail panel by clicking away, and make its close obvious Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cache the agent-worker dbt query, which needs the private feature to compile Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never fall back to a settings tab the embedder disabled Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preview dbt rows from the same project the graph was parsed from * feat: hide the script-kind selector for dbt projects * fix: pin a dbt row preview to the project its graph was parsed from * fix: pin a dbt row preview to the arguments its graph was parsed under * fix: keep dbt preview placeholders live while its vars stay pinned * fix: report a warehouse-less dbt parse's counts and flag stale preview args * fix: tell the pinned-vars case apart from a stale placeholder * chore: update ee-repo-ref to 59044635769f18f8ff5073236cfc7b5f41e917cc This commit updates the EE repository reference after PR #707 was merged in windmill-ee-private. Previous ee-repo-ref: 7e424384cdd4cef8653b55b04f17ad3f801bc50c New ee-repo-ref: 59044635769f18f8ff5073236cfc7b5f41e917cc Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-03 03:32:08 +02:00
Ruben FiszelandClaude Opus 5	eca24bdfb5	fix: report a missing worker tag instead of spinning in data table UIs (#10456) ... * fix: report a missing worker tag instead of spinning in db/datatable UIs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cancel the unpickable job, confirm the tag lookup, back off the poll Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: confirm an unserved tag over ~90s and never hide a failed refresh Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: leave the job queued so the autoscaler still sees the backlog Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cancel writes before reporting them, leave reads queued Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read the write's terminal state instead of trusting the cancel request Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never abandon a write, explain the wait instead of cancelling Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-02 13:29:07 +02:00
Ruben FiszelandClaude Opus 5	032300e28e	feat: run dbt projects as a first-class Windmill runtime (#10326) ... * fix: mount only the engine in the dbt jail, reject shadowed and malformed args Review round 42. The jail mounted the whole dbt cache directory, whose siblings of the engine are `repos/` and `packages/` — other workspaces' private checkouts and package trees, kept apart by cache key rather than by permissions. A jailed project could read them. It now mounts the engine's own directory, which the provisioner names; verified from inside the jail that `repos/`, `packages/` and `state/` are invisible while the engine stays usable. A `{{ placeholder }}` may no longer take the name of a run argument this runtime defines. It was silently dropped from the signature, so a descriptor like `value: "{{ select }}"` deployed and then could not be run at all: the built-in `select` is an array and the interpolation needs a scalar. Refused at parse, so the deploy says so. A `vars` override that is not an object is refused rather than ignored. Argument-schema validation is opt-in, so a string or an array silently ran the descriptor's own vars — against a different schema or alias than the caller asked for. `select` and `exclude` already refused theirs. * feat(dbt): the project is the script's module bundle, not a git checkout A dbt script now carries its whole dbt project as its module bundle. The descriptor is the script content; `<script>__dbt/` holds the project verbatim, so importing an existing project is `cp -r` plus `wmill sync push`, and the worker materialises the bundle into the job directory instead of cloning. Backend - `prepare_project` writes the script's modules and requires `dbt_project.yml` at the bundle root. `checkout`, the git-ssh command, the clone cache and the repository resource are gone, along with `repo`, `project`, `ref` and `git_ssh_identity` on the descriptor. - Run identity and the package cache key take a `project_digest` (sorted SHA256 over the bundle) where the commit used to sit, so an edited project cannot resume a previous run's `run_results.json` or reuse its `dbt_packages`. - The per-run graph re-ingest is now gated on `vars` placeholders and `$var:` env alone. - `capture_dependency_job` takes the script's modules so a dependency job, which has no generic module-writing step, materialises them itself. - `dbt deps` caching strips the git remote from every package it cached, not just the tree root: `packages.yml` can render a token into a `git:` URL. - `git_clone.rs` is dropped and `ansible_executor.rs` returns to its own copy of the clone helpers. CLI - `wmill sync pull` keeps a dbt script's lock beside its folder rather than inside it, so the folder holds nothing but the project. - Directories dbt generates (`target-path`, `packages-install-path`, `clean-targets` and the usual defaults, read from `dbt_project.yml`) are excluded from the bundle, from the sync diff and from staleness hashing. - A module-only edit now pushes its parent dbt script and is reported as a changed module rather than passing unnoticed. * fix: keep a script's modules in the worker's file-system cache The first fetch of a script version reads the database and carries its modules; every later fetch imports from the worker's cache directory, whose `RawScript::import` hard-coded `modules: None` and whose `export` never wrote them. A worker restart therefore started running the script without its own files, silently — for a dbt script, without its project, which fails with "carries no project"; for any other script with a module bundle, with the imports missing. `modules.json` is now written on every export and required on import, so an entry written by an older version fails to import and is refetched rather than serving a stripped script for as long as the directory lives. Also derives a dbt run's `project_digest` from the bundle the run actually carries: `handle_dbt_job` was passing `None`, which collapsed every project in a workspace onto one digest and let `dbt retry` resume a different project's `run_results.json`. * fix(dbt): give every phase the script's environment, bound the cache copies `dbt deps` ran without the script's environment variables on an unsandboxed worker, so a `packages.yml` resolving a private package URL through `env_var()` could not see them while the package cache key was still built on their digest. `with_invocation_env`, applied at three of the four call sites, is folded into `dbt_command` so no phase can be added without it, and `DBT_TARGET_PATH` is set after both environments rather than before. The package cache copies ran through a bare `Command::output()`: the tree is the project's, so a cancelled or timed-out job held its worker slot until `cp` finished. Both the restore and the publish now run under the job poller like every other phase. * fix(dbt): only offer commands whose writes match the graph, honour packages-install-path `dbt_command: run` is dropped from the allowed overrides. Asset dispatch fires a script's deploy-time writes on any successful job, and `dbt run` covers models only, so a project with seeds or snapshots notified consumers of relations the invocation left stale. That is the same reason `test` was already excluded. Narrowing what a run touches is `select`/`exclude`, which scope the graph too. `dbt deps` writes to the project's `packages-install-path`, so a project that moved it got no package cache at all: the publish found nothing at `dbt_packages` and every job resolved its dependencies over the network again. The path is read from `dbt_project.yml` and validated as project-relative, since both cache copies are rooted at it. Also states the sidecar's mutator contract at the module level: the dbt manifest tables carry no RLS and grant `windmill_user` full access, so a user-scoped transaction is not enforcement and every caller must have verified write access to the script itself. CLI: a module file is now grouped with its parent script for the push. Left in a group of its own it got its own `alreadySynced`, so a push touching several files of one bundle deployed the script once per file; the resulting versions raced, and the asset graph could end up describing none of them. * fix(dbt): seed a project for browser-created scripts, refuse a no-op retry A dbt script created in the browser only got a descriptor, and the runtime refuses a script whose bundle has no `dbt_project.yml`, so the advertised Create → Deploy → Run path always failed its dependency job. New dbt scripts now start with a project that builds: pointing `profile.resource` at a warehouse is the one edit, and growing it is `wmill sync pull` plus a local editor, which is where dbt development happens. `dbt retry` builds its graph from the previous run's error, fail and skipped nodes alone, so retrying an all-green run selected nothing and wrote nothing — and a job that succeeds having written nothing still dispatches every deploy-time write, waking every downstream consumer for relations no one touched. Refused, with the reason. CLI: a configured `target-path` or `packages-install-path` may be nested (`build/target`), and `clean-targets` has a block form as well as an inline one. Both are now parsed, and the exclusion compares the project-relative path rather than the top-level segment, so a nested generated tree no longer lands in the bundle and no longer makes a local `dbt run` look like a project change. * fix(dbt): lock a project once, find the parent on either path separator A dbt script's modules are its dbt project, not helper code with dependencies of its own, so the generic per-module lock loop is skipped for it: the parent lock already ran `dbt deps` and `dbt parse` over the whole project. Locking each file separately re-materialised the bundle and re-invoked dbt once per file, so a project of N files paid N project-sized passes and a large one timed the deploy out. The 13-file fixture went from 14 relock passes to 1. `pushParentScriptForModule` searched the raw path for `__dbt/`, so on Windows, where the folder is spelled `__dbt\`, a module-only edit returned without deploying its parent while the caller still recorded the file as synced. It now goes through `getScriptBasePathFromModulePath`, which normalizes separators. Also drops the last of the external-repository wording from the descriptor's module docs and from the `codebase` rejection a user can hit. * feat(dbt): infer the run form locally, keep test-only retries from cascading `windmill-parser-wasm-yaml` 1.770.0 carries `parse_dbt`, so the browser and the CLI derive a dbt script's run arguments from its descriptor instead of waiting for the deploy to hand back a schema. Pins bumped in both. `generate-metadata` was rewriting a dbt script's `lock` field on every run: a dbt lock comes from the dependency job on a worker, so nothing generates it locally and the resolved `!inline` reference was left inlined into the metadata or blanked. It is restored instead, and a push straight after `generate-metadata` is a no-op again. A retry now needs a failed node that materialises something. `dbt retry` builds its graph from error, fail and skipped nodes, and with `test_behavior: after_all` a failing test is what `run_results.json` ends up describing — so the retry reran tests, wrote nothing, succeeded, and still dispatched every deploy-time write. The dbt badge's destination is deterministic: writers outrank readers, and among several writers of one relation (which the backend permits) the smallest id wins, rather than whichever write edge arrived last. * feat(dbt): browse the project and read a run's per-node result Two views a dbt user expects and that the generic script surfaces do not give. **The project.** A dbt script's editor gains a Project tab beside its descriptor: the module bundle as the tree dbt itself expects, each file read-only with syntax highlighting. The existing module tab strip is a flat row built for a couple of helper files and does not survive a real project; a 13-file fixture already overflows it. Directories sort before files so it reads like the checkout on disk, and an empty bundle explains the `cp -r` instead of showing a blank pane. **The run.** `DisplayResult` renders a dbt invocation's per-node breakdown above the raw payload: totals, then a table of node, kind, target relation, rows and time, with failures and warnings sorted first and carrying their message. The data was already structured; it was being shown as JSON to scroll and PASS/WARN counts to find in the log. On a failed run the same JSON rides in the error message after the exit-status line, so it is parsed back out — that is the case worth rendering, since the failing node is what the user came for. * docs(dbt): say that profile.resource is what buys the asset graph The starter descriptor described `profile.resource` as the thing rendered into profiles.yml, with the project's own file as an equal alternative. It is not equal: the resource PATH is the warehouse's identity in the asset graph, so a project bringing its own profiles.yml runs fine and silently gets no assets, no lineage and no cascade. The deploy already says so in its log; now the descriptor a user starts from says it too, before they choose. * fix(dbt): authorize a resource used only for asset identity, clean up after failed installs A descriptor setting both `profile.profiles_yml` and `profile.resource` took its connection from the project's file but returned the resource path as the graph's warehouse identity without ever reading it. A script editor could therefore publish `table://<any resource>/...` writes, and wake that warehouse's subscribers, while connecting somewhere else. The resource is now read on that path too — reading is what authorizes it — so the combination keeps working for the case that wants it (keep your own profiles.yml, still get lineage) and fails closed otherwise. Provisioning cleaned up its staging directory only on the paths someone remembered, so a run of failed or cancelled first-use installs accumulated venvs, tarballs and installer scripts until the worker's disk was gone. All three engines now hold their scratch paths in a guard that removes them on drop, which is the one exit every path takes, cancellation included. Frontend: `partial success` is dbt's word for a node that built but whose tests failed, counted in `totals.error` and redone by a retry, so it ranks with the failures instead of rendering green with its message hidden. And the run panel now keys off the worker's engine discriminator rather than `{nodes, totals}`, which is a shape an ordinary script can return. Both pinned by unit tests on the extracted `parseDbtRun` helpers. * feat(dbt): show a run's models on the run page The run page is where you land on a running job, and until now it showed a dbt run as streaming text: the per-node table only renders once the job has produced a result, and the graph that moves per model lived on the pipeline page you had to navigate to. A Models section now sits above the result, scoped to the running script's own relations and its `ref()` lineage, polling while the job is in flight so nodes move as dbt walks the DAG. No `resolveGraph`: that merges drafts and live editor buffers into the persisted graph, and a run page has neither. * feat(dbt): retry failed nodes automatically, and from any worker **Node-level retry, in the job.** `retry_failed_nodes: {attempts, delay_seconds}` rebuilds only what a failed build left failed or skipped, before the job reports failure. dbt confines a failure to its own subtree and `dbt retry` resumes exactly that set, so a transient warehouse error costs those nodes rather than the project. Doing it in-job is what keeps the state question out of it: the previous attempt's `run_results.json` is still in the job directory, so there is nothing to persist and no worker to land back on. This is the granularity astronomer-cosmos gets from one Airflow task per model, without the ~6x that per-model tasks measured. A retry's `run_results.json` names only the nodes it redid, so it overlays the accumulated results rather than replacing them: the job's result has to be every node the job touched, or the nodes that succeeded before the retry settle no materializations. Pinned by a test. **Durable retry state.** `run_results.json` is now saved to `dbt_run_state` as well as the worker's local cache, so an explicit `dbt_command: retry` works from any worker of the group rather than only the one that failed. Only the results are stored: `dbt retry` also needs `manifest.json`, roughly sixty times larger and growing with the project (732 KB against 12 KB on the six-node fixture), but the manifest is a pure function of the project files, vars and env, all of which the stored identity already pins, so a worker restoring from the database re-derives it with a `dbt parse` of about a second. * fix(dbt): restore the sqlx cache, make retries cancellable and path-aware **SQLx cache.** A `cargo sqlx prepare` deleted 750 entries, including the enterprise queries CI needs under `SQLX_OFFLINE=true`, and the check that was supposed to catch it reported zero losses because it was run from `backend/` with a `backend/`-prefixed path, so its baseline was empty and it failed open. All 750 are restored; the branch now adds 19 and deletes none, and `SQLX_OFFLINE=true cargo check` passes. **Retry backoff observes cancellation.** `canceled_by` is only written by the job poller, which does not run between attempts, so re-reading it reported the state as of the failed attempt and missed every cancel issued during the wait — the whole window the check exists to cover. The wait now reads `v2_job_queue.canceled_by` each second, and the job's deadline is honoured before starting another dbt process. **Retry state follows its script.** `dbt_run_state` is path-keyed like the manifest sidecar but, unlike it, nothing regenerates it: a rename moves the row so a resumable failure survives, while archive and delete clear it, so a script later created at that path cannot inherit a stranger's failure and its arguments. **CLI.** `table` joins ducklake and s3object in the local graph's auto-trigger kinds, matching `is_auto_trigger_kind` and the frontend's set; without it a local graph and the generated docs omitted a cascade edge the deploy has. * fix(dbt): carry only the project files a bundle can hold, and say what it drops Exploring real and edge-case projects surfaced three frictions, all in the import path a user hits first. **A binary file broke the push, opaquely.** dbt projects carry images under `docs/`, stray `.DS_Store` files and occasionally a parquet seed. Read as text they become mojibake, and a NUL among them is rejected by Postgres with `unsupported Unicode escape sequence` — which `wmill sync push` then reported as success, exiting 0 with the script never created. Binary files are now detected the way `git` detects them, by a NUL in the first 8000 bytes rather than by extension, and skipped with the reason. **The size guard the docs promised did not exist.** Now it does: 5 MB per file, which only ever catches a committed dataset. Real dbt code is about 500 bytes median and 1.9 KB at p90. **Skipped files became a permanent phantom diff.** The push dropped them while the sync diff still offered them, so every push reported changes no push could resolve. One predicate now answers for the push, the staleness hash and the diff. Verified on a project with unicode filenames and content, CRLF endings, an empty model, an ephemeral model, a disabled model, a `.md` docs block, an extensionless README, six levels of nesting, a 7.6 MB seed and a PNG: it pushes, round-trips byte-for-byte through pull, deploys to 7 dbt nodes and 6 `table://` assets (ephemeral and disabled correctly absent), and runs green. * fix(dbt): resolve dbt-core against the adapter, settle partial success, unify status **Adapters could not be provisioned.** The 1.x engine pinned `dbt-core` to a fixed version independent of the adapter, but several adapters cap below it: `dbt-mysql` at `~=1.7`, `dbt-oracle` and `dbt-databricks` below 1.12, and `dbt-salesforce` has no package at all (it exists only inside Fusion). Those projects failed at provisioning with a uv resolver dump. The install now asks for a range and lets the adapter choose, and records what the resolver picked so the lock pins a version that adapter can take. The floor is the CLI this runtime invokes: resolving down to dbt-core 1.7 produced a working venv that then failed with `No such option '--target'`, which is worse than not resolving. An adapter with no release in range now fails naming itself and pointing at `dbt-core-2x` or `fusion`, instead of a resolver dump. Salesforce is refused up front with the reason. **`partial success` left a model stuck on `Running`.** It is dbt's word for a node that built and then failed its tests, and it was already treated as a failure when counting totals and deciding a retry — but the two sites that settle the RELATION fell through to "says nothing", so the tailer's `Running` was never replaced and a finished job showed a model still building. Six status comparisons had drifted apart, two folding case and four not, while dbt-core 1.x echoes the author's casing and 2.x uppercases; they are now one classifier. **Agent workers.** The durable retry state and the cancellation poll both need a database, which an agent worker reaches only through the API. The automatic node retry is refused there rather than running a wait it could not interrupt, and the docs say "any worker with a database connection" instead of overclaiming. Also clears `dbt_run_state` when a path stops being a dbt script, and moves `run_identity`'s contract onto `run_identity` from the digest helper below it. * feat(dbt): show the transform behind a model on the run graph The run page's graph carried a node for the script itself and drew every relation as a bare table. Both were wrong for that page: the graph there is already scoped to one script, so a node standing for it distinguishes nothing (on the pipeline page it separates one project from another, which is why it exists), and dbt's own DAG node is the model — the SQL and the relation it writes are one thing, so a graph of relations alone leaves out what a reader came to see. The script node is dropped, and selecting a model now shows its SQL underneath the canvas with its file path and materialization, read-only, the same view the pipeline details pane gives. * feat(dbt): move the graph with the run The worker has always recorded a state per relation as dbt walks the DAG — `running` when a model starts, `materialized` or `failed` when it ends — but nothing rendered it: the graph response carries what a relation IS, not what a particular run is doing to it, so the canvas had nothing to show and a running job looked identical to a finished one. `assets/run_progress/{job_id}` returns that state for one job, the run page polls it beside the graph, and the asset node carries a spinner or its outcome. Errors and retries need nothing extra: a failed node writes `failed`, and an in-job retry rewrites the same row, so the node returns to `running` and on to its new outcome by itself. `materialized_partition` holds a relation's CURRENT state keyed by relation, so filtering on `job_id` returns exactly what this run last touched — which is the question a run page asks, and why a superseded older run shows nothing. * feat(dbt): a dbt project is not a data pipeline Deploying a dbt script marked it `auto_kind = 'pipeline'`, which enrolled it in pipeline membership: the folder became a Pipeline entry on the home page, the script folded into it, and `/pipeline/<folder>` opened a canvas holding the project's whole model DAG next to the pipeline's own scripts. A folder holding both then read as two projects in one editor, and the pipeline editor offered to author transforms that are in fact authored in a local `dbt run` loop and pushed as the script's bundle. A dbt script is now never a pipeline member, and the pipeline canvas drops the dbt script node. Its models stay, with their `ref()` lineage: the relations are what a downstream pipeline script reads, and dropping them would break the cascade from a dbt run — the point of giving dbt models `table://` identity. Also drops a screenshot committed to this branch by accident. * fix(dbt): authorize run_progress through the job, drop dbt from the local graph `run_progress` read `materialized_partition` through `user_db` on the assumption that RLS would scope the rows. That table has RLS disabled and no policies, so any workspace member could pass a job id and read that run's relation paths, row counts and error text. It now joins `v2_job`, which does carry per-user policies, so a caller who cannot see the job sees nothing — the same pattern `v2_job_completed` reads need. Verified as a plain member: the old query returned 6 rows for another user's run, the new one returns 0, while the job's owner still sees all 6. The CLI's local graph still forced `in_pipeline` on every dbt script, so `pipeline docs --local` and `pipeline dev` kept presenting a dbt project as a pipeline the deploy no longer enrolls. It now skips them, matching the server. A dbt descriptor has no asset parser locally, so nothing is lost: its models come from the manifest the deploy derives. Declares `run_progress` in openapi.yaml so the frontend uses the generated client instead of a handwritten fetch; the generated `status` union also replaces a hand-rolled string mapping. * fix(dbt): drop the dbt node from the CLI's deployed pipeline views too `pipeline dev` and `pipeline docs` (without `--local`) read `/assets/graph` directly. That endpoint is asset-usage driven rather than membership driven, so it returns a dbt script like any producer — and both commands render every runnable, so a dbt project still showed up as a pipeline script there after the local builder stopped emitting one. `hideDbtRunnables` mirrors the frontend's projection of the same payload. It is generic over the graph shape so the bounded-cascade view (`BCGraph`, a narrower type over identical JSON) passes through without a cast. The relations stay: they are what a downstream pipeline script reads, and the node is what attributes them to a producer for every other consumer of the endpoint, so the filter belongs in the views rather than the query. * fix(dbt): narrow a selective run's cascade, settle the finished run graph Review-round fixes. A `select`/`exclude` run builds part of the project, but asset dispatch reads the deploy-time write set for the whole script, so a run selecting one model woke the subscribers of every other. Dispatch now intersects that set with the relations the run actually recorded as materialized, scoped to dbt because it is the only producer whose write set is decided per run. A run that recorded nothing still dispatches everything, so an agent worker whose reconciliation failed cascades as before. Verified both ways: `select: [extra_model]` no longer wakes the `fct_orders` subscriber, and a full run still does. `hideDbtRunnables` keyed its removal set on path alone while the graph keys runnables by `(usage_kind, path)`, so a flow sharing a path with a dbt script lost its node, edges and triggers too. Both copies now key on the pair. The run graph never took a final reading when a job finished, so the last state shown was whatever the tick before completion saw. Only `dbt-core-1x` streams node events; the other engines record every relation during end-of-run reconciliation, so their finished graph showed nothing until a reload. `DbtNodeOutcome::Inconclusive` collapsed statuses the tally has to tell apart, so two sites re-lowercased the status beside the classifier and `no-op` landed in `totals.error` — a clean run reporting an error in its own result. Split into Warn / Skipped / NoOp / Unknown so every site falls out of one match; `no-op` is kept out of the retry set, which dbt spells as error / fail / skipped. Also: reattach two doc comments to the items they describe, and correct the engine-distribution table — only dbt-core-2x is baked into the images, 1.x is a per-adapter venv provisioned on first use, and the default is compiled in rather than an instance setting. * fix(dbt): make the model chip inert where its project node is not on the graph The canvas passed `onDbtSelect` unconditionally, so the chip always rendered `cursor-pointer` and hover-highlighted — but the owner map is empty on both graphs this feature added, since the run page carries no runnables and the pipeline page hides the dbt node. The chip advertised a click that resolved to nothing. It now takes its handlers only when the relation has an owner on this graph, so it stays live on the surfaces that do show the project node. `classify_status` and `DbtNodeOutcome` were `pub` in a private module with no caller outside the file, unlike every neighbour. * fix(dbt): take the cascade's write set from the run's own result The previous narrowing read `materialized_partition`, which was wrong twice. That table keeps one row per relation and the newest writer takes `job_id`, so two overlapping runs over the same model erase each other's claim to it: the earlier job would dispatch a subset of what it built, or none of it. And an empty row set was read as "recording failed, dispatch everything" when it is also a real answer. A `select` matching no model, or one resolving to tests only, exits 0 having built nothing — and then woke every consumer of every model in the project, which is the opposite of what the narrowing exists to do and is reachable by a typo in a run argument. The run now reports the relations it materialized in its own result, which is immutable and per job. Absent means the producer said nothing (a job from before the field, a non-dbt producer) and the whole deploy-time set dispatches as before; present-and-empty means it built nothing and dispatches nothing. Verified on all three: an unmatched selector builds nothing and wakes nobody, a selector naming one unsubscribed model wakes nobody, and a full run wakes the subscriber. Also indexes `materialized_partition (workspace_id, job_id)` -- the run page polls that shape every 2s and no existing index leads with `job_id` -- corrects the selective-cascade section of the design doc, which still described the old deploy-time behavior, and reattaches `buildLocalPipelineGraph`'s doc comment. * docs(dbt): attach the CLI JSDoc to its function, correct the index rationale The `hideDbtRunnables` JSDoc ended up documenting the type declared beneath it — made while fixing the same mistake one function down. The migration's comment credited the cascade with a `job_id` lookup that the same commit replaced with a read of the job's own result. The run page's poll is the only reader keyed on that column. * fix(dbt): refuse graph publication for a removed script, allow test-only retries An archived or hard-deleted script could still republish its graph: the publication guard filtered `deleted` but not `archived`, and treated a missing row as "nothing newer exists" rather than "nothing left to publish for". A dependency job or dynamic run finishing after the removal put the asset, provenance and subscription rows back with nothing left to clear them. `dbt_command: retry` refused a run whose only failures were tests, which is precisely what `test_behavior: after_all` produces. That restriction existed because a successful job dispatched its whole deploy-time write set, so a test-only retry would have woken every consumer for relations no one touched — the cascade now dispatches what the run reports materializing, so it wakes nobody and the restriction only blocked a legitimate retry. * fix(dbt): gate run progress behind the job-read check, not RLS alone The endpoint joined `v2_job` so RLS would decide visibility, which it does — but `require_job_read_access` adds two things RLS does not: a scoped token's `if_jobs:filter_tags` restriction, and the app-embed cutoff that stops untrusted app JS from inheriting the viewer's broader job access. A scoped or embed token could therefore read relation names, statuses, row counts and errors for jobs the ordinary job endpoints deny it. That helper is private to `windmill-api`, which depends on `windmill-api-assets` rather than the reverse, so the endpoint moves to the job routes instead of the check being duplicated. It is job-scoped anyway: `/w/{ws}/assets/run_progress/{job_id}` becomes `/w/{ws}/jobs/run_progress/{id}`, and the frontend follows the generated client. * feat(dbt): a dbt run does not trigger downstream runs dbt orders its own DAG, so a cascade only ever adds one thing: waking a Windmill script that reads a mart. That edge is narrow, and only half of it can even be expressed — nothing outside dbt can declare a `table://` write, since `// materialize` accepts DuckLake targets only, so an ingestion script cannot wake a dbt project. Against that, dispatching correctly is not cheap. A run's `select` can build any subset of the project, so the deploy-time write set is not what ran; using it wakes consumers of relations the run never touched, and narrowing it needs a per-job record of what was built. The per-relation state table cannot supply one (it keeps a single row per relation stamped with the last writer), and the result field added for it made a run's own output carry the cascade's bookkeeping. So `asset_dispatch` returns early for `ScriptLang::Dbt`, before the producer gate. dbt still materializes, records per-model state and publishes its graph: models, `ref()` lineage and live run progress are unchanged, and a `# on table://<mart>` reader still renders beside the model it reads. It simply does not fire. Wiring it up later means deciding what a selective run should notify, which is the actual work. Verified: a full run of a 6-model project succeeds and starts nothing, where it previously triggered its subscriber; the run page still reports all 6 relations and the folder graph still carries 14 tables and 8 ref() edges. * fix(dbt): remove the cascade surface, settle stranded models, fix nested __mod Stopping dispatch left its surface behind. `table://` was still an auto-trigger kind, `persist_ingest` still derived subscriptions from a manifest's reads, and the deploy still accepted `# on table://` — so the canvas drew cascade arrows into scripts nothing could wake. All three are gone: the kind no longer derives, the ingest only deletes rows earlier versions wrote, and the deploy refuses the annotation with a message saying why rather than persisting a silent no-op. `DescriptorTriggers` went with them; every field it parsed was cascade config. A model marked `running` by the live tailer was never settled when the run did not finish: reconciliation only revisits nodes `run_results.json` names, and a cancelled or timed-out run has none for the model in flight, so the finished job showed a relation building forever. It is now settled on every exit path. Verified by cancelling a run mid-flight: 3 models `running` before, 3 `failed` after, none stranded. `getScriptBasePathFromModulePath` took the first matching suffix rather than the outermost boundary, so `proj__dbt/models/legacy__mod/a.sql` resolved to `proj__dbt/models/legacy`. dbt owns its directory names verbatim, so a folder ending `__mod` is legal inside a project, and a module-only sync would have looked for a descriptor that is not there and skipped the deploy. * fix(dbt): colour a finished run's models from its own result `materialized_partition` keeps one row per relation stamped with whichever job wrote it last, so reopening a run showed only the models no later run had touched since — down to none for an old run, which reads as a broken page rather than as stale data. Reproduced: a 6-model run reported 6 relations, then a second run rebuilt one shared model and the first reported 5. A finished run already carries the answer. Its result lists every node with a status, and the graph carries each asset's dbt `unique_id`, so the two join directly — no path derivation, nothing stored twice, and nothing a later run can overwrite. The endpoint stays for the live window, where the result does not exist yet, and as the fallback for a run that never produced one (cancelled or killed, whose relations the worker settles in the table instead). `relationOutcome` mirrors the worker's `classify_status` so the colour drawn over a record agrees with the record: `warn`, `skipped` and `no-op` leave the relation untouched and stay uncoloured, as do tests and analyses, which match no asset. Verified in the browser on the run whose model had been stolen: all six relations green again, both sources correctly uncoloured. * docs(dbt): record why only dbt-core 1.x has live per-model progress `emits_node_events()` reads as "the Rust engines produce no node events", which is false and would close off the option. They produce exactly the same events; they put them on the console and ignore `--log-format-file json`, which both accept. Measured on 2.0.0-alpha.5 and fusion 2.0.0-preview.202: 15 node events each on stdout, 0 in the file log, for a three-model project. Taking them means owning the job log's presentation to work around a flag that is documented and simply unimplemented, so the note records the measurement, the sample event, and that flipping the predicate is the whole change once either engine honours it. * fix(dbt): give HighlightCode a dialect-agnostic sql language `npm run check` had three errors the fast check does not reach: `"sql"` is not a value `HighlightCode` accepts. Every SQL dialect it knows maps to one grammar, but a dbt model is compiled by whichever adapter the project targets, so naming a dialect would be a guess — `sql` is now a value in its own right. `langOf` was typed `string` and returned `markdown`, `python` and `text`, none of which the component accepts either, so a dbt project's YAML and Python files rendered unhighlighted. It now returns the component's own prop type, which is what caught them, and `undefined` for what has no grammar rather than a name that silently means the same thing. Verified in the project panel: SQL 22 tokens, YAML 27, where YAML was plain. * fix(dbt): stop failing no-op models, drop table triggers client-side, keep cross-selection edges The sweep that settles a run's stranded relations was marking `no-op`, `warn` and `skipped` models FAILED on successful runs: reconciliation reports those nodes without settling their record, so they were indistinguishable from a model the run never reached. It now excludes every relation the run accounted for, so only the genuinely abandoned ones are settled. `table` was removed from the backend's auto-trigger kinds but left in both client mirrors, so the editor and `pipeline dev`/`docs` kept drawing cascade arrows the deploy will not create. `isModuleEntryPoint` scanned for the first `__mod/`, the same bug its sibling just had: a `legacy__mod/script.ts` nested in a dbt project — dbt owns those names verbatim — read as that script's entry point. Both now anchor on the outermost boundary. A script selecting a model whose parent another script builds dropped the parent entirely, so no `dbt_edge` could reach it and the two relations sat on the graph unconnected. The parent is now kept as an endpoint and recorded as a READ, since this script does not build it — splitting a project across selections only composes if the seam still draws. * fix(dbt): don't double-run after-all tests, count only models a script builds An `after_all` run whose test phase failed saves a `run_results.json` holding tests alone. Retrying it reran exactly those tests — and then the test phase ran the whole suite again, appending a second copy of every result: duplicate ids in the run table, doubled totals. A retry whose saved results are tests alone IS the test phase, so the suite is not run after it, and the two phases now merge by node id rather than concatenating. Keeping a selection's unselected parents as nodes made them count toward the `×N` badge, whose tooltip says "materializes N models" — a script selecting one mart claimed the staging models upstream of it, and the number grew with the seam. The count now comes from the relations the script writes. That change also made the cross-selection read block dead, with a comment asserting the inverse of what now happens; it is removed, and the test that covered it still passes on the new arm. The test I added landed between a neighbouring test's comment and its `#[test]`, orphaning the attribute so that test stopped running. Two display fixes: the run page no longer shows a relation's SQL when the provenance belongs to another project that materializes the same relation, and the editor no longer draws an explicit `# on table://` arrow the deploy refuses. The starter descriptor no longer promises the removed cascade. * fix(dbt): clear untouched models, reject unknown descriptor fields A `no-op` model was left `running` forever on a successful run. The previous attempt at this stopped the sweep marking such models FAILED but gave them no terminal state instead, so they simply never settled. Reconciliation now returns what it settled and what the run reported but did not build, and the two get opposite treatment: a relation the run left untouched has its row DELETED, which is what the finished run's own result says about it (`relationOutcome` colours a `no-op` nothing), so the live and settled views agree; only a relation the run never reached at all is failed. The descriptor accepted unknown fields, so `selcet:` was ignored and left an empty selection — building the whole project — and a misspelled `target` fell back to the profile's default. It rejects them now. That immediately caught two of our own test fixtures still passing `repo:`, a field removed with the git path, which is exactly the class of mistake it exists to stop. `isDbtModulePath` matched `__dbt/` anywhere in a path, the third site with that bug: `foo__mod/vendor/x__dbt/a.ts` read as a dbt project file, and the push then looked for `foo.script.yaml` and could skip the edit. A verbatim dbt bundle dropped any file named `*.lock` before it reached the module map, so an authored `uv.lock` never deployed and the unmodified-project round trip quietly lost it. The exclusion now applies only to `__mod` bundles, where `.lock` really is the script's own lockfile — in the walker that hashes modules too, or a change to such a file would not register as one. Also: `langOf` fell back to `undefined`, which HighlightCode resolves to TypeScript rather than to no highlighting, so seeds and Markdown were coloured as code; and four comments still gave the removed cascade as the reason for sharing an asset node, which is now lineage. * fix(dbt): retry failed tests too, anchor the last __dbt path check `retry_failed_nodes` only ran after the model phase, which fails before the `after_all` test phase exists — so a project whose models built and whose tests failed got no retry at all, exempting exactly the failure mode that separate phase produces. The loop is now a function, called after both phases. `isDbtGeneratedPath` matched `__dbt/` anywhere, the fourth site with that bug: `foo__mod/vendor/x__dbt/target/a.ts` counted as generated dbt output, so `ignoreF` excluded an ordinary module file and a module-only edit never deployed its parent script. `wmill sync push` still dropped an ADDED or DELETED `.lock` three branches before the module arm, so the earlier fix only covered a first push: adding a `uv.lock` to a deployed project was reported as a change forever and never applied, and deleting one left it deployed. Editing worked, which is what made the round trip look whole. Also removes a duplicate `#[test]` that was double-registering a test and detaching its neighbour's comment, and rewrites seven comments that still gave the cascade as the reason for behaviour that now serves lineage only. * fix(dbt): bound the excluded-file read, keep the retry budget job-wide `isBundledModuleFile` read a file in full before deciding it was too big or binary, so a project sitting next to a multi-gigabyte parquet seed loaded the whole thing only to reject it. It now takes the size from `stat` and reads at most the 8 KB the NUL check needs: a 191 MB file is rejected in 0.0ms at 82 MB RSS. Calling the retry helper after both phases gave each its own `attempts` budget, so a job could spend double what the descriptor asked for — the bound exists because every attempt is a real dbt invocation holding a worker slot. The budget is now the job's, spent across whichever phases fail, and the field says so. Extracting that helper had also placed it between `#[allow(clippy:: too_many_arguments)]` and `run_dbt`, taking the attribute off the 12-argument function it was written for. * fix(dbt): actually spend the retry budget `retry_failed_nodes` looped on `while *remaining > 0` and never decremented it, so a failing job reissued `dbt retry` — logging "attempt 1 of 3" each time — until the job's deadline instead of `attempts` times. The decrement existed briefly and was lost when the function was re-extracted by hand. Claiming and counting are now one operation, `claim_attempt`, because keeping them apart is exactly how the bound goes missing: the loop cannot iterate without spending the budget. Its test is bounded by its own `for` rather than by the function under test. An earlier version collected `std::iter::from_fn(|| claim_attempt(..))`, which against a non-spending `claim_attempt` is an infinite iterator — it allocated until the machine died. A test for a loop bound must fail an assertion when the bound regresses, not consume the host: it now reports `[1, 1, 1, …]` against `[1, 2, 3]` in 0.00s. * fix(dbt): ask before reading, not after Bounding `isBundledModuleFile` did nothing for the bundle builder, which read the whole file into memory and only then asked whether to keep it — so a multi-gigabyte seed beside a project was still loaded in full just to be skipped. The predicate is now consulted first, and the read happens only for files the bundle actually carries. * feat(dbt): animate the ref() edges feeding the model being built The nodes moved during a run but the edges did not, so the graph showed where dbt had got to without showing it flowing there. Reuses the canvas's existing rule rather than adding a second one: an edge animates when it touches what is happening. For a pipeline that is the running script; for dbt the unit of work is the model, so a `ref()` edge animates while its target builds. Same `animated` field, same visual language, no new styling. Verified mid-run on a 7-model project: of six `ref()` edges only the two feeding the model then building were animated, and none once the job finished. * feat(dbt): show what each model wrote, and say when its SQL is another project's Three things a reader wanted from the run graph and could not get. Row counts: the worker already records one per relation and `run_progress` already returned it, but the graph used only `status` and dropped the number. A model that built green having emitted zero rows is the failure that looks like a success, so the count is on the node. The relation's fully-qualified name, copyable: there is no table browser to open, so the next best affordance is the exact identifier to paste into a SQL client. It is parsed with `splitRelation`, which honours quoting the way the worker's `split_relation` does — splitting on every period renders `"wh"."analytics.v2"."orders"` as a relation `orders` in a schema `v2`, which does not exist. And when two projects materialize one relation, the graph keeps a single provenance winner, so the losing project's node carries the other's model. The SQL was already suppressed there — correctly, it is not this run's code — but silently, which reads as a dead click. It now says so. * fix(dbt): a finished run's graph is the models it built, not today's project `/assets/graph` is the current deploy, so an old run's graph drifted with the project: a model added after it appeared as though the run had built it, and the older the run the wronger the picture. A finished run's node set now comes from its own result, which named exactly what it touched. Sources survive the filter regardless — dbt never lists them in `run_results.json` because it does not build them, but they are the upstream the run read, and dropping them would leave the models hanging. The graph is still the current deploy's, so a model renamed or deleted since cannot be drawn at all. Rather than a silently shorter graph, the count is stated above it. Verified by adding a model after a run: the old run renders 7 models without it, a fresh run renders 8 with it. * feat(dbt): preview a model's rows with `dbt show` There was no way to see the data behind a node — only its SQL and its row count. `dbt show` selects from a model and returns rows, and every engine ships it, so the preview needs no adapter code of ours: no connection path, no dialect-correct quoting, no type coercion for ten warehouses. It runs against the profile the run already renders. It is a `dbt_command` rather than a new endpoint, so it inherits the whole job path — authorization, isolation, cancellation, logs, engine provisioning — and `limit` joins the run form beside it. That the allowlist can admit it at all is a consequence of dropping the cascade: while a successful job dispatched its deploy-time write set, a command that wrote nothing woke every consumer for relations nothing had touched. Read-only, and treated as such: no graph republish, no materialization records, no retry state, no test phase. Captured rather than streamed, like `dbt ls` — these rows are the result, not commentary, and the job-log writer is what `NO_LOGS_AT_ALL` discards. Verified: `{"dbt_command":"show","select":["stg_customers"],"limit":3}` returns three rows; a preview leaves `materialized_partition` untouched (62 → 62, 0 rows for the job); `clean` is still refused by the allowlist. * feat(dbt): preview a model's rows from the graph, and keep our locks out of dbt projects The run page could show a model's SQL and how many rows it wrote, but not the data. Selecting a model now offers "Preview rows", which runs the script with `dbt_command: show` and renders the result as a table. Explicit rather than on-select: a preview is a job, so it costs a worker slot and the engine's start-up, and previewing on every click would spend both on mere navigation. Sources are excluded — dbt shows what a model SELECTs, and a source is not one. Also: `updateModuleLocks` was the one module helper that never learned about verbatim bundles, so it walked a dbt project writing `foo.lock` beside `foo.sql`. None of those files is a Windmill script needing a lockfile, and the bundle promises to round-trip the project byte-for-byte — our artifacts have no business in it. Verified in the browser: selecting `stg_customers` and previewing returns the columns `id`/`src` and five rows from the warehouse. * fix(dbt): keep a run's models when another project owns their provenance Scoping a finished run's graph to the ids it named dropped relations whose provenance winner belongs to a different project — so a run of a project sharing a schema showed 3 of the 6 models it had built. An id that was never this run's package cannot be judged against its result, so it is kept: the relation IS one the run wrote, and hiding it understates the run. The same rule applies to the "no longer in the project" count, which otherwise reported deletions that were only provenance collisions. Previews are now cached per model and survive the selection moving. One was thrown away whenever the reader clicked elsewhere, which for a job costing a worker slot and an engine start-up meant re-running it to see it again — and the run continues in the background, so leaving and returning finds the rows there. The spinner also never span: `startIcon` takes the icon and its classes separately, so the animation has to be passed alongside. How long it took is shown with the rows. A preview is a job, and its cost should not be something the reader has to guess at. * fix(dbt): resolve argument references, clamp the show limit, flag renamed relations `handle_dbt_job` cloned `job.args` where every other executor calls `build_args_map`, so a `$var:` / `$res:` / `$encrypted:` argument reached dbt as the literal string. A placeholder holding a schema or an `enabled` flag would then build a different slice of the project than the caller asked for. `--limit` took any positive i64, and the worker buffers the whole of dbt's stdout to read the rows out of it — so a caller with only run permission could make it hold an unbounded allocation. It is clamped to a ceiling now, extracted as `show_limit` so the bound is pinned by a test rather than inline in an async function nothing can reach. And a model keeps its id when its alias or schema changes, so an old run's node showed today's relation while the run wrote another — the page asserting it had materialized a table that did not exist yet. The run's result carries the relation each node actually wrote, so the drift is detectable without a graph snapshot, and the count is stated above the graph. Rendering the run's own lineage still needs a per-job snapshot; this stops the page claiming otherwise. * fix(dbt): stop persisting resolved secrets, bound the preview by bytes Resolving `$var:` / `$res:` / `$encrypted:` for dbt — added in the previous commit — meant `save_run_state` wrote the resolved PLAINTEXT into `dbt_run_state.args` and the worker's `state.json`. The row outlives the job, so a secret stayed in the database and a later `dbt_command: retry` replayed it after the grant was revoked or the value rotated. The invocation now carries the args as submitted alongside the resolved ones, run state persists those, and the restore path resolves them again under whoever is retrying. Clamping `--limit` bounded the row COUNT, not the size: one column can hold a megabyte, so a thousand rows is a thousand megabytes, and `run_capturing` buffers all of it. The captured output has a byte ceiling now. `limit` became a built-in argument without joining `RESERVED_ARG_NAMES`, so a descriptor writing `{{ limit }}` was silently handed the preview control's default instead of being told the name is taken. Two display fixes: the relation-drift banner compared a canonicalized (lower case) asset path against the warehouse's own spelling, so it fired on every model of every finished Snowflake run; and caching a preview's failure left `Preview rows` dead for that model until reload. * feat(dbt): key the graph by script version so a run renders its own project The dbt graph was keyed by path alone, so a deploy overwrote the only copy and a run page could only ever show today's project — an older run rendered today's models, SQL and `ref()` lineage no matter what it had run. My previous attempt filtered that view to the ids the run named, which stopped it lying but could not show what was gone: the data no longer existed. `dbt_node` / `dbt_edge` now carry `script_hash` in their primary key, so each deployed version keeps its own graph, and the run page passes the version its job recorded. Per DEPLOY, not per run — ten thousand runs of one version share one graph — and a composite FK to `script (workspace_id, hash)` with ON DELETE CASCADE means a version's graph dies with it. Nothing pruned these before, because there was one copy per path; they would otherwise have accumulated with no sweep. Two deploys of one path now write disjoint rows, so the graph can no longer be lost to a race. `claim_graph_publication` remains only for what is still path-keyed — the `asset` usage rows — and an older deploy finishing late records its own graph before declining to touch those, where before it published nothing at all. A pinned request is scoped by the version's own nodes rather than by `asset`: that table describes the current deploy, so scoping through it would filter a model out of the very run that built it. Verified end to end: deployed v1 (8 models), ran it, deployed v2 with four models removed and one rewritten. The old run renders 8 models, 6 ref() edges and v1's SQL; a new run renders 4 and the v2 rewrite. * fix(dbt): scope graph cleanup to one version, bound the preview capture Archive and delete both act on a single `hash`, but the graph cleanup they called deleted every row for the path. Now that the graph is keyed per version, archiving an old version erased the live one's models, SQL and lineage, and nothing repaired it. Both callers have the path in hand, so the by-hash wrapper is gone and they use the version-scoped clear directly. `dbt show` checked its 8 MB ceiling after `wait_with_output` had already buffered everything, so the ceiling could not bound what the worker held. `run_capturing` now reads both pipes incrementally against a caller-supplied limit and kills the child on overflow. The read buffers are heap-allocated: as arrays they were baked into the future, which the job poller boxes several layers deep, and that overflowed the worker thread's stack — a `dbt show` run aborted the whole worker process. A retry's `dbt parse` ran on the arguments as submitted while the build ran on resolved ones, so a `$var:` shaping the graph parsed verbatim. The parse moves to the caller, after resolution. A run that names its own `select`/`exclude` now drops the descriptor's `selector`: dbt resolves `--selector` instead of `--select`, so passing both made a preview of one model return another's rows. Also: log instead of silently swallowing a `modules` column that fails to deserialize (pre-existing, but for dbt it means running with no project at all); keep the model SQL reachable once a preview has landed; render which node the rows came from; stringify object-valued cells; document `dbt_script_hash` in the OpenAPI spec. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: record the per-worktree dev environment and the backend-run check Three mistakes this guidance would have prevented, each of which cost a cycle: A worktree has its own database and ports, but AGENTS.md stated the single-checkout defaults as facts. Pointing `DATABASE_URL` at another worktree's database makes `cargo sqlx prepare` fail on every query touching a table your migrations added — and it deletes `.sqlx/` before it fails, so the cache is gutted rather than merely stale. Starting a backend on the wrong port leaves the UI up with every call 502ing, which reads as an application bug. Both values are now discoverable with commands that work as written. `prepare` is also documented as the wrong tool for a removal-only change: the cache is already complete for CI, and the only residue is orphaned entries that can be found by text-matching against the sources without a database. Nothing told a reader that `cargo check` does not exercise a worker path. A read buffer declared as an array inside an async block is baked into the future, and once boxed by the job poller it overflows the worker thread's stack — compiling and unit-testing clean while aborting the whole worker process at runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): scope the remaining path-wide reads and clears to one version Four places still spoke for a whole path after the graph became per-version: The relation-root drift check read `dbt_node` by path with an unordered `LIMIT 1`, so with v1 at root A and v2 at root B it could answer with v1's row, suppress the refresh v2 needed, and leave v2's graph naming relations the run does not build. It now reads this job's version. `dbt_dep`'s no-resource branch cleared the path, so a descriptor edited to bring its own `profiles.yml` emptied every earlier version's graph and with it every finished run's page. The ownership being given up is the path-keyed `asset` usages cleared beside it; the graph clear is now this version's. The graph was inserted before the publication claim checked the version was still live. Archive and delete only soft-update `script`, so the foreign key still accepted an in-flight dependency job's rows and the failed claim committed them — and because pinned queries deliberately serve archived versions, deleted model SQL became readable again. The write is now gated on a `FOR UPDATE` liveness check. `clear_dbt_run_state_by_script_hash` resolved a hash to a path and deleted the path's saved run. `dbt_run_state` is keyed by path by design — one saved run per script — so archiving one version discarded the live version's resumable failure. It clears only once no live version of the path is left; `identity` already refuses a resume whose project, warehouse or engine moved. "Preview rows" ran `runScriptByPath` while the SQL beside it was pinned to a hash, so an old run showed its own SQL over today's rows. Verified end to end: with v3 deployed, the v2 run's preview runs v2's hash and returns v2's rows. Also: keep the TAIL of a captured stderr, since dbt prints its summary last; one `$derived` for the parsed result rather than five; collapse three near-identical argument accessors onto one generic; fold the single-use `copy_dir_command` into its caller; and give `parseDbtRun.ts` one status classifier instead of spelling dbt's failure vocabulary twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(dbt): one JobCtx down the executor, one table per adapter Two changes aimed at the operations this code will keep having: adding a phase, and adding a warehouse. `JobCtx` already bundled the five values every phase needs, and nine functions took it — but the top of the executor threaded the fields apart and rebuilt the struct at each call, so the same literal appeared eight times and each new phase meant five more parameters. It is now built once per entry point and reborrowed. `prepare_project` goes from 21 parameters to 17, `retry_failed_nodes` from 15 to 11, and `run_dbt` drops below the lint threshold. The two remaining constructions are the worker boundary, where the pieces genuinely arrive apart. `DbtAdapter` answered five questions with five parallel matches over the same eleven variants, plus a sixth list of adapters kept by hand in a test. The facts now live in one `AdapterSpec` per adapter, reached through one exhaustive match, so adding a warehouse states its name, driver, package, port, database key and licensing together and the compiler demands the arm. Each arm spreads from a Postgres base, which makes the inheritance visible per adapter instead of hidden in the `_ =>` defaults `default_port` and `database_key` used to carry. `DbtAdapter::ALL` replaces the list the test kept separately. Verified by dumping all seven facts for all eleven adapters before and after: byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): job-keyed run progress, and stop path-wide reads and clears Six findings from the last round, in the order they bite. The publication liveness gate refused on `archived`, but `create_script` archives the parent on every redeploy — so deploying v2 while v1's dependency job was still parsing left v1 without a graph, permanently, which is the exact case the unconditional write existed to serve. It gates on `deleted` alone now; an explicit archive is still covered by the `FOR UPDATE` ordering. A project-owned `profiles.yml` trusted `profile.type` instead of reading the file. The Rust engines carry every adapter, so a CE script could declare `postgres` over a target that is `sqlserver` and have dbt connect with the enterprise adapter. The file is read whichever way, and a descriptor that disagrees with it is refused. Renaming a dbt script, or editing one so its newest version is no longer dbt, cleared the graph for the whole path — every older version's models, SQL and lineage, which their own finished runs still render. Neither needs it: graph queries join on `(path, hash)` through a `language = 'dbt'` CTE, so an old version's rows cannot attach to whatever lives at that path next. Live progress read `materialized_partition`, whose key is the relation and whose `job_id` is only the last writer. Two runs of one project took rows from each other. Progress now has its own job-keyed table; the relation table is untouched, because one row per relation is right for the pipeline canvas and fork defer. Verified with two overlapping builds: both keep 6 rows in the new table, while the old one attributes 6 to one run and 0 to the other. `Scratch::drop` removed a half-installed virtualenv synchronously from inside the job future, blocking a runtime thread; it goes to `spawn_blocking`, with a direct call when there is no runtime to hand it to. The E2E list asked for a `# on table://` subscription the deploy now refuses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(dbt): snapshot a dynamic descriptor's graph per run A `{{ }}` placeholder in `vars` can enable a different set of models per run, so those runs re-ingest the graph. Keyed by version alone, each re-ingest overwrote the last: reopening an older run showed the newer run's project, and a model only the older run built was gone entirely — no SQL, no lineage, and nothing the saved result could colour, since it can only tint nodes that are there. `dbt_node` / `dbt_edge` gain `job_id`. A run of a dynamic descriptor writes its own snapshot under its job id and its page reads it back; a static descriptor writes the version's graph once, under a zero-UUID sentinel, and every run of it reads that. The sentinel is a value rather than NULL because `job_id` is part of the primary key and Postgres does not treat two NULLs as the same key, so each re-ingest would add a row set instead of replacing one. `/assets/graph` takes `dbt_job_id` and prefers a snapshot when one exists, falling back to the version's graph otherwise — so a run page passes it unconditionally and static descriptors are unaffected. Snapshots age out after 30 days, pruned by the runs that write them, so no background sweep has to learn about these tables. Verified end to end: one deploy, two runs of it with `extra=yes` and `extra=no` gating a model's `enabled`. The version's graph holds 6 models, run 1's snapshot 7 including `opt_extra`, run 2's 6 without it; the endpoint returns each run's own and falls back to the version's when the parameter is omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(dbt): only snapshot a run whose graph differs, and prune from every run Two costs the per-run snapshot carried, both found by measuring it rather than by reading it. A snapshot was written for every run of a dynamic descriptor, but marking one dynamic is conservative: `graph_is_per_run` is true whenever `vars` holds a `{{ }}` placeholder or `env` holds a `$var:`, which says the arguments reach dbt and not that they change which models exist. The usual case is a date var, whose graph is identical run after run, so the table filled with copies of an unchanging picture — around 1 KB per model per run, which is a gigabyte or so a month for a 200-model project on an hourly schedule. A row set now carries a digest of its nodes, edges and relation root, and a run whose digest matches the version's writes nothing; the read already falls back to the version's graph, so those pages are unchanged. Only a run whose model set really differs pays. The prune was hung off the progress reporter, which exists only for engines that emit node events — so a Fusion or dbt-core-2x instance accumulated snapshots and never deleted any. Retention that stops working because of an engine choice is not retention; it runs detached from every dbt run instead. Verified against a descriptor with a var-gated model: the version's graph holds 8 rows, a run that resolves to that same graph stores none at all, and a run that enables the extra model stores its own 9. Both pages still render their own project — 6 assets without the extra model, 7 with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): scope every dbt_node join to the chosen snapshot `job_id` joined the key, but only the scoping CTE and `dbt_edge` were taught to filter on it. The outer node SELECT and the parent/child joins in the edge query were not, so each model came back once per retained snapshot plus once for the version's graph, and each edge matched every combination of the two — the model count multiplied and the edge join fanned out quadratically. Measured against one stored snapshot: 17 node rows where 8 are wanted, and 28 edge pairs where 7 are. The response dedup hid the edge blow-up from the payload, not from the plan, and the run page refetches the graph every two seconds. The progress table gained writers it was missing. `terminalize_running_relations` settled only the relation-keyed table, so a cancelled or killed run — the case that function exists for, since it leaves no `run_results.json` — showed every in-flight model still spinning on the run page for as long as the row lived. An agent worker cannot write the new table at all, having no database of its own, so the read falls back to the relation-keyed one when a job has no rows there. Also: a wrapped string literal missing its backslash put eighteen spaces in the middle of the profile-disagreement error; a comment still described concurrent runs of one dynamic version overwriting each other's graph, which is what keying by job removed; and the `materialized_partition` index justified itself by a run-page poll that has since moved to another table, though the closing sweep still earns it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): give a graph snapshot a marker row, and scope what reads it Five findings, four of which are the same mistake in different places: a snapshot's identity was inferred from its contents. Existence was inferred from a `dbt_node` row, so a dynamic run that disabled every model — a legitimately empty graph — read as "no snapshot" and its page showed the deployed models instead. The digest was a column repeated on every node and read back with a `LIMIT 1` carrying no `job_id`, so a run could compare itself against another run's digest and suppress a snapshot it needed. The relation-root drift check read the same rows unscoped, so after a drift it could find a previous run's root and conclude nothing had moved. `dbt_graph_snapshot` holds one row per stored graph — path, version, job, digest, timestamp. Existence is that row, the digest lives there once, the drift check reads the deployed row explicitly, and the retention sweep deletes markers first and then the rows no marker stands for. The digest is SHA-256 rather than `DefaultHasher`, whose output is documented as unstable across Rust releases: this value outlives the process that computed it, so a toolchain bump would have silently stopped every comparison matching and quietly reinstated the duplicate snapshots the digest exists to prevent. `/run_progress` ignored the view token, so a share-link viewer got the graph and was refused the progress that colours it. A preview sent only its own three arguments, so a descriptor with a required `{{ }}` var could not be previewed at all and an overridden one previewed a different relation than the page was showing. The run's arguments go first now, with the preview's three overriding. Verified on a project whose only model is var-gated: the deploy stores a marker with zero nodes, a run with the var set stores a marker with one, and the endpoint answers 0 and 1 respectively rather than showing the deployed models for both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(dbt): squash the runtime's migrations into one Ten migrations reshaping the same three tables is a history no installation ever had. `dbt_node` gained `script_hash`, then `job_id`, then `ingested_at`, with its primary key rebuilt twice; `graph_digest` was added by one migration and dropped by the next after the digest moved to its own table. On a fresh database all of that replays to arrive at a shape the schema can simply state, and this feature has never shipped, so there is no upgrade path to preserve. One migration now creates `dbt_node`, `dbt_edge`, `dbt_graph_snapshot`, `dbt_run_state` and `dbt_run_progress` in their final shape, carrying forward the rationale each of the replaced migrations recorded. The enum additions stay in `add_dbt_lang`, since a value cannot be added and used in one transaction, and the `materialized_partition` index stays separate because it belongs to a table this feature did not introduce. Verified by rebuilding: dropped the five tables, replayed from the single migration, and confirmed the result is identical — same primary keys, the same two composite `script` foreign keys, the same seven indexes. Every `sqlx::query!` in the workspace then compiled against it, which checks each column's name, type and nullability, and a deploy plus run on the rebuilt schema produced 8 nodes, 7 edges, a snapshot marker and 6 progress rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): authorize snapshot reads, bound the prune, give the marker a lifecycle `dbt_job_id` is caller-supplied and selected straight from `dbt_graph_snapshot`, which carries no RLS — so a caller who could see the script could read any run's model set and relation paths, which a dynamic alias or schema can encode. Both graph queries now require the job itself to be visible, in the authed transaction, the same gate `raw_code` already applies to the script that produced it. The drift check compared against the deployed graph alone, which misses the way back: a run at root B republishes the path-keyed `asset` usages at B, and returning the profile to A then matches the deploy and skips the refresh, leaving those usages at B while dbt builds A. It reads the most recent ingest for the version instead — the one that last wrote them — ordered rather than an arbitrary `LIMIT 1`. The prune anti-joined every non-deployed node and edge with no age predicate, so each run scanned the whole retained sidecar and concurrent runs duplicated it. All three deletes share one age bound again, with the sentinel spelled as a literal so the partial indexes apply — a bound parameter cannot be proven to match the index predicate. `dbt_graph_snapshot` was the one dbt table nothing in the script lifecycle deleted: no `script` foreign key and absent from both `clear_dbt_manifest*` sites. A marker outliving its rows is read as a snapshot with no nodes, and its digest still answers the suppression check, so an identical run would write nothing and then render an empty graph. It cascades like the rows now and both clears take it. Also: the preview cleared `exclude` rather than inheriting it, since previewing a model the run excluded reached dbt as `--select m --exclude m`; and `terminalize_running_relations` no longer claims to cover a killed worker, which never reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): quote profile names, record where usages were published, stop polling the graph A profile name comes from the project's own `dbt_project.yml` and a target from the descriptor, and both were interpolated into `profiles.yml` as bare YAML — including as mapping keys. A name like `prod # hidden` truncates the mapping and a newline opens a sibling key of the author's choosing. Both are rendered as quoted scalars now, as are the BigQuery keyfile's keys, with a test that asserts the document still parses to exactly the keys we wrote. The drift check read the most recent ingest, which latches: a run that returns to the deployed root re-ingests but stores no snapshot (its digest matches the version's), so the moved run's rows stay newest and every later run pays an extra parse and ingest. The publisher now records the root it published the path-keyed usages at, which is the only thing that answers "where do the current usages point" — the deploy's own root goes stale as soon as a run republishes. The run page polled `/assets/graph` every two seconds alongside progress, so it re-sent every node's SQL for the length of a run — hundreds of KB a tick on a real project, for a graph that a dynamic descriptor re-ingests exactly once before the build. It fetches once more shortly after mount and then polls progress alone. Node results carry `outcome` beside `status`. `status` stays dbt's own word, but dbt owns that vocabulary — 1.x and 2.x differ on casing and `no-op` arrived in a minor release — so publishing only it would force a break or a lie the first time it moves. `outcome` is the stable half a downstream script branches on. Also: the worker's dbt entry points are `pub(crate)`, since nothing outside the crate calls them and they resolve secrets and launch processes; and the snapshot gate records that it is RLS-only where `/jobs/run_progress` also honours a share-link token, which is a gap in what a shared page shows rather than in what it protects. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(dbt): pin the graph storage invariants against a real database Every defect review found in this area was DB-shaped — which row set a read resolves to, which rows a clear takes, whether a snapshot exists at all — and none of it is reachable from a unit test on a pure function. Four rounds established these answers and nothing guarded them, which is why each round kept finding another. Six cases, on the harness the repo already uses for schema-shaped behaviour: an identical run stores no snapshot and leaves no marker; a differing run keeps its own while the version's is untouched; an empty run graph is still a snapshot rather than an absent one; clearing one version leaves the others whole; the path-wide clear takes the markers with it; and the sweep ages out run snapshots while never touching a version's own graph. `IngestedNode` gains `Default` so a test can state the two fields a case is about rather than the eighteen it is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(dbt): bound a script's stored graphs by deploy count Run snapshots expire on a clock, but a VERSION's graph could not: its reader is every finished run of that version, and a run page is as old as its job. So nothing reclaimed them — a deploy graph went only when its `script` row was hard deleted, which Windmill does not routinely do. A CI deploying on every commit added a full model set with SQL bodies per commit, forever: roughly 200 KB a deploy for a 200-model project, which is gigabytes a year across an instance. Bounded by COUNT instead of age, since age is the thing that cannot be right here. The newest 50 deploys per path keep their graph and older ones are reclaimed, making growth `versions x models` rather than unbounded in time. Generous on purpose: reaching the bound empties that version's run pages, so it exists to stop unbounded growth rather than to be hit in normal use. Ordered by the script's own `created_at`, so a late-finishing job re-ingesting an old version cannot promote it. Pinned by a test that deploys past the bound and asserts both halves: the count holds, and the newest version is always among the survivors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): let the run page know when its snapshot has landed The one-shot graph refetch was wrong: a dynamic descriptor's ingest happens before the build but after cloning, dependency install and parse, so a fixed delay either fires too early — and the run page then shows the deployed models for the whole run, never that run's own — or keeps re-sending the whole graph for the length of it. Neither is a timing problem to tune; the page had no way to tell "the snapshot is not written yet" from "this run has none". `/assets/graph` answers that directly: `dbt_snapshot_job` is the job the dbt half resolved from, when one was asked for and found. The page polls the graph until that is its own job, and stops. A static descriptor never snapshots, so an attempt cap ends it there rather than polling for the run's duration. `dbt_node.relation_root` is gone. The drift check moved to the marker's `published_relation_root`, which left the column written on every node and read by nothing. `outcome` was published as the stable half of the result contract, but the in-tree consumer still ranked and coloured from dbt's own word — so the field existed and nothing used it. `statusRank` takes it, `DbtRunResult` passes it, and `classifyStatus` is documented as the fallback for results that predate it and for the live event stream, which carries dbt's word alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(dbt): record what a share-link viewer actually sees The comment at the snapshot gate said the graph "falls back to the deployed set", which is only the rarer half of it. A share link is an extra grant for a logged-in user who lacks access to the job, so the usual case is no read on the script either — and then the `live` CTE matches nothing and the whole dbt half comes back empty. A blank Models panel over working progress rows, not a fallback. `docs/dbt-runtime.md` now carries the analysis a follow-up needs: that relaxing this leaks nothing, because `v2_job_completed.result` already gives that viewer every node's `unique_id` and `relation_name` — the graph's only incremental exposure is `raw_code`, which is gated separately on seeing the script. And the shape of the fix: `OptViewToken` and `validate_view_token` are self-contained enough to move into `windmill-api-auth`, which `windmill-api-assets` already depends on, after which the gate can honour a token for that job's snapshot alone while `raw_code` stays where it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): keep model SQL behind the scripts:read scope, and unbreak CI `/assets/graph` is authorized as `assets:read`, and RLS decides whether the caller can see the script that produced a node — but RLS is not a scoped token's grants. A token deliberately narrowed to `assets:read` could therefore read model source and repository paths for scripts outside its `scripts:read` paths. The same `build_scope_path_predicate` the macro endpoint already applies now gates `raw_code` and `original_file_path`; the relation's shape is unaffected, only its body is withheld. `DbtAdapter::ALL` exists for the tests that must cover every adapter, so it is dead in a release build and `-D warnings` failed all four backend checks on it. It is `#[cfg(test)]` now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): stop the graph poll at the ingest, and type `limit` in the schema The poll's stop condition was a snapshot appearing, with a 40-attempt cap behind it — so a STATIC descriptor, which never snapshots, took the cap every time and re-fetched the whole graph forty times. That is most of what removing the poll was meant to save, and static is the common case. The ingest runs BEFORE the build, so the first model to report progress proves it has already happened: a snapshot absent by then is one this run never writes. Progress arriving is now the second exit, and the cap is only a backstop for a run that reports none at all. `limit` is declared `Typ::Int` but `dbt_arg_schema` had no integer arm, so the run form and the generated clients saw an untyped default and offered no numeric control for a value the worker clamps. Covered by the schema test. `relationOutcome` still re-derived from dbt's word while `statusRank` had moved to `outcome`; both read it now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): make a retry prove its arguments still resolve the same The saved arguments are the ones SUBMITTED, so a `$var:` in them is re-resolved on retry. The identity did not cover the resolved values, so a variable that changed between the failed run and the retry was accepted — and which graph the retry then used depended on WHERE it landed: a worker holding the local snapshot replays the saved manifest, while a database restore reparses with the new value. Placement decided whether the resumed failures described the relations being built. The identity gains a digest of the resolved arguments, and is compared in two halves because resolution happens between them. Project, warehouse, engine and env are checkable up front; the arguments are not, because a retry request carries only `dbt_command` and the ones to compare are the SAVED arguments after this caller has re-resolved them. Comparing the whole string up front would have refused every retry — which is what the obvious version of this fix does. A row written before the digest existed has no last segment, and still restores rather than being refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): keep pre-upgrade retries working, and stop losing a late snapshot Splitting the identity on its last `|` read a pre-upgrade row's env digest as an arguments digest and left only `<run_identity>` as the prefix, so every saved failure on an upgraded instance became unretryable — a regression the previous commit's own test missed by using an identity with no `|` in it at all, which is not what an old one looks like. The digest is tagged (`|args=`) rather than positional, and the test now uses a real pre-upgrade identity. The graph poll gave up after a bounded number of tries, but provisioning and `dbt deps` precede the ingest and can outlast that on a cold worker — and the engines that emit no node events never produce the progress that ends it early. A finished run now reloads the graph unconditionally, and the poll's own exit issues one last load: progress proves the ingest happened, not that the previous tick saw it, and dbt's compile window is wider than one tick. A `dbt retry` restores the failed run's arguments inside the worker and they are never written back to the retry job, whose own args are just `{"dbt_command": "retry"}` — so previewing a row on a retry's page ran without the vars the run used. The result now carries the invocation's arguments as SUBMITTED, so a `$var:` stays a reference and no resolved value is published. The deploy-count sweep ran instance-wide on every dbt run: `FROM script WHERE language = 'dbt'` has no index to stand on, and both orphan deletes are the complement of every partial index here. It is scoped to the running script's `(workspace_id, path)` — which `index_script_on_path_created_at` serves — and the orphan deletes only run when a marker actually went. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): hide dbt from module-less pickers, and stabilise the retry digests `processLangs` feeds every language picker, including flow steps and app inline scripts. Those are raw bodies with nowhere to carry a module bundle, and a dbt script IS its bundle — so choosing dbt there produced a job that could only fail once the worker looked for `dbt_project.yml`. Those two surfaces use `processInlineLangs`, which drops the languages that need modules; a flow still reaches dbt the way it reaches any script, by path to a deployed one. `graph_digest` moved to SHA-256 because it is persisted and compared by a later worker, and `DefaultHasher` is documented as unstable across Rust releases — but the retry identity's own digests were left on it, and they are persisted in `dbt_run_state.identity` for exactly the same comparison. A toolchain bump would have refused every saved failure as a different project. All three go through one `stable_digest`, length-prefixed so no split of the same bytes collides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): enforce the tag scope on snapshot reads, reset state between runs A tag scope is an orthogonal hard restriction: a token limited to some tags must not read a job outside them however else it is authorized. The snapshot lookup went through `v2_job` RLS alone, which knows nothing about tags, so such a token could still retrieve a run's model set and its dynamic relation paths. The same predicate `require_job_read_access` applies for the progress half of the page is applied here — `get_scope_tags` is already public in `windmill-api-auth`, and it is `None` for an unscoped caller, so a normal session pays nothing. SvelteKit reuses the run graph between run ids, and `graphTries`, `polled` and `raw` all describe the previous job: a spent retry count stopped the next run's snapshot poll before it began, and stale progress coloured its models with another run's statuses. All three reset when the graph key changes. Also a wrapped string literal missing its backslashes, which put two ~22-space runs in the middle of the retry-refusal message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(dbt): put each digest helper's rationale on its own function Inserting `stable_digest` above `split_identity` split that function's doc, so five lines describing where the identity divides ended up introducing the hasher. Each is back on the function it describes, stated once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): read the run-pinned graph through the job, not the asset graph Pinning the asset graph to one run is job-scoped data, but `dbt_job_id` sat on `/assets/graph`, authorized as `assets:read`. The job-read contract — `require_job_read_access` — is five parts that pull in opposite directions (tag scope restrictive, `created_by` permissive, app-embed restrictive-overriding, view token permissive, RLS underneath), so plain RLS is neither a stricter nor a looser approximation of it. Restating the parts near the graph query kept leaving one out: first the job check entirely, then the share-link asymmetry, then the tag scope, and the app-embed restriction was still missing and failing open. The helper cannot be called from `windmill-api-assets`, because `windmill-api` depends on that crate. So move the read instead of the check: the run-pinned graph is now `GET /w/{w_id}/jobs/dbt_graph/{id}` in `windmill-api`, on the same gate as the `run_progress` it colours, and `/assets/graph` has no `dbt_job_id` parameter at all. - `asset_graph_for` takes the job as an argument from an already-authorized caller; the route handler passes `None`. - Extract the graph response into a `AssetGraph` component schema, now that two paths return it. - The run page fetches the job route when it has a job id. * fix(dbt): charge assets:read on the run-graph route, trust the job gate in SQL Round 13 findings on the route moved last commit. The scope domain comes from the URL segment, so putting the read under `/jobs` asked a scoped token for `jobs:read` alone while returning asset-graph data that `/assets/graph` charges `assets:read` for. A token narrowed to polling run status could read workspace topology, and the missing-job fallback made it cheaper still — any random UUID skipped the job gate. Both scopes are now required: the job gate reaches this run, `assets:read` reaches asset data at all. The `chosen` CTE re-decided job visibility under plain RLS after the caller had already passed `require_job_read_access`. It could only disagree, and did so silently by falling back to the deployed graph — a share-link viewer entitled to the run was shown a different run's model set. Dropped; the contract is that a job reaching `asset_graph_for` is already authorized. Also: the flow editor's `+` insert menu still offered dbt (the third `processLangs` caller, missed when the other two moved to `processInlineLangs`), the docs still described the deleted `dbt_job_id` parameter, and the new handler had again been inserted between `get_run_progress`'s doc comment and its function. * fix(dbt): resolve a pinned run's version from the job row, not script RLS Local codex review of the branch. A share-link viewer is entitled to the run and usually has no grant on the project — that is what the link works around. The graph's `live` CTE resolved the version by selecting `script` inside the viewer's RLS transaction, so it answered for their access to the project rather than for the run they were given: the Models panel came back blank beneath working progress rows. A pinned run now takes its path and hash from the job row the handler already read after authorizing the job, so `live` does not consult `script` at all. `raw_code` keeps its own `EXISTS` against `script`, so the model bodies stay behind access to the project. Verified under RLS as an unprivileged role: the shape query goes 0 rows -> 1, the `raw_code` gate stays 0. Taking the version from the job also means a caller can no longer pin one project's version while naming another's run, since `dbt_script_hash` is ignored when a job is given. The run page's graph fetch is a raw `fetch`, which bypasses the interceptor that adds `X-View-Token` to generated-client calls, so a shared page was refused before any of this mattered; it goes through `appendViewToken` now. Also trims three comments to the AGENTS.md limit, dropping drafting-history rationale that belongs in docs/dbt-runtime.md. * fix(dbt): snapshot vars-overridden runs, carry the pinned version everywhere Second local codex pass. A `vars` run argument overrides the descriptor's, and vars drive `enabled`, alias, schema, database and materialization — so such a run builds relations the deployed graph does not describe. It now snapshots under its own job id, which per-job keying makes safe: the version's graph stays for runs that did not override. The old comment claimed gating on it would strand the override's graph for the next default run, which was true only when the write went to the deployed slot. Two sites still read the caller's `dbt_script_hash` instead of the version resolved from the job, so `/jobs/dbt_graph/{id}` without that redundant parameter dropped models the run's version had and a later deploy removed. The `dbt_snapshot_job` marker re-checked `v2_job` under RLS — the recheck the graph query itself drops. A share-link viewer got the right graph and a null marker, so the run page refetched it 40 times before giving up. `wmill script preview` read the bundle with the generic `__mod` suffix and script-module parsing, so previewing a `.dbt.yaml` omitted the project and failed on the missing `dbt_project.yml`. It uses the same suffix and verbatim read as deploy. * fix(dbt): key retry state by principal, not by script path alone `dbt_run_state` held one row per (workspace, script path), and a retry replaces the caller's arguments with the saved ones. Anyone able to run the script could therefore retry whoever ran it last, replaying that run's literal `select` and `vars` against the warehouse and publishing them as their own job's `invocation_args`. Running the script was already theirs to do; seeing another principal's arguments was not. `permissioned_as` joins the key, so a retry resumes only state written under the same authority. Two runs sharing an authority can already act for each other, so this is the boundary that matches the rest of the job model. * test(dbt): pin what a caller without access to the project sees of its run The share-link case had no regression guard, and every fix in this area touched one of its two halves: the graph's SHAPE has to survive a caller who cannot read the script, and the model SQL must not. Two cases against a real database, calling `asset_graph_for` as a member with no grant on the project's folder: pinned to a run, the models render and `raw_code` is withheld; unpinned, the same caller sees nothing of it, so making the first work did not relax the second. Both assertions were checked by mutation — reverting the `live` bypass empties the graph, and dropping the `raw_code` script gate leaks `select 1` — so neither passes on the code it is meant to catch. * fix(dbt): key the worker-local retry cache by principal too Keying `dbt_run_state` by `permissioned_as` left its worker-local twin keyed by workspace and script path alone, so the boundary held only where the database row was consulted. An agent worker never reads that table — `Connection::Http` leaves `latest_job` as `None` — so there the local cache was the whole boundary and it had none: the next principal to retry the script on that worker restored the previous one's `select` and `vars`. Also records the sqlx `--all-targets` trap in the update-sqlx skill: it is needed for queries inside tests, and in a CE checkout it aborts on `tests/otel.rs` (EE-only `otel_ee`) after having already emptied the cache. * fix(dbt): log a dropped retry-state save, correct the run-progress contract Saving retry state is best-effort — losing it costs a retry, not the run that just finished — but `.ok()` dropped the reason too. The only symptom was `dbt retry` reporting nothing to resume, which reads as a bug in retry rather than a failed write. Found by running a real failing build against a worker whose binary predated the `permissioned_as` column: the insert violated NOT NULL and said nothing. The run-progress endpoint's OpenAPI description promised an empty list for a caller who cannot see the job. It is refused instead; an empty list means the job recorded nothing yet or is unknown here. * fix(dbt): return the retry-state write failure the warning was added to report `save_run_state` discarded the insert result, so the caller's warning could never fire and a lost retry row stayed silent — the symptom being `dbt retry` finding nothing on another worker. The error is held rather than returned at once: the worker-local copy is what an agent worker resumes from, so a failed insert must not cost that too. Every exit after it surfaces it, including the ones that give up on the local save. * fix(dbt): decide a retry's graph from its restored args, keep local state in step Three from the seventh local review. A retry submits only `dbt_command`, so the vars-override check ran against an empty argument set and left `graph_is_per_run` false. The failed run's arguments are restored afterwards, and those are what the retry builds with — an overridden one wrote no snapshot for its own job and its page fell back to the deployed graph, showing the wrong enabled models, aliases and schemas. The decision is re-asked once the restore has happened. A failed durable write no longer publishes the worker-local generation either. `restore` accepts a local generation only when the database row names it, so publishing one the database never recorded made this worker reject its own newest state and resume the previous run's — its selection and vars, or "nothing to retry" if that one had succeeded. An agent worker attempts no durable write, so it keeps its local copy as before. A rename that also converts away from dbt moved the old path's retry state onto the new one, reinstating what the conversion had just cleared and leaving one user's arguments and results under a path no dbt script occupies. It moves only while the destination stays dbt, and clears the source otherwise. * fix(dbt): drop retry state when a run produced none, let module pushes fail loudly Three from the eighth local review. A run that never wrote `run_results.json` — cancelled, timed out, or dead before dbt got there — left the PREVIOUS run's state authoritative in both the database and the local pointer, so a later `dbt retry` resumed that older invocation's failed nodes. Producing nothing resumable now clears both copies, so neither can answer for the other. `wmill sync push` wrapped the descriptor lookup and its deployment in one try/catch meant for a missing parent. Any API failure or invalid descriptor was reported as "no parent found" and swallowed, so a module-only push exited zero with the remote project unchanged. Only the lookup is tolerated now. Also condenses a comment that narrated how earlier status comparisons behaved. * fix(dbt): forget retry state on pre-build exits too, drop cascade claims A dynamic run whose pre-build `dbt parse` or graph ingest fails returns before the save that clears stale state, so the previous run stayed authoritative in both the database and the local pointer and `dbt retry` resumed ITS failed nodes — writing relations the run that just failed never touched. Both exits now invalidate, through one helper shared with the no-artifact case. Two frontend comments described dbt producer rows as driving cascade dispatch. The executor returns before dispatch for every dbt job and deployment rejects `table://` subscriptions, so they promised behaviour that cannot occur; they describe the lineage and ownership that is actually retained. * fix(dbt): let the database decide retry state where it is reachable A SQL worker treated "no `dbt_run_state` row" as no opinion and accepted any worker-local generation. But no row is the authoritative answer that the last invocation left nothing resumable, so a local pointer that outlived it — an unlink that failed, a process killed between the delete and the removal, a stale cache — resurrected a replaced run and let `dbt retry` write relations it never touched. An agent worker keeps accepting its local copy: it has no authority to consult. Invalidation failures are logged rather than dropped, since a silent one is exactly what leaves the pointer behind. * docs(dbt): record how to run an agent worker locally, keep archived graphs Every step of standing one up fails as something else: a normal build cannot start one at all, the server's routes need a separate feature, and all three token mistakes surface as a bare 401 on the agent with the reason only in the server log. Written down with the error each produces. Also keeps a dbt script's graph when it is ARCHIVED rather than deleted. The pinned read resolves versions through a CTE that already skips archived rows, so clearing bought nothing and emptied the Models panel of every completed run of the project. Deletion still clears it. * feat(dbt): let an agent worker publish its graph, through one endpoint An agent worker was refused any dbt script whose profile comes from a Windmill resource — the common case — because it could neither read the stored relation root to check for drift nor re-ingest a corrected one. Those look like two needs but collapse into one: verification exists only to decide whether the stored graph still describes reality, so a worker that can PUBLISH never has to ask. It stores what it just parsed. `POST /api/agent_workers/dbt_graph/{workspace_id}` is the whole addition. It wraps the same `replace_dbt_manifest` the SQL path calls, so digest suppression, the marker write and retention cannot drift between the two transports, and it refuses a job the token's tags do not cover. `IngestedManifest`/`IngestedNode` gain Deserialize to cross the wire. Two guards go, both now false: the pre-build refusal, and the `Connection::Sql` gate added earlier to stop a `vars` override grounding an agent run. Live progress stays SQL-only — that is a per-model event stream, and routing it through the API would mean a round trip per node. * docs(dbt): warn that a differing cargo feature set swaps the shared binary * docs(dbt): record the verified agent-worker behaviour and the tmpfs quota trap An agent worker now runs a dbt job end to end, retries, and publishes its graph — confirmed with a dynamic descriptor whose per-run snapshot came back through the new endpoint. The doc said it was refused; that was true before the endpoint existed. Also `WINDMILL_DIR`: on a dev box the job dies with `Disk quota exceeded (os error 122)` writing the project's files while `df` shows free space AND free inodes, because /tmp is a tmpfs carrying a per-USER quota. Point the worker at a real disk rather than trying to clean up beneath it. * chore(dbt): pin the EE revision carrying the agent graph endpoint * fix(dbt): bind the published graph to the job, break the completed-page poll loop Five from the thirteenth local review. The EE endpoint took `script_path` and `script_hash` from the payload and checked only that the supplied job carried one of the agent's tags, so an agent holding any matching-tag job could name another script and replace its graph. Both are read from the verified queue row now and the request carries only the job id. A raw preview has no version, so it no-ops rather than 422ing before dbt runs. `IngestedManifest`/`IngestedNode` take `#[serde(default)]`: they were serialize-only, and a field the serializer skips made the whole manifest unparseable on the receiving side. A completed run page fetched the graph forever — `load()` assigns `raw`, which recomputes `settled`, which re-entered the same effect. The final fetch is keyed to the job by a plain (non-reactive) variable, and `settled` is read untracked. The pin now names a revision that compiles: the previous one still called `authed.tags()`, a method that does not exist, because both that fix and the JSON response landed after it was committed. * fix(dbt): keep a run snapshot out of the script's deployed ownership Everything `persist_ingest` writes after the manifest is keyed by PATH — one row set per script, describing what is deployed there. A run snapshot was still reaching it, so a one-off `vars` override republished that invocation's relations as the script's ownership and the workspace graph stayed on the override's schemas and aliases: an ordinary run of a static descriptor never ingests again to correct it, so only a redeploy would. A snapshot now stops after recording its own rows. The row preview also selected a bare model name, which dbt resolves across every installed package while `show` takes a single node — a project model sharing its name with a package's was previewed wrongly or refused. It selects the package-qualified FQN. * fix(dbt): forget stale retry state when preparation itself fails `prepare_project` runs before every path that could clear it, and it fails for reasons unrelated to the saved run — a profile that stopped resolving, a provision cancelled, packages that will not install. The invocation still left nothing resumable, so the previous one must not stay authoritative: a repaired project would otherwise let `dbt retry` rebuild an older run's selection and write relations the latest invocation never reached. A retry is exempt, since it is trying to use that state and failing to prepare says nothing about it. Also corrects the runtime doc, which still described agent workers as unable to run dynamic descriptors or Windmill-resolved profiles. They publish their graph through the API now; what they do not get is live progress and a durable retry row, and the doc says so. * fix(dbt): spell the whole FQN for preview, bound retained retry generations The FQN selector added last commit was `<package>.<name>`, but a dbt FQN is the resource's path within its package and the matcher must consume the selector and end on equal lengths — so it matched nothing for a model under `models/marts/`, which is the layout most projects use and the one this repo's own complex fixture has. The middle segments come from `original_file_path`, whose first element is the resource root the FQN excludes. Without a path it falls back to the bare name: ambiguous across packages, but a selector dbt resolves rather than rejects. Tested on a nested model, which is the input that separates the three spellings. Superseded retry generations were removed only when a later run published one, and never inside the hour-long grace period — so a burst left a manifest and a results copy per run with nothing afterwards to collect them. At most four now sit in the grace window, oldest evicted first. * fix(dbt): scope preview state to the run, seed the project on a language switch Previews are keyed by `unique_id`, which is the same string for the same model in every run, and the run-change effect reset only the graph and progress. Opening a second run of one project therefore showed the previous run's rows immediately, and `runPreview` treated them as cached and refused to fetch. A generation counter also drops a preview that resolves after navigation, which the reset alone cannot catch. The dbt project was seeded only by the empty-script bootstrap, but dbt is in the ordinary language picker: reaching it by switching a draft produced a script with no `dbt_project.yml`, which the runtime refuses to deploy or run. Both entry points seed now, and neither touches modules that already exist. * chore(dbt): cache the agent graph endpoint's query for the EE offline build * fix(dbt): publish the graph a moved profile built A run snapshot stopped before everything `persist_ingest` keys by PATH, which is right for a one-off `vars` override and wrong for the other two reasons a run re-ingests. `graph_is_per_run` was one bool for all of them, and the profile drift check both sets it and reads back what the publisher recorded: a profile moved A->B was detected by every run forever, each paying a `dbt parse` for a snapshot nobody reads while the asset rows went on naming schema A. The reason is carried now (`GraphRefresh`), and it decides both writes. Drift is the version's own move, so it rewrites the VERSION's graph and republishes the ownership that ends the drift; a dynamic descriptor snapshots under its job id and still publishes; anything the CALLER scoped — an overridden `vars`, a narrowed `select` — snapshots and publishes nothing, so one invocation's subset can neither stand as what the script owns nor drop the models it left out from the version's graph. Where they meet the caller wins, and the next ordinary run settles the drift. A restore also rebuilt `run_results.json` by copying the generation directory a second time, so a burst of saves pruning it mid-restore left `dbt retry` with nothing to resume and a job that reported success. It is written from the bytes the restore already read; a manifest that went the same way falls back to the parse a database restore pays anyway, and a generation that vanished before either read falls back to the database's row for that same run instead of reporting there is nothing to retry. * fix(dbt): select a row preview by package, not by file path The preview built dbt's FQN by dropping one segment of `original_file_path`, which assumes the model root is `models/`. A project setting `model-paths: ["src/models"]` turned `src/models/marts/orders.sql` into `pkg.models.marts.orders`, and dbt's matcher — equal lengths, compared from the front — resolves that to nothing: the preview came back empty for every model in the project. It selects `<name>,package:<pkg>` instead. The comma is dbt's intersection operator, so this names the node by its own name and the package it belongs to, which is what the FQN was reaching for and needs no knowledge of the resource root. Verified on dbt-core 1.12, dbt-core 2.0.0-alpha.5 and fusion 2.0.0-preview.202, including a package shipping a model whose name the root project also uses. * fix(dbt): refuse a lockfile version that is not one, keep a named selector Two things a preview reaches that a deploy does not vouch for. A raw preview submits its own `lock`, so `engine_version` arrives from the caller and was interpolated straight into the engine cache path — `../..` in it made the download, extraction and rename land anywhere the worker can write, and provisioning runs on the host rather than inside the dbt jail. Both it and `adapter_version` (a pip requirement) are now accepted only as a plain version token. `effective_selector` also read any submitted `select`/`exclude` as an override of the descriptor's named selector. The generated run form posts a default back for every field the caller left untouched, and a selector descriptor's `select` default is `[]` — so pressing Test, saving a schedule or firing a webhook built the WHOLE project instead of `--selector nightly`. An override is now one that DIFFERS from the descriptor's own value; a run that wants the whole project despite the selector asks with `["*"]`. * fix(dbt): let a moved profile settle, from the runs that actually happen Two ways the drift check could never come to rest, both verified against a real run of a real project on a normal worker. `add_caller_args` read any submitted `select`/`exclude` as a caller's narrowing. The generated run form posts a default back for every field left untouched, so every run from the UI, a schedule, a webhook or a flow step carried them and was marked caller-scoped: with the profile moved A->B, each one stored its models under its own job id and left the workspace graph — and the root the check reads back — at A. Since no UI run omits the field, the "an ordinary run settles it" escape hatch was unreachable. Both this and `effective_selector` now ask one question, `selection_is_overridden`: DIFFERENT from the descriptor's, not merely submitted. The root was also recorded beside the path-keyed publication rather than beside the graph it describes, so a version that cannot claim the path — an older one run by hash, a deploy overtaken by a newer one — rewrote its graph at the moved root and recorded nothing. Its next run then compared against a root that was absent or two moves stale and skipped the refresh its own run page needed. It is written wherever the deployed row's graph is. Verified end to end: same UI-shaped arguments before and after, the moved profile now republishes (asset rows and version graph both move to the new schema), a second run detects nothing and re-parses nothing, and moving the profile back settles it again. `prune_dbt_run_graphs` also ran from runs alone, while a deploy writes a whole node set of its own, `raw_code` per model included: a project redeployed on every push by CI and run nightly kept one full graph per push until the next run, and one deployed but never run kept them for good. * fix(dbt): drop a self-dependent effect in the run graph `previewGen` was `$state` written by the effect that also reads it, three lines under a `finalLoadFor` that is a plain `let` for exactly that reason. Nothing reactive reads it — the only reads are inside `runPreview`, a plain async function — so it becomes a plain `let` too. * fix(dbt): pin a retry to the engine versions it resolved `run_identity` carried the engine KIND but not the version it resolved, nor the dbt-core 1.x adapter's. Redeploy an unchanged project after a release and it locks a newer dbt or adapter while the saved `run_results.json` still passes the check, so `dbt retry` feeds one version's artifacts to another — the exact reproducibility the lockfile exists to hold. Both resolved versions are in the identity now; a real failure and retry still resumes. Also drops three comments that outlived what they describe: two said `[]` clears a descriptor's selector, which `selection_is_overridden` reversed, and one pointed at an agent-worker guard that no longer exists — the agent path reaches the ingest deliberately and publishes through the API. * fix(dbt): discard a run graph the page has already navigated away from The component is reused across runs, so a slow `/jobs/dbt_graph` or progress response could land after the reset and put the previous run's models, statuses and failure state on the current run's page, where nothing would fetch again to correct it. Every response is now checked against the generation it was requested under — the counter the preview path already used, renamed for what it means. The graph poll also backs off. Neither of its stops is reachable for a whole class of runs — `dbt_snapshot_job` never matches a static descriptor, and `polled` stays empty for the engines that emit no node events — so an ordinary run walked to the cap, re-sending every model's SQL 40 times in two minutes. * fix(dbt): forget the previous run when the durable save fails, keep quoting `save_run_state` returns the database error when its upsert fails, which leaves run N-1's row and local generation in place: same project, same arguments, so a `dbt retry` matches them and resumes an older attempt's failed nodes against this checkout — the outcome the no-results branch twelve lines above calls `invalidate_run_state` to prevent, reached by another door. It now goes through the same call. Best effort, since the delete goes to the database that just refused a write, but the local pointer is what a retry landing back here reads. The run page also rejoined a relation's parts after `splitRelation` stripped their quotes, so the one name the button exists to paste — `"wh"."analytics.v2"."Order Items"` — was copied as something no client resolves. It copies `relation_name` verbatim. And a source on a finished run was called another project's: the check that guards against two projects claiming one relation asks whether this run executed the node, and a run executes no sources — they appear in no `run_results.json`. Nothing materializes a source, so that warning could never be true of one. Docs: the `vars`-override paragraph still said such a run does not refresh the graph, which the table above it contradicts — it refreshes under its job id and publishes nothing. * fix(dbt): keep the asset rows and the version's models describing one graph The workspace graph takes an asset's relations from the path-keyed `asset` rows and its models, SQL, tests and lineage from the version's `dbt_node`/`dbt_edge`. A dynamic descriptor published the former while storing the latter under its own job id, so a placeholder that moved an alias or a schema left the current graph with assets no model stands behind — nothing dbt contributes to them survives. Ownership is published exactly when the VERSION's graph was written now, which is the only state in which the two agree. Two cases are settled elsewhere by design: an override's relations are a one-off, and a dynamic descriptor at a moved profile keeps the deploy's ownership until a redeploy — its runs each show their own models and it re-parses regardless, so the undetected drift costs it nothing it was not already paying. An agent worker has no durable row, so its local `current` pointer is the whole of what a retry reads — and every local publication failure returned success with the PREVIOUS run's pointer still in place. Where a row exists that is harmless (`restore` takes a local generation only when the row names it), so the abandonment is scoped to the agent case. A failed `dbt show` also cleared the retry state: the preparation-failure exempts `retry` but not a read-only command, and the run page's row preview is exactly that, run as the principal the state is keyed by — so a preview that could not provision took the retry away from the run being looked at. Frontend: the run-change reset left `loading` and `failed` behind, so the gap before the next run's answer rendered "no models in the asset graph" — a claim about the descriptor — over a project that is fine. And three derivations argued from "the graph is the current deploy", which the pinned endpoint made untrue; each is still needed, for the version-graph rewrite and retention reasons now written down. * fix(cli): let --skip-scripts cover a script's module files The module shortcut in `elementsToMap` maps the file and `continue`s before every skip filter, and a module is deployed as part of its parent script — so `wmill sync push --skip-scripts` still pushed the script whenever one of its modules changed, and pull still overwrote them locally. Harmless while a module was a rare helper file; every file of a dbt project is one of these now. * docs(dbt): a dynamic descriptor's ownership stays the deploy's * fix(dbt): read the run out of a failure whose message has braces of its own `parseDbtRun` anchored on the FIRST `{` in the error message and parsed everything after it. The worker appends the structured result after the error text, and dbt's errors carry braces — a Jinja template, the compiled SQL, an adapter's own JSON — so the failures most worth reading were the ones whose summary and per-node outcomes the run page dropped. Every brace is tried now, bounded, and the first that parses as a run wins. Pins the EE revision that gives the agent publish endpoint the deleted-version guard the SQL path takes: deletion is soft, the foreign key still accepts graph rows, and the pinned graph query serves non-live versions, so an agent finishing during a delete put a deleted project's model SQL back on screen. The query is byte-identical to `persist_ingest`'s, so the offline cache already covers it — verified with a full-EE `SQLX_OFFLINE=true` check. * fix(dbt): seed a project when a modular draft switches to dbt `seedDbtProject` returned whenever the draft carried any module at all, so a modular script holding a `helper.ts` reached dbt with none of what dbt needs: the project view is read-only, and the worker refuses a bundle without `dbt_project.yml`, so that draft could neither run nor deploy. Keyed on the project file now, and the seed goes in under whatever is already there — the previous language's helpers are inert to dbt and the user's to remove. Also records this runtime's schema in `backend/summarized_schema.txt`: the `table` asset kind, the `dbt` script language, the five dbt tables and the `materialization_status` enum the progress table uses. * docs(dbt): move the pipeline-membership rationale out of the deploy path * fix(dbt): gate a pinned run's model SQL on the version it belongs to The `EXISTS` against `script` is the only thing standing between a share-link viewer and the project's source, and it matched the workspace and path alone. `extra_perms` is a grant on a ROW: archive a version that granted someone access, recreate the path with narrower permissions, and that stale grant satisfied the probe while the query returned the NEW version's `raw_code`. Both probes name the hash now. The regression test drives exactly that shape and fails without it, returning `select 2` to a caller granted only on the archived version. * fix(dbt): keep a delimiter an identifier escaped by doubling Every dialect these relations come from escapes its own delimiter by doubling it, and both split functions closed the quoted section on the first half and reopened on the second: `"schema"."a""b"` came out as `a.b`. The manifest keeps the real spelling, so the run wrote its per-model status and row counts under an asset path no graph node has — the node simply never moves, which is the failure mode this splitter exists to prevent. Fixed in the worker and in its frontend mirror, which have to agree, with a case per delimiter on both sides. * fix(dbt): key retry state by the caller, not only by the principal it runs as An `on_behalf_of` script executes every caller's job as its owner, so `permissioned_as` names one principal for all of them and the retry state — the durable row and the worker-local generation both — collapsed onto a single entry. After one caller's run failed, the next could submit `dbt_command: retry` and resume it: their arguments replayed against the warehouse, and handed back through `invocation_args`. Nothing else separated them, and on an agent worker the local directory is the whole boundary. `created_by` joins the key in both places. For an ordinary script it changes nothing — `permissioned_as` is already that caller — and a run that was itself superseded was never resumable anyway. Includes the offline cache for the four changed queries and the three the pinned-graph regression test added last commit, which had none: `prepare` without `--all-targets` does not compile test targets, so CI's `SQLX_OFFLINE=true ... --all-targets` would have failed on them. * fix(dbt): compare the schema too when reporting a relation that moved `relationDrift` compared the leaf name alone, and the move it exists to report — a profile repointed at another schema, which a later run then writes into the version's graph — leaves every model's name exactly where it was. So the one case that reliably produces a graph naming relations this run did not write was the one case the notice stayed silent for. The schema segment joins the comparison, qualified against qualified: an unqualified one means the target's own database, which the relation names anyway, so comparing that would report a move on every node. * fix(dbt): bound the retry state now that it is keyed per caller Keying by `created_by` fixed one caller resuming another's run and created a growth problem doing it: a shared `on_behalf_of` script kept one row and one worker directory for everyone who had ever run it, and the generation prune only bounds files INSIDE a directory. Three bounds, none of them new machinery. A run with nothing failed or skipped saves nothing — `dbt retry` builds from those nodes alone, so that state could only ever be refused — while still clearing what the previous run left, since its failures are no longer what last happened here. The rows expire on the same 30-day clock as a run snapshot, swept per path by the prune every dbt job already spawns. And the worker-local directories are swept there too, by the age of the pointer a save rewrites, because their digest names neither the script nor the caller. * docs(dbt): the retry state is worker-affine only on an agent worker * fix(auth): only the server may set a token label that names a user `create_token_internal` wrote `NewToken.label` verbatim, and the auth layer reads some labels as an IDENTITY: `username_override_from_label` maps `ephemeral-script-end-user-<name>` to exactly `<name>`, which then becomes `created_by` on every job that token pushes. The label is free-form request input, so any member could mint a token that speaks as somebody else — the shape `require_job_read_access` already works around when it refuses to trust `username_override` and falls back to an RLS probe, and the one that made dbt's retry-state key (`created_by`) forgeable for an `on_behalf_of` script. The labels are refused where request input enters: the member-facing `tokens/create`, and `impersonate`, which names its subject in `impersonate_email` and has no business renaming the caller too. The legitimate producers are unaffected — a job's own token comes from `create_token_for_owner` in the worker, and native triggers and app-embed tokens build their labels themselves rather than accepting one. `Ephemeral lsp token` stays allowed: its override is the fixed sentinel `lsp`, not a name the caller chose, and the editor mints exactly that label through this endpoint for its language server. The test pins the two lists together, so an arm added to `username_override_from_label` that lets a label choose a name fails until it is reserved too. * Revert "fix(auth): only the server may set a token label that names a user" This reverts commit efaa498d82. * fix(auth): only the server may set a token label that becomes a bare username * revert(dbt): key retry state by the execution principal again Reverts the per-caller key and the retention it needed. `created_by` cannot carry an isolation boundary: it is `display_username()`, which a token LABEL supplies, so two callers can share one value and — before the guard two commits back — one could name a third person. GHSA-8x8x-88qc-qp4r settled that class by refusing to trust the name for authorization, and keying on it here was the same mistake in a new place. It also cost a migration, a PK column, two sqlx cycles and a retention sweep to defend. Back to `(workspace, script_path, permissioned_as)`, which is derived from the authenticated username and cannot be chosen by a request. What that boundary is, and the one case it does not cover, is now written down where the retry is specified rather than left to be re-derived: anyone entitled to run the script as that principal may resume its last failure — the same capability as re-running that job, since running it requires the read access that already shows them the run and its arguments — except for a run pushed `invisible_to_owner`, whose arguments a retry still returns. Closing that means having a retry NAME the job it resumes and authorizing it as a job read, which is a change to the run argument, not to the key. * fix(dbt): give a caller's own selection a graph, and load a finished run once A run that overrides `select`/`exclude` was marked caller-scoped but not per-run, so it ingested nothing and its page fell back to the deployed graph. That is only right when the override NARROWS the descriptor's selection — `["*"]`, or any model outside it, builds relations the deployed graph never had, and those are exactly the ones whose progress, SQL and lineage had nothing to draw. It ingests its own graph now, still caller-scoped, so the subset stays out of what the script owns. `relationDrift` also read a schema whose own name contains a period as a move: a segment holding one is ambiguous — an overridden database, or a schema really called `a.b` — so either spelling now counts as unmoved. And a finished run fetched the whole graph twice on mount, every model's SQL included: the reset effect and the finished-run effect both fired in the same tick. The reset only loads while the run is in flight; the other owns the finished case, because a snapshot can land after the run ends. * fix(dbt): classify a restored retry by its resolved arguments `add_caller_args` was handed `raw_args` on the retry path — the arguments as SUBMITTED, where a `select` spelled `$res:` is still a string. `arg_list` then refuses it as "must be a list of strings" and the retry dies before parsing, for a reference that resolves to the very list the failed run built with. The resolved map decides it now, matching the selection resolver and the build; `raw_args` stays what is persisted and published, so no resolved secret outlives the job. Verified against the shape it breaks on: a `select` from a resource, failed, then resumed. Two docs that outlived their code: `dbt_script_hash` is described as a fallback for a job naming no deployed script rather than as the pinning mechanism, since a script job's version comes from the job row and the query value is ignored (spec plus the generated client, which carries the same sentence); and `GraphRefresh`'s fields no longer claim a caller's selection only ever narrows, which is the premise the previous commit disproved. * fix(dbt): a hidden run keeps no retry state The state is keyed by the execution principal, which every caller of an `on_behalf_of` script shares, and a retry publishes the arguments it restored — so for a run the other callers cannot read, that retry was the one way to see them. The equivalence the key rests on ("resuming it is the same capability as re-running it") holds only while the run IS readable, and exactly one run is not: one pushed `invisible_to_owner`. Those now save nothing, and clear whatever the previous run left, since this invocation happened. A hidden run therefore cannot be resumed by anyone, its author included — the cheaper half of the trade, and the reason having a retry NAME its source job is the design that would give it back. Verified: a visible failure still saves and resumes; a hidden one leaves no row, and the retry after it refuses without returning the hidden `vars`. Also collapses `GraphRefresh::caller_scoped`, which stopped distinguishing anything once a selection override became per-run: both writers set both flags, so `snapshot_job` and `publishes_ownership` are now one field and its negation — which is what lets the agent payload carry a single `per_run` bit, recorded there. `is_reserved_token_label` says why `ephemeral-` is reserved (a client-chosen system label is a token its owner cannot list or revoke) rather than restating a username rule its second prefix does not follow. And the module-cache import comment states the invariant once, without the shape of the cache that preceded it. * fix(dbt): a pre-build failure leaves the saved run alone Three exits cleared the retry state before `dbt build` ever ran: a preparation failure, a failed `dbt parse`, a failed ingest. None of them touches a relation, so the warehouse is exactly what the previous run left and its failures are still the accurate description of it — clearing there just costs a resumable failure, and a resume that no longer fits is refused by `run_identity` and the arguments digest regardless. The reachable one is a cancellation during provisioning. Only an interrupted BUILD invalidates, which the save at the end of the job already decides. Verified: a failure saves state, a run whose selection matches no node fails in the ingest and leaves it, and the retry after that still resumes the original failure. Agent-published graphs also had no sweep — `Connection::Http` snapshots every run and spawns none — so the EE endpoint prunes too (pinned at f58bf06), and `prune_dbt_run_graphs` now names every writer rather than only the runs. Drops `ephemeral-webhook-` from the reserved token labels: it yields the label verbatim, not a bare username, so the check now matches the rule its doc and its test state, and the reason given for it was wrong about what `is_user_token` costs an owner. And the result contract lists `invocation_args` — the field most needing it, being another invocation's arguments on a retry. * fix(assets): decode a doubled delimiter when canonicalizing a table key `split_relation` decodes it on the worker side, so the canonicalizer had to as well: a relation whose identifier escapes its own delimiter — `"sales""east"` — was filed by the ingest under `sales"east` and by a hand-written `table://` annotation under `sales""east`. Two nodes for one table, no edge, which is the exact split this key exists to prevent (decision 11). Fixed for both the name and each half of a database-qualified schema segment, with a case per dialect. `invalidate_run_state`'s contract also still promised the pre-build behaviour the previous commit removed; it now names the cases that do invalidate — an interrupted build, and a run hidden from the script's owners. * fix(dbt): reclaim package trees no project asks for any more The cache key covers the whole project digest — a `local:` dependency's content is in no manifest, so nothing narrower is safe — which means every edit of a project that declares packages publishes another full dependency tree, and nothing ever removed one. A worker that lives through a hundred deploys held a hundred trees until an operator cleared the entire cache by hand, and the disk that fills fails every job on that worker, not only dbt's. Swept by last USE, not by publication: a hit dates the tree, so a project unchanged for months is not evicted from under the jobs still running it. The marker is a sibling rather than a file inside the tree, which the restore copies into the project. Staging directories go after a day — one belongs to a single job and is removed when it ends, so an older one is from a worker that died mid-publish. Verified by planting a 30-day-old tree and a stale staging dir: one run reclaimed both and left a fresh tree alone. Narrowing the key to the declared `local:` paths would fix the churn as well, and is deliberately not done here: getting it wrong runs a project against another revision's packages, which is worse than the disk it saves. * docs(api): declare the dbt half of the asset-graph response `AssetGraph` is the response of both `/assets/graph` and the new `/jobs/dbt_graph/{id}`, and the spec described neither `assets[].dbt`, `runnables[].dbt`, `dbt_edges` nor `dbt_snapshot_job` — so every client generated from it saw a graph with no dbt metadata, no `ref()` lineage and no snapshot marker, which is why the run page reaches the endpoint with a raw `fetch` and a hand-written cast. `frontend/src/lib/gen` is gitignored, so the spec is the only tracked description of this surface. Declared, and the client regenerated from it now carries all four (verified: the regen is otherwise a byte-for-byte no-op). * fix(dbt): stop the package sweep from racing the restore it protects Three faults in the sweep the previous commit added. The `.last_used` marker was written AFTER the copy, and the tree it matters for is the one at the retention edge — where the first use in a fortnight is also when the sweep fires. So the source looked stale for the length of the restore and a concurrent job's sweep could remove it mid-copy, failing a run that should merely have refetched. Marked before the copy, and a restore that loses the race anyway is treated as a cache MISS: `dbt deps` resolves the tree again, costing a fetch rather than the run. The sweep also sat inside `if let Connection::Sql`, though it is a walk of the worker's own disk that needs no database — so an agent worker, which has none, published a tree per project edit and reclaimed nothing. That is the state the doc claimed was fixed, and the same argument the graph sweep makes two lines above: retention that depends on the connection is not retention. And the canonicalizer's two halves disagreed after `5890436`. `unquote_identifier` requires the quote at both ends, so a lone `"` in an already-decoded name survives; the schema walk treated one as opening a quote and dropped it, filing the ingest's `sa"les` under `sales` while an annotation's `"sa""les"` decoded to `sa"les` — the split this canonicalization exists to prevent, in the function I had just touched. The halves share one rule now, and the test drives the decoded spelling against the quoted one rather than asserting only the annotation's. * fix(cli): track a dbt project's parent descriptor for every authored file `buildTracker` decides whose top hash `wmill-lock.yaml` refreshes, and it reached the module branch only for files matching a Windmill script extension. A dbt project is mostly files that are not: `dbt_project.yml`, `packages.yml`, schema YAML, seed CSVs. Editing any of them left the descriptor untracked and its module-inclusive hash stale, so the lock disagreed with the bundle that was pushed. Module paths are now handled ahead of that gate. The parent was also derived by searching the RAW path for `__dbt/`, which finds nothing on Windows, where the folder is spelled `__dbt\` — so even a model edit was skipped there. It goes through `getScriptBasePathFromModulePath`, which normalizes separators, and which the sibling helpers already used for this exact reason. The regression test drives all three shapes. With the old gate reinstated it fails on the non-script files and on the backslash path, and passes only for `.sql`. * fix(dbt): re-resolve unlocked dependencies, and reclaim retry state nobody writes A project with no checked-in `package-lock.yml` asked dbt to RESOLVE its ranges and mutable git revisions, and dbt does that on every run. A cache hit skipped `dbt deps` entirely and every hit refreshed the marker, so the first resolution was pinned for as long as the project stayed in use — a version range that moved was never picked up. An unlocked tree is now a miss once it is a day old; a locked project keeps its tree, since the lock is in the key and pinning is what it asked for. Retry state: `invalidate_run_state` removed the pointer and left the generations, and the whole state root was swept by nothing after `a236972` — so a script that stops running, or a principal who stops running it, kept its last generations (a `manifest.json` each) for good. Forgetting the state now prunes its generations, and a 30-day sweep of the state root runs beside the package sweep, for the reason that one already gives: retention that runs only for the script being run is not retention for the ones that are not. This restores one half of what `a236972` reverted — the local sweep, not the per-caller key or the row retention — because it never depended on that key. The graph orphan sweep also committed its marker delete separately from the two deletes it gates, so an error or a restart in the gap left graph rows whose marker was gone — and since the sweep only runs when a marker went, every later call skipped it and those rows were unreachable for good. One transaction now. `parseDbtRun` scans from the LAST brace: the payload is appended, so backwards finds it immediately where forwards parsed the whole message once per brace in the error text. And `reject_reserved_label` moved into `create_token_internal`, the one path every caller-supplied label reaches, so the invariant sits where a new route would break it. * fix: undo two of my own regressions, and agree with the backend everywhere `2e34dd48` hoisted the module check above the extension gate, which is right, but did not bring the entry-point rule with it: a folder-layout script's METADATA is `<base>__mod/script.yaml`, an entry-point path, and pushing it as a content file makes the metadata pass ask for the language of `.yaml` and abort the whole command. Reached by editing the summary of ANY modular script — pre-existing, nothing to do with dbt. Metadata resolves to its content file now, with a test. `d3716a09`'s unlocked-dependency refresh is reverted. It measured staleness against the last-USE marker, which every hit rewrites, so an actively used project never reached the threshold — and when it did fire, the re-publish `rename` cannot land on a populated directory, so the freshly resolved tree was discarded and the stale one re-stamped: a network resolve thrown away per idle day. Doing it properly needs a publication timestamp and a publish that can replace a tree; the limitation is recorded where the cache is keyed instead of half-implemented. `parseDbtRun` no longer counts braces in either direction. The payload is appended pretty-printed, so its `{` is the only one at column zero — forwards blew the cap on an error full of braces, backwards on one `{` per node, which a few hundred nodes reaches. Test drives a 400-node failure. And `parsePipelineAnnotations` was the third copy of the quote rule, still splitting a doubled delimiter: the live canvas keyed `"sales""east"` as `saleseast` where the deploy keys `sales"east`, so an annotation pointed at a node the deployed graph does not have. All three agree now, including on a lone delimiter in an already-decoded name. * chore(dbt): leave cache retention and the migration's comment out of this PR Two deliberate subtractions, so what ships is the set that has been verified rather than the set that was written. The package-tree and state-directory sweeps are gone (recoverable on `dbt-cache-retention-followup`). They delete directories on a worker, they were the newest code here, and one of them already needed a second pass for racing an active restore — while what they buy is disk hygiene, not correctness: today an operator's `cache_clear` reclaims these caches, exactly as it does for every other language. Both limitations they addressed are now recorded where the cache is keyed. `invalidate_run_state` still prunes the generations it orphans, since that reuses the existing pruner and its grace window rather than sweeping a root by age. And the migration is back to its pre-PR bytes. The only change left in it was a corrected comment, which every database that already applied this migration would have paid for with a checksum mismatch at startup — the explanation lives in `prepare_project` and in the design doc, which is where it is read. * fix(dbt): pin the resolved package tree to the deployed version A cache hit skipped `dbt deps`, so an unlocked range or mutable git revision stayed on whatever the first worker resolved, and a retry could feed one resolution's run_results.json to another. The deploy now records the digest of the generated package-lock.yml, the package cache is keyed by it, and it joins the run identity; a worker that resolves anything else is refused rather than run. Also carries the agent-wire round trip test for an ingested manifest, and the EE pin for the agent publish endpoint. * docs(dbt): state the dependency contract the deploy pins The design doc still described the package cache as keyed by the project digest alone, and carried an open question about refreshing unlocked dependencies that the deploy-time pin answers: resolution happens once, at deploy, and every run of that version installs it or is refused. Records what that costs and buys — committing package-lock.yml makes deploys cache-hit, deploying again is what picks up a newer range — and that these worker-local caches are reclaimed by cache_clear, as every other language's are. * fix(dbt): four findings from the review round - An interrupted retry republished the results it had only restored. A retry starts with the previous attempt's run_results.json in place; cancelled before dbt rewrites it, the save dated those failures to this job, so the next retry rebuilt nodes this one had already redone. Verified by A/B: without the check the row survives, carrying the cancelled retry's id. - The deploy refused a committed package-lock.yml that dbt itself updates, which it does whenever the sha1_hash it recorded for packages.yml no longer matches. Only a run has a resolution to be held to; the deploy establishes one. - A second graph load for the SAME run could land out of order, leaving a finished run showing the deployed fallback for good. - Directory nodes in the project tree carried no path, so two of one name at the same depth shared a collapse key and folded together. * fix(dbt): name every adapter, and stop guessing which project owns a model The invalid-adapter message listed four of the eleven accepted spellings. And a relation may have several script producers, none of which the provenance record identifies — prefixing the first one names a __dbt folder that does not exist, so an ambiguous relation now shows the path inside the project alone. * refactor(dbt): name the graph-snapshot column for when it is written `published_relation_root` described a design persist_ingest does not use: the root is recorded by every ingest that is not a run's own snapshot, including one that publishes no ownership — deliberately, since a version that cannot claim the path would otherwise record nothing and compare against a stale root forever. The column comment said the opposite and the name followed it. Safe to edit the migration in place because this PR introduces it: no database outside a checkout of this branch has ever applied it. One that has needs UPDATE _sqlx_migrations SET checksum = decode('<sha384 of the file>','hex') WHERE version = 20260725084314; alongside the column rename, or a fresh database. * style(dbt): bring comment blocks within the four-line guidance AGENTS.md asks for each invariant at the place someone would break it, in at most four lines. The new dbt files carried 60 inline blocks past that, several of them three rationales deep at one site. Nothing durable is dropped: what covered several constraints at once is split to the lines it constrains, and eight comments that had drifted above the wrong test — six stacked over one profile test, two over the wrong executor test — are reattached to the tests they describe. * docs(dbt): carry the snapshot-column rename into the summarized schema The rename reached the migration, both queries, the sqlx cache and the design doc, but not the compact schema summary — which is the file agents read instead of the migrations, so it was the one copy that could mislead silently. * docs(dbt): bound the retry-state and package-pin residuals precisely Two claims in the design doc were true but not precise enough to act on. Retry state: the equivalence between resuming a failure and re-running the job holds because folder read grants job read, which is also what grants execution. It does NOT hold for a u/<owner> script shared through extra_perms, where no folder policy applies and a grantee cannot read even their own on-behalf run. The package pin: the refusal is per worker. A ranged dependency with no committed lock reproduces its resolution only from a cache hit, so it keeps running where the tree is warm and fails on the first cold worker. * fix(dbt): refuse a retry of a run its caller cannot read Retry state is keyed by execution principal, so every caller of an on_behalf_of script shares one saved run. Under a folder that matches job visibility exactly, but a u/<owner> script shared through extra_perms has no folder policy: a grantee can read neither the run nor their own, while a retry published its arguments. The restore now applies require_job_read_access's rule — you can always read a job you launched, otherwise the row must be visible under your own RLS. Deliberately NOT job_perms, which carries the identity the job runs AS: for an on_behalf_of script that is the owner, and probing with it authorizes everyone. Verified with two members on one shared principal: bob is refused alice's run on the u/ path and her marker never reaches his result, alice resumes her own, and bob still resumes alice's run of the same script under a folder. * fix(dbt): close the retry check's own gaps The check landed with three holes and four rough edges, all found by review: - A pruned source run was allowed. dbt_run_state outlives job retention, so that handed the last failure's arguments to whoever asked next. It now fails closed, and says so rather than claiming the run was someone else's. - An agent worker skipped the check entirely: it reaches no database. The saved generation now records who launched it, and an agent authorizes the half it can prove — the launcher may resume. A generation written before this field is not resumable there. - The row was authorized on one read and restored on another, so a run completing in between was restored unauthorized. restore_from_db now takes the job it was authorized against. Also: restore_run_state's doc comment had drifted onto the new helper, the not-a-member comment described the saved run rather than the caller, and the refusal message carried a run of spaces from its line continuation. * fix(dbt): keep the descriptor in --json sync, and fail closed on a stateless row A dbt script's content is `<name>.dbt.yaml`, and `elementsToMap` drops every .yaml when metadata is JSON — so a --json workspace tracked the metadata, the lock and the whole project bundle but not the descriptor: a descriptor-only edit pushed nothing and a fresh pull wrote a script with no source. Pinned by a test that fails without the exemption. And the retry check read job_id flattened, which merged "no row" with "a row naming no job" and skipped authorization for the second. Unflattened, the row that cannot be authorized is refused. Also graph_digest: serde rather than {:?} for edges, and delimited parts. * docs(dbt): state that retry authorization is by identity, not token scope The section claimed the read-access equivalence was enforced. It is, for identity — but the worker never sees the submitting token, and neither v2_job nor job_perms records a scope, so a token denied jobs:read can still resume its own principal's last failure. What it gets is the arguments as submitted, so a reference is re-resolved under whoever retries rather than disclosed. Also names the cases the check refuses outright, which were spread across three commit messages and nowhere a reader would look. * chore(dbt): repin EE after rebasing the agent-graph branch onto EE main * fix(dbt): say why a test-only run has no models, instead of blaming the profile A selection of tests alone builds nothing, and a test is an assertion rather than a relation, so the ingest keeps nodes with no asset to hang on and the graph comes back empty. The run is correct and its results render — but the empty state named the one cause it is not, a project with no warehouse identity, and sent the reader to their profile. The graph itself is unchanged: retaining a selected test's attached model would have a test-only script claim a read on every model it asserts against, which is an asset-graph semantics change and wants its own review. * fix(dbt): authorize a retry by RLS alone, and eight review findings Retry authorization drops the created_by grant that mirrored require_job_read_access's "you launched it". That name is a display name derived from a token label, so a worker cannot tell a launcher from a collision — the objection Codex raised on four heads. Visibility under the caller's own RLS is the whole rule now. An on_behalf_of script under u/<owner> shared by extra_perms is resumable by the owner alone; under a folder every caller keeps the resume. Also, and each verified against the code first: - The local generation is revalidated after the file work, so a newer run publishing state mid-restore no longer lets the superseded one resume. - Both dbt zip lookups in sync normalize the OS separator, as the resource lookup beside them already did: on Windows a pulled dbt project was laid out as __mod, the one layout dbt cannot run from. - A resource storing port as a string no longer silently connects to the adapter default; a value that is not a port is refused. - An oversized but readable project file fails the deploy instead of shipping an incomplete project that compiles and fails at run time. - cache_clear does NOT reach the dbt caches: it clears cache/ while these live under cache_nomount/, as bun's do. The doc said the opposite, and that claim was the justification for dropping the retention sweeps. - A dbt unit test counts as a test for the empty-state message, as it already does for the worker's test-phase detection. - Per-node results keep the catalog segment when rows disagree on it. - The project panel copies through the app's clipboard helper, .dbt.yaml resolves back to dbt in EXTENSION_TO_LANGUAGE, and two test blocks clean up their temp directories. A comment describing a race against a sweep this PR does not ship is gone. * fix(cli): keep the oversized-file check to a bounded read The refusal statted the file and then read it whole to tell text from binary, which is the cost the two comments above it and isBundledModuleFile exist to avoid: a multi-gigabyte seed would be loaded just to be refused. Same 8 KB head read as that predicate. * revert(dbt): accept the retry residual instead of gating it Resuming grants no capability a caller lacks: they may already run the script as that principal, and a plain run builds a superset of what a retry rebuilds. The one thing a retry adds is information -- the result echoes the resumed run's submitted arguments, for the row preview -- and in almost every shape the caller could already read that run: a folder grants job read alongside execution, and an ordinary script's runs are keyed under the caller's own principal. It takes a user-path script AND extra_perms sharing AND on_behalf_of for the two to diverge. Gating it needed an identity the worker does not have. created_by is display_username(), which a token label supplies: trusting it authorizes a collision, and resolving it as a username denied every labelled token -- a CI token is `label-<name>`, no workspace member, so its own retry was refused. That regression cost more than the exposure. Kept, because none of it is about the caller: - a cancelled retry no longer republishes the results it only restored, - the restore is pinned to the row it chose rather than re-reading it, - a newer run publishing mid-restore no longer lets the superseded generation resume. The three sqlx entries the gate needed are gone with it. * fix(dbt): select unit tests, and say why the state key is the boundary `dbt ls` enumerated five resource types and omitted `unit_test`, which dbt treats as its own type rather than a flavour of `test`. A descriptor selecting `test_type:unit` or a unit test by name therefore resolved to the empty set, which the deploy refuses outright with "matched no dbt nodes". Verified both ways on a fresh path: the deploy fails without this and succeeds with it. All three engines list `unit_test` among the accepted values. The restore's state lookup now carries the reason it keys on the principal and not the caller. That decision was in the design doc, the commit messages and a PR thread — everywhere except the line someone would change to "fix" it, which is where AGENTS.md asks for it and where I twice went wrong myself. Also: the `GraphRefresh` doc claimed a dynamic descriptor publishes its graph as the script's ownership, while `per_run_models` snapshots it under the job id and `publishes_ownership()` returns false — only a moved profile republishes. And `fn n` in dbt_profiles.rs had no callers left after `port_of` replaced it. * feat(dbt): offer the resume on a failed run, where the choice is relevant "Run again" prefills the arguments the run just used, so the obvious action after a failure rebuilds the whole project while the cheap resume stays invisible unless the reader knows dbt_command has a retry value. A finished run with failed or skipped nodes now says how many, and links to its own run form with the command already switched. A link rather than a submission: the caller still presses Run, so nothing about permissions or arguments changes. Deliberately not automatic. A resume is not a re-run — models are usually rebuilt because upstream data moved, not because they failed — so a schedule that resumed after a failure would leave every model that succeeded carrying yesterday's data while reporting success. * feat(dbt): put the command first, and offer the resume where the rebuild is Three things made rebuilding the whole project the path of least resistance after a failure, when resuming its failed and skipped nodes is what dbt offers: - `dbt_command` was LAST in the generated signature, under `limit`. The schema's `order` drives the run form, so the argument that decides what the run does sat below the ones that only narrow it. It leads now, and the signature test asserts the order rather than the set, with the reason it is pinned. - "Run again" prefills the arguments of the run it came from, so for a failed run it reproduces the rebuild. It now carries `dbt_retry_hint`, and the form shows what resuming would do instead -- without choosing it, so the caller decides. - Its dropdown offers "dbt retry with same args" above the plain re-run, for a failed dbt job only: a run that succeeded has no saved failure to resume. Verified in the browser: both dropdown labels render unclipped, the alert appears only on that path and disappears once the command is switched, and the schema of a redeployed script lists dbt_command first. * feat(dbt): describe the run arguments, and hide the ones a command ignores The form offered six fields at once with no indication that most of them apply to one command each. `retry` reuses the arguments of the run it resumes, so every override is ignored; `limit` belongs to `show`; `full_refresh` to `build`. Each argument now carries what it is for, and a `showExpr` for when it applies, which SchemaForm honours by hiding the field AND dropping its value from the payload. Selecting `retry` leaves the command alone on the form. Not modelled as a `oneOf`, which is the more precise shape: Windmill renders that as a variant selector whose value is a nested object keyed by a discriminator, so the arguments would stop being flat -- and the worker reads them by name, as do schedules, webhooks, the CLI, saved inputs and the retry state's own copy of them. * fix(dbt): a retry may name the run it means to resume The retry actions I added an hour ago appear on any failed run, but the saved failure is keyed by script and principal rather than by job — so from an older run's page they resumed whatever failed most recently, under a label promising "same args". Both now pass `dbt_retry_job`, and a retry naming a run the state no longer holds is refused with the id it does hold. Verified: with failures A then B saved, a retry naming A is refused and names B in the message, while a retry naming B resumes. Also from the same review: the `select` help advertised `state:modified`, which needs a comparison manifest this runtime supplies no `--state` for, and the openflow note still described a dbt script as a descriptor naming a git repo rather than the module bundle this ships. * fix(dbt): make both retry actions actually name their run `dbt_retry_job` is not a form field, so the banner's link to the run form dropped it and the retry resumed whatever failed last — the bug the argument exists to prevent, reintroduced by the affordance meant to use it. The banner submits the run itself now, as the "Run again" dropdown item already did. And an agent worker never applied the check at all: its `latest_job` is always None, so the local generation was accepted unconditionally. Both reviewers found this independently. The decision moved into `chosen_generation`, which the agent path shares, with the refusal naming the run the worker does hold. Pinned by a test that fails without the check, on the connection kind where it was missing rather than on the one that already worked. * fix(dbt): let an unchanged push be a no-op, and stop advertising a retry the form cannot aim Two of a dbt script's fields are DERIVED by the server after the no-op check runs: the lock, which only a dependency job can produce, and the schema, which comes from the descriptor and which no client can derive (windmill-parser-wasm has no dbt arm). The check compared what arrived instead of what would be stored, so every unchanged `wmill sync push` created another version and another dependency job -- the sync churn skip_if_noop exists to prevent. Both are now compared as stored, with the parent required to hold a lock so a deploy whose dependency job failed can still be retried by pushing again. Verified by A/B on a faithful payload: without this the same unchanged push creates a version, with it the existing hash comes back and the count holds. The run form's retry hint is gone. `dbt_retry_job` is not a form field, so a retry started there resumes the last failure of the script rather than the run the message pointed at -- the two buttons name their run, that path could not. Better to not offer it than to offer it wrong. Also reunites a comment with the derivation it describes, 40 lines below where an earlier edit of mine left it. * docs(dbt): an unchanged push no longer re-resolves dependencies The section told the reader that a byte-identical deploy re-pins a ranged dependency, which was true only because no-op detection was broken for dbt. Now that an unchanged push is skipped, moving a pinned resolution takes an actual change. * feat(dbt): offer the resume only on the run it would actually reach * fix(dbt): a refusal names both the run asked for and the one held * docs(dbt): say that show previews one node when several are selected * feat(dbt): a retry names the run it resumes, and the form can fill it in * refactor(dbt): make the run's command a oneOf carrying its own overrides * fix(dbt): refuse a command block that names no command, and keep state off retired paths * fix(assets): let copilot and the assets filter see warehouse tables * fix(dbt): drop the unused db handle from the retry-principal lookup * fix(dbt): stop a preview poll when the run page moves on * fix(dbt): refuse an escaping packages-install-path, re-arm the graph poll on navigation * docs(dbt): concurrent runs of one script are the script's to serialize * fix(dbt): bound what the log tailer holds in the worker process * fix(dbt): a stale prefill answer must not aim another script's retry * fix(dbt): a rename leaves the old path archived, so state must not be rewritten there * fix(dbt): keep a retry's routing inputs, and stop an app-embed token probing resumability * docs(assets): state the authorization the graph helper expects of its callers * fix(dbt): merge the retry into the fetched arguments, not the too-big placeholder * fix(dbt): the command block's label decides the command, not map order * fix(cli): an oversized dbt project file must not read as generated output * fix(dbt): the prefill names a run only when the caller may read it * fix(cli): refuse an oversized dbt file before anything reads its body * fix(dbt): fold table paths as the deploy does, and preview with fetched args * fix(dbt): stream an engine archive to disk instead of holding it in the worker * refactor(assets): the dbt asset kind is dbt://, keyed on the relation * chore: regenerate the auto-generated prompts for the dbt asset kind * fix(assets): rename the kind at the callers that request it, and refuse an unknown one * feat(dbt): show the project's models under the run form on the script page * fix(dbt): the rename missed the dbt-edge membership keys, so the DAG lost every edge * fix(dbt): a preview must run the dbt writer, with the form's arguments * feat(dbt): the models graph takes the bottom of the script page, as a flow's does * fix(dbt): an unknown outcome is not a pass, and a preview keys on its arguments * feat(dbt): a warehouse is configured on the workspace, named like the lake * feat(dbt): asset identity keys on the warehouse name, resolved for agents too * feat(dbt): warehouses are configured in workspace settings * feat(dbt): the descriptor lives in the project and is optional * docs(dbt): the warehouse is a workspace setting and the descriptor is optional * fix(dbt): the warehouse route names its capture, and the editor stores the map * test(dbt): pin the warehouse setting round-trip and the job-scoped route * fix(dbt): a project-owned profile must authorize the warehouse it names * fix(dbt): the settings tab imports TextInput from where it lives * fix(dbt): a warehouse name is validated wherever one enters, and an absent descriptor is not a diff * fix(dbt): the settings inputs pass their placeholder the way TextInput takes it * fix(dbt): pushing a project that has no descriptor is not an error * chore(dbt): keep the module suffix private to its module * fix(dbt): warehouses survive a fork or rename, and an empty descriptor is never a file * fix(dbt): a workspace rename carries its dbt graph with it * fix(dbt): a rename takes the run state with it, not only the graph * chore(dbt): drop the query-cache entries the settings edits orphaned * docs(dbt): state the three rules that keep an absent descriptor absent * fix(dbt): an unreadable descriptor is an error, and an emptied one is a deletion * feat(dbt): the warehouse is unpermissioned, like the workspace bucket * feat(dbt): an agent worker's run reports its per-model state too * docs(dbt): how an agent worker reports what it ran * feat(dbt): a project-owned profile reports its database, so its models share nodes * style(dbt): rustfmt the files this branch touched * fix(dbt): drop the job id write_profiles no longer reads a resource with * fix(dbt): a preview reads the graph's edges and sends the model, not the cache key * fix(dbt): a deleted producer explains itself instead of a bare 404 * fix(dbt): one warehouse resolution path, interpolated against the job * fix(dbt): a project with no descriptor pulls and opens in dev mode * feat(dbt): one file tree, editable, and Test builds the model you have open * fix(dbt): a preview asks the writer's own graph, and lfs keeps its omission * fix(dbt): metadata stays beside the project, and only a model narrows a build * fix(dbt): a descriptor-less project previews, and dev reloads the whole bundle * fix(dbt): forks carry the graph, and a vanished project stops the push * fix(dbt): no-auth resolves warehouses, and a preview runs the version it read * fix(dbt): a retry recognizes its own run when the profile carries a job token * fix(dbt): the project marker is not deletable and the descriptor path is reserved * fix(dbt): a templated adapter type is refused, and a removed project archives * perf(dbt): the manifest lands in batches, and .py models are creatable * fix(dbt): only a 404 excuses a failed archive, and edges are covered * fix(dbt): a python model narrows the build, and a failed removal is not silence * fix(dbt): a settings name may be any word, and the reserved path is stated once * fix(dbt): an absent descriptor needs its project, and the form matches the run * fix(dbt): a bundle is dbt by language, and both job routes agree who may call * docs(dbt): every identity doc names the warehouse, as the code does * perf(dbt): an agent posts its run's progress once, and show cannot touch a seed * fix(dbt): bound what a warehouse may be named, and say who may ask * fix(dbt): a run's graph withholds what the project's author wrote, and dbt_project.yml is rendered before it is read * feat(dbt): withhold dbt from the language picker until it has its own editor * fix(dbt): refuse a path claimed by both a dbt project and an ordinary script * test(dbt): the bundle assertion is a set, not an order * fix(dbt): a project's generated dirs are re-read when its config changes * test(dbt): assert the synthesized descriptor without assuming a separator * fix(dbt): both push paths refuse a shared path, and every writer sweeps its progress rows * fix(dbt): preview only what dbt show can select, and a retry keeps its command block * fix(dbt): an identifier carrying a path separator gets no asset key * fix(dbt): no engine ships in an image, and show may name only one node --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-01 11:58:05 +00:00
Ruben FiszelandClaude Opus 4.8	010059a449	feat(hub): surface data pipelines in deploy-to-hub drawer (#10299) ... * feat(hub): surface data pipelines in deploy-to-hub drawer The predeploy step listed a folder's scripts with no indication that some form a data pipeline. Add a "Data pipeline" summary row (step count + "View pipeline graph" drawer rendering the asset-graph cascade) and tag pipeline-member scripts with a Pipeline badge in the item list. Fixes WIN-2238 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): drop misleading node-inspect hint from pipeline graph drawer The read-only graph doesn't wire node selection, so the "Click a node to inspect it" copy promised interaction that doesn't happen. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub): render pipeline graph inline above the list, collapsed by default Replace the "View pipeline graph" drawer with an inline collapsible panel above the deploy item list for pipeline folders. Collapsed by default so the selection list stays the first thing in view; expanding reveals the folder's asset-graph cascade in place. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): don't let the inline pipeline graph capture the drawer scroll Add an opt-in scrollZoom prop to AssetGraphCanvas (default true, preserving the full-height editor/player). The inline deploy-to-hub panel sets it false so wheel gestures over the 420px graph scroll the surrounding drawer instead of zooming the canvas and swallowing the scroll. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-24 13:29:36 +02:00
Ruben FiszelandClaude Opus 4.8	65e504146d	feat: data-pipeline recorder, interactive player, and deploy-to-hub recording (WIN-2156) (#10055) ... * feat(frontend): add data-pipeline run recorder and interactive player Adds a recorder/player for data pipelines, mirroring the existing flow and script recorders. Arm "Record" on a pipeline, run it, and the resulting cascade is captured into a downloadable JSON that the /replay player can rerun fully offline. Because a pipeline run is a cascade of independent jobs (not a single root SSE job like flows), the recording captures three things: the resolved asset graph, the per-node cascade status timeline (from the orchestrator's onUpdate), and each node's job stream (opened via getupdate_sse on launch). The player renders the graph read-only, animates the recorded node transitions in real time, and lets you click any node to inspect its recorded args, logs and result — reusing the same JobLoader replay path the flow/script players use (setActiveReplay + isReplay gating), so no network calls are made during replay. - recording/types.ts: PipelineRecording, PipelineTimelineFrame, RecordedNodeState - recording/pipelineRecording.svelte.ts: createPipelineRecording() store - recording/PipelineRecordingReplay.svelte: the player component - replay/+page.svelte: dispatch type === 'pipeline' - pipeline/[folder]/+page.svelte: Record toggle + Download recording; capture the whole-pipeline / bounded cascade run Fixes WIN-2156 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): capture DuckLake/datatable data samples in pipeline recordings Follow-up to the pipeline recorder/player: asset nodes are now inspectable offline in the player, showing what each table held after the recorded run. At record finalization, for each ducklake/datatable asset in the pipeline the recorder samples the table (up to 100 rows + columns + row count) reusing the exact live-preview query path (loadAllTablesMetaData + getRows), so a replayed sample matches what the asset-detail pane would have shown. Captures are best-effort and per-asset — a missing/unconfigured table is stored as an error marker, never thrown, so the recording still completes. The player renders the sample as a read-only typed grid when an asset node is clicked (script nodes keep their logs/result/args detail). - recording/types.ts: PipelineAssetSample + assetSamples on PipelineRecording - recording/pipelineAssetSample.ts: capturePipelineAssetSample() helper - recording/pipelineRecording.svelte.ts: recordAssetSample() + assetSamples - recording/PipelineRecordingReplay.svelte: asset-node data-sample panel - pipeline/[folder]/+page.svelte: sample each asset in finalizePipelineRecording Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * recorder * feat(hub): record data pipelines in deploy-to-hub with interactive player Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): match editor cascade timeout, warn on cycles, reset badge on re-run Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): address review — finalize race, stale replay timers, /replay redirect, bounded sampling, jobs validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): structural recording validation, guard-clear + SSE cleanup on throw paths Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate nested graph arrays and timeline frame statuses Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): scope recording to bundle membership, fail cyclic runs, validate recording elements Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub): prune recorded graph + asset samples to bundle membership Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate graph.triggers array and per-job initial_job/events shapes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): guard non-object payloads, event elements, and asset-sample/code maps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): render error boundary + validate trigger_kind and non-empty sample error Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): validate event.data and recorded-job shapes for all replay types Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): make the replay event timer crash-proof against malformed events Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): await replay completion and boundary-wrap all three players Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(recording): guard flow Play handler, cap ?src= download size, trim comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-23 18:31:40 +02:00
+414 people	30eedf9ee1	feat: Add section to deploy projects to hub (#9332) ... * feat: add Deploy to Hub workspace settings tab * Init record logic * Fix wordings * Add publish-app drawer with per-app rate limit mock - Publish drawer on raw_apps/apps exposes public URL, copy-iframe, unpublish - Inline per-app rate limit config (req/min, burst, per-IP toggle) - Rename workspace settings "Default app" tab header to "Apps" to cover both default app and public rate limiting Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Simplify publish drawer to show workspace-wide rate limit only Drop per-app rate limit fields (req/min, burst, per-IP) — none of these are supported by the backend. The drawer now shows the existing workspace-level rate limit read-only with a link to edit it in Workspace settings → Apps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Rename publish-app drawer wording to 'Share as iframe' 'Publish publicly' was ambiguous (publish to Hub vs make public URL). Use 'Share as iframe' for the button and drawer title, and 'Generate iframe' for the confirm action. Intro text now explicitly mentions iframe embedding use cases (Hub, docs page, own site). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire DeployToHub to real workspace data - Fetch apps, raw_apps, flows, scripts, resources via their services - Fetch workspace rate limit via WorkspaceService.getSettings - Share-as-iframe flips app policy.execution_mode to 'anonymous' via AppService.updateApp and resolves the real public URL via getPublicSecretOfApp + computeSecretUrl - Detect already-public apps from listApps execution_mode field - Filter out app_theme resources (noise, present in every workspace) - Hub bundle/version push and recording remain mocked (no backend yet) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Wire recordings to real jobs with run-preview UX - Recording flow now fetches the real schema, runs the job, and polls getCompletedJobResultMaybe to surface success/failure before saving. - Drawer shows a sticky status box (loader / success / failure) with a result preview, a job link, and an in-context Save CTA. - Only successful runs can be saved as a recording. Failures show the error and offer re-run. - Filter cache/state/app_theme internal resource types (mirrors workspaces_export.rs filter). - Added "What is a recording?" explainer banner above the items list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add draft/review state machine and submission gating - Phases: predeploy → draft → under_review → live, with workflow step indicator and contextual footer actions per phase - Bundle drawer collects name + readme before pushing the draft - draftItems snapshot frozen at deploy time; workspaceItems keep refreshing without affecting the draft - Folder MultiSelect lets users scope the bundle to one or more folders; empty = whole workspace - Submit-for-review disabled until every script and flow in the draft has a recording (progress bar + counter) - Recordings now run the real job and poll for success/failure; only successful runs can be saved - under_review phase locks editing, sharing, and recording - Dark mode variants on every coloured banner - Steps card shows the full 3-step process always, highlighting the current step Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Make recordings optional, encourage them for discoverability - Submit for review no longer gated on full recordings - Footer hint now frames recordings as boosting approval speed and public Hub featuring, not as a hard requirement - Progress card label switched from 'Recordings needed' to 'Recordings recommended' - Items without a recording display a yellow 'No recording' badge in every phase so the gap stays visible after submission Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Allow per-item selection inside the bundle scope - Items in predeploy now have checkboxes (all selected by default) - Select all / Deselect all act on the current folder filter - manualDeselected resets when the folder filter changes - Bundle button uses the selected count, disabled when zero - Draft snapshot keeps only the selected items - Checkboxes hidden in draft / under_review / live phases Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add diff button once approved by admins * Small fix * Nits * fix(deploy-to-hub): paginate workspace list and cancel stale record polls - loadWorkspace fetches all pages instead of capping at 100 items per kind - pollJobUntilComplete now bails when recordRunSeq advances (new record target, re-run, or drawer close), preventing late completion of a previous run from overwriting current state Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(deploy-to-hub): parallelize public-app URL resolution resolvePublicUrl now runs once per anonymous app via Promise.all instead of serially inside the items loop, removing N round-trips from initial tab load. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(indexer): tell admins when ingress routes search to wrong pod (#9274) * [ee] fix(indexer): tell admins when ingress routes search to wrong pod When the IndexReader is absent on the pod handling a search request but another pod is actively holding the indexer lock, the EE handler now returns a tailored error pointing at the ingress/load-balancer configuration instead of the generic "indexer not running" message. The indexer status endpoint reads the DB lock so it reports "running" from any pod, but search endpoints need the in-memory IndexReader that only exists on the lock holder. In multi-replica deployments this looks like the indexer is healthy but every search 404s. Companion: windmill-labs/windmill-ee-private#TBD Fixes WIN-1968. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 This commit updates the EE repository reference after PR #586 was merged in windmill-ee-private. Previous ee-repo-ref: 7dd43d1850813071cc18ba49ba090583e7321f4b New ee-repo-ref: eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * feat(cli): add `wmill init prompts` and custom override slot (#9266) * feat(cli): add `wmill init prompts` and custom override slot Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): replace init prompts with refresh prompts + AGENTS.md/AGENTS.cli.md split Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): dedupe claude skills via @-includes and add prompts freshness check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): drop migration-choice flags from `refresh prompts` Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(cli): add 'Running and previewing local changes' section to AGENTS.cli.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): write full skill content to .claude/, drop @-include wrapper Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): reconcile CLAUDE.md the same way as AGENTS.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(cli): address PR review nits — argv parsing, lazy import, comment detection, error propagation Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add yolo mode for ai chat tools (#9258) * feat: add yolo mode for ai chat tools * nit * fix: align chat footer controls * feat: add ai chat autonomy modes * feat: add autonomy mode dropdown * fix: highlight yolo autonomy icon * fix: auto accept flow edits * fix: hide unsupported autonomy modes * fix: handle auto-accept flow editor races * fix(debugger): add non-root user support to Dockerfile (#9277) Mirrors the main Windmill Dockerfile pattern: creates a windmill user (UID/GID 1000) and makes cache/work directories world-writable so the image runs cleanly under Kubernetes securityContext.runAsNonRoot or runAsUser: 1000 without permission errors on Bun, pip, or windmill cache writes. Fixes WIN-1969 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path (#9276) * fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path The AI proxy handler accepts an X-Resource-Path header to override the configured workspace AI provider. When supplied, the handler loaded the resource value from the resource table using the root DB pool with no resources:read scope check, so any authenticated workspace user could point X-Resource-Path at a restricted AI resource (e.g. one in a folder they cannot read) and the proxy would use that resource's provider credentials for the outbound AI request. For user-supplied resource paths, now require resources:read:{path} scope and fetch the resource through user_db.begin(&authed) so RLS enforces the same folder/group boundary as the resource API. The RLS- scoped $var: resolution stays in place as defense in depth. The admin-configured workspace/instance ai_config path is unchanged. Fixes WIN-1971 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai): regression test for X-Resource-Path RLS enforcement Cover all four cases: - non-admin pointing X-Resource-Path at a restricted resource is rejected - non-admin pointing it at a resource they own still works - admin can point it at any resource - workspace-configured proxy flow (no X-Resource-Path) is unchanged Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add userdraft listing primitives (#9268) * feat: add userdraft listing primitives * fix: cancel stale userdraft discard writes * docs: remove global ai userdraft plan * feat(nsjail): optional disk-backed /tmp via instance setting (#9272) * feat(nsjail): optional disk-backed /tmp via instance setting * test(nsjail): unit-test tmp mount resolver and narrow visibility * refactor(nsjail): switch tmp backing to select + conditional UI * ui(nsjail): make tmpfs the visible default in /tmp backing select * fix(nsjail): refuse preexisting jail_tmp to block symlink escape * fix(nsjail): allow jail_tmp reuse on sequential nsjail calls Codex flagged that python/ruby/rust executors invoke nsjail twice per job_dir (install then run). The previous resolver treated any preexisting jail_tmp as hostile and silently fell back to tmpfs on the second call, so disk-backed mode never reached the main script run for those langs. Use symlink_metadata().is_dir() to distinguish a real directory left by an earlier call in the same job_dir (safe to reuse) from a symlink or other entity (still refused, as the codebase-tar escape requires). Also loosen the frontend visibility predicate: only hide nsjail settings when job_isolation is explicitly 'none' or 'unshare', so deployments that enable nsjail via DISABLE_NSJAIL=false with no DB setting can still see the controls. * chore(main): release 1.706.0 (#9270) * chore(main): release 1.706.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix(nsjail): gate unix-symlink test behind cfg(unix) for Windows build (#9280) The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls std::os::unix::fs::symlink directly, which doesn't exist on Windows targets. Without a cfg gate, `cargo check --tests` fails on Windows with E0433. Other symlink call sites in this crate (php_executor, bun_executor, rust_executor, etc.) already follow this pattern. Fixes WIN-1972 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Reduce slim image vulnerability surface (#9279) * Reduce slim image vulnerability surface * chore(docker): drop apt-get upgrade -y from slim images apt-get upgrade hurts build reproducibility (same Dockerfile + same commit at different times produces divergent images) and trips hadolint DL3005. The freshness it buys is dominated by simply rebuilding against the periodically-refreshed debian:bookworm-slim base image. The --no-install-recommends and apt-list cleanup wins are kept. --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev> * fix(git-sync): bump to hub/28234 with stateless gpg.program wrapper (WIN-1974) (#9282) * fix(git-sync): revert LATEST_GIT_SYNC_SCRIPT_PATH to hub/28230 to restore GPG-signed deploys (WIN-1974) hub/28231 (PR #9230) is the "thin" script that hands the actual `git commit` to the CLI's hidden `sync git-deploy`. The hub script still does the GPG setup (import key into a fresh GNUPGHOME, dummy `gpg -bsau` to warm the agent passphrase cache, then `git config user.signingkey` + `commit.gpgsign` locally), but the commit no longer runs in the same `git_push` flow — it runs minutes later inside the CLI after workspace API resolution, zip pull, file extraction, and lockfile autofill. By the time the spawned `git commit` asks gpg-agent for the cached passphrase, the cache state is no longer reliable (or the spawned `gpg` ends up talking to a fresh agent), so signing fails non-interactively with `gpg failed to sign the data`. hub/28230 is hub/28217's in-script logic rebuilt with windmill-cli@1.703.3: the GPG setup and the in-script `sh_run("git commit ...")` happen back-to-back in `git_push`, so the cache is always fresh. It preserves wm_deploy / fork branch behavior, the EE deployment-callback `main()` signature is unchanged, and the only min-version check in EE (`is_script_meets_min_version(28103)`) is comfortably below 28230 — so this revert is safe. Forward fix (separate PR): publish a new thin script that, alongside the existing GPG setup, writes a `gpg.program` wrapper using `--pinentry-mode loopback --passphrase-file` so signing is independent of the agent's cache state. Re-bump past 28231 then. Fixes WIN-1974 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): check in source-of-truth for the next hub script (gpg.program wrapper) This is the script that will be published to hub.windmill.dev once verified on a customer GPG-signed deploy. It replaces hub/28231's agent-cache pre-warm (`gpg -bsau` with --passphrase) with a stateless gpg.program wrapper + chmod-600 passphrase file. Every git-invoked gpg call goes through the wrapper, which always uses --pinentry-mode loopback (and --passphrase-file when a passphrase exists). Signing no longer depends on gpg-agent having a cached passphrase by the time the CLI's `git commit` runs — which closes WIN-1974. Not wired in yet: LATEST_GIT_SYNC_SCRIPT_PATH stays on hub/28230 until this script is uploaded and the new hub id is known. This file is checked in so the diff is reviewable, future bumps have a source of truth, and a CLI regression test can `cat` it for fixture parity. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(frontend): skip format/pattern validation for $var/$res/$jsonvar references in ArgInput A resource field with a `pattern` constraint (e.g. the gpg_key.private_key field, whose pattern enforces a `-----BEGIN PGP PRIVATE KEY BLOCK-----` prefix) rejects values like `$var:u/me/gpg-private-key` with an "invalid format" error in the resource editor — even though `$var:`/`$res:`/`$jsonvar:` are placeholders the backend resolves at runtime, not the actual string that needs to match the regex. Bail out of all format/pattern checks (email, ipv4, ipv6, uuid, custom pattern) when the value is one of these references. Required/numeric bounds/array checks still apply since they're shape-level, not regex. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(git-sync): bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28234 (gpg.program-wrapper fix) hub/28234 is the forward fix for WIN-1974: replaces hub/28231's agent-cache pre-warm (which became stale by the time the CLI's `git commit` ran) with a stateless `gpg.program` wrapper that uses `--pinentry-mode loopback` (and `--passphrase-file` when a passphrase exists) on every gpg invocation. Bundled CLI is windmill-cli@1.705.0. Verified via reproducer at /tmp/git-sync-diff/test-gpg-fix.sh: deliberately killing gpg-agent between GPG setup and `git commit` reproduces the customer's `gpg failed to sign the data` error verbatim under the old flow, and the wrapper signs through it. Holds for passphrase-protected keys, split-subkey [C]+[S] layouts, and unprotected keys. Drops the local source-of-truth copy (`hub-scripts/`) — hub is canonical now that 28234 is published. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(git-sync): drop verbose comment above LATEST_GIT_SYNC_SCRIPT_PATH The git history (this PR) carries the why; the constant name + value carry the what. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(cli): wmill sync git-deploy stops committing; caller owns commit+push (#9284) Single contract for the deployment-callback path: the CLI does branch checkout + pull, the caller (hub script in production, test in test) does git add + commit + push. This restores the WIN-1974 invariant — GPG setup and `git commit` run back-to-back in the same process, so the agent's pre-warmed passphrase cache is still warm at sign time — without needing a `--skip-commit` flag for the hub case and a default "also-commit" for everything else. Same behavior in every call site. Changes: - sync.ts: drop the gitSyncDeployPush call from pull()'s deploy path (both the onlyCreateBranch fast-return and the post-pull commit). `gitSyncDeployPush` stays exported for any caller that wants the same commit/push semantics — just not invoked by the CLI subcommand. - gitsync_promotion.test.ts: e2e test now does its own git add + commit + push after `wmill sync git-deploy`, mirroring what the hub script does in production. Same regression coverage (wm_deploy branch created in Case A, main untouched; main updated in Case B, no new wm_deploy). CLI typecheck unchanged (two pre-existing TarAsZip errors at lines 2578/3307, present before this PR). All 743 unit tests still pass. The accompanying hub script (option-C — CLI for branch+pull, script for commit+push) lives at /tmp/git-sync-diff/sync-script-to-git-repo-windmill.option-C.ts. Once published, a follow-up bumps LATEST_GIT_SYNC_SCRIPT_PATH to its id. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * bump git sync to 28236 * fix: fork compare visibility for non-admins and stale-token superadmins (#9283) * fix: use fork-scoped authed for fork visibility in compare_workspaces * test: add EE end-to-end repro for fork rename visibility * chore: restore concurrency_locks sqlx cache lost in cleanup * test: add regression for stale-superadmin-token fork visibility bug * chore: update sqlx cache for new test queries * chore(main): release 1.706.1 (#9281) * chore(main): release 1.706.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: add wmill job rerun subcommand (#9275) * feat: add wmill job rerun subcommand * feat: add wmill job restart subcommand for flow restart-at-step * chore(system_prompts): point plugin skills sync at plugins/windmill/ (#9287) * chore(system_prompts): point plugin skills sync at plugins/windmill/ The plugin checkout's plugin folder is being renamed from `plugins/windmill-code-plugin/` to `plugins/windmill/` to shorten the slash-command namespace and align with the matching Cursor plugin layout. Paired with windmill-labs/windmill-claude-plugin#8. That PR must merge first so the next sync run finds the new folder. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(system_prompts): update plugin-dir example to plugins/windmill Co-authored-by: centdix <centdix@users.noreply.github.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> * fix(cli): wmill sync pull updates wmill-lock.yaml for raw apps (#9289) * fix: flow recording teardown crash + rename package to @windmill-labs/components (#9288) * fix: guard against null recording during FlowRecordingReplay teardown Navigating away from a flow recording inside a workspace file-tree view threw `TypeError: Cannot read properties of null (reading 'flow')` from FlowGraphViewer once during the teardown tick. Svelte 5 compiles child component props as live getters that close over `$$props.recording.flow`. When `recording` flips to null on the parent's navigation, an outer `{#if !recording?.flow}` doesn't stop those getters from firing one more time as derived effects re-evaluate before the unmount lands — so the getter dereferences null and throws. Fix at the two layers where the deref actually happens: - FlowRecordingReplay: use `recording?.flow` at the binding sites (FlowViewer + graph-snippet FlowGraphViewer) so the compiler emits an optional-chained getter, and guard the snippet branch with `{:else if recording?.flow}` so it doesn't mount when there's nothing to show. - FlowGraphViewer: finish the optional chaining the rest of the file already used everywhere else (`flow?.value?.skip_expr`, `flow?.value?.cache_ttl`, `flow?.schema`). When the upstream binding returns undefined during teardown, the graph degrades to an empty frame instead of crashing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rename package to @windmill-labs/components - frontend/package.json: rename `windmill-components` → `@windmill-labs/components` - frontend/publish.sh: drop the in-place sed rename dance; the checked-in name now matches what's published, so `npm run package && npm publish` is enough - frontend/package-lock.json, system_prompts/auto-generated/prompts.d.ts: regenerated by `npm run package` under the new name Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(flows): restore Variables and Resources in flow editor prop picker (#9290) The design system overhaul in 888837431c accidentally dropped the fallback condition that displayed the Variables and Resources sections in the prop picker by default. After that commit, these sections only appeared when the user typed `variable.` or `resource.` in their expression, which meant they effectively disappeared from the flow editor's prop picker for most users. Restore the previous behavior by showing the sections when no input match is active (the equivalent of the old `!filterActive` clause). Fixes WIN-1976 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(auth): tighten token-owner fallback for unscoped tokens (WIN-1978) (#9293) * fix(auth): reject unscoped tokens with cross-workspace forged owners (WIN-1978) An unscoped token (workspace_id IS NULL) whose `owner` field references a user, group, or unprefixed value that is not present in the target workspace must not authenticate. The previous fallback in the `u/<username>` branch granted `(is_admin=false, is_operator=true)` when no `usr` row matched in the target workspace, letting a token holder who could mutate the `token` table cross workspace boundaries with operator privileges. The `g/<groupname>` branch likewise silently accepted any group name as a "group user", and the no-prefix branch granted operator state from arbitrary owner strings. Both are now rejected unless the owner matches a real user/group membership in the target workspace. Adds an integration regression covering all three forged-owner shapes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: drop integration regression for auth fallback The test added in the previous commit relies on a sqlx::query! that requires offline-cache regeneration; removing per code-review preference to keep this PR scoped to the auth-layer fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ResourceEditor): don't reset state when `selected` reverts to undefined (#9295) The bootstrap effect tracked `selected` via its early-return check, so any time `selected` flipped back to `undefined` it would re-run and reinitialize `states[effectiveWorkspace]` to empty — wiping user input. This happens in the React SDK consumer: reactify re-syncs all Svelte props on every React render, and since `selected` isn't passed through, `$props()` reverts it. Move the `selected !== undefined` check inside the existing `untrack` so the effect only tracks `effectiveWorkspace`. Bootstrap still runs once on mount; subsequent `selected` flips no longer retrigger it. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(secret-backend): pass DB to Vault migrations + show failure details (#9292) * [ee] fix(secret-backend): pass DB to Vault migrations + surface failure details Companion to windmill-ee-private fix for WIN-1977. The HashiCorp Vault migration always failed under JWT/OIDC auth because the migration constructed VaultBackend without a DB, so every secret hit "Database connection required for JWT authentication". Creating new secrets worked because the runtime path passes the DB. Frontend: when failed_count > 0, the toast and console now show the per-secret failures (path + error, capped at 5 with "...and N more") instead of just aggregate counts. Fixes WIN-1977 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 14315067c083d3361512de621b12e41dbe3b017d This commit updates the EE repository reference after PR #587 was merged in windmill-ee-private. Previous ee-repo-ref: 390ed6c851b1915f0b492897c663f8058477680f New ee-repo-ref: 14315067c083d3361512de621b12e41dbe3b017d Automated by sync-ee-ref workflow. * fix(secret-backend): escape failure fields and use <br> in migration toast Address CI review on PR #9292: - P1 (cubic/codex): backend-supplied workspace_id/path/error are now HTML-escaped before being interpolated into the migration toast, which renders through {@html processMessage(...)} in Toast.svelte. This prevents stored XSS via secret paths or backend errors that contain markup. '/' is intentionally left intact so the toast's path-highlight regex still tags workspace paths. - P2 (pi): swap '\n' for '<br>' so multi-line failure lists actually break in the toast instead of collapsing to a single run-on line. - Extend the same per-secret failure surfacing (toast + console.error) to the Azure Key Vault and AWS Secrets Manager migration handlers via a shared reportMigrationFailures() helper so all six migration paths report identically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * nit react-sdk resource editor * sdk_resource * make `selected` resilient + snapshot args for React (#9298) * fix(ResourceEditor): make `selected` resilient + snapshot args for React Two issues surfaced via the React SDK (reactify wrapper re-spreads Svelte props on every host re-render): 1. The bindable `selected` prop transiently resets to undefined on each re-spread, flipping `current` through undefined and unmounting the form (input loses focus on every keystroke). Rename the prop to `selectedProp` and derive `selected = selectedProp ?? effectiveWorkspace` so the fallback insulates the component without effects. 2. The onChange dispatch passed `current.args` (a `$state` proxy) directly, so React consumers diffing by reference or JSON.stringify saw the same value forever, and the effect only tracked the args reference (not nested mutations). Wrap with `$state.snapshot` to deep-track and emit a plain object. The bootstrap effect is also restructured: it no longer writes `selected` (the derived handles defaulting) and now guards on `selected in initialStates` so workspace flips remain idempotent. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ResourceEditor): declare effectiveWorkspace before use in selected Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * remove unused workflow * feat(typescript-client): add deleteS3File + optional workspace arg on S3 helpers (#9300) * feat(typescript-client): add deleteS3File + optional workspace arg on S3 helpers Customer-requested ergonomics for the TypeScript SDK: - New `deleteS3File(s3object, workspace?)` wrapper around the existing `HelpersService.deleteS3File` (backend endpoint is already there). Saves callers from having to either hand-roll `denoS3LightClientSettings()` + AWS SDK calls, or wire up `HelpersService` directly. - `denoS3LightClientSettings`, `loadS3File`, `loadS3FileStream`, `writeS3File`, and the new `deleteS3File` all gain an optional trailing `workspace?: string` parameter that falls back to the `WM_WORKSPACE` env var via `getWorkspace()`. Mirrors the calling convention customers already expect from helpers like `getVariable` / `runScript`. `build.sh` and `build.jsr.sh` are updated to export `deleteS3File` from both the NPM and JSR entry points. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate system_prompts auto-generated for new S3 helpers `python system_prompts/generate.py` after adding deleteS3File and the optional workspace param to the existing S3 helpers, so the agent-facing docs (CLI skills, TS SDK prompt, script skills) reflect the new signatures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(github-app): hide cloud-only UI on self-managed + admin assignment UI (#9299) * feat(github-app): hide cloud-only UI on self-managed + admin assignment UI Two related UX fixes for the GitHub App self-managed (GHES) integration: 1. On self-managed instances, the per-installation Export button and the "Import installation from other instance" section in the workspace UI both hide. Both round-trip a JWT carrying only {installation_id, account_id} with no github_base_url, so they would produce broken cloud-style installs on a self-managed instance. The previous Export attempt also failed with "No JWT token received from server" because self-managed installs store an empty JWT by design. 2. New "Workspace assignments" panel in instance settings (GhesAppSettings.svelte) that auto-discovers installations of the configured GHES App and lets the super-admin assign them to specific workspaces. Workspace users without GitHub permissions no longer need to install the App themselves — the admin provisions the link from instance settings. Admin-provisioned installs show a "Provisioned by admin" badge in the workspace UI and can only be removed by the super-admin from instance settings. Backend support is in the EE companion PR windmill-labs/windmill-ee-private#588. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to da5189cf69a453de3855057f41be0d84e5910707 This commit updates the EE repository reference after PR #588 was merged in windmill-ee-private. Previous ee-repo-ref: d959b83ce413ad531e9cc28e0f8199cdecb73a31 New ee-repo-ref: da5189cf69a453de3855057f41be0d84e5910707 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * chore(main): release 1.707.0 (#9285) * chore(main): release 1.707.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat(queue): per-workspace fairness cap on the shared cloud worker pool (#9303) * feat(queue): cloud-only per-workspace fairness cap on the shared worker pool On `app.windmill.dev` the cluster runs a single default worker group, so a single workspace flooding the queue can degrade quality of service for everyone else. This adds an opt-in mechanism that caps any single workspace at a configurable share of the shared worker pool when it has been dominating cluster activity for more than a configurable window. Detection signal counts both currently-running jobs and jobs completed in the rolling window, so it catches workspaces hogging slots with long jobs **and** workspaces spamming many tiny jobs (where no individual job's started_at is old, but throughput share dominates). Refresh is coordinated cluster-wide via a single UPDATE on `background_task_state`: the `WHERE updated_at < now() - interval` predicate combined with row-level locking means only one process per refresh cycle actually runs the aggregation, regardless of fleet size. Every other process gets the freshly written value in the same round trip via `UNION ALL ... LIMIT 1`. Heavy aggregation rate stays at ~0.2-0.5 qps for the whole cluster. Pull queries are split: the existing query string and its bind shape stay bit-identical to today, so the planner keeps using the same indexes when fairness is off or no workspace is currently capped. A separate `WORKER_PULL_QUERIES_FAIRNESS` adds `AND workspace_id <> ALL($2::text[])` and is only materialized while the feature is enabled. Hard-gated to `CLOUD_HOSTED=true` + BASE_URL host == app.windmill.dev at three layers: frontend `cloudonly: true`, API setter rejection in `set_global_setting_internal`, runtime check in `fairness_active`. Settings are exposed under Jobs in the instance-settings UI; defaults are off so the change is a no-op for self-hosted. Two-pass pull guarantees no worker idling: if every queued job belongs to a capped workspace, the second pass uses the unmodified pull queries. Cap re-asserts on the next refresh. Fixes WIN-1982 * fix(queue): address CI review findings on workspace fairness Six fixes from the four-reviewer cross-check on #9303: 1. **Aggregation evaluation (Codex P1).** The previous `INSERT ... ON CONFLICT DO UPDATE WHERE updated_at < ...` had the heavy `v2_job_queue ∪ v2_job_completed` aggregation inlined into `VALUES`, which Postgres evaluates for every contender to build the proposed row — losing the "one heavy aggregation per cycle cluster-wide" property the design advertises. Split into three small statements: (a) cheap claim with constant `VALUES`, (b) winner-only `UPDATE ... SET value = jsonb_build_object('overloaded', <agg>)` (Postgres only evaluates `SET` per row matching `WHERE`, so losers never compute the aggregation), (c) read for everyone. Heavy query now truly runs ~0.2-0.5 qps cluster-wide regardless of fleet size. 2. **Numeric setting wraparound (cubic P1).** `u64 as u32` and downstream `u32 as i32` could silently flip sign and feed `make_interval(secs => -N)`, making `now() - interval` a future timestamp and disabling the completed-jobs half of the activity signal. Clamp `duration_secs` to [1, 86400] and `min_total_jobs` to [0, u32::MAX] before storing. 3. **`/instance_config` bypass (cubic/Claude/Codex P2).** Bulk config endpoint sidestepped `set_global_setting_internal`'s gate; a self-hosted superadmin could persist `workspace_fairness_*` rows via the bulk path. Mirror the per-key check in `set_instance_config` upsert flow. 4. **DB error coerced to false (Claude P2).** `load_workspace_fairness_enabled` collapsed `Err(_)` to `false` and unconditionally swapped the atomic — a transient DB blip during notify-event propagation toggled the feature off cluster-wide (and triggered a `store_pull_query` rebuild precisely when load is highest). Now propagates the error so the atomic stays at its prior value. 5. **Refresh failure cooldown (Claude P2).** Storing `0` removed the rate limit entirely; every subsequent pull spawned a new refresh task. Leave `LAST_REFRESH_MICROS` at `now_us` (already written by the CAS) so the natural interval acts as the cooldown. 6. **Visibility + duplication (Pi P2).** Mark `make_pull_query_fairness` as `pub(crate)`. Move the duplicated `BASE_URL host == app.windmill.dev` parser into `windmill-common::worker::is_cloud_production_host` and share it between the API setter and the runtime path. Verified locally: - `POST /api/settings/global/workspace_fairness_enabled` → 400 (per-key gate) - `PUT /api/settings/instance_config` with fairness key → 400 (bulk gate) - `cargo check --workspace --features=private,enterprise,quickjs` — clean Refs WIN-1982. * fix(queue): second round of CI review nits on workspace fairness Three issues raised by the Codex/Claude re-review of commit 0b38ff2: 1. Non-cloud deletes were rejected (Codex P2). The cloud gate ran before the Null / empty-string deletion branches in both `set_global_setting_internal` and the bulk `set_instance_config`. A self-hosted instance that inherited stale `workspace_fairness_*` rows from a cloned cloud DB couldn't clear them through the API — the rows stayed in `global_settings` and continued to show up in the YAML export. Now the gate only blocks upserts; Null / empty-string deletes pass through on any host. 2. Deleted numeric knobs kept stale runtime values (Codex P2). When a cloud admin cleared `workspace_fairness_max_percent`, `..._duration_secs`, or `..._min_total_jobs`, the notify-event fired but the numeric loaders ignored `Ok(None)` and left the previous in-memory value pinned until process restart. Loaders now distinguish three outcomes: - `Err(_)`: transient — leave atomic alone (preserves the previous-round fix). - `Ok(None)` / `Ok(Some(invalid))`: reset to the documented default. - `Ok(Some(valid))`: clamp and store. Defaults are extracted to `WORKSPACE_FAIRNESS_*_DEFAULT` constants kept in sync with the `AtomicU32::new(...)` initialisers in `windmill-common/src/worker.rs`. 3. `fairness_active` was `pub` with no cross-crate caller (Claude nit). Tightened to module-private. Verified locally on this non-cloud instance: POST .../workspace_fairness_enabled body=null → 200 (delete passes) POST .../workspace_fairness_enabled body=true → 400 (set blocked) PUT .../instance_config {} → 200 (no-op passes) PUT .../instance_config with fairness key → 400 (bulk set blocked) Skipped the partial index on `v2_job_queue WHERE running = true` that Claude flagged as a residual nit — queue stays under 50k rows per the operator's measurement, so the seq-scan cost (~10 ms × 0.5 qps = ~0.5% of a DB core) is well below the noise floor and the index isn't worth the maintenance cost on job transitions. Refs WIN-1982. * chore(main): release 1.708.0 (#9304) * chore(main): release 1.708.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * feat: add copy button to Path component (#9311) * feat: plug global chat drafts into userdraft (#9291) * refactor: move global chat drafts to userdraft * feat: share script and flow drafts with editors * feat: share trigger drafts with editors * feat: share raw app drafts with editor * feat: share resource drafts with editors * docs: rename global chat drafts copy * feat: add global chat draft discard tool * fix: resolve global chat editor draft paths * fix: remove editor draft path resolver * feat: track live editor drafts in userdraft * fix: snapshot live userdraft reads * chore: checkpoint pending global draft changes * fix: address global draft review issues * fix: defer raw app draft persistence * docs: remove pr investigation docs * fix: persist live global draft writes * refactor: move bedrock proxy handling to windmill-ai (#9309) * refactor: move bedrock proxy handling to windmill-ai * docs: track ai refactor follow-ups * fix(auth): filter resource/variable listings by token scope (WIN-1981) (#9302) A token scoped to a single resource (e.g. `resources:read:u/alice/foo`) could call `GET /api/w/{w}/resources/list_search` and receive `path` and `value` for unrelated resources in the workspace. Route-level scope checks only validate `domain:action`; per-resource handlers do a `check_scopes` against the path, but the listing endpoints did not — leaking integration credentials, API keys, and other secrets stored as resource values to narrowly-scoped tokens. Add `build_scope_path_predicate` to `windmill-api-auth` (mirrors `check_scopes` semantics but parses the token's scopes once, suitable for filtering many rows). Apply it to `list_search_resources`, `list_resources`, `list_names` (resources) and `list_variables` (non-secret value leak), so a scope-restricted token only ever sees the paths it is authorized to read. Unscoped tokens and tokens whose only scopes are `if_jobs:filter_tags:*` are unaffected. Includes regression tests covering: unscoped, tag-filter-only, single-resource, wildcard, wrong-domain, and write-implies-read. Fixes WIN-1981 Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * audit-log workspace-fairness cap transitions (#9306) * feat(queue): audit-log workspace-fairness cap transitions When the cloud per-workspace fairness mechanism adds a workspace to the capped set or releases one, write `workspace_fairness.capped` / `workspace_fairness.uncapped` audit-log entries to the affected workspace. The cluster admin can review the full timeline from the `admins` workspace audit view with `all_workspaces=true`; per-workspace owners see their own events in their normal audit list. Only the per-cycle refresh winner emits entries (matching where the heavy aggregation runs), so a fleet of N workers does not produce N duplicates per transition. The diff is computed against the value already in `background_task_state` rather than the winner's in-memory cache, so a freshly-restarted process winning the claim does not spuriously emit "newly capped" entries for workspaces that were already capped before it started. Audit writes are best-effort: failures are logged via tracing and do not abort the refresh cycle. Fixes WIN-1984 * feat(queue): scope fairness audit to admins workspace + queue-metrics pane - Write `workspace_fairness.capped` / `workspace_fairness.uncapped` to the `admins` workspace (was: per-affected-workspace) with the affected workspace_id moved to the `resource` field. Cluster admins now get the full timeline in one place without `all_workspaces=true`. - Add `GET /workers/workspace_fairness_events` returning the last 100 events. Cloud-gated (returns `[]` on non-cloud) and devops-only. - Add a `WorkspaceFairnessEvents` Section to the Queue Metrics drawer, rendered only when `isCloudHosted()` is true. Shows time / event badge / workspace / parameters with a refresh button. Fixes WIN-1984 * feat(ai-chat): expand chat question answers (#9310) * feat(ai-chat): align footer bar + DropdownV2 mode/autonomy selectors (#9308) * feat(ai-chat): align footer bar, use DropdownV2 for mode/autonomy selectors Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dropdown): add `selected` item prop rendering a trailing check Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(ai-chat): add small spacing between chat input and footer bar Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai-chat): always offer the 3 autonomy options in the auto-accept picker Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ai-chat): default autonomy mode to auto-accept on Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(ai-chat): use Button component for footer dropdown triggers Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(ai-chat): use a hand icon for the auto-accept-off autonomy state Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(ai-chat): use subtle Button variant for mode and model selectors Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(ai-chat): tighten spacing between input and footer bar Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai-chat): reword autonomy levels as ask/auto-accept/bypass permissions Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(button): add 2xs unified size with tighter padding Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ai-chat): compact footer bar — 2xs buttons, AtSign context icon, short Yolo label, discreet model Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style(ai-chat): widen the permission selector dropdown Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(dropdown): group shortcut + selected check to avoid ml-auto collision Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(ai-chat): cover getPersistedAutonomyMode default; clarify default comment Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(raw_apps): tab-based editor surface with split-with-preview (#9273) * feat(raw_apps): custom tab system for source / runnable / preview Replaces the fixed split-pane layout with a tab bar inside the editor area. Each frontend file is a tab, each selected runnable is a tab, and the Preview is pinned to the right (non-closable). Tabs are an alternative discoverability surface to the sidebar — both stay functional, but tabs make navigation viable on small screens with the sidebar collapsed. A "Split with Preview" toggle in the tab bar's trailing slot pairs the active tab with the preview side-by-side for wide-screen multitasking. The toggle hides when Preview is already the active tab. The UI Builder, runnable editor, and preview iframe all stay mounted across tab switches (toggled via `display`) — no bundler restarts, no preview state loss, no editor remounts. - New common/tabs/DraggableTabs.svelte: reusable tab strip with drag-reorder (@windmill-labs/svelte-dnd-action), pinned-left/right slots excluded from the drag zone, hover-revealed X close, middle- click close, keyboard navigation (arrows / Enter / Backspace), and a `trailing` snippet for inline toolbar add-ons. - raw_apps/RawAppEditor.svelte: - Tab state (`tabs`, `activeTabId`, `splitWithPreview`) lives in Windmill. Persisted in localStorage keyed by workspace + app path. - Sidebar file clicks (`handleSelectFile`) and runnable selection (`selectedRunnable` via `bind:`) are mirrored into tabs via an effect — the sidebar interaction is otherwise untouched. - Listener augmented: `setActiveDocument` backfills tabs for files VS Code opens by itself; `setFiles` / `runnables` updates drop stale tabs. - Bundler / inspector / rebuild toolbar moves into the tab bar's trailing slot — always visible regardless of active tab. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(raw_apps): modern tab styling + resizable split-with-preview Two polish passes on the new tab system: DraggableTabs styling: - Remove the bottom border on the tab strip + the accent-coloured border-b-2 on the active tab. The active tab now shares the surface background with the content area below it, so the boundary visually "disappears" — modern IDE-style tabs. - Inactive tabs sit on the darker surface-secondary tab strip and get a subtle right separator so they don't blur into each other. Split-with-Preview is now a real resizable Splitpanes: - The content area is rendered as a Splitpanes (always), with the source/runnable slot on the left and the preview iframe on the right. The user can drag the divider to adjust the ratio when the "Split with Preview" toggle is on. - Iframes never remount across single↔split toggles — pane sizes are driven reactively from (activeTabKind, splitWithPreview), not by adding/removing the Splitpanes itself. - The user's preferred split ratio is remembered while they're dragging and reapplied next time split is enabled. - The inner splitter is CSS-hidden in single mode so the toggle button stays the single canonical way to flip layouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): split mode moves preview tab into the right pane Cleaner mental model for split-with-preview. Instead of "split the active tab + always keep the Preview tab around", the Split toggle now physically moves the Preview tab out of the bar and into a permanent right pane. When the user toggles split off, the Preview tab reappears in the bar like any other tab. - New `displayedTabs` derived: filters out the Preview tab when splitWithPreview is on, so the user sees only file/runnable tabs in the bar and a dedicated preview pane on the right. - `toggleSplit` redirects the active tab to the most recent file/runnable when the user toggles split on with Preview active, so they don't end up staring at an empty left pane. - Split toggle is now always visible — the user can flip both ways. The button label flips between "Pin preview to the right" and "Move preview back into a tab" to reflect what's about to happen. - reorderTabs preserves the Preview tab in the underlying `tabs` array even though it's filtered out of the drag set in split mode. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(raw_apps): VS Code-style "Preview" header on the right pane In split mode, the right pane now shows a small "Preview" tab-styled header anchored at its top-left — making the layout read like a real VS Code editor split, where each group has its own tab bar. - Header appears only when `splitWithPreview && activeTabKind !== 'preview'` (i.e. when the right pane is meaningfully separate from the left's content). In single mode with preview active, the right pane is the only thing visible and the main tab bar already labels it. - The header uses the same styling as an active tab: `bg-surface` on a `bg-surface-secondary` strip, h-8, text-xs, no border. - An X button next to the label toggles split off — equivalent to closing the editor in VS Code's split view (preview goes back to living as a tab in the main bar). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): VS Code-style symmetric tab bars per pane Restructure the editor area so each pane is a self-contained "editor group" with its own tab bar at the top. The Splitpanes is now the topmost element — the divider runs floor-to-ceiling, splitting both the tab bars and the content. Layout (left pane = source / runnable, right pane = preview): - Left pane top: DraggableTabs (file/runnable tabs, Preview tab when split is off) + Split-toggle in the trailing slot. - Right pane top: a custom preview header — "Preview" label styled like an active tab on the left + the preview-affecting toolbar (bundler, inspector, rebuild) on the right. - Each pane independently sized via Splitpanes; iframes + the runnable panel stay mounted and toggled via `display` so state survives every transition. Trade-off: in single-mode with Preview active (paneA=0), the left tab bar is hidden along with the left pane. To switch back to a file tab the user uses the sidebar — which is exactly the discoverability surface tabs were meant to complement, not replace. Button placement by semantic ownership: - Layout control (Split toggle) — left side, with the editor. - Preview-affecting controls (bundler, inspector, rebuild) — right side, with the preview. No close-X on the right; the Split toggle on the left is the canonical way to flip layouts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(raw_apps): keep tab bar visible when Preview is active in single mode The "VS Code-style" restructure put the tab bar inside the left Pane. When activeTabKind became 'preview' in single mode, the left pane collapsed to width 0 and the entire tab bar disappeared with it — leaving the user with no way to switch back to a file tab except via the sidebar. Move the main tab bar back above the inner Splitpanes (full width, always visible). The preview pseudo-header stays inside the right pane, carrying the bundler / inspector / rebuild toolbar. The splitter only goes through the content area below the tab bar, which is acceptable given how much friction the disappearing-tabs edge case caused. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): per-pane tab bars with mirrored single-mode lists Replace the single tab bar above the inner Splitpanes with one DraggableTabs per pane. Splitter now goes floor-to-ceiling through tabs AND content in split mode. In single mode both bars mirror the full tab list, so the visible pane always carries every tab — fixes the bug where activating Preview hid the tab strip. Clicking Preview while in split mode is a no-op (Preview is permanently visible in the right pane). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): polish tab strip and sync editor font to text-xs * feat(raw_apps): move logs overlay onto the preview pane * refactor(splitpanes): extract pixel-aware minSize helper * fix(raw_apps): tab hydration loads correct file; closeTab in split mode * fix(raw_apps): lazy-mount UI Builder iframe + add dev:ui-builder script * feat(raw_apps): default split view, blue preview tab, fix dnd ghosting * fix(raw_apps): remove 1px splitter sliver beside preview in single view * fix(raw_apps): tab scrollbar on hover, fix thumb height + resize staleness * refactor(raw_apps): don't persist tab/split layout in localStorage * refactor(raw_apps): derive pane sizes + binding setter instead of effects * style(raw_apps): trim verbose comments * feat(raw_apps): accept appendLogs delta from the UI Builder iframe * fix(raw_apps): exit inspect mode on Escape * fix(raw_apps): Escape clears lingering inspector selection after pick * style(raw_apps): accent-selected styling for active tab, bg-surface strip * fix(raw_apps): address PR review nits (drop debug log, timer/reorder/pane-setter, dev script restore) * fix(raw_apps): clear inspector overlay on the preview iframe, not the source * style(raw_apps): neutral tab look (surface-tertiary/text-emphasis selected, text-hint idle) * chore(raw_apps): bump bundled ui_builder to 61b6fdd --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(raw_apps): bump bundled ui_builder to b4f6219 (#9314) * skip workspaced-route duplicate checks on cloud (#9305) * fix(settings): skip workspaced-route duplicate checks on cloud The pre-write validation hooks for `app_workspaced_route` and `http_route_workspaced_route` query the DB for cross-workspace duplicates and fail the save when any are found. On cloud both `custom_path_exists` (apps) and `route_path_key_exists` (HTTP triggers) already scope lookups by `workspace_id` regardless of these settings, so duplicates across workspaces are expected and the validation has no runtime meaning. The result was that any cloud super-admin attempting to save instance settings with these toggles set to false received `Duplicate HTTP route paths detected` even though the setting has no effect on cloud routing. Fixes WIN-1983 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(error): render JsonErr as readable text and return 400 `Error::JsonErr` previously rendered through `#[error("Error: {0:#?}")]`, leaking Rust's `Debug` output (`Object { "error": String(...), "details": Array [...] }`) into the HTTP response body, and was bucketed into the catch-all 500 branch in `IntoResponse`. The result was a 500 status with a wall of Rust debug syntax in the toast — confusing and user-hostile. - Bucket `JsonErr` into 400 (Bad Request): every current call site (workspaced-route duplicate checks, OAuth client errors, etc.) is a client/validation issue, not an internal server fault. - Add `format_json_err_message` which surfaces the `error` field as the headline, summarises `details` (with a `- key=value` per entry), and pretty-prints the rest as JSON for unknown shapes. The frontend toast now reads e.g. Duplicate HTTP route paths detected - route_path=a, workspace_id=admins, http_method=post - route_path=a, workspace_id=starter, http_method=post Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toast): preserve newlines and escape HTML in multi-line errors The toast renders via `{@html processMessage(message)}`, so server-side error bodies that span multiple lines (e.g. the duplicate-route response from the settings endpoint) collapsed into a single line because HTML treats consecutive whitespace (including `\n`) as a single space. When the message contains a newline, escape HTML first (defends against injected markup in server error bodies) and convert `\n` to `<br />` so multi-line errors stay readable in the toast. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup: address CI review feedback - toast.ts: escape HTML unconditionally. The previous gate on `\n` left single-line server error bodies unsafe under {@html}, which cubic flagged as P0. The path regex below only inserts a `<span>` around a `u/...` or `f/...` capture that can't contain HTML metacharacters, so escaping the whole input is the simpler and correct fix. - error.rs: add unit tests pinning the rendered shape of `format_json_err_message` (error+details, error-only, truncation cap, non-object fallback to pretty JSON). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(service-accounts): allow choosing role at creation time (#9307) * [ee] feat(service-accounts): allow choosing role at creation time Previously, service accounts were hardcoded to operator and could not be used as the CLI sync user since they had no write access. They also only counted as 0.5 seat each. This change: - Extends `NewServiceAccount` to accept optional `is_admin` / `operator` (defaults to `operator=true` for backward compatibility). - Exposes a role picker in `AddUser.svelte` when creating a service account (Operator / Developer / Admin). - Lets admins update a service account's role from the user list (it used to be locked to "Operator" with a tooltip). - Updates the OpenAPI spec + regenerates the frontend client. A developer/admin service account counts as 1 seat under the existing seat-cap logic (operators stay at 0.5). Companion PR on windmill-ee-private updates the `INSERT INTO usr` to honour the chosen role. Fixes WIN-1985 * [ee] feat(service-accounts): wm_deployers opt-in for Dev role When creating a service account with role=Developer, surface a toggle "Add to wm_deployers" (recommended). Members of wm_deployers can deploy on behalf of other users — the typical setup when the service account is used as the CLI sync / CI deploy identity. - `NewServiceAccount` gains an optional `add_to_deployers` flag. - Frontend defaults the toggle to on but only shows it under Developer (admins have it implicitly; operators can't deploy). - Tooltip links to docs.windmill.dev "Run on behalf of". Companion EE PR updates the handler to INSERT into usr_to_group for wm_deployers when the flag is set. Refs WIN-1985 * chore: update ee-repo-ref to 974ed42067d9f63acb42332b671b8c01ffd4b625 This commit updates the EE repository reference after PR #589 was merged in windmill-ee-private. Previous ee-repo-ref: f7dbc3cc2ba21c396f4828881e3b9d9ab6f50c69 New ee-repo-ref: 974ed42067d9f63acb42332b671b8c01ffd4b625 Automated by sync-ee-ref workflow. * [ee] fix(service-accounts): unhardcode role in superadmin user list Two review issues from the merged #9307 / #589: 1. P1 — The global Users tab in #superadmin-settings still pinned every service account to "Operator". Now it shows the actual role (Admin / Operator / Developer), derived from the SA's usr row. - `list_users_as_super_admin`: replaced `true as operator_only` with the real `operator` value, and added `is_workspace_admin` from the row (NULL for password users since their admin status is per-workspace). - `global_whoami`: when the email belongs to a service account, look up its real `operator` / `is_admin` instead of pinning to operator. - `SuperadminSettingsInner.svelte`: drop the hardcoded "Operator" badge; render Admin / Operator / Developer using the new fields, matching the workspace-level view. 2. P2 — Regenerate the bundled `openapi-deref.{yaml,json}` so the `createServiceAccount` body (now exposing `is_admin`, `operator`, `add_to_deployers`) and the new `GlobalUserInfo.is_workspace_admin` field show up at runtime in `/api/openapi.{yaml,json}`. Bumps `ee-repo-ref.txt` to the EE follow-up that adds the offline seat-cap check on `create_service_account`. Refs WIN-1985 * chore: update ee-repo-ref to b7a6068c1f3dc845e012959268b2426f0de4d697 This commit updates the EE repository reference after PR #590 was merged in windmill-ee-private. Previous ee-repo-ref: 0b1307c21d1bfd6fb43a03c2ba39d2a8bf8e6470 New ee-repo-ref: b7a6068c1f3dc845e012959268b2426f0de4d697 Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> * fix(jobs): authorization bypass in only_result job updates (WIN-1980) (#9301) * fix(jobs): enforce anonymous-only guard on `only_result` job updates The `jobs_u/getupdate/{id}` and `jobs_u/getupdate_sse/{id}` endpoints accept `only_result=true`. In that branch, `get_job_update_data` queried the result solely by (workspace_id, job_id) and skipped the `created_by == "anonymous"` check that the non-only_result path and adjacent unauthenticated endpoints apply. An unauthenticated requester who learned a private job UUID could therefore retrieve that job's output. Hoist the guard to the top of `get_job_update_data` so both branches are covered. Fixes WIN-1980 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: fold `created_by` check into existing only_result queries Avoids the extra `SELECT created_by` round-trip per call by joining `v2_job` once in the two queries that handled the unauth path and checking inline. Behavior is identical to the prior commit; the SSE polling loop now does one query per poll instead of two for unauthenticated callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: cache anonymous_verified across SSE polls Replace the LEFT JOIN approach with an upfront `SELECT created_by` guarded by a new `&mut bool anonymous_verified` parameter that mirrors `early_return_suppressed`. The SSE polling loop now performs the auth check exactly once per stream rather than per poll, and the data SQL reverts to its original form so authenticated callers pay no extra cost. `created_by` cannot change after job creation, so caching the verification across polls is safe. Cost matrix: - Authed (any path): 0 extra queries - Unauthed one-shot: 1 extra query (unavoidable) - Unauthed SSE: 1 extra query at stream start, 0 per poll Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: scope anonymous check to only_result branch The non-only_result branch already enforces the `created_by` check via its main query, so a top-level hoisted check duplicated work for unauthenticated default-path callers. Move the check inside the `if only_result.unwrap_or(false)` block — exactly where the bypass lives — and leave the non-only_result path untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(raw_apps): surface UI Builder build errors over the preview pane (#9316) * feat(raw_apps): surface UI Builder build errors over the preview pane Companion to the matching change in the UI Builder repo (see linked PR), which stops rendering the build-error overlay over the VS Code editor iframe and instead emits a `buildError` postMessage on every build (message: undefined on success to clear). Listen for that message on the existing window message handler (already source-gated by the UI Builder iframe), store it in a `buildError` $state, and surface it in two places: * A red banner over the preview iframe, sibling to the existing logs overlay (`top-12 left-2 right-2 z-20` so it clears the tab bar) — failures appear right where the user looks for the rendered output. * The Preview tab's icon and label tint red (`text-red-600 dark:text-red-400`, matching the existing error convention in raw_apps) — important in single-tab mode where the preview pane is collapsed to 0px and the banner would be hidden. Done by mapping `leftPaneTabs` / `rightPaneTabs` through a small `tintPreviewOnError` helper so the source-of-truth `tabs` array is untouched (DnD, ordering, fallback selection keep using the original previewTab object). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): use Alert component for the build-error banner Replace the hand-rolled red div with the shared `Alert` component (`type="error"`, `title="Build failed"`). The error text stays in a `<pre>` child so multi-line bundler output keeps its formatting, with `max-h-60` so a long error never takes over the whole preview pane. The absolute-positioned wrapper (`top-12 left-2 right-2 z-20`) and the `role="alert"` move to that wrapper so the Alert component itself stays unstyled at the call site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(raw_apps): solid bg-surface backing behind build-error Alert The Alert's error background is semi-transparent in dark mode (`bg-red-900/40` in `common/alert/model.ts`), so the preview iframe shows through when the banner is laid over it. Add a `::before` pseudo on the Alert root with `bg-surface` (matched `rounded-md`, `-z-10` so it sits behind the red bg) to give it a solid plate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(raw_apps): isolate banner stacking context, DRY tab tint chain Two small follow-ups from review: * Add `isolate` to the build-error banner wrapper so the `before:-z-10` pseudo's stacking context is pinned locally — it works today because `position: absolute` + `z-20` creates one, but `isolate` makes the dependency self-documenting and survives a future refactor that removes the explicit `z-20`. * Extract `tintTabs = (ts) => ts.map(tintPreviewOnError)` so the two `$derived` blocks for leftPaneTabs / rightPaneTabs read identically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(raw_apps): trim build-error overlay comments Per review feedback. Keep only the load-bearing facts (bg-surface backs the Alert's translucent red, isolate pins the pseudo stacking, the `message: undefined` clear convention) and drop the prose context that duplicated what the code already shows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(raw_apps): bump bundled ui_builder to 00c9834 Brings in the postMessage emission from windmill-labs/windmill-code-ui-builder#9 (merged) so this PR's host listener actually receives `buildError` events. SHA verified against the R2 artifact. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(main): release 1.709.0 (#9312) * chore(main): release 1.709.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * add cli-sync workspace snapshot/load scripts (#9322) * feat(fixtures): add cli-sync workspace snapshot/load scripts * fix(fixtures): address review nits (env var password, mktemp, dead refs) * fix(fixtures): address CI review (SIGPIPE, JSON escaping, doc/code drift) * feat(queue): stochastic admission + EE availability of workspace fairness algorithm (#9321) * refactor: unify AI provider credentials (#9317) * refactor: use provider credentials for worker builders * refactor: resolve api proxy credentials directly * fix: lazy load frontend eval modes * fix(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (#9324) * feat(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (WIN-1988) `tokio_tungstenite::connect_async` opens a raw TCP socket and ignores the standard outbound-proxy env vars, so deployments behind a forward HTTP proxy can't reach the WebSocket endpoint and Test Connection times out after 30s. Add a small `proxy` module that resolves the right proxy URL for the target host (HTTPS_PROXY for wss://, HTTP_PROXY for ws://, NO_PROXY exclusions, ALL_PROXY fallback, lowercase variants), opens an HTTP CONNECT tunnel when one applies, and hands the resulting TcpStream to `client_async_tls_with_config` for the TLS + WS handshake. Direct connect remains the default when no proxy env is set. Unit tests cover NO_PROXY matching, proxy URL parsing (including IPv6 literals and basic-auth userinfo), and the CONNECT handshake itself against an in-process fake proxy (success, basic-auth header, 407 rejection). Fixes WIN-1988 * refactor(websocket-trigger): reduce blast radius and reuse existing logic Follow-up to the proxy support change. Three things: 1. Skip the new code path entirely when no proxy is configured. `connect_async_with_proxy` now checks the env-var snapshots up front and delegates straight to `tokio_tungstenite::connect_async` if neither `HTTP_PROXY` nor `HTTPS_PROXY` is set. Same fall-through applies when proxy env is set but `NO_PROXY` excludes the host or the proxy URL doesn't parse. Non-proxied deployments now exercise exactly the previous code path. 2. Move the `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` env-var snapshots from `windmill-worker::worker` into `windmill-common`. The worker's `PROXY_ENVS` static now reads from there, and the websocket trigger reads from the same source — one place reads the env, one source of truth for both call sites. 3. Replace the hand-rolled proxy-URL parser with `url::Url::parse` (already a workspace dep, used across the codebase). Half the LoC and handles edge cases (userinfo percent-encoding, IPv6 literals, path/query stripping) via the well-tested crate instead of by hand. All 13 proxy unit tests still pass. `cargo check` is clean. * fix(websocket-trigger): unbreak EE build + trim proxy tests - Re-export `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` from `windmill-worker::worker` (via `pub use windmill_common::...`) so the EE `otel_tracing_proxy_ee` module's `use crate::{HTTPS_PROXY, ...}` resolves like it did before. Fixes the `check_ee_full` / `cargo_test` CI failures from the previous commit. - Trim the proxy tests to one un-ignored canary (`http_connect_tunnel_sends_well_formed_request_and_unwraps_stream`) that exercises the actual on-wire CONNECT handshake plus byte-perfect tunnel passthrough. The NO_PROXY-matching, URL-parsing, and edge-case tunnel tests are kept under `#[ignore]` for manual debugging (`cargo test -- --ignored`) since they're either delegated to `url::Url::parse` or trivial string matching — low ROI on every CI run. * chore(main): release 1.710.0 (#9323) * chore(main): release 1.710.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * fix: improve workspace fairness * chore(main): release 1.710.1 (#9327) * chore(main): release 1.710.1 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> * prevent windows backend tests from running out of disk space (#9325) * ignore flaky fairness regression tests in CI (#9328) `fairness_ignores_zombie_running_rows` and `fairness_ignores_concurrency_suspended_rows` panic intermittently in CI (both Linux and Windows runs). Mark them `#[ignore]` until the underlying flakiness is resolved. * feat(cli): add object-storage commands and flow test-step (#9326) * feat(cli): add object-storage commands and flow test-step * docs(cli): clarify flow test-step doesn't recurse into aiagent tools * fix(cli): correct failure step id in docs, handle bare flow.yaml path * refactor(cli): fold flow test-step into flow preview --step (#9330) * fix(queue): duration-weighted workspace fairness signal (#9329) * fix(queue): bump EE ref to include worker_ping fairness signal The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge commit on the asset-graph-view-ee branch) instead of ddc9e80, which contains the workspace-fairness fix that switches the active-share signal from v2_job_queue.running=true to worker_ping. As a result cloud was still computing overload off the legacy signal, so a workspace with many in-flight/suspended flows (lancom01-prod, with 799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397 running-true rows) was flagged as 95% of cluster activity despite consuming zero worker slots. Bumping to ddc9e80 picks up the worker_ping-based signal, which naturally excludes (a) suspended jobs (no worker pinging them), (b) zombie running-rows from dead workers, and (c) flow/flownode orchestration rows that never run on a worker in the first place. * test(queue): seed v2_job rows + realistic durations for fairness helpers The new duration-weighted fairness algorithm joins v2_job_queue and v2_job_completed to v2_job for the `kind` filter (excluding flow bookkeeping) and reads `duration_ms` for the completed contribution. Update the test helpers to mirror that schema: * `insert_completed` now inserts a matching v2_job row (kind=script) and writes `duration_ms = 1000` with a 1-second [started_at, completed_at] interval, so each completed row contributes ~1 worker-second when fully inside the refresh window. * `insert_queued` likewise pre-inserts v2_job, sets `started_at` to NOW() - 1s when running=true (so running rows contribute ~1 worker-second by the time the refresh runs), and seeds v2_job_runtime.ping so the running side accrues real-time worker seconds (the algorithm bounds end-of-interval by ping). The zombie/suspended insert helpers are intentionally left without v2_job rows — the new algorithm's INNER JOIN excludes them, so they still correctly contribute zero worker-seconds. * chore(queue): bump EE ref to duration-weighted fairness algorithm Companion to windmill-ee-private#<TBD>: switch the EE workspace fairness aggregation from a count-based UNION (worker_ping snapshot + v2_job_completed count) to a worker-seconds aggregation sourced directly from v2_job_queue and v2_job_completed, with kind/suspend filters mirroring handle_zombie_jobs and per-row defenses against zombie inflation on both halves. * chore(queue): bump EE ref for fairness perf fix (inline window_start) * chore(queue): bump EE ref for fairness perf rewrite (driver-side flip) * update ee ref * feat(hub-publish): add backend proxy routes for hub publishing New workspaced router /api/w/:ws/hub/* forwarding to the Hub: - POST /publish_draft → POST {HUB}/workspaces (slug/name/summary/readme) - POST /scripts → POST {HUB}/scripts/add (workspace_slug + content) - POST /flows | /apps | /raw_apps → corresponding hub endpoints - POST /scripts/:ask_id/recording, /flows/:flow_id/recording → recording uploads Auth uses HUB_DEV_TOKEN env var (dev shortcut). All bodies are serde-typed; the helper forward_to_hub centralises the HTTP call. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): wire frontend to backend hub proxy Replaces the mocked deploy flow with real backend calls: - confirmBundle() POSTs /hub/publish_draft with sanitized slug, name, summary and readme. - deployAll() pushes selectedItems one by one via pushItem(), fetching the live content (Script/Flow/AppService + raw_apps get_data) before forwarding to /hub/{scripts,flows,apps,raw_apps}. - saveRecording() builds the replay-shaped payload expected by the Hub (initial_job + events with type: 'CompletedJob') and POSTs to /hub/{scripts,flows}/{hub_id}/recording. - Adds bundleSummary state + TextInput in the drawer. Hub item ids (ask_id / flow_id) returned by the create calls are cached client-side to wire later recording uploads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(hub-publish): add /resources proxy route Forward workspace resource stubs (path + type) to the hub's /workspaces/{slug}/resources endpoint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): auto-detect resource dependencies from selection Derive resource dependencies from the $res:/res:// references in the selected scripts/flows/apps instead of a manual resource list, sync them as empty stubs, and show them read-only (chip per type, hover for path + which items use it). Aborts item publish if dependency sync fails to avoid broken fork references. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): add project-bundle closure + path-rewrite logic Pure, unit-tested module (projectBundle.ts) backing the "project = folder" Hub bundle: - extractScriptRefs / extractFlowRefs / extractAppRefs: structural detection of $res: references (code, static step inputs, script-by-path), hub refs classified separately. - classifyPath / buildPathMap: relocate external u/.. and f/other/.. paths under f/<slug>/, with deterministic _2/_3 collision suffixes. - rewriteContent / rewriteFlowValue / rewriteAppValue: rewrite every ref to its relocated path, leaving hub/.. untouched. - buildProjectBundle: walk the transitive closure of a seed selection (scripts pulled in recursively, resources pulled as stubs), returning the rewritten items + resource stubs + unresolved list. 14 vitest cases cover classification, extraction, collision suffixing, partial-match safety, deep-clone, and the closure orchestrator. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): publish as relocated project bundle + resource drawer - deployAll now builds a self-contained project bundle (buildProjectBundle), pushing resource types, empty resource stubs at relocated f/<slug>/ paths, and the rewritten items — so a fork's references resolve inside the project. - Resource-dependency detection is unified on the same bundle: the UI list (dependencyTypes) is derived from the bundle preview, guaranteeing what's shown matches what's pushed. Removes the duplicate in-component detection (extractResRefs/refsForItem/resolveResourceSet/typeForResource). - Input-type deps (schema format: resource-<type>) are synced as types and conventional f/<slug>/<type> stubs alongside hardcoded ones. - Replaces the hardcoded-path warning/fix/block machinery with a read-only "Resource dependencies" drawer: per-type usages tagged input vs hardcoded path, with an info popover explaining the portability tradeoff. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): gate hub publish endpoints + harden trigger detection - Add ApiAuthed + require_admin to all hub publish handlers; previously any workspace-authenticated session could trigger Hub-side writes attributed to the URL workspace via the shared HUB_DEV_TOKEN. - Track per-kind trigger fetch failures (triggerLoadErrors) so an EE-gated or transiently failing trigger service no longer silently maps to "0 triggers"; UI surfaces an amber badge listing the missing kinds and a toast warns the operator before publish. - Add workspaceLoadSeq cancellation so the parallel loadWorkspace + loadTriggers stop bleeding stale data when the workspace switches mid load. - Drop the silent effectiveSlug fallback to sanitizeSlug(hubName) when the Hub response can't be parsed; abort the publish instead so items don't land under a slug the Hub never locked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): thin /triggers proxy to forward trigger bulk-sync to Hub Mirrors the existing /scripts, /flows, /apps thin proxies. Forwards { triggers, workspace_slug } to Hub's POST /workspaces/[slug]/triggers bulk-replace endpoint, with the same require_admin + HUB_DEV_TOKEN guardrails. Lets the frontend push trigger stubs in a single round-trip after the items they reference have landed on the Hub. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): push trigger stubs as the final bundle step After scripts/flows/apps land on the Hub, pushTriggers() builds a relocation map for the trigger paths, strips operational metadata (workspace_id, edited_by/at, enabled, last_*/captured_*, capture data, error_handler_path/args, permissioned_as*) from each config, resolves script_ask_id / flow_id via the hubItemIds map produced by step 3, and POSTs the whole set to /api/w/:wsp/hub/triggers. Triggers whose runnable didn't publish are skipped with a warning rather than emitted as broken stubs. Also drops the per-kind trigger-load error surfacing: feature-gated services (Kafka, NATS, ...) 404 on instances that don't enable them, and the banner was lighting up on every load for nothing. Errors are swallowed silently again, matching the pre-review behaviour. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(hub_publish): rename Hub-facing fields and URLs from workspace to project Matches the windmillhub rename: every body now carries `project_slug` instead of `workspace_slug`, the draft creation forwards to `/projects`, and the resource_types/resources/triggers proxies hit `/projects/{slug}/...`. `HubWorkspaceBody` becomes `HubProjectBody`. The instance-side `Path(workspace)` extractor and the `workspace` URL parameter stay because that's still the source tenant's identifier. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ui(deploy-to-hub): user-facing rename from "workspace" to "project" The Hub-deploy surface now talks about *projects* (the bundle published to the Hub) instead of *workspaces* (which still means the source tenant). Tab is "Publish project", header copy mentions "project", the Hub URL in the breadcrumb points to /projects/<slug>, payload field is `project_slug`. Internal state names (`workspaceItems`, `workspaceStore`, `WorkspaceService`, …) stay — they refer to the instance workspace the items are read from, which has not been renamed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * ui(deploy-to-hub): open-in-tab affordance on each dependency and trigger Adds a small ExternalLink icon at the far right of every row in the Resource dependencies drawer (script / flow / app / raw_app) and the Triggers drawer (per trigger kind, opens the matching list page — /routes, /schedules, /websocket_triggers, /kafka_triggers, …). Both buttons open in a new tab scoped to the current $workspaceStore. Sized to sit after the role badge so the dominant signal (input vs hardcoded path, script vs flow) stays read first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): proxy raw app embed to the Hub Add POST /w/{workspace}/hub/raw_apps/{id}/embed forwarding to the Hub so a shared (public) raw app's external_embed_url can be set/cleared. null is forwarded (not skipped) so unpublish clears the embed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): bundle raw apps, share live iframe, folder-scoped bundles - Detect modern raw apps (app table, raw_app=true) and push them to the Hub as raw apps: fetch source files + runnables + the compiled bundle (via the latest-version bundle secret) and shape them into the raw payload RawAppView expects. Fail loudly when no compiled bundle exists. - Capture the Hub id for raw apps and wire "Share as iframe"/"Unpublish" for them (post-bundle, like recordings); re-sync the embed on re-bundle for already-public apps. Factor the publish/unpublish flow into setAppShared + pushRawAppEmbed helpers. - Scope bundles to a single required f/<folder>/ (Select instead of MultiSelect) so relocated paths stay predictable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub): send item path when publishing a project to the hub Include each item's newPath in the script/flow/app/raw_app publish payloads so the hub can store the relocated Windmill path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub): accept path on publish and proxy project export Add an optional path field to the publish bodies and a GET /projects/{slug}/export route that proxies the hub export (admin-only, authenticated with HUB_DEV_TOKEN). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(projects): add project install page New /projects/install page pulls a hub project's export and re-creates it in the selected workspace. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): let user pick target folder on project import Add a FolderPicker to the project install page (defaulting to the project slug, with create-new-folder support) and retarget every `f/<slug>/` prefix in the bundle — item paths, $res:/script refs, schedule runnable paths — to the chosen folder in one pass. Ensures the target folder exists before creating items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix wording * fix(hub-publish): bind Hub publish/export to the trusted workspace via source_id Hub publish endpoints ignored the {workspace} path and addressed the Hub project purely by client-supplied project_slug, forwarding with an instance-wide HUB_DEV_TOKEN. Any workspace admin could mutate or export another workspace's Hub project by passing its slug. Stamp the server-trusted workspace from the path onto every forwarded request as source_id (body for mutations, query param for export) so the Hub can enforce that the targeted project belongs to the calling workspace. Requires the matching Hub-side source_id ownership check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): reset draft/publish state on workspace switch The workspace-switch effect only reset load-derived state, so phase, draftItems, recordings, hub/bundle metadata, hubVersion, deploymentStatus, effectiveSlug and hubItemIds survived a switch — a draft built in one workspace could publish its items/slug under the next workspace's auth. Reset the full publish session on switch. Also drop explanatory comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): pull sub-flows referenced by type: flow steps into the bundle extractFlowRefs only emitted refs for type: script steps, so a flow calling an external sub-flow by path was never followed and the published project was silently incomplete. Add a 'flow' RefKind, emit it for type: flow steps, recurse on it in buildProjectBundle, and rewrite its path on relocation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): fall back to slug when import folder is whitespace-only (folderName || slug).trim() let a whitespace-only folder bypass the slug fallback and trim to an empty target, producing invalid f//... paths and a failed import. Trim first, then fall back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-publish): validate project slug before interpolating into Hub path slug/project_slug are caller-controlled and were interpolated straight into the Hub request path; a crafted value (e.g. ../../admin) could reach an unintended Hub endpoint after URL normalization. Validate against the frontend charset (lowercase alphanumerics + hyphens, 3-50 chars) in the four handlers that put the slug in the path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): use ?tab query param to link to the Apps settings tab The "Edit in Workspace settings → Apps" link set window.location.hash, but the settings page derives the active tab from ?tab=..., so the link was a dead affordance. Navigate with goto('?tab=default_app') instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): point Open in Hub link at the project slug, not the workspace hubSlug was derived from $workspaceStore, so the Open in Hub link and badge used the workspace id instead of the published project slug — navigating to the wrong (or nonexistent) Hub project. Derive hubSlug from the actual project slug (effectiveSlug, falling back to sanitizeSlug(hubName)). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Sync hub to instance * feat(deploy-to-hub): rehydrate project state, wire review flow, bundle trigger resources - Rehydrate the publish panel from the Hub by source_id on load (phase, slug, metadata, items, hub ids, recordings) so refresh no longer loses the draft. - Map Hub project status to the draft/under_review/live phase; submitForReview now persists to the Hub instead of a local stub; drop the unused v{n} version display (status is the source of truth). - Send source_path (original workspace path) per item for recording round-trip. - Detect resources referenced by triggers, add them to the bundle closure (extraResourcePaths) so they appear in dependencies, get stubbed/relocated, and rewrite the trigger config path via the full bundle pathMap (no leaked private path); show trigger usages in the dependency drawer. - Review fixes: Array.isArray guards on trigger topic/subject lists; snapshot relevantTriggers in deployAll to avoid a mid-deploy folder-switch race; index-key the usage list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): keep item summary on rehydrated draft, drop placeholder diff button Rehydrated draft items now carry their summary (from the Hub) so step 2 shows the summary like step 1 instead of falling back to the path. Remove the "Diff vs submitted" button: it only toasted add/remove counts with no view, which read as broken. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): New draft returns to the folder-picker step instead of erroring In the live phase the folder picker is hidden, so startNewDraft's selectedFolder guard always failed with "Pick a folder..." and the user had no way to pick one. Now New draft goes back to step 1 (predeploy) with the project's folder pre-selected (inferred from the item paths) so the user can re-bundle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub_publish): return 500 not 400 when HUB_DEV_TOKEN missing * fix(deploy-to-hub): keep internal subfolder paths identity-mapped when bundling * fix(projects-install): never overwrite existing resources; isolate invalid raw app json * fix(deploy-to-hub): route raw_app to apps_raw/get and guard openRecord schema race * refactor(hub_publish): extract hub_token helper, drop duplicated env lookup * refactor(projects-install): route raw-app and unsupported-trigger failures through record() * fix(deploy-to-hub): refresh review status from Hub and use configured hub base url * fix(hub_publish): return 400 not 500 when HUB_DEV_TOKEN is unset Missing config is a client/config error, not a server fault. Restores the BadRequest class lost when hub_token() was extracted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub_publish): scope Hub projects per folder via workspace:folder source key * feat(deploy-to-hub): publish per-folder projects from the Folders page * ui(deploy-to-hub): move phase CTA to the top-right header * Fable review * feat(hub_publish): forward the caller's token to the Hub instead of HUB_DEV_TOKEN * style(windmill-api): cargo fmt fallout in build.rs and lib.rs * Nit fixes * Nit fix * fix: structural project-ref rewrite and deploy-to-hub state fixes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: deterministic draft phase fallback when post-deploy rehydrate fails Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): skip EE-only native trigger calls on CE to avoid console 404s Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(deploy-to-hub): fork all trigger kinds on project install The project install (fork) flow recreated only schedule triggers and rejected every other kind with "not supported yet". Recreate all trigger kinds instead, imported disabled (enabled: false → mode disabled). Kafka, NATS, SQS, GCP and Azure require an Enterprise license, so they are gated behind enterpriseLicense and reported as "requires Enterprise" on CE rather than firing backend calls that 404. http, websocket, postgres, mqtt and email are recreated on CE. The kind-specific config (with retargeted resource paths) is spread into the create body; explicit path/script_path/ is_flow/summary/enabled win over it. Also carry the schedule summary through. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): list CE trigger kinds without an Enterprise license loadTriggers wrapped http, websocket, postgres, mqtt and email list calls in eeList, so on CE (no enterpriseLicense) they resolved to [] and never made it into deploy state — those triggers silently disappeared from the Hub publish set. Only Kafka, NATS, SQS, GCP and Azure are EE; switch the CE kinds back to safeList so they are always listed and published. Mirrors the EE gating used on the project install (fork) side. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): reload triggers when EE license hydrates late loadTriggers captures enterpriseLicense at call time and the main reload $effect only depends on workspace/folder, guarded by lastLoadedKey. When the license store hydrates asynchronously after loadTriggers already ran, the EE trigger kinds (kafka/nats/sqs/gcp/azure) stay empty until the workspace or folder changes. Add a dedicated $effect that re-fetches triggers on the license false→true transition, mirroring the sidebar's license-race handling. prevHadLicense is seeded from the current value so a license already present at mount doesn't trigger a redundant reload. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): token loadTriggers so a late EE reload can't be clobbered The license-late reload calls loadTriggers with the same workspaceLoadSeq as the original license-less load, so the workspace guard alone lets both assign workspaceTriggers. If the earlier (EE-empty) request resolves last, it overwrites the newer license-aware result and the EE trigger kinds disappear again. Add a per-invocation triggerLoadSeq token and only let the latest load assign (and toggle triggersLoading), so a slow earlier request is discarded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): use mode 'disabled' for forked non-schedule triggers Non-schedule triggers expose `mode` (TriggerMode), not the deprecated `enabled` flag, in their create body. `enabled: false` happens to still map to disabled today via the backend's legacy BaseTriggerData field, but relying on a deprecated path is fragile. Set `mode: 'disabled'` explicitly so imported http/websocket/postgres/mqtt/native triggers stay disabled. Schedules keep `enabled: false` (NewSchedule uses the enabled flag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): snapshot workspace + key bundle by kind:path Address three P1 review findings: - Project install (fork) read the reactive `workspace` ($derived) across many sequential awaits, so a workspace switch mid-import could create the folder in one workspace and later items in another. Snapshot the target workspace once at the top of install(). - DeployToHub.deployAll re-read $workspaceStore after confirmBundle had already created the Hub draft bound to a specific workspace's source_id, so a switch during draft creation could publish items to a different workspace. Pass the workspace captured by confirmBundle into deployAll instead. - buildProjectBundle keyed its fetched/queued maps by bare path, silently dropping one of two distinct-kind items at the same path (script vs flow). Key by `${kind}:${path}` and derive item paths from the fetched values, keeping path relocation separate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(deploy-to-hub): condense comments Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): surface backend error body on failed project import record() only showed `e.message`, which for API errors is the generic status text ("Bad Request"). Prefer the ApiError `.body` (plain-text reason for Windmill 4xx) so a failed import reports the actual cause — e.g. a path or route_path collision — instead of a bare "Bad Request". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): close mid-request workspace-switch races Two follow-ups to the workspace snapshotting: - confirmBundle captured `workspace` but read selectedItems / relevantTriggers / hubSlug only inside deployAll, after the publish_draft await. A workspace switch during that request resets those to the new workspace, so deployAll would push the new workspace's items into the old workspace's Hub draft. Capture workspaceLoadSeq before the request and abort (with a toast) if it changed before publishing. - install() snapshotted `workspace` but still read the reactive `data` after the createFolder await; load() can replace `data` on a workspace switch, so retarget() could run against a different export than `folder` was derived from. Snapshot `data` up-front and use it throughout install(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(deploy-to-hub): guard stale load response and mid-publish status writes - install load() assigned `data`/`folderName` unconditionally, so a slow /export for an old ?hub= could overwrite a newer project after navigation. Add a load token + captured slug/workspace and only assign if still current. - deployAll wrote deploymentStatus/hubItemIds incrementally and only checked the workspace at the very end. Bail at the top of the per-item loop when the active workspace changed, so a mid-publish switch can't keep writing the old workspace's item statuses and Hub IDs into the new workspace's live view. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): generate and apply datatable migrations on project publish/install (#9977) * feat: add datatable_migrations table * feat: add route to run datatable migrations * feat: sync datatable migrations as .up.sql/.down.sql files * feat: add datatable migrate up/down commands and post-push run prompt * feat: add datatable migrate new command to scaffold migrations * feat: add datatable migrations management UI * feat: prompt to create migration on DDL in datatable SQL editors * feat: support running a single specific datatable migration * feat: view migration content, run single migration, fix stacked modal * feat: per-row revert button with out-of-order warning * fix: avoid migrations list flicker on refresh after an action * feat: generate initial datatable migration via pg_dump * fix: surface datatable migration API error details in toasts * fix: revert created migration if create-and-run fails to run * fix: include postgres error detail in migration run/rollback failures * feat: sync datatable migrations as files via the workspace export * refactor: move datatable migrations to migrations/datatable/ path * fix: drop redundant datatable_migration label in sync output * fix: exclude datatable migration sql files from script metadata generation * feat: run datatable migrations as user-permissioned labeled jobs * feat: reject invalid datatable migrations on sync push * feat: datatable migrate up/down default to all datatables, --datatable to target one * fix: surface postgres error detail when datatable migrations fail to run * chore: regenerate CLI docs for datatable migrate commands * feat: default new datatable migration to a BEGIN/END transaction template * fix: validate datatable migration name and datatable at the API boundary * fix: ensure detected DDL ends with semicolon when wrapped in transaction * fix: re-prompt instead of stripping DDL when new-migration modal is cancelled * feat: refresh datatable schema after running a migration from the SQL REPL * feat: record db manager DDL on data tables as migrations * feat: make datatable migrations opt-in per data table * fix: make migration view editor read-only so its code can scroll * fix: don't re-prompt DDL guard when creating a migration without running * feat: generate down migrations for db manager DDL (postgres) * fix: correct down migration for db manager alters (no double-wrap, serial) * feat: explain migrations purpose with a tooltip in the migrations modal * compare paeg * feat: add datatable_migration kind to workspace diff pipeline * chore: point ee-repo-ref at datatable_migration git-sync companion * fix: harden datatable migration version allocation and initial-migration bookkeeping, add tests * feat: deploy and run datatable migrations on workspace merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Refactor + handle datatable setting delete/rename * refactor: move datatable migration rename/delete cascade into module Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(windmill-utils-internal): bump to 1.7.1 for datatable migration deploy provider methods Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(db-manager): add Migrations button to top bar, make Refresh icon-only Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * BEGIN/END placeholder in down migration * feat: autofocus migration name input and flag it red when empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(datatable-migrations): allow non-admins to create/run/revert migrations, gate only opt in/out Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * border nits * refresh db manager schema on migrations * BEGIN/END scaffold in CLI * feat(cli): push local datatable migrations before running on migrate up Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: flag invalid migration name with red border, not just empty Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop random slug from auto-generated migration names Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: offer revert-and-delete when deleting an installed migration Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: record fork merge as a migration when target datatable opts in * nit * clone migrations on fork * windmill-utils-internal * fix(datatable-migrations): serialize run/rollback with a per-db advisory lock Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(db-manager): fail closed when migrations-status check errors on DDL apply Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix generate_initial migration ordering comment to match code * chore(datatable-migrations): remove unused update_datatable_migrations endpoint Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: run DDL migration guard on the script editor Test button Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * split * ee-repo-ref * chore(frontend): sync package-lock with package.json (@emnapi deps) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(datatable-migrations): never resolve instance credentials into migration job args datatable_database_arg eagerly resolved instance data-table credentials (including the shared instance-wide Postgres password) and passed them as the migration job's plaintext `database` arg, landing in v2_job.args. Since the run route has no admin gate, a non-admin could run a migration and read args.database to recover the password, granting cross-workspace psql access to all instance data-table DBs. Pass a `datatable://<name>` reference for both resource-backed and instance data tables instead; the pg executor already resolves it to real credentials server-side at run time, so nothing sensitive is ever stored in the job args. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * fix: handle dollar-quoting and comments when splitting SQL statements * feat: deploy datatable migrations on merge with explicit opt-in error * fix(frontend): sync package-lock with npm 11 peer-dep resolution npm ci failed with 'Missing: @emnapi/core@1.11.2 / @emnapi/runtime@1.11.2 from lock file'. @napi-rs/wasm-runtime declares @emnapi/core|runtime ^1.7.1 as peerDependencies while @rolldown/binding-wasm32-wasi pins them to exactly 1.10.0. Newer npm (bundled with node 24 in CI) installs the peer deps at the highest match (1.11.2) alongside rolldown's nested 1.10.0, so the ideal tree needs both versions; the committed lock only had 1.10.0. Regenerate the lock with npm 11.18 so it carries both 1.11.2 (top-level, for the peer deps) and 1.10.0 (nested, for rolldown's pin). Verified npm ci passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit npm publish * fix: fail closed on migrations-status error in fork schema merge * nit CI emnapi/core version * prevent initial_datatable_migration if migrations already exist * fix(datatable-migrations): validate persisted data table names as path segments edit_datatable_config only validated rename segments, not the actual settings.datatables keys, so a data table could be saved directly under a name like '..' or one containing '/'. Since new tables default to migrations_enabled = true, generate_initial_datatable_migration would then insert a migration row and the sync export would build migrations/datatable/<name>/... paths from that name, producing malformed or directory-escaping export paths. Validate every persisted data table name in edit_datatable_config (alongside the existing rename checks) and add validate_datatable_path_segment to generate_initial_datatable_migration for defense in depth. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope datatable _wm_migrations by data table and cascade renames/deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(system_prompts): resolve nested local command groups in CLI docs generator The CLI docs generator anchored on the first `new Command()` in a file and never resolved locally-defined command groups passed as `.command("name", localCmd)`. For datatable this flattened the nested `migrate` group: it emitted `datatable new/up/down` plus a bare `datatable migrate`, and mislabeled the datatable command with the migrate group's description. jobs was broken the same way (its description was pull's, and pull/push rendered empty). Anchor block extraction on the `export default`ed command, recurse into locally-defined `const x = new Command()` groups mounted as subcommands, and render nested sub-subcommands. Regenerated docs now show `datatable migrate new/up/down` and `jobs pull/push` with their real options. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: drop unreleased _wm_migrations legacy-upgrade handling Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: return datatable migration SQL from getItemValue for the diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(frontend): use windmill-utils-internal 1.8.2 for migration diff drawer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * nit * fix: handle datatable migration renames on push and dedupe timestamps * fix: reject rewriting an already-applied datatable migration on upsert Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): add missing @emnapi/core and @emnapi/runtime lockfile entries Resolves npm ci EUSAGE failure: the optional cpu:wasm32 @rolldown/binding-wasm32-wasi declares deps on @emnapi/core@1.11.2 and @emnapi/runtime@1.11.2 that had no resolved lockfile entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): datatable migrate up/down default to main datatable, not all Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail closed when applied status unreadable on datatable migration rewrite Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: surface full error detail in Database Manager DDL/query errors * "See migration" button in the toast * feat: add Enter shortcut to Create-a-migration in the DDL guard * fix(frontend): warn before running a newly-created datatable migration out of order The row-level Run action warns when earlier migrations are still pending, but the create-and-run paths ran a just-created migration with `only` directly, applying it ahead of older pending migrations without that confirmation. Reuse the same "Run migration out of order" confirmation across all create-and-run paths via a shared helper (datatableMigrationUtils): - NewDataTableMigrationModal "Create and run" (and the DDL guard path) - DatatableSchemaDiff fork→parent merge - dbOps schema ops (DB manager create/alter/drop) — the pure factory throws a MigrationRunCancelled sentinel on decline, which DBTableEditor treats as a silent cancel Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep renamed datatable migrations visible in compare view * fix: record per-migration deployment on datatable migrations disable * fix(cli): run deployed datatable migrations after workspace merge The merge command upserted datatable_migration definitions into the target workspace and reported the item as successfully deployed, but never ran the migrations. For forked datatables backed by separate databases, this left the target schema unchanged until someone manually ran `wmill datatable migrate up`, while the CLI reported a successful merge. Collect the datatable migrations deployed (not deleted) into the target and, after the deploy loop, offer to run them via the existing offerToRunNewMigrations helper — the same post-deploy run prompt the push/sync path uses (interactive only; `--yes`/non-TTY skip the mutating run, matching push behavior). Export parseDatatableMigrationDeployPath so the merge path can parse the deployed items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): serialize datatable migration edits/deletes with the run lock A migration run snapshots a migration's code_up from datatable_migrations and only records its version in the data table's _wm_migrations after the job succeeds. upsert_datatable_migration checked _wm_migrations before allowing an edit but took no lock, so a concurrent edit could read "not applied yet", rewrite code_up/code_down, and then the in-flight run would record the version for the old SQL — leaving _wm_migrations pointing at SQL that was never applied (migrate up then skips it; rollback runs a down that doesn't match). Serialize definition rewrites and deletes with the same per-database advisory lock the run/rollback paths use: - Factor the connect+advisory-lock into lock_datatable_migration_runs and the applied-versions read into read_applied_versions_on_client. - run_datatable_migrations now snapshots the definitions AFTER taking the lock, so code_up can't change between snapshot and version-record. - upsert (when changing an existing def) and delete take the lock across the applied-check and the write; delete now rejects deleting an already-applied migration (would orphan its _wm_migrations record), symmetric with upsert. Both fail closed if the data table database is unreachable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): stack the out-of-order migration confirm above the DB editor preview Creating a table on a migrations-enabled data table opened the DB table editor's "Confirm running the following" preview modal, whose confirm triggers applyDdl, which then asks for out-of-order confirmation. Both are ConfirmationModals with a hardcoded z-[9999]; the out-of-order one lives in DBManagerContent (mounted before the editor), so it rendered behind the still-open preview modal. Add an optional zIndexClass prop to ConfirmationModal (default z-[9999], backward-compatible) and give the DB-manager out-of-order confirm z-[10000] so it stacks on top. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): generate and apply datatable migrations for projects Detect datatable assets in a project's scripts/flows/raw apps when publishing to the Hub, generate a best-effort CREATE TABLE migration per data table from the source workspace's live schema, and let the publisher edit/toggle them in the bundle drawer. On import, offer to run the shipped migrations: recorded (datatable_migrations + _wm_migrations) when the target data table opted into migrations, otherwise as a one-off preview job. Missing target data tables are surfaced and skipped. - backend: POST /hub/migrations proxy forwarding to the Hub - frontend publish: projectMigrations.ts detection + generation, new "Data table migrations" section in DeployToHub - frontend import: run/skip modal + missing-datatable confirmation - extract pure SQL-gen from DatatableSchemaDiff.svelte into datatableSchemaSql.ts so plain .ts modules can import it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): close datatable migration table set over foreign keys Pull a referenced table's FK targets into the generated migration transitively, so it creates every table it references (ordered by FK dependency), and drop any FK whose target still isn't in the set so the generated SQL always runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): show Data table dependencies in the publish view Detect data table usage off the predeploy bundle preview and surface it as a "Data table dependencies" summary right after "Resource dependencies", mirroring how resource types and triggers are shown. The editable migration itself stays in the bundle drawer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): explain un-generated migrations with SQL comments When a table can't be found in the schema, a data table is referenced as a whole, or the schema can't be loaded, write a `--` comment describing the problem into the migration instead of leaving it blank. Partial migrations keep the CREATE TABLEs that did generate and comment the rest; comment-only migrations stay disabled. The bundle drawer now always shows the SQL box so those comments are visible and editable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): review/edit migrations on import + rollback down migration Replace the plain "run migrations?" confirmation with a review drawer that previews each runnable migration, lets the user edit the SQL and toggle which to run, before the import proceeds. When recording an imported migration, also record a down migration (DROP TABLE of the created tables, in reverse order) derived from the up SQL, so it can be rolled back; the derived rollback is previewed in both the publish bundle drawer and the import review drawer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * feat(hub-projects): editable Up/Down Monaco editor for migrations Replace the plain textarea with a Monaco SQL editor split into Up/Down tabs. The down migration is now generated once as best-effort (DROP TABLE in reverse creation order) and is fully editable — no longer parsed back out of the up SQL. The down is threaded through publish → Hub → import (new project_migration.sql_down) and recorded as code_down when an imported migration is applied. - projectMigrations: GeneratedMigration.sql_down generated from the table set - MigrationSqlEditor.svelte: shared Up/Down tabbed Monaco editor (re-keyed on regeneration since Monaco ignores external code changes) - DeployToHub + install review drawer use it; sql_down pushed/applied - backend: PublishMigrationBody carries sql_down Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): generate CREATE TABLE IF NOT EXISTS for project migrations The FK closure pulls a referenced table's parents into the same transaction (e.g. `orders` drags in `customers`); those shared parents often already exist in the target, so a plain CREATE TABLE aborted the whole migration on the first collision. Emit CREATE TABLE IF NOT EXISTS for project migrations (via a new opt-in flag on generateMigrationSql, leaving the schema-diff behavior unchanged) so a pre-existing parent is skipped instead of failing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): key FK ordering by schema-qualified table name orderByFkDependency keyed its dependency graph by bare table name (and resolved FK targets with .split('.').pop()), so two same-named tables in different schemas collapsed and one was dropped from the ordered set and never created. Key by schema.table like the rest of the pipeline, resolving FK targets through resolveTable. Also let resolveTable fall back to the bare table name when a schema-qualified ref's schema doesn't match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): comment out generated down-migration DROP statements The generated down migration listed DROP TABLE for every table in the FK closure, including shared parent tables that may have pre-existed in the target — a rollback could drop a table the project never created (data loss). Emit all DROP statements commented out with a note, so nothing is dropped by default; the publisher uncomments the tables this migration actually owns. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): disable Import button during migration review planMigrations awaits the review / missing-datatable modals before setting installing = true, so the Import button stayed enabled during review and a second click launched a concurrent install() (second review drawer, duplicated item creation). Track a planningMigrations flag, disable the button on it, and early-return install() if already installing or planning. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): toast when migration generation fails regenerateMigrations cleared the drafts on error, showing "No data table usage detected" — indistinguishable from a genuine schema-load failure. Add a toast on the catch so the publisher can tell the two apart. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): honor cancel on the missing-data-table warning planMigrations awaited missingDatatableModal.ask() but ignored its boolean, so cancelling the "some data tables are missing" warning still proceeded with the import — the cancel affordance did nothing. Show the warning first and abort the whole import when the user cancels (planMigrations returns null; install() early-returns), so they can create the data table(s) and re-run. Confirming still imports without the missing migrations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): detect data tables from low-code app DB-table config Low-code apps don't carry a persisted asset list, but the DB-table component declares its data table and table explicitly: a `oneOf` `type` config with `selected === 'datatable'` holding `datatable://<name>` and the table. Walk the app value for those configs so an app that reads a data table is picked up by the Data table dependencies detection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "feat(hub-projects): detect data tables from low-code app DB-table config" This reverts commit 9c43ebd512. * fix(hub-projects): detect data tables from full-code apps' declaration Full-code (raw) apps explicitly declare the data tables/tables they use in value.data.tables (refs like main/customers or main/schema:table), which the "Data table dependencies" detection missed — it only looked at inline-script assets. Read the declaration via extractDataConfig/parseDataTableRef. The bundler previously dropped value.data (kept only files + runnables); include it so detection sees it and the imported app keeps its declaration, and pass it through on import. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): recompute app policy on project import Apps imported from a Hub project were created with an empty triggerables_v2 policy, so running any inline component script failed at runtime with "Path rawscript/<sha> forbidden by policy". The policy is computed client-side on deploy and stored verbatim by the backend, and import skipped that step; retargeting also rewrites inline-script content (changing its sha), so a copied policy would not match either. Recompute the policy from the retargeted value at import, mirroring the deploy path: updatePolicy for grid apps, updateRawAppPolicy for raw apps, defaulting execution_mode to publisher. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit fix --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): retarget plain trigger resource paths on import Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(hub-projects): reset migration drafts on workspace/folder switch Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(hub-projects): bundle http auth resources, pin drafts during deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(hub-projects): make generated data table migrations idempotent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reapply "feat(hub-projects): detect data tables from low-code app DB-table config" This reverts commit 112844deea. * fix(hub-projects): create all tables before FK constraints in migrations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(hub-projects): reset install state when the hub slug or workspace changes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "Reapply "feat(hub-projects): detect data tables from low-code app DB-table config"" This reverts commit 14abefb4f6. * fix: dedupe args state duplicated by main merge in AssetGraphDetailsPane Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(deploy-to-hub): extract session class keyed by workspace+folder All DeployToHub state and async operations move into DeployToHubSession (deployToHubSession.svelte.ts), an immutable-(workspace, folder) state class. A workspace/folder change replaces the instance and remounts the UI via {#key} instead of manually resetting ~20 state vars, and in-flight async work writes to the discarded object instead of racing the new scope. The workspace-scoped seq counters (workspaceLoadSeq/triggerLoadSeq for lifecycle, migrationsSeq) collapse into a dispose flag plus intra-session tokens only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqqWYQR46tcunvPVRfidZS * refactor(triggers): single shared module for all-kind workspace trigger listing TRIGGER_KINDS (badge/route/note/resourceField/eeOnly + list call), listAllWorkspaceTriggers, triggerResourcePath, stripTriggerConfig and triggerDetails move to $lib/components/triggers/workspaceTriggersList.ts, so EE-license gating per trigger kind is declared once instead of being re-decided at each call site. DeployToHubSession consumes it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WqqWYQR46tcunvPVRfidZS * refactor(hub-publish): route every endpoint through one validation choke point HubPublishCtx (a FromRequestParts extractor) is now the only way a handler reaches the Hub: it performs the admin check, resolves and validates the workspace:folder source key, and carries the forwarded token — a new endpoint cannot skip any of it. Project slugs become a ProjectSlug newtype whose only constructor is validating deserialization (body field or path segment), so every slug that reaches a Hub URL or payload is valid by construction; the previously unvalidated slugs in publish_draft/scripts/flows/apps/raw_apps/ embed/recording bodies are now checked too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * refactor(hub-projects): shared bundle format module + per-kind project installer The Hub export format (types + retargetProjectExport/buildRetargetMap) moves into projectBundle.ts so publish and install share one definition, with unit tests for retargeting. projectInstall.ts owns the import: one importer per item kind with per-item error capture, and trigger creation goes through createWorkspaceTriggerDisabled in the shared trigger module, which encodes the per-kind disable semantics (schedules use enabled:false, everything else mode:'disabled') and EE gating once. The install page shrinks to orchestration and UI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): AMQP kind, trigger handler bundling, import containment Review-round fixes: register the AMQP trigger kind (CE) in the shared registry so it lists/bundles/imports like every other kind; stop stripping error_handler_path/args from trigger configs and bundle + relocate handler runnables (including schedules' script|flow-prefixed on_* refs) with the project; resolve full schedule rows on listing (listSchedules is slim) and spread the exported config on import so cron_version, retry, handlers and no_flow_overlap survive; refuse per-item any export path that escapes the selected f/<folder>/ target; and gate the install page's results/done writes on the load sequence so a stale import can't mark a newly loaded project as imported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): schedule config hygiene and complete handler bundling Strip email/is_draft/paused_until from exported trigger configs (the full schedule row carries owner and runtime state that must not reach the Hub); bundle and relocate dynamic_skip handler scripts (schedule creation refuses a missing one, so an unrelocated path breaks the import); exclude and report a schedule whose detail fetch fails instead of silently exporting the slim row with default behavior; and seed migration detection with the same handler-augmented item set as deployment so data tables used only by bundled handlers get their migrations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(deploy-to-hub): single guarded publish path, gated on trigger load publishBundle() owns draft creation + deployment under one synchronously-set deploying flag, so a double-click can't start two interleaved publishes, and it refuses to run while triggers are still loading — snapshotting an incomplete relevantTriggers list would permanently omit triggers, their handlers and handler-only migrations from the draft. The bundle CTAs disable while trigger discovery is in flight. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): block publish on failed trigger discovery, strict schema-qualified table resolution listAllWorkspaceTriggers now distinguishes a feature-gated 404 (kind not compiled into the instance — legitimately empty) from a real listing or detail-fetch failure: failures are surfaced, recorded per kind, and the session blocks publishing with a visible retry until discovery completes cleanly, so an incomplete trigger snapshot can't be bundled silently. resolveTable no longer falls back to a same-named table in another schema when a qualified ref misses — that generated a migration for an unrelated table; the miss now produces the existing commented warning instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * docs(openapi): document the 15 hub publish proxy routes All /w/{workspace}/hub endpoints (draft/items/recordings/resource types/resources/triggers/migrations/export/submit/by-source) enter the public API contract with their body schemas derived from the serde structs, a shared HubProjectSlug schema encoding the slug validation, and passthrough text responses matching the proxy behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): bundle $res refs nested in trigger configs, nullable trigger payload fields Trigger dependency collection now scans the full stripped config for $res:/res:// tokens (schedule args, on_*_extra_args, error_handler_args — e.g. the built-in Slack handler's channel resource) in addition to the kind-specific resource field, so those resources enter the bundle path map, get relocated by rewriteTriggerConfig, export a typed stub, and show up in the dependency pane. PublishTriggerBody's summary/description/ script_ask_id/flow_id become nullable in the OpenAPI contract, matching what the publisher actually sends and the Rust Options accept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): flow preprocessor/env refs, no cloud provisioning on import, config containment Flow extraction and rewriting now cover preprocessor_module (walked like any other module) and flow_env $res: values, so those dependencies are bundled and relocated instead of keeping source-workspace paths. GCP/Azure triggers are refused at import with an actionable message — their create endpoints manage cloud subscriptions before storing the trigger, even disabled, so auto-creating them from an import could mutate external infrastructure. The import containment guard now also validates everything a trigger config binds to (kind resource field, handler runnables incl. hub/ refs, nested $res: tokens), closing the path where a crafted export binds a trigger to assets outside the chosen folder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * refactor(hub-projects): per-kind config allowlists from a full trigger-field audit Every trigger kind's boundary-crossing config is now an explicit per-kind allowlist (configFields in TRIGGER_KINDS), derived from a field-by-field audit of every create type: portableTriggerConfig replaces the blocklist and is applied on export AND import, so an upstream field addition is dropped until consciously admitted (no more email-style leaks) and a crafted export can't inject fields like permissioned_as into create calls. The audit also surfaced unbundled websocket runnables — $script:/$flow: URLs and initial-message runnable_result paths are now collected and relocated — and drops GCP/Azure provisioned identities (subscription ids, delivery_config with the source instance's endpoint) from exports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): bundle $res refs nested in JSON flow_env values The worker resolves $res: references inside nested JSON flow_env values (transform_json walks the full value), so extraction and rewriting now scan the env's full serialization instead of only top-level strings. Also: the install-page Enterprise note includes GCP/Azure, the trigger-discovery Retry button binds to the loading state so clicks can't stack requests, and extractTriggerConfigResourceRefs no longer splits rewriteTriggerConfig from its doc comment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): scope $script:/$flow: relocation to the websocket url field The runnable-url form is only meaningful in that one field; remapping it on every nested config string could corrupt a literal payload that happens to look like one (e.g. a websocket initial raw_message). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): nested static-transform refs, shared flow walk for migrations, top-level-only url remap Static input transforms accept arbitrary JSON and the worker resolves $res: refs nested inside them — extraction and rewriting now scan the full serialization, preserving the value's type. projectMigrations reuses projectBundle's allFlowModules instead of carrying its own module walk, so the preprocessor module (and any future module class) can't diverge between bundling and migration detection. The websocket $script:/$flow: url remap applies only at the config's top level, leaving nested url keys in args or handler payloads untouched. Schedule tag stays excluded by design (a source instance's worker-group name; a foreign tag queues jobs forever) — now documented in the allowlist contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): remap prefixed runnable refs only in their known config fields script/<path> and flow/<path> forms are now rewritten only in the top-level schedule handler fields (on_failure/on_recovery/on_success), joining the url field treatment — shape-based remapping on arbitrary strings could rewrite a literal payload that merely looked like a handler ref. Bare-path exact matches and $res: tokens remain position-independent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): abort stale-session imports after review, walk failure-module descendants Confirming a migration review whose project/workspace was switched away from now aborts with a toast before any write — previously the writes went to the old workspace with all feedback suppressed by the session guard. And allFlowModules puts the failure module in the root list so its nested children (loops/branches inside a failure handler) are expanded like every other module, for both bundling and migration detection. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): preserve app share state on Hub draft rehydration `rehydrateFromHub()` rebuilt `draftItems` from the Hub project payload, which carries only draft membership, so it dropped each app's `published`/`publicUrl` and app-table origin. Outside `predeploy` the UI reads `draftItems` exclusively, so reopening a draft showed a still-public app as unshared and removed its Unpublish control. Merge the live workspace-item state onto matching drafts after both `#loadWorkspace` and `rehydrateFromHub` (they race). Also gate the Share-as-iframe action on `canShareAsIframe`: legacy raw apps live only in the `raw_app` table, but that flow drives `AppService` (the `app` table) and fails with "App not found" for them, so the action is now hidden for legacy entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): stale-identity import guard, live publish state on drafts, iframe action gating The install session check now also compares the live slug and workspace to the captured ones — loadSeq only advances when a new load starts, so navigating away (workspace or ?hub becoming empty) previously left the stale migration review able to import into the captured workspace. Draft items are decorated with the live workspace item's shared-iframe fields (published/publicUrl/appTable) so a public app still shows as public after reopening a draft, settling reactively regardless of load order. The share-as-iframe action is offered only for apps and app-table raw apps — legacy raw_app entries have no AppService representation and the action could only fail. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * refactor(deploy-to-hub): drive share-state merge from one reactive derived The rebase left two parallel fixes for the same rehydration gap: an imperative mergeShareState call after each racing load, and a read-time derived. Keep the pure, tested mergeShareState as the single implementation and invoke it from the derived — no load-completion call sites to maintain, and the merge settles whichever load finishes last. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018DoHaGJdACgE7RvknRDAb6 * fix(hub-projects): block publish on unresolved refs; contain imported item refs Address two Codex findings: - Publish continued after `buildProjectBundle` reported unresolved references (a selected root or transitive runnable that failed to fetch, or a resource with no resolvable type), shipping a project whose items silently vanished or still pointed at the publisher's private source-workspace path. `#deployAll` now aborts before any Hub write when the bundle doesn't close, and the bundle drawer surfaces the unresolved list and disables "Create bundle". - `installProject` validated only each item's own path, so a crafted or incomplete export could place a script/flow/app inside the target folder while its `$res:`/script/flow reference stayed bound to an existing `u/...` or other `f/...` asset. Extract each item's live references and reject any that escape `f/<folder>/` (hub/ script refs allowed), mirroring the existing trigger-config containment check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): contain imported $var refs; dedupe unresolved list Follow-up to the publish-blocker and import-containment fixes: - `$var:` references (flow static inputs, flow_env, app values, and trigger config fields such as SQS queue_url) were not caught by the containment check, which only recognized `$res:`/runnable refs. Retargeting leaves them unchanged, so an export with `$var:u/admin/token` imported an item that resolves a variable outside the target folder under the runnable's permissions. Scan each imported flow/app/trigger for `$var:` tokens and reject out-of-folder ones. Scripts are skipped: `$var:` is resolved in job args, not script source. - `buildProjectBundle` stored bare paths in `unresolved` while keying missing items by kind:path, so a script and flow sharing a missing path produced a duplicate string. The new keyed unresolved list in the bundle drawer then hit Svelte's duplicate-key runtime error instead of rendering the publish blocker. Dedupe `unresolved` at the source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): contain $var/$jsonvar imports; retryable partial publish; iframe rollback Address four Codex findings: - `$jsonvar:` (secret JSON args) was not contained on import, and scanning the serialized flow/app for `$var:` tokens falsely rejected inline-code literals. The worker only substitutes a variable when an argument value *is* the reference (whole value, walking nested JSON), never a token embedded in code. Replace the token scan with a structural whole-value walk (`$var:`/`$jsonvar:`) and reject out-of-folder refs in flows, apps, and trigger config. Scripts carry no variable args, so they are skipped. - A partial publish (failed item/trigger/migration write) still transitioned to the submit-ready `draft` phase. Stay in the retryable `predeploy` state on any failure, keeping the failed items visible, so nothing incomplete can be submitted and re-publishing retries every idempotent write. - `#setAppShared` flipped a raw app public before checking its Hub item id or syncing the embed, so a missing id or a failed embed sync left the app publicly accessible while reporting failure. Validate the Hub target up front and roll the policy back if the embed sync fails. - `buildProjectBundle` could emit duplicate unresolved paths (a script and flow sharing a missing path), breaking the keyed publish-blocker render. Deduped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): full-set trigger sync; count iframe re-sync + URL failures Address three Codex findings, two of them refinements of the incomplete-publish gate and iframe-rollback fixes: - `#pushTriggers` returned early on an empty set, so re-deploying a project after removing all its triggers left the previous Hub triggers intact. Always post the trigger list (an empty one clears them), mirroring the migrations full-set sync. - A raw app's post-deploy iframe re-sync failure only toasted; it now increments `failures`, so a public app left with a stale embed keeps the draft out of the submit-ready phase. - `#setAppShared` skipped the embed and still returned success when the public URL couldn't be resolved, leaving the app anonymous with no usable link. It now rolls the policy back and throws when a share has no resolvable URL, alongside the existing embed-failure rollback (factored into one helper). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): count URL-less iframe re-sync; evict failed preview caches Two Codex findings, both refinements of earlier fixes: - The post-deploy iframe re-sync skipped a published raw app whose public URL was missing (URL resolution had failed) without counting it, so the re-bundle left the app public with a cleared Hub embed yet the draft still became submit-ready. Treat a published raw app with no resolvable URL as an incomplete publish and count it like a push failure. - The bundle-preview dependency caches memoized promises that resolve to undefined after transient item/resource fetch failures, so fixing or retrying a dependency could never clear `bundlePreview.unresolved` and the Create bundle button stayed disabled until the session was recreated. Evict a cache entry once it resolves to undefined so a later rebuild re-fetches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): keep Unpublish for a public app whose URL didn't resolve The iframe controls required both `published` and `publicUrl`, so an anonymous app whose public-URL lookup failed rendered as unshared with only a Share action and no way to unpublish. Branch the Public badge and Unpublish on `published` alone, gate the URL-dependent Open/Copy-iframe actions on `publicUrl`, and offer a Retry link that re-resolves the URL when it is missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(hub-projects): make $var/$jsonvar dependencies portable on import Variable references were neither retargeted nor materialized, so a published project that used a variable broke on import: a renamed-folder import rejected the containing item (the `$var:` kept the old folder prefix), and a same-folder import left the reference dangling (the target variable never existed). Treat variables like resource stubs, fully on the import side (their `$var:`/ `$jsonvar:` refs already travel inside the exported item values): - `buildRetargetMap` now also relocates the internal variable paths embedded in the export's flows/apps/triggers, and `rewriteContent` rewrites `$var:`/ `$jsonvar:` tokens (kind preserved) for any path in the map — so the publish map, which omits variables, is unaffected. - `installProject` creates an empty secret placeholder for each in-folder variable ref, conflict-safe via `existsVariable`, for the importer to fill. Values are never shipped. External refs stay rejected by containment. Custom resource-type definitions (the sibling finding) are intentionally left to the standardized official Hub resource types, so no schema import is needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): relocate $var refs structurally, never in inline code Routing variable retargeting through `rewriteContent` also rewrote `$var:`/ `$jsonvar:` tokens embedded in script source, inline rawscript, and serialized app strings, so an inert literal sharing a real variable's path was silently altered on a renamed-folder import — contradicting the whole-string runtime-reference rule. Relocate variables with a structural walk (`rewriteVarRefsInValue`) that rewrites only whole-string `$var:`/`$jsonvar:` values (the sole form the worker resolves), applied to flow/app/trigger values in `retargetProjectExport`; `rewriteContent` is back to `$res:`-only. Inline code literals are left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): relocate $var refs into the slug at publish Import-side retargeting assumed exported `$var:`/`$jsonvar:` refs already began with the project slug, but `buildProjectBundle` never relocated them from the source folder. Publishing `f/source_folder/...` as slug `my-toolkit` therefore exported `$var:f/source_folder/key`; import (fromSlug=my-toolkit) left it unchanged and containment rejected the item. Collect each item's runtime variable refs, feed them through the same path map that relocates items/resources into `f/<slug>/`, and structurally rewrite the whole-value refs — symmetric with the import retarget. The export is now slug-relative whatever the source folder, and inline-code literals stay untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(hub-projects): relocate trigger config $var refs at publish Item variable refs were relocated into the slug, but triggers publish through a separate path (`#pushTriggers` → `rewriteTriggerConfig`), which doesn't touch `$var:`/`$jsonvar:`. Publishing `f/source/...` under a different Hub slug left schedule args and other config refs pointing at `f/source/...`, and import containment then rejected the trigger. Collect each trigger config's whole-string variable refs (`#triggerVarPaths`), feed them through the bundle path map via a new `extraVarPaths` arg to `buildProjectBundle`, and structurally rewrite the config on publish. Symmetric with the item and import-side handling; the import retarget already relocated trigger vars. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(hub-projects): correct varContainmentViolation retargeting contract The comment claimed retargeting doesn't rewrite variable refs; it now relocates a project's own refs into the target folder, and containment rejects only those left outside it. Describe the current behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: hugocasa <hugo@casademont.ch> Co-authored-by: centdix <40307056+centdix@users.noreply.github.com> Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> Co-authored-by: Aldrin Jenson <aldrinjenson@gmail.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com> Co-authored-by: Diego Imbert <70353967+diegoimbert@users.noreply.github.com> Co-authored-by: Guilhem <guilhemlemouel@gmail.com> Co-authored-by: Diego Imbert <diego@windmill.dev>	2026-07-23 13:47:28 +02:00
Ruben Fiszel	1f912f4104	feat(ai): enable data pipelines in AI sessions with alpha notice (#10273) ... * feat(ai): enable data pipelines in AI sessions with alpha notice * test(ai): assert real pipeline instructions returned in session * fix(ai): await pipeline editor registration in open_preview to end build_pipeline_node race * test(ai): drop historical narration in pipeline instruction assertion * docs(ai): clarify S3 asset-URI storage forms and literal-key rule in pipeline prompt * docs(ai): document data_upload S3Object input mechanism in pipeline prompt * test(ai): guard open_preview pipeline tool-registration wait seam * feat(ai): report inferred asset lineage in build/edit_pipeline_node results * test(ai): assert open_preview(pipeline) awaits async handler registration * docs(ai): qualify S3Object as wmill.S3Object in pipeline prompt examples * fix(ai): report failure when a backgrounded session's pipeline tools never register * fix(ai): include output_kind-seeded outputs in build_pipeline_node lineage feedback * fix(ai): report only body-detected lineage, never the output_kind seed placeholder * fix(ai): include materialize target in lineage feedback, condense comments * docs(ai): use default-storage s3:/// in canonical pipeline read example	2026-07-23 12:55:59 +02:00
3 people	7fb8a2e390	fix(parsers): keep s3 asset path suffix verbatim to preserve storage distinction (#10241) ... * fix(parsers): keep s3 asset path suffix verbatim to preserve storage distinction Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01An2pTqSmqJd2XwnagvX4kM * package json * fix(pipelines): preserve named storage in generated TS/Python S3 URIs The TS/Python templates emitted `s3:///${s3Key(path)}`, stripping the leading slash and pinning the URI to default storage. For a named-storage asset path (`secondary/key`) that produced `s3:///secondary/key`, which resolves to the default storage with key `secondary/key`, dropping the named-storage dependency and reading/writing the wrong object. Emit the path verbatim after `s3://` (matching the DuckDB template) so a named-storage input/output keeps its storage; identical to the previous output for default-storage paths. Removes the now-unused `s3Key` helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(cli): align bun.lock parser versions with frontend The PR bumped windmill-parser-wasm-asset (1.749.0→1.753.0) and windmill-parser-wasm-regex (1.692.0→1.764.0) in package.json and the npm package-lock.json for both cli and frontend, but cli/bun.lock was left pinned to the old versions. Sync it so the CLI's wasm asset parser (used by localGraph inference) matches the frontend and deploy-time parser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-07-21 16:25:06 +00:00
3 people	68debab877	feat(triggers): add AMQP (RabbitMQ) trigger via lapin (#10230) ... * feat(triggers): add AMQP (RabbitMQ) trigger using the lapin library Fixes WIN-2214 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(triggers): defer AMQP cross-workspace deploy pending utils-internal publish Revert the amqp_trigger additions to the shared windmill-utils-internal TriggerDeployKind and the frontend cross-workspace deploy adapter: the frontend installs the published npm package, which lacks the new kind until a release is cut. AMQP create/edit/delete/list/sync/capture are unaffected (they use local types); only cross-workspace deploy/merge of AMQP triggers waits on the package bump. Also document the at-most-once ack in the consumer loop. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): address AMQP review — at-least-once ack, workspace cascade, contracts - ack AMQP deliveries only after successful dispatch; nack+requeue on failure - add ON DELETE CASCADE workspace FK so amqp_trigger rows are cleaned on workspace deletion (and the listener stops) - fix the /amqp_triggers/test OpenAPI body and add amqp_trigger to WorkspaceDiffRow.kind - register AMQP in the generated workspace trigger tool (create_trigger) - drop banned $bindable defaults on optional props in the config section - add build_uri unit tests (encoding, ports, vhost) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): stop AMQP poison-message loop and reconnect on transient drops Chaos testing against a live RabbitMQ broker showed the previous nack(requeue) + immediate re-poll spun a tight redelivery loop (~1000 critical-error reports/sec) on a poison message, and any connection blip permanently disabled the trigger (lapin has no built-in reconnect). - on dispatch failure: nack+requeue then stop consuming; the listener framework re-lists the trigger after its ping goes stale (~15s), backing redelivery off to that cadence instead of a tight loop (verified: rate dropped from ~1000/s to ~1 per ~26s, message preserved) - on connection/stream error: stop and let the framework reconnect instead of disabling; persistent failures are still disabled via get_consumer (verified: a forced connection close now auto-reconnects and resumes) - finish the AI create-trigger action wiring for AMQP: add amqp to CreatedResourceTriggerKind, the action-card registry, and the drawer registry so the result card renders and its "Open" action works Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP frontend registries and defer merge rows - add amqp to capturableTriggerTypes (so AmqpCapture mounts), the Runs jobTriggerKinds filter, and CLOUD_DISABLED_TRIGGER_TYPES - wire AMQP into global AI chat mode: TRIGGER_KINDS, the request union, writeTriggerSchema, triggerServices, and the draft adapter - stop emitting actionable AMQP fork-comparison rows (revert amqp_trigger from TRIGGER_OR_SCHEDULE_TABLES) since cross-workspace deploy is deferred until windmill-utils-internal is published — avoids a deploy that fails with "Unknown kind: amqp_trigger" - use design-system TextInput instead of raw <input> in the config section Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP session/draft registries and constrain prefetch - add amqp to the session-deploy, draft-compare, preview-router, and copilot workspace-item registries so AMQP drafts/deploys/nav/path resolution work - include amqp_count in the MoveDrawer attached-trigger rename warning - replace the raw prefetch <input> with a design-system TextInput bounded to an integer 1-65535 (backend u16) and block save on invalid values Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): make AMQP disconnect/reconnect consistent with the Kafka trigger lapin, like rdkafka, has no transparent reconnect, so the AMQP listener now mirrors the Kafka trigger's explicit reconnect loop instead of relying on the framework re-list (which disabled the trigger once get_consumer failed on a sustained outage): - get_consumer returns cheaply; consume owns a (re)connect loop that retries with a 30s backoff, reports a critical error every 10 failed attempts, and reports a recovered critical error once it reconnects — never disabling the trigger on a connectivity failure - a consumer/stream error breaks out to reconnect rather than disabling - dispatch failure still nacks+requeues (at-least-once) with a short backoff to avoid a tight poison-message loop, keeping the connection alive Verified against a live RabbitMQ broker: killing the broker keeps the trigger enabled and retrying (attempt N), and restarting it auto-reconnects (logs "reconnected after N attempts") and resumes dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): complete AMQP capture registries and constrain prefetch contract - add the 'amqp' case to triggerKindToTriggerType so opening the AMQP editor from a capture button no longer throws "Unknown TriggerKind: amqp" - register AmqpIcon in CaptureTable's icon map and add an AMQP entry to the script/flow CaptureButton menu - bound the OpenAPI prefetch_count to an integer 1-65535 (matches the Rust u16) and regenerate clients/prompts - require a non-empty exchange name when the exchange binding is enabled - build_uri: fall back to "/" on a blank vhost and bracket IPv6 hosts (+ tests) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(triggers): wire AMQP into pipeline graph, git-sync, and preprocessor types - asset_graph: discover attached amqp_trigger rows and emit an AMQP TriggerEdge so AMQP triggers render (and can be opened/deleted) on the data-pipeline canvas - frontend pipeline graph: add amqp to NativeTriggerKind, the add-trigger menu, node presentation, event-trigger set, annotation keywords, and the editor/service registrations - git-sync: add the amqp_trigger include pattern (+ test) so an AMQP git-sync deployment stages only its .amqp_trigger.* file, not an unrelated same-path object - preprocessor starters: add the AMQP event to the generated TS/Python/PHP trigger event types (kind/payload/exchange/routing_key/queue_name/redelivered/ delivery_tag) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): finish AMQP pipeline/parser wiring, prefetch validation, source lists - fix a stray edit that corrupted the pre-existing MqttTriggerEditor import ($lib/... path) in PipelineTriggerEditors.svelte - reject prefetch_count = 0 server-side in validate_config (RabbitMQ treats 0 as unlimited) and defensively skip basic_qos(0) in build_consumer (covers the capture path that bypasses CRUD validation) - recognize `// on amqp` in the canonical parser (TriggerSpec::Amqp) and add amqp to the CLI non-autorun/event-trigger sets so a pipeline cascade never runs an AMQP-only node as a manual root without an event - add amqp to the preprocessor intro lists and both pipeline AI instructions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(triggers): reject zero AMQP prefetch in all paths and finish guidance lists - extract a shared validate_amqp_options used by both CRUD validate_config and build_consumer, so capture configs (which bypass CRUD validation) also reject prefetch 0 instead of silently connecting with an unlimited buffer (+ unit tests for 0/1/65535/None) - add AMQP to the main script-writing preprocessor-sources prompt and the CLI triggers-skill guidance list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(triggers): de-duplicate AMQP prefetch comment and fix GET response text - keep the zero-prefetch rationale only on the shared validate_amqp_options doc; drop the redundant call-site comments - correct the getAmqpTrigger OpenAPI 200 description ("deleted" -> "retrieved") Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to aaa6cb89b05b76139252c64f057e53b94d12ac60 This commit updates the EE repository reference after PR #680 was merged in windmill-ee-private. Previous ee-repo-ref: 5da5fd65aca9594b2611837a52e4677b544b0380 New ee-repo-ref: aaa6cb89b05b76139252c64f057e53b94d12ac60 Automated by sync-ee-ref workflow. * chore(migrations): consolidate the four AMQP migrations into one The table and the three enum ADD VALUE statements (trigger_kind, job_trigger_kind, draft_kind) are one atomic feature. ALTER TYPE ... ADD VALUE runs inside the migration transaction on PG >= 14 (Windmill's minimum) since the amqp_trigger table doesn't reference those enum types, so they can share a single migration instead of four. Verified applying cleanly in a single transaction on a fresh DB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-21 15:10:29 +00:00
Ruben Fiszel	fd51d40f12	feat(pipelines): catalog declared measures and dimensions (#10190)	2026-07-21 07:34:44 +02:00
GuilhemandClaude Opus 4.8	8828341a2b	fix(frontend): scope session pipeline trigger editors to the session workspace (#10032) ... * fix(frontend): scope session pipeline trigger editors to the session workspace Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): hoist triggerWorkspace decl above GCP init-time getBaseUrl call Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): scope nested trigger pickers to the session workspace The triggerWorkspace resolver scoped direct trigger CRUD calls to the session's (forked) workspace, but nested pickers still defaulted to the nav `$workspaceStore`: in a fork session, resource lists/creation, variable creation, and path-existence checks ran against the parent workspace while save/delete targeted the fork — misleading options and false path-validation failures. Thread `wsId` into the nested controls of the 9 pipeline-canvas kinds: - `<Path workspaceOverride={wsId}>` (8 editors) — path + folder checks - `<ResourcePicker workspace={wsId}>` (6 config sections; add the resolver to MqttEditorConfigSection, which lacked `wsId`) - SQS `<VariableEditor workspace={wsId}>` — variable creation Also drop the per-site `wsId` rationale comment repeated across ~20 files; the invariant is documented once in triggerWorkspace.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): scope trigger runnable picker to the session workspace The runnable picker (`ScriptPicker`, reached via `TriggerRunnablePicker` and directly in the schedule editor) listed scripts/flows/apps from the nav `$workspaceStore` with no override, so a forked session offered the parent workspace's runnables when attaching a script/flow to a trigger. Add an optional `workspace` prop to `ScriptPicker` (defaults to `$workspaceStore` → no change for existing callers), pass it through `TriggerRunnablePicker`, and wire `wsId` from the 7 trigger editors that use it plus the schedule editor's 3 direct pickers. Completes the nested-picker workspace scoping; the excluded kinds (azure/http/websocket/native) keep their own ScriptPickers unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): scope trigger error-handler, folder default, and runnable actions Follow-up to the trigger-workspace scoping: three subtrees still read the nav `$workspaceStore` in a forked session. - ErrorOrRecoveryHandler (via TriggerRetriesAndErrorHandler in the 7 in-scope editors): add a `workspace` prop (defaults to `$workspaceStore`) and route handler lookup/schema, Slack/Teams settings, test jobs, and run links through it, so the error handler is resolved/tested/saved in the session workspace instead of A while the trigger lives in B. - useFolderDefaultPermissionedAs: accept an optional workspace getter so a `f/...` trigger's default permissioned-as is read from the session workspace, not the nav one (PermissionedAsLine passes `() => wsId`). - ScriptPicker actions: scope the View drawer (`getScriptByPath`) and `FlowPathViewer` to `effectiveWorkspace`, and carry `?workspace=` onto the Edit/View routes when an explicit override is set (the layout consumes the param, same mechanism as editInFork). The param is only appended when a workspace override is passed, so existing callers' links are unchanged. Also consolidate the repeated workspace-scoping comment in PipelineTriggerEditors (the invariant lives in triggerWorkspace.ts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): scope error-handler/schedule nested consumers to the session workspace Address CI Codex review on #10032 — the error-handler and schedule subtrees still had nested consumers reading the nav workspace: - ErrorOrRecoveryHandler: pass the resolved workspace to its own nested `ScriptPicker` (custom-handler list + View/Edit) and add a `workspace` prop to `ChannelSelector` (Teams channel listing); carry the acting workspace onto the "create from template" link. - ScheduleEditorInner: pass `workspace={wsId}` to the error/recovery/ success `ErrorOrRecoveryHandler` panels, `workspaceId={wsId}` to `WorkerTagPicker`, and the workspace query param onto the dynamic-skip template link. Template-link and picker overrides only diverge from `$workspaceStore` when a session override is set, so non-session callers are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 19:48:08 +02:00
Ruben FiszelandClaude Fable 5	f7efb646bf	fix(pipelines): live materialize/dataset editing — stale graph, phantom drafts, stale Save-all deploys (#9990) ... * fix(pipelines): live materialize/dataset edits reflect on the graph; no phantom draft after deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): Save all deploys the open pane's live buffer, not the stale draft snapshot Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): pin deployedFromPane to the shipped content so mid-deploy keystrokes still promote to a draft Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): draft rename ping-pong loop, stale rename deploys, inactive-draft input lineage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): dedupe inferred-lineage overlay against accumulated edges; first draft teardown still captures reads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): record an authoritative empty read capture on uncaptured draft entries Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): teardown skip compares lineage too, so access-only overrides still persist Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * style(pipelines): compress persist-back guard comments to the invariant Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-07 22:00:39 +02:00
Ruben FiszelandClaude Opus 4.8	51e1eba1cc	icon-only muted-read badge + lighter DuckDB template (#9972) ... Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 22:11:11 +02:00
Ruben FiszelandClaude Opus 4.8	3dcd3949a1	feat(pipelines): auto-derive cascade edges from ducklake/s3 reads (+ muted-read badge) (#9963) ... * feat(pipelines): auto-derive cascade trigger edges from ducklake/s3 reads Within a `// pipeline`, a read of a ducklake table or s3 object now auto-wires its cascade trigger edge straight from the FROM clause, so `// on <asset>` is only needed for edges inference can't see (dynamic SQL) or to carry per-edge opts. Two opt-outs: `// mute <asset>` suppresses a single derived edge (a lookup / SCD input read every run but not cascaded on), and `// mute all` opts the script out of derivation entirely (back to explicit-`// on`-only). Explicit `// on` still wins the dedup. Scoped to ducklake + s3 reads; resource/datatable/volume stay explicit. Read-write (RW) and write inputs are excluded so a self-referential merge can't loop-trigger itself; ambiguous (None) access is skipped. - parser: `mute` / `mute_all` in PipelineAnnotations (Rust + TS mirror) - deploy: derive_pipeline_asset_trigger_refs → script_trigger rows - frontend: resolveGraph mirrors derivation for the live edit-mode canvas - tests: shared parity corpus + derive-helper units + resolveGraph overlays Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): mark auto-derived cascade edges with a persisted derived flag + "auto" badge Persist script_trigger.derived (deploy: true for ducklake/s3-read derivation, false for explicit // on) and return it from the asset-graph endpoint so the canvas renders a Sparkles "auto" badge on auto-wired edges — the inference is now visible on both the deployed graph and the live edit canvas, not just implied. Dispatch (fetch_subscribers) ignores the flag, so a derived edge fires identically to an explicit // on. Also copy derived in the workspace-clone trigger copy, and backfill muteAssets/muteAll into two empty PipelineAnnotations literals the base commit left stale (check:fast). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): derive cascade edge from effective (alt-fallback) asset access derive_pipeline_asset_trigger_refs gated on the raw parser access_type, but the persisted asset.usage_access_type and the frontend canvas both use access_type.or(alt_access_type). An ambiguous parse with a manual read override was persisted/drawn as a read yet derived no edge, so the auto edge silently vanished on deploy. Gate on the effective access type for parity. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): badge muted reads instead of auto-derived edges Auto-derivation is the default now, so badging every derived cascade edge is noise. Drop the "auto" badge and the persisted `script_trigger.derived` flag (migration + insert param + graph field + clone copy) that only powered it, and instead badge the exception: a ducklake/s3 asset a script reads but does NOT cascade — `// mute <asset>` / `// mute all`. `computeMutedReadKeys` marks a read-only ('r') supported read with no cascade trigger and no self-write; the canvas renders a bell-off "muted" badge on that read edge. Also fixes two review parity nits: - TS `// on` parser now strips trailing `key=value` opts (e.g. `debounce=60s`) like the Rust `split_trailing_kv_opts`, so the ref dedups against inference. - A `// materialize` producer reading its own target is upgraded to `rw` (deploy) / excluded via the materialize write refs (canvas), so it neither self-cascades nor shows as a muted read. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): drop redundant // on for auto-derived reads; gate muted badge to pipeline scripts - Templates no longer scaffold `// on <asset>` for a ducklake/s3 input the body reads — the read auto-wires the cascade now that derivation is the default. Kept for datatable/resource (not auto-derived) and native triggers. The discoverability hint now mentions `// mute` (the newly relevant annotation). - computeMutedReadKeys only badges reads by `// pipeline` scripts. A plain script or flow reading a ducklake/s3 asset never had an auto trigger to suppress, so it must render as ordinary lineage, not "muted" (Codex review). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): only drop template // on when the body actually reads the input The redundant-`// on` removal assumed the generated body reads the ducklake/s3 input, but postgres/bash/generic bodies (and `data_upload`, which reads the picker file) ignore `input` — dropping `// on` there left the asset-created script with no cascade at all. Gate the drop on READS_INPUT_LANGS (bun/deno/python/duckdb) so non-reading templates keep the explicit trigger. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 18:12:12 +00:00
3 people	43044c2e28	feat(pipelines): wm_partition macro for grain-agnostic partition filters (#9950) ... * wip: partial work before earlyoom-recovery relaunch * fix(pipelines): scaffold the strftime {partition} filter idiom (frontend-only) The DuckDB materialize scaffold and the AI pipeline prompt now teach the grain-agnostic `WHERE strftime(<ts_col>, '<fmt>') = {partition}` filter instead of the naive `= TIMESTAMP {partition}` cast. `{partition}` substitutes to the partition IDENTITY string (`2026-07-05T23`, `2026-W27`, `2026-07`), which is not a valid DuckDB TIMESTAMP literal for any non-daily grain — so the naive form raises a `Conversion Error` for hourly/weekly/monthly (only daily parses). Adds a frontend unit test asserting the hourly scaffold emits the strftime idiom (`%Y-%m-%dT%H`) for every grain and never scaffolds the naive TIMESTAMP cast as executable SQL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): scope strftime partition idiom to time grains Review nit: `dynamic` partitioning's identity is a caller-supplied key, not a timestamp, so `strftime` doesn't apply. Scope the scaffold + AI prompt claim to time grains and add a `dynamic` example that filters on the user's own key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): wm_partition macro for grain-agnostic partition filters The materialize runtime now injects a `wm_partition(ts)` temp macro as the first setup statement of a time-partitioned script, so filtering the source to the active slice is one grain-agnostic line — `WHERE wm_partition(<ts_col>) = {partition}` — instead of a hand-written `strftime` format the author must keep in lockstep with the resolver, or the `= TIMESTAMP {partition}` cast that only parses for daily and Conversion-Errors for hourly/weekly/monthly. The macro's format comes from `PartitionKind::default_time_format` in windmill-parser, the same source the EE resolver reads to stamp the `{partition}` identity, so the two can't drift. `dynamic` partitions get no macro (their identity is a caller-supplied key → `WHERE <key_col> = {partition}`). Replaces the earlier 9-line strftime comment block in the scaffold with the single macro line; AI pipeline prompt and design doc updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(pipelines): verify wm_partition strftime parity vs chrono through real DuckDB Runs the bundled DuckDB engine in-memory and asserts strftime renders every grain format (daily/hourly/weekly `%G-W%V`/monthly) byte-for-byte identically to chrono — the engine the resolver uses to stamp the `{partition}` identity — across ISO-week year boundaries (2027-01-01 → 2026-W53 etc.). Also proves the injected `wm_partition` macro buckets the whole slice and that the naive `TIMESTAMP '<weekly|monthly identity>'` cast Conversion-Errors. Closes the one cross-engine assumption the pure-Rust/frontend tests couldn't reach (flagged by CI review for weekly ISO-week rendering). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 0de2412ff0734b11e12ba378c9bcc373ff9ae800 This commit updates the EE repository reference after PR #649 was merged in windmill-ee-private. Previous ee-repo-ref: ad6c6685689d7741058e7d2c9ecbe95d982e6268 New ee-repo-ref: 0de2412ff0734b11e12ba378c9bcc373ff9ae800 Automated by sync-ee-ref workflow. * fix(pipelines): classify CREATE TEMP MACRO as a DuckDB prepare-path setup statement The FFI prepare/diagnostics pass only EXECUTES statements recognized by is_setup_statement (ATTACH/USE/INSTALL/…); everything else is merely prepared. `CREATE [OR REPLACE] TEMP MACRO` wasn't recognized, so on a `-- prepare` run of a partitioned materialize the injected `wm_partition` macro was never created on the connection, and the later generated `CREATE TABLE … SELECT … WHERE wm_partition(...)` failed to bind ("function does not exist"). The same latent gap affected the workspace-macro splicer, which injects TEMP MACRO blocks too. Classify CREATE [OR REPLACE] TEMP|TEMPORARY MACRO as setup so it's executed before dependent blocks and excluded from the PrepareQueryResult count (persistent CREATE MACRO stays a user statement). Adds a prepare-path test that fails without the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-06 10:26:53 +02:00
Ruben FiszelandClaude Opus 4.8	a1c5b7aa3e	feat(pipelines): require data uploads before running a pipeline (#9953) ... * feat(pipelines): require data uploads before running a pipeline Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): require every S3Object filled for data-upload readiness Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipelines): Run pipeline defaults schedule-triggered scripts to their schedule args Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: correct why only schedules default their args in Run pipeline Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): close cascade double-start race and gate data-upload readiness on full-schema validity Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: expose ScriptEditor validity via callback, not banned bindable-with-default Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 10:26:19 +02:00
Ruben FiszelandClaude Opus 4.8	6eabb96ae7	fix(pipelines): make node & pipeline-level run affordances always visible (#9948) ... * fix(pipelines): make node & pipeline-level run affordances always visible Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): don't leave node Run disabled by stale form validity The always-visible node Run button read `isValid` directly, but that flag is only meaningful while PipelineRunForm is mounted to set it. On a same-path re-resolve (the component is keyed on script.path) from an input-carrying script to an input-less one, the form unmounts leaving `isValid=false`, which wrongly kept the empty-args Run button disabled. Gate validity through a `runValid` derived that is true whenever no form is rendered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): run with {} when node has no form, not stale args The always-visible node Run button dispatched $state.snapshot(args) unconditionally. `args` persists across a same-path re-resolve (component keyed on script.path), so a script that once had inputs/partition args and is re-resolved as input-less would run with the stale hidden args instead of {}. Send {} whenever no form is rendered, matching the no-form run intent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 03:06:31 +02:00
Ruben FiszelandClaude Opus 4.8	b13113964a	fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect (#9939) ... * fix(pipelines): canonicalize S3 asset keys so SDK writes and DuckDB reads connect The SDK object forms — TS `writeS3File({s3:"exports/x"})` and Python `write_s3_file(S3Object(s3="exports/x"))` — resolve to the URI `s3:///exports/x` (empty default storage), whose parsed asset path was `/exports/x` (leading slash). DuckDB `read_csv('s3://exports/x')` and the `// on s3://exports/x` trigger form yielded the bare `exports/x`. The same object thus produced two asset identities, so a DuckDB consumer never connected to a TS/Python producer in the pipeline graph. `parse_asset_syntax` (shared by the native backend parsers and the wasm parser that drives `frontend/src/lib/infer.ts` and the CLI `localGraph`) now strips a single leading slash from S3 paths, so `s3:///key`, `s3://storage/key`, DuckDB `s3://…`, and `// on` all canonicalize to one key. Both deploy-time inference and editor/CLI inference agree, and the producer's write edge and the consumer's read/trigger edge share a node. Only one leading slash is stripped, so `s3:///` triple-slash default-storage keys collapse to the bare key while Hive-partition keys (`s3://bucket/y=2024/f.parquet`) and explicit-storage `s3://storage/key` paths are untouched. Non-S3 asset kinds (res://, ducklake://, …) keep their paths verbatim. Note: existing deployed pipelines that recorded `/key` paths need a redeploy to pick up the canonical `key`; the fix is forward-consistent for anything parsed after this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pipelines): mark S3 asset-path normalization (item 6) resolved The open-issues list still flagged the SDK-form leading-slash vs bare-URI no-slash mismatch as "Still open", contradicting the fix in this PR. Mark it resolved to match the updated Language-coverage prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs+test(pipelines): disclose S3 explicit-storage vs default-storage-nested-key aliasing Collapsing to one canonical key means `s3://storage/key` (explicit storage) and `s3:///storage/key` (default-storage nested key) now alias to the same node `storage/key`, though they name different objects. Low-probability (needs a storage config named to match a default-storage prefix) and inherent to a best-effort lineage graph that doesn't split the first segment as a storage name, but previously undisclosed. Document the tradeoff and pin the intended aliasing with a test so it's intentional, not a latent surprise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): mirror S3 leading-slash strip in frontend live-preview parser The pipeline graph live preview parses `// on` annotations client-side via the hand-written `parsePipelineAnnotations.ts` (a TS mirror of the Rust annotation scanner), NOT the wasm parser. Its `parseAssetSyntax` still returned the raw suffix, so `// on s3:///exports/x` yielded `/exports/x` while the deploy-time and wasm parsers now canonicalize to `exports/x`. `resolveGraph` synthesizes trigger edges from that path, so the browser preview could still render disconnected `/exports/x` and `exports/x` nodes for the exact triple-slash case this PR fixes at deploy time. Mirror the S3-only single-leading-slash strip in the TS parser and extend the shared parity fixture corpus (run by both the Rust and TS parity suites) with the triple-slash trigger case, so Rust/TS drift on this is now caught. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): seed slashless S3 template asset paths to match canonical key `autoOutputAsset` seeded new S3 template outputs with a leading slash (`/pipelines/…`), which the old parser required to match `s3:///key` writes. This PR made `parse_asset_syntax` strip that slash, so the seeded draft asset (stored as `outputAssets`, used by `resolveGraph` for inactive-draft node identity) no longer matched the body-inferred identity `pipelines/…` — the live preview could render a duplicate `/pipelines/…` node and a phantom post-deploy drift warning. Seed the canonical slashless key instead, and switch the DuckDB body's S3 URIs from `s3://${path}` to `s3:///${path}` so the generated runtime URI stays the triple-slash default-storage form byte-for-byte (the SDK sites already build `s3:///` + bare key). Add a pure-logic parity test asserting, for every language and S3 output kind, that the seeded asset path is slashless and that every S3 URI the generated body emits is triple-slash and canonicalizes back to that seeded path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): canonicalize S3 keys in CLI + frontend bounded-cascade resolvers Two more hand-written S3-URI sites returned the raw suffix, so `s3:///exports/x` stayed `/exports/x` while native/wasm parsers now canonicalize to `exports/x`: - `cli/src/commands/pipeline/localGraph.ts` — the no-wasm fallback `// on` scanner (go/bash/ruby). A fallback consumer's `// on s3:///x` would not connect to a wasm-inferred `x` producer in `wmill pipeline show/run --local`. - `boundedCascade.ts` `assetUriToNodeId` (duplicated in the CLI and the frontend AssetGraph engines, kept in sync) — `--to s3:///exports/x` / a cascade bound token would not resolve against the canonical graph node `s3object:exports/x`. `resolveToken` delegates here, so it is covered too. Mirror the S3-only single-leading-slash strip in all three, and add `s3:///` tests to the CLI local-graph fallback suite and both bounded-cascade suites (explicit-storage and Hive-partition keys asserted untouched). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pipelines): phrase S3 template test comment as a current invariant Describe the slashless-seed requirement as the invariant it is, not as change history, per the AGENTS.md "describe the code as it is" rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): strip all leading slashes from S3 keys so trigger refs round-trip `parse_asset_syntax` stripped only one leading slash, so `S3Object(s3="/x")` — which resolves to the quad-slash URI `s3:////x` — parsed to path `/x`. But `trigger_spec_to_row` rebuilds a stored trigger ref as `s3://<path>` = `s3:///x`, which `parse_asset_trigger_ref` then parses back to `x`. The producer recorded `/x` while its consumer trigger resolved to `x` → a broken edge. The same asymmetry affects every `s3://`+path reconstruction site (backend refs, frontend `assetUri`, page refs) whenever a path starts with `/`. Strip ALL leading slashes so a canonical S3 path never starts with `/`; naive `prefix + path` reconstruction then round-trips everywhere. Applied uniformly across all six S3-URI sites (Rust `parse_asset_syntax`, the TS live-preview parser, template `s3Key`, and the frontend+CLI `assetUriToNodeId` and CLI fallback scanner). The pathological leading-slash key collapses to the bare key — acceptable for a best-effort lineage graph that never split storage anyway. Tests: a windmill-common round-trip test (parse → trigger_spec_to_row → parse_asset_trigger_ref) over every URI form incl. the quad-slash case; a `s3:////x` shared parity fixture (Rust + TS); and quad-slash assertions in the Rust parser test and both bounded-cascade suites. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(pipelines): align S3 template parity helper with strip-all canonicalization The template seed/body parity test's `canonicalS3Key` helper (and its comment) still stripped a single leading slash, so it no longer mirrored the parser it claims to pin. Strip all leading slashes to match `parse_asset_syntax` and the frontend/CLI mirrors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 01:18:29 +02:00
Ruben FiszelandClaude Opus 4.8	2d3a773441	feat(pipelines): mid-DAG selective execution (dbt model+) for pipeline runs (#9945) ... * feat(pipelines): mid-DAG selective execution (dbt `model+`) for pipeline runs Relax the root-only constraint on bounded-cascade starts so `--from` can name any node in a pipeline — not just a schedule/manual root. A mid-DAG start runs that node plus its transitive downstream and never re-runs upstream, giving dbt's most common gesture (`dbt run --select model+`) a direct form: wmill pipeline run f/orders --from fct_orders_daily Previously this errored with "Starts must be schedule-triggered or manual roots". The bounded-run engine already computed downstream/path-between sets generically; only the eligibility gate was root-only. - Shared engine (`boundedCascade.ts`, CLI + frontend mirror): add `validFromStarts` — every autorun-able script (roots AND mid-DAG asset subscribers / pure readers), excluding only event/input-only handlers (kafka/mqtt/…/webhook/data_upload) that can't run with empty args. - CLI: `--from` accepts any `validFromStarts` node; asset `--from` and non-autorun handlers still rejected (the latter runnable via `--upload`). An explicit mid-DAG start is protected from the barrier cut. Help text + regenerated system_prompts describe the new surface. - Frontend graph UI parity: any node with downstream now offers "Run + downstream…" (was roots-only). With no end picked the bounded-run bar runs the full downstream closure (`model+`); picking end(s) still bounds the path-between set. - Unit tests for the new selection semantics in both engines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): address CI review — scheduled-root --from regression + pick-mode barrier parity Codex review findings on #9945: - P1: explicit `--from` rejected a scheduled root that also carries a secondary non-autorun trigger (e.g. `// on schedule` + `// on data_upload`), even though it stays a valid IMPLICIT start. `validFromStarts` excluded anything in `nonAutorunTriggerScripts`; now it unions in `validStarts` (which lets the schedule identity win over the secondary trigger), so a scheduled root is `--from`-eligible in both CLI and the graph UI. Regression tests added in both engines. - P2: bounded-pick mode built `eligible` (pickable end bounds) from raw `descendants`, so an event handler — or a node only reachable through one — could be clicked as an end yet be silently dropped from the barrier-cut run. `eligible` is now the barrier-cut closure, so those nodes are dimmed and non-pickable. The highlighted `bounded` ring now also reflects the actual (barrier-cut) run set, including the no-ends "Run + downstream" case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): frontend barrier set must exclude all valid roots, not just the picked start Codex review follow-up: the frontend `boundReachable` barrier set only protected the picked start (`id !== boundPickStart`), while the CLI protects every valid root (`!starts.has(id)`). So a scheduled root that also carries an event trigger, reached downstream from another start, was wrongly treated as a barrier — the UI dimmed/skipped it and its downstream, diverging from the CLI run set. Exclude `validStarts` from the barrier set too (a scheduled/manual root runs on its own identity even with a secondary event trigger). Regression test asserts a scheduled-event root and its downstream stay reachable from an upstream start, and that the naive (start-only) barrier set would have dropped them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): frontend must exclude webhook/data_upload as mid-DAG autorun starts Codex review follow-up: the frontend `validFromStarts` only excluded `EVENT_TRIGGER_KINDS`, so a mid-DAG `webhook`/`data_upload` subscriber was added by the new eligibility loop — the UI would offer "Run + downstream" and launch it with empty args (no uploaded S3Object / webhook body). The CLI mirror already excludes these input-only kinds. Add a frontend `NON_AUTORUN_TRIGGER_KINDS` (event kinds + webhook + data_upload), mirroring the CLI, and use it in both `validFromStarts` (exclude such mid-DAG handlers from starts) and `nonAutorunTriggerScripts` (cut them as barriers). When the marker is visible (editor overlay / draft) these are now handled exactly as the CLI does; the deployed-graph blind spot (no webhook/data_upload rows) remains the documented pre-existing `validStarts` limitation. Regression test: a `data_upload`/`webhook` mid-DAG subscriber is not an eligible start and is barrier-cut (with its exclusive downstream) when running from an upstream root. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 00:11:25 +02:00
Ruben FiszelandClaude Opus 4.8	475b072987	feat(pipelines): partition run-arg picker + first-run setup signpost (#9943) ... * feat(pipelines): partition run-arg picker + first-run setup signpost Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): honor partition start= anchor in picker default and per-input upstream hint Addresses CI review (Codex): - P1: never seed/offer a pre-start bucket — the worker takes an explicit partition arg verbatim, so seeding today's bucket before start= would materialize early. defaultBucket now clamps to the start bucket and drops pre-start recent-missing chips; a hint explains the start anchor. - P2: upstream-missing hint checks each partitioned ducklake input separately instead of unioning, so a fan-in where one input has the bucket no longer masks another that lacks it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): reseed partition picker on header change and fail safe on bad tz/start Addresses second CI review (Codex): - P1: run form is now keyed on the parsed partition spec as well as the schema, so editing the // partitioned header (same schema) remounts the picker and reseeds/re-strips instead of keeping a stale bucket that re-bypasses the start anchor. - P1: malformed metadata is fail-safe in parity with the backend — invalid tz= no longer throws in Intl (falls back to UTC for display), invalid start= (e.g. 2026-02-31) is rejected via round-trip check, and neither auto-seeds an explicit partition (which would bypass the worker's own tz/start validation). A warning hint points at the header. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 00:10:28 +02:00
Ruben FiszelandClaude Opus 4.8	6ae8dd37b1	fix(pipelines): pipeline-level run control, tables label, data-test rollback + fork badges (#9944) ... * fix(pipelines): pipeline-level run control, tables label, data-test rollback + fork badges - Add always-visible "Run pipeline" header control (edit mode) that runs every script in dependency order via the bounded-cascade engine, so a run no longer requires hovering a node's play button. - Header summary counts ducklake/datatable assets as "tables" (and s3object as "files") instead of the raw kind, collapsing shared nouns. - Surface a data-test outcome badge on guarded asset nodes: EE shows a rolled-back (previous version left live) state, CE shows published-despite- failure — driven by the producer's last run state and the edition. - Make the fork data-environment marker a prominent labeled chip (⑂ fork / ↗ parent) instead of a bare icon. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): address CI review — scope Run pipeline to members, anchor guard badge, spin loader - Run pipeline now filters to `in_pipeline` script runnables, so it never launches dependency-only endpoints the graph shows for context (macro libraries, custom data-test scripts, out-of-folder producers). - Data-test guard badge only attaches to the producer's declared `// materialize` target, so a multi-output producer no longer badges its other ducklake writes. - Spin the Loader2 icon in the "Run pipeline" button while a run is in progress (startIcon classes), matching every other loading affordance. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): data-test badge copy speaks to write policy, not failure cause producerFailed is a generic job-failure signal, so the failed-state tooltip no longer claims the run "failed its data tests" (it could be a runtime/worker error). It now states the edition's behavior on any failed materialize: EE rolls back (previous version left live), CE may leave a failing write live. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipelines): Run pipeline keeps independent branches running after a failure runSelection used a single global fail-fast flag, so once any node failed it refused to schedule *any* newly-ready node — a failure in one branch could strand an unrelated healthy branch as 'skipped' depending on job timing. Now a failure poisons only its transitive descendants; independent branches finish. Add regression tests: independent-branch-survives-failure and join-node-skipped -when-one-upstream-fails. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 00:08:24 +02:00
Ruben FiszelandClaude Opus 4.8	574d3ac9ff	fix(pipelines): link SCD2 <dim>_current view to its producer across all graph surfaces (#9933) ... * fix(pipelines): link SCD2 <dim>_current view to its producer across all graph surfaces An SCD2 producer (`// materialize … history`) creates the base table AND a `<dim>_current` view at runtime. The deploy path already registered both writes, but the CLI `--local` graph and the frontend live-editor graph only emitted the base write, so a consumer reading only `<dim>_current` orphaned there. Centralize the companion derivation in `MaterializeSpec::write_targets` / `scd2_current_target` (+ TS `scd2CurrentTargetPath` mirror), emit the `_current` write in every surface, and mark the companion node `derived_from` the base so the canvas renders it as a derived "current view" instead of an unrelated table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): keep scd2 _current write edge when editing a saved producer Addresses Codex CI review (P1): opening a deployed scd2 materialize producer for editing dropped its persisted `<dim>_current` write edge. `liveRefKeys` (the set of asset keys a saved-script edit preserves against stale-filtering) only added the base materialize target, so the companion `_current` write was judged stale and filtered — orphaning consumers of only the view mid-edit. Add `scd2CurrentTargetPath(m)` to `liveRefKeys` too; covered by a new saved-edit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 22:35:46 +02:00
Ruben FiszelandClaude Opus 4.8	46be39dfb7	fix(pipelines): order data_test relationships refs before the tested script in a cascade (#9934) ... * fix(pipelines): order data_test relationships refs before the tested script in a cascade Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): key custom-test reads by (usage_kind, path) to avoid same-path flow collisions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 22:34:58 +02:00
Ruben FiszelandClaude Opus 4.8	0ad174fa49	feat(pipelines): self-teaching custom data_test errors + scaffold (#9937) ... Custom `// data_test <path>` scripts must be a single SELECT reading the freshly-materialized target via the internal `_wm_target.<table>` alias — neither was documented or scaffolded. Make the codegen errors name the exact violation (multi-statement, non-SELECT, wrong alias, empty) and append a copyable `SELECT * FROM _wm_target.<table> WHERE <condition>` example. Add a DuckDB-only 'Data test' pipeline output kind that scaffolds that starter body. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-05 22:29:35 +02:00
Ruben FiszelandClaude Opus 4.8	5769b6036c	fix(pipelines): activity-axis label clarity + select failed node on cascade failure (#9931) ... * fix(pipelines): clarify activity-window axis label + select failed node on cascade failure Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): clear active draft so failed-node focus is not masked PipelineGraphEditor gives an open draft (activeDraftPath) priority over selection via openScriptPath, so the cascade-failure focus set only selection and stayed masked while a draft pane was open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-05 17:10:06 +02:00
Ruben FiszelandClaude Fable 5	39eb9de1bc	feat(pipelines): fork data environments for ducklake materialization (dev data) (#9915) ... * feat(pipelines): fork-scoped ducklake namespaces with read-defer to parent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork graph indicator + fork ducklake namespace cleanup endpoint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork_views-keyed view transition, fork lineage clone, design doc Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review hardening - fork DATA_PATH last-wins, registry cache TTL, defer tests Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): per-lake isolated/shared choice at fork creation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): chain-aware defer discovery + per-location fork namespace registry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lake-scoped fork schemas, catalog identity in registry, chain-aware graph chips Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): cleanup deletes fork data from the registered storage identity Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): collapse fork data-path segment to one component (slash-safe ids) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): per-catalog ancestor checks, ancestor extra_args passthrough, test compile fix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): invalidate fork ancestor-chain cache on lineage mutations Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): sweep descendant ancestor-chain caches on delete/reparent Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): run fork ducklake cleanup inline in delete_workspace Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): resolve fork cleanup credentials pre-commit, destroy post-commit Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): shared dev-workspace authz gate for namespace drop, invalidatable registration cache, segment-boundary delete filter - extract require_prod_admin_for_dev_workspace, used by both delete_workspace and drop_forked_ducklake_namespaces so the gates cannot drift - key FORK_DUCKLAKE_REGISTERED per workspace and invalidate it in cleanup_fork_ducklake_namespaces so a same-id fork recreated within the TTL re-registers its namespaces - filter listed object locations to the segment boundary before deletion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): keep orphaned wm-fork-* workspaces ducklake-isolated parent_workspace_id is ON DELETE SET NULL, so a fork can outlive its parent with an empty ancestor chain while its cloned config still points at the shared lake. Key the isolation gate on the wm-fork- prefix as well as the chain (mirroring workspace_is_fork): orphaned forks get the write redirect, registration and cleanup with zero ancestors (no defer), and keep their 'fork' graph chips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): attach orphaned wm-fork-* ancestors at their fork namespace Chain position alone classified the last ancestor as a root, but an orphaned wm-fork-* ancestor (its own parent deleted, SET NULL) ends the chain the same way while its data lives in its fork namespace — its descendants' defer views bound the dead root's lake instead. Key the root-vs-fork decision on the wm-fork- prefix too, matching the resolution gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): never inherit shared lake opt-out; durable cleanup ledger for failed fork deletions - fork creation strips cloned fork_behavior stamps before applying the request's shared_ducklakes list: sharing is a per-creation choice, a fork of a shared fork defaults back to isolated - fork_ducklake_namespace loses its ON DELETE CASCADE FK: rows are the durable cleanup ledger and outlive the workspace when physical cleanup fails post-commit; fork creation retries leftover rows for the reused id and refuses to create while a metadata schema still cannot be dropped (data-file leftovers alone are inert once the schema is gone and are swept by the next successful same-prefix cleanup) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): make orphaned-namespace cleanup retries independent of deleted fork resources - ledger rows gain a schema_dropped phase flag: set when the schema drop succeeded but data cleanup failed, so later retries skip the schema phase and need no catalog credentials at all; registration resets it on re-attach (ON CONFLICT DO UPDATE) since attaching recreates the schema - retry-path $res: resolution falls back to the workspace being forked (the deleted fork's resources were clones of a parent's); live paths (delete_workspace prepare, drop endpoint) pass no fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): fork tables from failed-after-commit runs stay fork-owned in defer and graph A failed materialization must not disguise a physically existing fork table as deferred: CREATE VIEW IF NOT EXISTS silently yields to the table, so reads hit fork data while the graph claims parent defer. - record_mat upsert preserves the last committed snapshot_id on failure - defer discovery and graph chips treat fork rows with a committed snapshot as fork-owned even when status is failed - inspect_fork_catalog also lists live fork tables (same round trip) and the defer list is filtered against them — covers rows recorded before this fix and tables created by raw SQL - drop stale FK-cascade wording in the design doc and sidebar comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): fork-mode ducklake settings — per-lake isolated/shared chips + banner, fork_behavior round-trip The workspace-settings ducklake editor had no fork awareness: no reminder of each lake's isolated/shared choice and no warning about what edits mean in a fork. It also rebuilt each lake explicitly on save, silently dropping fork_behavior — any settings save in a shared fork flipped the lake back to isolated. - fork detection mirrors the backend gate (parent link or wm-fork- prefix) - info banner explaining isolated vs shared semantics in a fork - per-lake chip (emerald 'isolated' / amber 'shared with parent') with tooltips, matching the pipeline graph chip colors - fork_behavior added to DucklakeSettingsType and preserved through convertDucklakeSettingsToBackend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-05 09:58:06 +02:00
Ruben FiszelandClaude Fable 5	5ad2de91a2	feat(sdk): enforce s3:// URIs for string S3 params + ingestion (EL) docs (#9912) ... * feat(pipelines): ingestion (EL) templates + docs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): review nits — draft collision guard, template-mode selection reset, invariant test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): lead the insert menu with ingestion templates Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(pipelines): ingestion story as docs-only — drop editor template UI The insert-menu template section mixed two selection grammars in one popover and confused more than it helped. The three E2E-verified example pipelines now live verbatim in docs/pipeline-ingestion.md; the Python bare-string S3 key fix in pipelineTemplates.ts stays. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(sdk): bare string S3 keys in py/ts clients + asset parsers A plain string passed where an S3Object is expected is now a bare key in the default storage — previously the py client silently degraded it to s3="" (auto-generated key) and both asset parsers canonicalized it without the leading slash, splitting lineage. parseS3Object moves to s3Types.ts so it is unit-testable without the generated services. The pipeline template fix from the earlier commit is superseded (bare strings are the supported spelling again); docs examples flipped to bare keys. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(sdk): enforce s3:// URIs for string S3Object params Bare strings now raise/throw with a hint pointing at the s3:///<key> spelling instead of being treated as keys (previous commit) or silently degrading to an empty key (original behavior). One string spelling everywhere: SDK calls, // on annotations, and DuckDB SQL all use s3:///<key>. TS regains the s3://-template-literal type; the asset parsers record no asset for a bare string (the call can only error); templates emit the URI form. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(pipelines): move ingestion (EL) guide to windmilldocs, keep design constraints User-facing how-to (engine choice, cursor recipes, schema drift, worked examples) moves to windmilldocs core_concepts/63_pipelines (windmilldocs#1462); the repo keeps only the design constraints future feature work must not break, as a section of ducklake-materialization.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: regenerate system prompts after parse_s3_object docstring change Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sdk): reject empty-key s3 URIs; align asset parsers with the runtime rule Addresses CI review: s3:/// and s3://bucket/ now raise (an empty key would fall back to the auto-generated-key path the strict contract exists to prevent); the asset parsers' string branch applies the same valid-URI-with-non-empty-key rule so no R/W edge is recorded for a call that can only error (the generic URI-literal scan still records ambiguous access-None assets, by design); comments rephrased as current constraints per AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-04 18:07:39 +02:00
Ruben FiszelandClaude Fable 5	42e11c6570	feat(pipelines): schema contracts — save-time consumer checks vs captured schemas (#9917) ... * feat(pipelines): schema contracts — save-time consumer checks vs captured schemas Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor: move schemaContractContext above schemaCanEvolve doc comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: emit scd2/on_schema_change in CLI local graph, address review notes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: gate editor _current ignore-suppression on scd2, matching backend Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-04 10:40:00 +02:00
3 people	5d7fb6deca	feat(pipelines): asset freshness — fresh/stale badge (CE) + watchdog (EE) (#9909) ... * feat(pipelines): passive asset freshness tracking on the graph Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(pipelines): drop dead freshness-enforcement stub, document query ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(pipelines): freshness watchdog (EE) — auto re-run stale producers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): watchdog review fixes — archived workspaces, badge kind parity, scan index Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — no singlestepflow in freshness, +N parity, completion-time fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pipelines): CI review — history completedAt, freshness/asset trigger UI metadata Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 6f5fe0f7f56696fbef5a8349da38496c32e71666 This commit updates the EE repository reference after PR #643 was merged in windmill-ee-private. Previous ee-repo-ref: 1f13380354bf591ae25a2c20d36917534bcc5459 New ee-repo-ref: 6f5fe0f7f56696fbef5a8349da38496c32e71666 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-04 06:23:44 +02:00
Ruben Fiszel	bcb4dd8d27	docs link	2026-07-03 11:40:34 +02:00
Ruben Fiszel	84141add1d	feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890) ... * feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud	2026-07-03 00:59:30 +02:00
Ruben Fiszelandwindmill-internal-app[bot]	53bbb92953	feat(pipeline): backfill a range of partitions from the asset drawer (#9885) ... * feat(pipeline): backfill a range of partitions from the asset drawer (ee) * feat(pipeline): cancel in-flight backfill job and show cancelling state * refactor(pipeline): move backfill range logic behind private feature * fix(pipeline): close backfill cancel-launch race and record dispatch intent * docs(openapi): producer_path also covers SDK write-edge producers * chore: update ee-repo-ref to c3852ecb36bd0be1a74c63169e513888f3347850 This commit updates the EE repository reference after PR #641 was merged in windmill-ee-private. Previous ee-repo-ref: 7c1450ef89fbc9e844a121b39cafe0d7235d704b New ee-repo-ref: c3852ecb36bd0be1a74c63169e513888f3347850 Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-02 12:50:55 +02:00
Ruben FiszelandClaude Fable 5	b92a86b8b3	fix: polish pipeline graph view (layout, viewport, minimap, lineage, timestamps) (#9883) ... * fix(frontend): keep pipeline graph layered when lineage has cycles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): fit pipeline graph to visible canvas on initial load Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): style pipeline minimap so it reads as a minimap Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(parser): don't infer s3 reads from bare string-literal mentions in sql Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(duckdb): render temporal values as ISO strings in job results Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(frontend): key pipeline viewport fit on the loaded graph's folder Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(parser): treat list/named read-fn arguments as definitive s3 reads Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 11:48:11 +02:00
Ruben FiszelandClaude Fable 5	20351a6b4c	feat(frontend): pipelines index page and sql editor hint (#9881) ... * feat(frontend): surface pipelines in sidebar nav, index page and sql editor hint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(frontend): remove pipelines sidebar nav item Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-02 11:14:19 +02:00
Ruben FiszelandClaude Opus 4.8	74f579e6d9	feat(pipeline): local development for data pipelines (CLI --local + pipeline dev preview) (#9840) ... * feat(pipeline): local development for data pipelines (CLI --local + pipeline dev preview) Add the local edit→preview→run loop for data pipelines (folders of `// pipeline` scripts), the analog of `wmill dev` / `wmill app dev`, usable from a code editor or an agentic loop — without deploying. No backend changes: full body inference comes from the same wasm the frontend uses (windmill-parser-wasm-asset), which returns assets + pipeline annotations in one call; local runs reuse runScriptPreview with _wmill_skip_asset_dispatch. - localGraph.ts: wasm-backed working-tree → asset-graph builder (the enabler) - pipeline show/run --local; new pipeline docs (PIPELINE.md/AGENTS.md) subcommand - pipeline dev watcher + /pipeline_dev page (PipelineDevView) rendering the same PipelineGraphEditor from the pushed local graph, run via preview - cascadeRun.ts: reusable run primitives extracted from the route page - regenerated CLI agent docs See docs/pipeline-local-dev.md for the full design, test steps, and handoff TODOs. The live `pipeline dev` browser preview is implemented but not yet stack-verified. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): improve local dev preview (run, activity, responsive) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): dev-preview args, multi-root run, ws auto-reconnect Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): connect managed-materialize producer in local dev graph The CLI pinned windmill-parser-wasm-asset ^1.728.1, which predates managed-materialize support (added in 1.733.1); the frontend already pins 1.740.0. The CLI's wasm therefore never emitted `// materialize`, so the producer had no output edge and showed disconnected from its `// on` consumers. Bump the CLI to 1.740.0 (matching the frontend) and translate the parsed materialize target into the producer's write edge + materialize_target, mirroring frontend resolveGraph.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): harden local-dev CLI (bare-.sql crash, defaultTs, docs clobber) Review fixes, complementary to the dev-preview/materialize/multi-root work already on the branch (none overlap those commits): - localGraph: a bare `.sql` (no dialect) made inferContentTypeFromFilePath throw and abort the whole graph build — and wedge `pipeline dev` at startup. Skip the unclassifiable file instead. Also map `bunnative` → parse_assets_ts and add ruby/rlang/nu/powershell to the `#`-comment fallback. - show/run/docs/dev: thread the resolved `wmill.yaml` defaultTs into the graph builder so `.ts` infers under the workspace's runtime (bun vs deno) instead of always bun — `opts.defaultTs` was always undefined (no such CLI flag). - dev: wrap the startup graph build so a half-written file can't abort the watcher. - docs: don't clobber a user-authored AGENTS.md/CLAUDE.md — only (over)write the pointer when absent or already a generated `@PIPELINE.md` pointer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): bind dev WS to loopback + local-graph regression tests - pipeline dev WS broadcast the folder's full script source (scripts[].content + temp_script_refs) unauthenticated on 0.0.0.0:3201 — bind 127.0.0.1 so it's not LAN-reachable (webview localhost + SSH/devbox port-forward still work). - Add regression tests for the just-landed local-graph fixes: bare .sql is skipped (was a build/dev-startup crash), defaultTs threads into .ts runtime inference (bun vs deno), and #-comment languages (ruby) use the # annotation fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): --frontend flag for pipeline dev page origin wmill pipeline dev opens <remote>/pipeline_dev, but that route only exists in this build's frontend, so it 404s against a remote whose deployed frontend predates it. --frontend <origin> points the page at a locally-run frontend (REMOTE=<remote> npm run dev) while the API/token still target the remote — enabling the live preview against a real backend before the PR is deployed. No behavior change when omitted. Regenerated CLI agent docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): WS session token + details-pane live-reload refresh Addresses CI review (Codex/Pi/Claude): - dev WS: a browser tab could open ws://localhost:<port>/ws and receive the folder's full source (browsers don't enforce same-origin on WS, loopback bind alone doesn't help). Gate the upgrade on an unguessable per-session token carried in the dev-page URL (verifyClient → 401 without it). Verified: no-token/bad-token connections get 401 with no bundle. - details pane: scriptRes keyed on [workspace, selection, draftScript] didn't re-run on a pipeline dev live-reload (same selection), so the open pane showed stale source. Thread a localScriptsVersion (the pushed bundle) into the key. Verified: editing a selected node's file updates the pane source without reselect. - docs/pipeline-local-dev.md: refresh the stale 'not yet exercised' status + done TODOs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): emit volume: annotation assets in local dev graph Addresses CI review (Codex P1 / Pi P1): the wasm body parser doesn't surface `// volume: <name>` annotations — the frontend (infer.ts:parseVolumeAnnotations) and backend (asset_inference.rs) parse them separately and merge as rw volume assets. localGraph didn't, so a `# volume: cache` producer had no write edge and showed disconnected from its `// on volume://cache` consumer (and pipeline run --local wouldn't schedule downstream). Mirror the leading-comment-block scan (SQL excluded, matching both reference parsers) and merge into inferScriptAssets. Regression test added; verified producer -> volume://cache -> consumer connects. Also (Codex P2): docs/pipeline-local-dev.md manual browser URL omitted the new ws_token param — without it the WS upgrade is rejected and the page sits disconnected. Doc now says to copy the URL the CLI prints (carries wm_token + ws_token) and recommends --frontend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): runAll excludes event roots + review polish Addresses CI review (Codex P1, Claude P2/P3): - pipeline run runAll: derive the whole-pipeline selection from validStarts + descendants instead of all runnables, so an unqualified 'pipeline run <folder>' no longer fires event-trigger roots (kafka/mqtt/…) with empty args/side effects. Verified: a kafka root is excluded from the plan. - cascadeRun.ts runBoundedCascade: use buildLineageDownstreamMap (read-aware) so a pure-reader runs after its producer, and return cyclic — parity with the route page's bounded run (the file is meant to be THE shared correct primitive). - PipelineGraphEditor: storedRightPaneSize starts at 0 so the orientation-aware default (55% stacked / 40% side-by-side) actually applies on first open. - localGraph fallbackParse (go/bash): scan only the leading comment header (no body-comment phantom triggers) and strip key=value options from the asset URI; regression test added. - docs: reject '..' in the folder arg (it writes files under f/<folder>). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): route local previews to the // tag worker Addresses CI review P1: the local graph/bundle dropped the parsed `// tag`, so a node annotated `// tag gpu` ran on the default worker in both `pipeline run --local` and `/pipeline_dev`, while the deployed pipeline routes it to that worker tag. Carry the tag through LocalScript / the pushed bundle / LocalScriptContent and pass it to runScriptPreview at all three launch sites. Verified: a duckdb node tagged `bash` produces a job tagged `bash`; regression test added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): add asset partitions/schemas routes to OpenAPI, use generated client The ducklake asset panels (PartitionStatusGrid, SchemaHistoryPanel) hit /assets/partitions and /assets/asset_schemas via raw fetch with cookie-only auth, because those backend routes were never added to openapi.yaml so the generated client had no methods for them. On /pipeline_dev (token-via-URL, no session cookie) the raw fetches 401'd. Add both GET routes + MaterializedPartition/AssetSchemaVersion schemas to openapi.yaml and call them through AssetService, which injects the bearer token, types, and cancellation automatically. Verified: Partitions + Schema tabs load in /pipeline_dev. (backfill stays a raw fetch — it's an EE-only route not in the OSS spec — with the token added inline.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(cli): regenerate bun.lock for windmill-parser-wasm-asset package.json / package-lock.json carry windmill-parser-wasm-asset@1.740.0 but the tracked bun.lock (the CLI installs/builds/tests via bun) was stale, so fresh bun installs would resolve a different graph than the committed lock. Regenerated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): show asset producer + its runs in the dev-preview panel Selecting a ducklake/asset node in /pipeline_dev showed 'No producer for this asset' because selectionProducers wasn't passed (it's derived from the deployed graph on the route page, absent here). Compute it from the local graph's w/rw write-edges (incl. the // materialize target) and pass it through, mirroring the route page — so the panel shows the producing script and its (preview) runs, including data-test failures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): carry annotation metadata onto local-graph runnables The local graph emitted only path/usage_kind/in_pipeline/materialize_target per runnable, so /pipeline_dev and pipeline show --local weren't the same surface as the deployed graph for annotated scripts — missing the badges/lineage the shared canvas renders. Map the wasm-parsed partition_kind, freshness, tag, retry, data_tests, column_lineage, and materialize_strategy (derived append/merge/replace) onto each runnable, mirroring the deployed AssetGraphRunnableNode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): exclude event handlers that are lineage descendants from runAll The runAll guarantee ('never fires an event handler with empty args') only held for event ROOTS — validStarts excludes them, but runAll then unions in descendants(dag, start), so a kafka/mqtt/... handler that also reads an upstream pipeline asset (a lineage descendant of a valid start) still landed in the plan. Add eventTriggerScripts() and subtract it from the selection after the descendant union. +unit test. Also: docs/pipeline-local-dev.md recipe used 'pipeline docs demo_pipeline' without --local (default queries the deployed graph → hits the empty hint); add --local. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): whole-pipeline run cuts at event handlers (drop their downstream too) The prior runAll fix subtracted event handlers from the selection but left their downstream: for manual_root → asset_x → kafka_handler → asset_y → consumer, deleting only kafka_handler left consumer selected, and topoOrder then ran it as a root with missing/stale event-derived inputs. Replace the descendant-union+delete with reachableCutting(dag, validStarts, eventHandlers): traverse from valid starts but treat event handlers as cut points, so a node reachable ONLY through an event handler is dropped while one reachable via a non-event path stays. +unit test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): recover // tag in the go/bash annotation fallback The wasm path carries out.tag, but the go/bash fallback (and the wasm-error degradation path) only recovered pipeline + on, so a // tag gpu on a bash/go node — or a temporarily-unparseable ts/py/sql node — silently routed the local preview to the default worker while the deployed pipeline routes to the tag. Scan for // tag in fallbackParse too. +test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): extract shared assetProducers helper The 'who writes this asset' write-edge derivation was copied verbatim in PipelineDevView and the pipeline route page — two copies that would drift. Extract assetProducers(graph, selection) into graphTraversal.ts and use it from both, keeping the dev view and route page in lockstep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): only overwrite AGENTS.md/CLAUDE.md when it's the exact generated pointer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): wire local-dev runs into the selected-node runs pane Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): exclude data_upload/webhook entrypoints from auto CLI runs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): --upload binds an object to a data_upload/webhook entry point Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): add "Run + downstream" to the dev preview detail form Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): cut non-autorun triggers on all run paths; multi-binding --upload Address CI review: apply the data_upload/webhook/event barrier cut to the single-root and bounded (--from/--to) paths, not just whole-pipeline; accumulate repeatable --upload bindings per script (were overwritten); scope dev upload keys by script+param to avoid basename clobbering; drop <script> from help text. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): reseed dev run form when a local edit changes the script's args The read-only pane is keyed on script.path only, so in /pipeline_dev the selected node re-resolves on every WS bundle without remounting; PipelineScriptView cloned script.schema once, so adding/removing args left the run form on a stale schema (could run with missing inputs). Extract PipelineRunForm (owns the SchemaForm clone) and key it on the serialized schema: a real arg change reseeds the form, an unchanged re-resolve keeps in-progress input. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): don't cut a scheduled/manual root that also has a non-autorun trigger Address Codex P1: the barrier set subtracted only --upload-bound scripts, so a script with both `// on schedule` and `// on data_upload` resolved as the start yet was also a barrier — reachableCutting skipped it, giving an empty run plan. Subtract all valid starts (schedule/manual roots + bound handlers) from barriers: a legitimately-scheduled root runs on its schedule path even if it also carries a caller-input trigger; pure input-only roots stay cut. Adds a regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): deployed non-autorun enrichment, s3:// storage, --to cut accounting, tag regex Address CI review (Codex P1/P1/P2, Pi P2): - Deployed `pipeline run` recovers marker-only data_upload/webhook/email triggers from script bodies (like the `show` path) so input-only entrypoints are cut instead of auto-run empty on the deployed graph. - `--upload s3://<storage>/<key>` keeps the named storage (authority) instead of folding it into the key, matching the S3Object round-trip convention. - Bounded `--to` targets cut by a barrier are reported in droppedEnds (+warning), not reachableEnds. - fallbackParse `// tag` matches a single token (\S+), rejecting multi-word prose. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): header-only deployed marker scan, fail-closed enrichment, default-storage s3 keys Address CI review (Codex P2, cubic P1/P1/P2): - Deployed marker recovery scans the LEADING comment header only (shared recoverHeaderMarkers helper, reused by the show enrichment too) so a body comment `// on data_upload` can't inject a phantom trigger and over-cut. - Deployed run enrichment fails CLOSED: a script-body fetch error aborts the run instead of silently letting an input-only entrypoint run with empty args. - Revert `--upload s3://` to default-storage whole-path keys (matching pipeline `s3://` asset-URI semantics); named-storage authority-splitting broke nested default keys like `s3://raw/2026/events.csv`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): reject trailing content on fallback native markers; trim s3:/// key Address CI review (Codex P2, cubic P3): - fallbackParse now requires a native marker (`// on data_upload`) to stand alone; a line with trailing content (`// on data_upload f/foo`, `# on kafka topic`) is rejected, matching the canonical parser and keeping local/deployed parity. - s3UriKey trims a leading slash so the canonical empty-authority default form `s3:///key` doesn't leak a leading slash into the object key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): persist dev WS token per-port so reconnect survives a CLI restart Address Codex P2: the /pipeline_dev auto-reconnect reuses the ws_token from the page URL, but `pipeline dev` minted a fresh random token each start, so a restart on the same port left the open page rejected by verifyClient forever. Persist the token per-port under the user-private config dir (0600) and reuse it on restart, so an already-open page reconnects — matching the reconnect behavior's intent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): scope persisted dev WS token by workspace+folder+port Address cubic P2: keying the persisted token by port alone let a stale browser tab from a previous folder's session on the same port reconnect and receive a different folder's source. Scope the token file by workspace+folder+port so a same-session restart still reconnects, but a different folder on the same port gets a distinct token that rejects stale cross-folder tabs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): caller args can't override skip-dispatch guard; hash the dev token key Address CI review (Codex P1, cubic P2): - makeLaunch / CLI run build args with `_wmill_skip_asset_dispatch` LAST (and drop any caller-supplied copy) so a run-form/`--upload` arg can't re-enable backend asset dispatch while the client orchestrates the cascade (double-run / running deployed subscribers from a local preview). Adds a cascadeRun guard test. - Dev WS token file key is a sha256 of NUL-delimited workspace+folder+port, so different folders (`a/b` vs `a_b`) can't collide onto the same token file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): canonical s3://storage/key --upload parsing; scope dev token by remote+root Address Codex P1/P1: - Restore canonical S3Object URI parsing for `--upload` s3 sources, matching the frontend's `parseS3Object` (`s3://<storage>/<key>`, empty authority ⇒ default, `s3:///key`/`s3:///nested/key` for the default store). `s3://secondary/k.csv` → `{ s3: "k.csv", storage: "secondary" }` so a named-storage object is read from the right store. (This is the canonical convention; the default-storage nested key is served by the `s3:///` form.) - Scope the persisted dev WS token by remote+workspace+root+folder+port (was workspace+folder+port), so two profiles on different remotes (or local checkouts) with the same workspace/folder/port don't share a token — a stale tab can't reconnect across a workspace/remote boundary. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 11:41:36 +02:00
Ruben FiszelandClaude Opus 4.8	5a661279a3	feat(pipelines): add managed SCD2 history materialize strategy (#9850) ... * feat(pipelines): add managed SCD2 history materialize strategy `// materialize ducklake://... key=<col> history [track=...]` (alias: `scd2`) upgrades the keyed merge to SCD type 2: diff the current snapshot against live rows, close changed versions (valid_to/is_current) and open new ones in one transaction, keeping full history. Adds a consumer-convenience <dim>_current view; effective-dated joins via native ASOF JOIN >= valid_from. Managed, so // data_test and schema capture work (unlike manual mode). Non-partitioned v1, soft-delete on absence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pipelines): document scd2 track= spacing, reserved _current suffix, schema-freeze Addresses non-blocking CI-review nits on the new SCD2 public surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): null-safe scd2 key matching + create _current view inside txn Addresses CI review: (1) Codex P1 — NULL natural keys were flagged as changed but silently dropped because `key IN (...)` never matches NULL; close/open now match with `IS NOT DISTINCT FROM` via correlated EXISTS. (2) cubic P2 — the `<dim>_current` view was created after COMMIT and CREATE VIEW advances the DuckLake snapshot, so the summary recorded the view's snapshot instead of the data write; the view is now created inside the write transaction. Validated both against a real DuckLake (NULL key materialized; one snapshot per run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): create scd2 _current view with IF NOT EXISTS to keep no-change runs no-op Addresses CI review (Codex P2): CREATE OR REPLACE VIEW advances the DuckLake snapshot every run, so an unchanged rerun still minted/recorded a snapshot. The view definition is static, so IF NOT EXISTS creates it once (folded into the first data-write snapshot) and is a true no-op thereafter — verified an unchanged rerun keeps max(snapshot_id) constant. Also softens the reserved-name collision: IF NOT EXISTS skips silently instead of erroring. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipelines): add scd2 deletes=close (hard-delete-close) Opt-in `deletes=close` closes the current version of a key that disappears from the snapshot (dbt's hard_deletes=close); default stays soft-delete. Codegen adds a vanished-key temp set (current keys EXCEPT snapshot keys) + a second null-safe close UPDATE with no reopen; a reappearing key opens a fresh version (validity gap = correct SCD2). Wired through both parsers with parity fixtures/tests, worker derivation, unit + codegen tests, and docs. Verified end-to-end against a real DuckLake incl. delete-close + reactivation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): align materialize deploy precedence warning with runtime (scd2>append>merge) The deploy-time conflict warning only knew append>key, so warned 'append wins' while the runtime (duckdb_executor) runs SCD2 (history wins). Warn for history+append (history wins, append ignored) before the append+key case, mirroring the runtime strategy precedence. (Pi review P2.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): register scd2 _current view as a produced asset for cascade dispatch The docs present the companion <dim>_current view as a subscribable produced asset (// on ducklake://.../<dim>_current), but deploy registered only the base table as a write asset, so a subscriber on the view would never be dispatched (the cascade fans out from deploy-time asset rows). Register <dim>_current as a produced write asset when scd2 so those subscribers fire. (Codex review P1.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipelines): don't register _current asset for manual+history (no view created) Manual mode short-circuits before the scd2 codegen, so no <dim>_current view is created; gate the produced-asset registration on !manual so a contradictory // materialize manual ... history doesn't leave a false write edge dispatching subscribers on a nonexistent view. (Codex review P2.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-01 10:15:00 +02:00
Ruben FiszelandClaude Opus 4.8	c91027824b	feat(pipeline): AI-chat data-pipeline editor (route + in-session) + home surfacing (#9805) ... * feat(pipeline): AI chat tools to build pipeline nodes with diff/approval Add a data-pipeline AI chat experience modeled on the flow editor and surfaced through the dev-gated global chat (no new chat panel). The /pipeline editor registers PipelineAIChatHelpers on the AIChatManager; while it is open the global mode layers pipeline tools, a pipeline prompt section, and the helpers on top of the full global tool set (behavior is unchanged when no pipeline editor is open). New tools (frontend/src/lib/components/copilot/chat/pipeline/core.ts): - get_pipeline_graph / read_pipeline_node — read the live graph and bodies - build_pipeline_node / edit_pipeline_node — stage changes as AI-pending drafts - remove_pipeline_node — drop a staged proposal - test_pipeline_node — preview-run a node (requires confirmation) Tools never deploy: they stage drafts flagged aiPending, rendered on the canvas with an accent ring and reviewed via Accept all / Reject all (the flow editor's GlobalReviewButtons). Accept commits the drafts; Reject reverts to a pre-AI snapshot, preserving earlier accepted drafts. Auto-accept is gated on the chat autonomy mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): teach the global/session chat to author data pipelines Without an open /pipeline editor the session chat had no pipeline concept, so "create a data pipeline" loaded flow instructions and built a flow. Add a first-class pipeline authoring path: - system_prompts/base/pipeline-base.md — what a data pipeline is (a DAG of annotated scripts wired by storage assets, NOT a flow) and how to author the // pipeline / // on / // materialize annotations; wired through generate.py as getPipelinePrompt() (regenerated prompts.ts/index.ts). - global/core.ts — new get_instructions subject "pipeline", and a global-prompt rule disambiguating data pipelines from flows so the model routes correctly. - ai_evals/cases/global.yaml — two global cases (single node, two-node chain) asserting pipeline-annotated script drafts and forbidding write_flow, guarding the pipeline-vs-flow conflation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): show & build pipelines in the AI session preview Add a 'pipeline' session preview target so the session AI can show the data-pipeline graph for a folder and build nodes in-pane: - open_preview now accepts kind="pipeline" (path = folder); SessionTarget / EDITOR_TARGET_KINDS widen accordingly. The slot/codec load model stays flow|script|raw_app — pipeline bypasses it with its own fetch/draft state. - New PipelineEditorView.svelte mounts in the session pane: fetches the folder graph, overlays AI drafts, renders AssetGraphCanvas + the Accept/Reject review buttons, and registers PipelineAIChatHelpers on the *session-scoped* manager (via getAiChatManager) so build_pipeline_node / edit_pipeline_node + the diff/approval work inside the session too. - System prompt nudges the model to open the pipeline preview and use the staging tools while building. Verified end-to-end with a real model: the session AI called open_preview, the graph mounted in the side panel, then build_pipeline_node staged a node on the session canvas with its schedule trigger and ducklake output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): share the AI editor logic between route page and session Consolidate the duplicated data-pipeline AI logic onto a single shared layer so the route editor and the in-session preview behave identically and the session gains the full code editor. - New pipelineAiHelpers.ts: createPipelineAiHelpers(deps) owns the propose/edit/ remove/accept/reject/test staging + the per-turn snapshot bookkeeping that powers Reject. Callers inject accessors for their own draft Map and graph. - Route page (/pipeline/[folder]) drops its ~250-line inline AI-helper block and wires the shared factory via deps (folder/workspace/graph/drafts + focus, ensureEditable, run-started). Its shell — persistence, navigation guard, activity, cascade, trigger drawers — is untouched. - Session PipelineEditorView uses the same factory and now renders the real AssetGraphDetailsPane (code editor + live overlays + test), so a node built in a session opens with its source, matching the route editor. Verified: route page hydrates/renders drafts unchanged; in a session the AI opened the pipeline preview, built a node, and its code showed in the details pane. check:fast clean, 197 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): externalize editor state into PipelineEditorState (step 1) Introduce PipelineEditorState — the data-pipeline analogue of the flow editor's flowStore. It owns the draft Map, the live editor overlays, and the selection, with callback-safe methods (handleDraftPersist / handleAnnotationsChange / … ), so a single editor can be rendered by both the route page and the session. This commit lands the store and points the in-session PipelineEditorView at it (no behaviour change — the session already had these inline). Next steps move the route page onto the store and a shared <PipelineGraphEditor>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): point the route editor at PipelineEditorState (step 1) Move the route page's draft Map, live editor overlays, selection, and the draft-persist / live-change handlers onto the shared PipelineEditorState (`pe`), referencing them as `pe.*` in place. No behaviour change — persistence, graph resolution, run dispatch, AI staging, and deploy all stay on the page and now read/write the externalized state. This is the data-pipeline analogue of the flow editor's flowStore: the route page and the in-session preview now share one source of editor truth, setting up the shared <PipelineGraphEditor> in the next steps. Verified: the page hydrates its DB draft, renders the overlay graph, the toolbar counts (Save all (N)) track pe.drafts, and selecting a node opens it in the details pane. check:fast clean, 84 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): render the route editor via shared PipelineGraphEditor (step 2) Extract the canvas + details-pane editor body into PipelineGraphEditor.svelte, the data-pipeline analogue of FlowBuilder. The route page now delegates its Splitpanes block to it, passing the externalized PipelineEditorState plus its run/cascade/trigger/deploy callbacks; the component owns pane sizing, selection/details-open derivation, and the canvas+details rendering. Root-caused the earlier ts2769 "$props() No overload" to a prop named `state` colliding with the `$state` rune (`let x = $state(...)` parsed as a store auto-subscription on the prop) — the prop is now `editor`. Net: the route page sheds ~310 lines of template/state; behaviour preserved. Verified: the page hydrates its DB draft, renders the graph, opens the draft in the details pane (live code editor + Test), pane sizing works. check:fast clean, 24 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): move draft autosave into PipelineGraphEditor (step 3) Fold the per-user `data_pipeline` DraftService bundle autosave (hydrate + debounced persist + localStorage crash mirror) into PipelineGraphEditor, gated by a `persistDrafts` prop — FlowBuilder's parameterized-autosave shape. The route page passes `persistDrafts` + `folder` and reads `editor.loadedFromDbDraft` for its AutosaveIndicator; the in-session preview will leave persistence off. Also restores the `untrack(...)` wrapping on the pane-sizing $effect (dropped when the editor body was extracted in step 2). Without it the Pane `bind:size` feedback loops the effect and pegs the main thread when the details pane is closed — a latent hang in the step-2 commit. check:fast clean, 24 pipeline tests pass. Note: browser revalidation was not possible this session (the Playwright MCP browser was reset); the autosave is a verbatim port and the untrack fix is the original working form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): render the session preview via shared PipelineGraphEditor (step 4) Point the in-session PipelineEditorView at the shared PipelineGraphEditor instead of its own inline canvas + details pane. The session now renders the exact same editor body as the route page — gaining the full details/code pane — while opting out of persistence (persistDrafts=false) and the run/cascade/trigger/bounded affordances (their callbacks are omitted, so those controls hide). Building nodes + the Accept/Reject diff still work via the AI helpers. Also fixes issues surfaced by a full `svelte-check` while wiring this up: - PipelineGraphEditor: edit mode opened the details pane unconditionally (a step-2 regression); restored the route's "open only on selection/draft" behaviour. - Route page passed an `isOperator` prop the component doesn't accept (step-2; caught only by full check, not check:fast). - SessionItemNotFound: narrow its `kind` to exclude `pipeline` (pipeline targets never slot-load, so they can't 404 through it) — closes the SessionTarget-widen fallout. - PipelineEditorView: cast the resolveGraph base to AssetGraphResponse. Full `svelte-check` now clean across all pipeline/session files; 137 unit tests pass. (Browser revalidation still pending — Playwright MCP was unavailable this session; see the smoke-test note on the PR.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): stop an infinite microtask loop when persisting a no-output draft handleDraftPersist short-circuits when the open draft's content + inferred writes are unchanged. The writes check compared `d.outputAssets?.length === writes.length`, but a no-output draft has `outputAssets: undefined` (so `?.length` is `undefined`) while the details pane infers an empty `writes: []` (length 0). `undefined === 0` is false, so it never short-circuited: every persist re-wrote the drafts Map with an equivalent object, which gave `activeDraft.script` a new identity → the pane re-emitted its overlays → the graph re-derived → persist fired again. A self- sustaining microtask loop that pegged the renderer and froze the tab on any pipeline carrying a no-output draft (e.g. hydrating one from the saved data_pipeline draft on load). It hangs rather than throwing effect_update_depth_ exceeded because it cycles across microtasks, not within one reactive flush. Fix: coalesce the undefined length to 0 so "no outputs" compares equal to an empty inferred-writes list. Adds pipelineEditorState.test.ts covering the idempotency (fails without the fix) plus the change/no-change cases. Root-caused by instrumenting the reactive churn: every iteration reassigned drafts/liveContent/liveBodyAssets/liveAnnotations/displayGraph with identical values — pure reference churn off the drafts re-write. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): make the agent open the pipeline editor before building nodes In a session, the GLOBAL system prompt only *advised* opening the pipeline preview ("show its graph with open_preview ... prefer those tools once it is open"), so the agent routinely skipped it: on a plain "build a data pipeline" request it reached for write_script and staged plain script drafts, and the canvas editor never opened. build_pipeline_node / edit_pipeline_node are only registered once the preview is open, so skipping open_preview also loses the canvas-staged Accept/Reject diff-approval flow entirely. Make the guidance imperative: open_preview(kind="pipeline", path=<folder>) is the FIRST step before creating any node (an empty or not-yet-created folder is fine — create_folder first if needed), and pipeline nodes go through build_pipeline_node / edit_pipeline_node, never write_script. This also clears the agent's "the folder might not exist" hesitation that pushed it toward write_script. Verified live (same plain prompt, before/after): before it used write_script with no editor; after, the agent opens the editor first and stages a canvas-highlighted node with Accept all / Reject all. The guidance is gated on previewTools (session-only), so it doesn't affect the non-preview global eval cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): preserve in-session pipeline drafts across editor hide/show The session preview's PipelineEditorState lived in the PipelineEditorView component with persistDrafts=false. Hiding the editor sets editorVisible=false, which makes `hasEditor` false and the `{#if hasEditor}` block unmount the view — discarding its component-local store. Showing it again remounted a fresh, empty one, so the pipeline the AI had built in the session vanished. Move the PipelineEditorState onto the per-session SessionRuntime (like the flow / script / raw_app editors, which already host their state there and take {runtime}), so it survives the pane unmount on hide and across session switches. The runtime is keyed by session id and only dropped on session deletion. Because the instance is now reused, guard against a retarget to a different folder: PipelineEditorView resets the state when `path` changes to a new folder (a same-folder remount keeps the drafts). Adds `folder` + `reset()` to the store. Verified: build a node in a session → Close editor → Show editor → the staged node, its wiring, the details-pane code, and Accept/Reject all re-appear. Full svelte-check clean; 139 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline-ai): clearer diff + persistent review banner on the canvas The AI review affordance had two problems on the pipeline canvas: - The floating Accept-all / Reject-all bar sat bottom-center, where it collided with the minimap once the canvas narrowed on node selection — reading as "the buttons vanished when I select a node". - Every staged draft rendered with the same blue ring, so it wasn't clear what the review would actually change (a plain manual draft looked the same as an AI proposal). Replace the floating bar with a top-left review banner (z-30, clear of the controls and minimap) that stays put regardless of selection and spells out the pending counts. Color the diff per node: a proposal that adds a node that isn't deployed rings green with a "new" chip; one that edits an already-deployed node rings amber with an "edited" chip. Plain manual drafts keep the neutral gray dashed border, so only the green/amber nodes read as part of the Accept/Reject set. aiPendingKind is resolved in resolveGraph (deployed runnable present → modified, else added) and forwarded through the canvas to the node. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): persist in-session pipeline proposals across reload/switch Staged AI proposals lived only in the per-session runtime's in-memory PipelineEditorState (persistDrafts=false), so a page reload — and an LRU-evicted runtime on session switch — dropped them, leaving the canvas and the Accept/Reject review empty even though the chat still showed the nodes as staged. Enable the same per-folder DB-draft persistence the route page uses for the in-session editor. To keep hide/show cheap and race-free, hydration is now gated per editor instance (PipelineEditorState.hydratedFromDb) rather than per component mount: the runtime-hosted instance hydrates ONCE when fresh (reload / evicted runtime) and then keeps its in-memory drafts across the editor pane unmounting on hide — re-reading the DB on every remount would race a not-yet-flushed autosave and drop a just-staged draft. A folder retarget resets the flag so the new folder re-hydrates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): make Reject all work for rehydrated proposals rejectAll only reverted paths tracked in the in-memory aiSnapshots map, which is rebuilt empty on each editor mount. After a reload (or session switch into a fresh runtime) the proposals are restored from the persisted draft but have no snapshot, so Reject all was a no-op on exactly the nodes it should discard. Sweep any still-pending draft without a snapshot and discard it (revertPath with no snapshot deletes the path; for an edit of a deployed node that correctly falls back to the deployed body). Adds unit coverage for accept/reject including the no-snapshot case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): keep proposals visible while the graph reloads on switch The session editor pane is LRU-capped (MAX_WARM_EDITORS), so returning to a session whose pane was evicted remounts PipelineEditorView with a fresh graphRes resource (loading=true, current=undefined). The deployed-graph loading spinner gated the whole canvas, so the staged proposals and the Accept/Reject review banner vanished until the re-fetch resolved — read as "the proposal disappears when I switch sessions". Only show the loading/error placeholder when there are no drafts to display. When the runtime already holds staged drafts, render the editor immediately: resolveGraph overlays them on an empty base so the proposals + banner stay visible, and the deployed nodes fill in when the fetch completes. Verified with a 4s-delayed graph fetch — proposals render through the load with no spinner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(pipeline-ai): apply AI node edits directly as drafts, no approve/reject The canvas-level Accept all / Reject all review (aiPending proposals, the green/amber diff ring + "new"/"edited" chips, and the review banner) didn't fit the pipeline editor. Match the flow/script editor instead: build/edit apply directly as ordinary unsaved drafts on the canvas, which the user then deploys — there is no separate approval step. Removed across the surface: - aiPending / aiPendingKind on the runnable node + resolveGraph seeding + canvas forwarding; AI-built nodes now render with the existing plain unsaved-draft dashed styling. - the review banner, count derivations, and hasAiPending/onAccept/onReject props from PipelineGraphEditor and both consumers (route page + session view). - acceptAll/rejectAll/hasPending and the per-turn snapshot bookkeeping from the shared helpers; removeProposedNode now just discards the unsaved draft at a path (undo a build). acceptAllProposals/rejectAllProposals/ hasPendingProposals dropped from the PipelineAIChatHelpers interface and the manager's auto-accept hook. - accept/reject language from the tool descriptions, return messages, and the system-prompt section. Tests updated; pipeline + AssetGraph suites pass (142). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(drafts-diff): support data_pipeline diffs + fix blank empty-summary row Two issues in the session "Drafts" diff drawer (DraftDiffDrawer): - Clicking a `data_pipeline` bundle row threw "Draft diff not supported for kind data_pipeline" (utils_draft_deploy.ts) — there was no handler for the kind, so it fell to the OVERLAY_GETTERS lookup and errored. The bundle has no deployed counterpart (each node deploys individually as a script), so diff it node-by-node: surface each node's draft body keyed by path, folding in the deployed body as the "before" when a node edits a deployed script. - A draft row whose summary is an empty string (e.g. the app draft) rendered with no title at all: WorkspaceItemRow's single-line branch used `summary ?? secondary`, and `??` doesn't treat '' as absent, so it showed the empty summary instead of the path. Use `||` so an empty summary falls back to the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(drafts-diff): explode data_pipeline bundle into per-node subitems A data_pipeline draft is a bundle of node-script drafts, so a single row diffed the whole thing as one blob. Explode it in DraftDiffDrawer into one script row per node, nested under the bundle's `…/data_pipeline` folder so they read as the pipeline's subitems — each with its own path and a proper script Content/Metadata code diff. The node's draft body is the "after"; its deployed body (when the node is already deployed) is the "before", so edits show as line diffs and new nodes as added. A single bundle row (via the getDraftDiffValues data_pipeline fallback) is kept only for the case where the bundle can't be read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(pipeline-ai): simplify — drop vestigial approve/reject scaffolding & redundant field Review pass over the PR, removing complexity left from the approve/reject removal and the shared-component refactor (all behavior-preserving): - Inline the `acceptPendingEdits` pass-through into `acceptPendingFlowEdits` and revert the now-inert `autoAcceptEditsAvailable` GLOBAL+pipeline widening (pipeline edits are direct drafts — nothing to auto-accept). - Fix the global system prompt: pipeline tools "apply directly as unsaved drafts (no accept/reject)", not "proposals the user Accepts or Rejects". - Collapse the redundant `outputAsset` (singular) into `outputAssets`, removing a whole resolveGraph fallback tier; simplify propose/editNode. - Drop the single-field `PipelineAiHelpersHandle` wrapper (callers just destructured `{ helpers }`); inline the misleading `isoNow()` helper. - Remove the now-unreachable `data_pipeline` branch in getDraftDiffValues (the drafts drawer explodes bundles per-node; an unreadable bundle is skipped) and the "Step N consolidation" drafting narration. - Un-export internal-only types; reuse `storageKey`; refresh stale comments that still referenced proposals / the review banner / diff-approval. svelte-check clean; 141 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline): tooltip clarifying the Create/Save button deploys The accent button in the asset-graph details pane ("Create" for a new script, "Save" for an existing one) is really a deploy, but had no tooltip explaining that. Add a title — "Deploy this new script to the workspace" / "Deploy your changes to this script" — keeping the create-vs-update label distinction while making clear both deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): document the `materialize` annotation in the pipeline prompt The model invented "materialize run" because the prompt only mentioned `// materialize <uri>` in passing. Spell out what it is in both the in-app pipeline prompt (getPipelinePromptSection) and the base prompt (pipeline-base.md, regenerated): a MANAGED output where the runtime writes the table around a single SELECT (no manual CREATE/INSERT); replace (default) vs `append` vs `key=<col>` strategies; `manual` to opt out (track-only); and its pairing with `// partitioned …` (runs once per partition, `{partition}` token substituted at run time). Explicitly: materialize is an output declaration, not a command — there is no "materialize run". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline-ai): trigger drawers in the AI session preview Bring the route page's native-trigger affordances to the in-session pipeline editor by reusing the shared <PipelineTriggerEditors> (no duplication of the drawer UI). Clicking a "Schedule · Missing — no trigger row" node (or edit/delete on an attached trigger, webhook, data-upload) now opens the same drawers the full editor uses, instead of doing nothing. Draft nodes get the same "save the script first" guard (a trigger row needs a deployed script). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline-ai): run buttons + live run state in the AI session preview Wire the per-node Run button and live run-state badges into the in-session pipeline editor, reusing the shared folder-scoped job poll (useActiveRunnableIds) the route page uses — node badges, the event log, and the zero-latency "running" hint all come from it. The session runs one node at a time (preview for an unsaved draft, the deployed version otherwise), skipping the route page's cascade/deploy-queue machinery the AI-session UX doesn't need. Verified: a node's Run button dispatches a job and the badge updates live from the poll. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline): label the node deploy button "Deploy" (was Create/Save) Users read "Create" and asked whether it deploys. It does — and the main script editor's DeployButton already says "Deploy", so this is the consistent term. Use "Deploy" for both the new-script and existing-script cases; the new-vs-changes nuance stays in the button's tooltip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(home): surface data pipelines as units, including bundle-phase drafts Treat a data pipeline as one home entry instead of scattering its member scripts: - The home "Pipeline · f/<folder>" entry now also covers bundle-phase pipelines — a folder that so far only exists as a `data_pipeline` draft — not just deployed ones, so a pipeline shows up the moment its first node is drafted (union listPipelineFolders + data_pipeline draft folders). - Pipeline-member scripts (`auto_kind='pipeline'`) are filtered out of the individual scripts list; they're represented by their pipeline's entry. - Tree view injects pipeline folders so they (and their "Pipeline" entry) still appear when their only scripts are hidden members or they have none deployed yet. Verified in both list and tree view: app_groups (deployed member folded) and a draft-only nyc_transit both show as pipelines; the member script no longer lists individually. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(scripts): compute auto_kind for draft-only pipeline nodes A never-deployed pipeline node (a script draft starting with `// pipeline`) had no script row, so list_scripts synthesized it with `auto_kind: None` — and the home page therefore couldn't tell it was a pipeline member, listing it individually instead of folding it into its pipeline. Parse the draft content the same way the create path does (`parse_pipeline_annotations(...).in_pipeline`) and set `auto_kind = "pipeline"` on the synthesized draft-only row, so draft nodes fold into their pipeline like deployed members. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(search): hide pipeline-member scripts from global search The Ctrl+k global search listed pipeline-member scripts (`auto_kind='pipeline'`) individually. Filter them out — they're reached through their pipeline, matching the home page. Deployed members carry auto_kind from the script row; draft-only members now do too (computed from draft content in list_scripts), so both are excluded here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address PR review findings Session run dispatch (the one real bug): - runNode now passes `_wmill_skip_asset_dispatch: true` for a single-node run of a deployed node unless the user chose "run + downstream" (cascade) — previously a single Run could fan out to downstream deployed scripts via the backend asset dispatcher and fire side-effecting production runs. - onRunProducer guards `kind === 'script'`; onTestStateChange only clears the run hint for the script the pane finished (not a different in-flight node); clear the hint on folder retarget; gate the background poll on isActiveSession so hidden warm panes don't poll; note the PipelineTriggerEditors workspace coupling. Home page pipeline surfacing: - Fold pipeline-member folders into `pipelineFolders` (captured in loadScripts) so a members-only / draft-only-`// pipeline` folder still shows its pipeline entry instead of vanishing; and don't render the empty-state when only pipelines remain (they aren't part of the text filter). - Insert injected tree folders in name order instead of prepending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): make clear `// materialize` is DuckDB + DuckLake only The model put `// materialize` on a python3 node, which deploy rejects ("only supported for DuckDB scripts"). The prompt only implied SQL ("write the body as a single SELECT") without stating the hard constraint. Spell it out in both the in-app prompt and pipeline-base.md: `// materialize` is DuckDB-only and its target must be a DuckLake table; for python3/bun/postgresql nodes, write the output via the SDK instead and let it be inferred — reach for duckdb when a node should materialize a DuckLake table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): fix stale comment — session now wires run + trigger affordances Addresses review: the comment still claimed the session 'opts out of the run/cascade/trigger/bounded affordances', but run buttons + trigger drawers were wired in. Describe the current state (wires run + triggers; omits only cascade/bounded/add-script). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address Codex review — test_pipeline_node dispatch + tree search - [P1] testNode (the test_pipeline_node tool) ran a deployed node via runScriptByPath without `_wmill_skip_asset_dispatch`, so previewing one node could fan out to downstream deployed subscribers and run side-effecting scripts. Add the skip flag (test is always single-node) + a regression test. - [P2] Home tree view injected pipeline folders — and rendered their Pipeline row — even during a text search, surfacing unrelated pipelines. Gate both the TreeViewRoot injection and TreeView's hasPipeline on `!isSearching`, matching the list view which hides pipeline rows on a query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): keep the pipeline prompt after update_user_instructions rebuildGlobalSystemMessage (called by the update_user_instructions tool) rebuilt only the base Global prompt, dropping the pipeline-editor section that configureGlobalMode appends. So after the chat remembered an instruction, the next GLOBAL turn lost the active /pipeline/<folder> context + direct-draft/ materialize guidance while pipeline tools stayed registered. Re-append the pipeline section here when a pipeline editor is registered. Addresses Codex review [P2]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(home): gate pipeline entries by kind/archived/owner filters Codex review [P2]: pipeline rows/folders rendered independently of the item filters, so a pipeline still showed under the Flows/Apps tabs, in the archived view, and outside a selected owner. Add `visiblePipelineFolders` applying the same gates the items get (kind ∈ {all, script}, not archived, owner-prefix match) and route the list rows, tree injection, and empty-state check through it. Pipelines are always `f/<folder>`, so the user-folder toggle and kind=script keep including them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address review — route folder-switch state, AI node guards, diff identity claude[bot] [P1]: the route page's in-app folder switcher navigates same-route (no remount), but nothing reset PipelineEditorState — so folder A's drafts displayed under B and autosave persisted them into B's bundle, and B never hydrated. Reset pe on folder change (mirror the session retarget), and guard the shared hydrateDrafts against a stale folder result landing after a retarget. codex/claude [P2]: build_pipeline_node (proposeNode) only checked drafts.has — now rejects a path outside the open folder and one colliding with an existing deployed node (model should edit_pipeline_node). + 3 regression tests. codex/claude [P2]: exploded pipeline-node diff rows shared `script/<path>` with a standalone script draft at the same path, colliding in the {#each} key + value cache. Add an explicit unique `key` (the distinct bundle-nested path) on DiffRow; pipeline nodes set/look up by it while `path` stays the real edit target. claude [P2]: session AI test_pipeline_node now arms the live run badge (onRunStarted), matching the route page. nit: pipelineAiHelpers.test uses afterEach(restoreAllMocks) instead of an unreachable inline mockRestore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): harden AI node mutations + close home label-filter / rename gaps Codex [P1] (AI mutations trust model paths) — fully scoped now: - editNode validates the open folder too (proposeNode already did), via a shared assertInFolder; an edit_pipeline_node for f/other/* no longer persists an unrelated script into the current folder's data_pipeline bundle. - both build_pipeline_node and edit_pipeline_node now require the `// pipeline` annotation (assertPipelineAnnotation) so a staged draft is definitionally a pipeline member, not a silently-non-member script. + tests. (proposeNode's folder + deployed-collision guards landed in the prior commit.) Codex [P2] home label filter — visiblePipelineFolders ignored labelFilter, so a label selection still showed every pipeline (and the empty-state fell through to render pipeline rows). Pipelines carry no labels, so a label filter hides them. Codex [P2] session rename — PipelineEditorView now wires onScriptRenamed (repoint selection + refetch), matching the route page; a persisted-script rename no longer leaves the canvas on the old path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): language-specific comment prefix for annotations Codex [P2]: the tool schema and prompt told the model to write `// pipeline` / `// on` / `// materialize` regardless of language, and pipeline-base.md grouped SQL with `#`. A `//` (or `#`) annotation line is invalid in a DuckDB/Postgres node — it passes the frontend parser (which strips `//`/`--`/`#`) but is a SQL syntax error at deploy/run. Make the guidance language-specific everywhere: `--` for SQL (duckdb/postgresql), `#` for python3/bash, `//` for bun/TS — the `//` in examples is the TS form to translate. Regenerated the prompt outputs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): re-scope Global prompt on folder switch + language-aware base prompt Codex [P2] x2: - The route page resets editor state on an in-app folder switch, but the Global chat's system message kept the old `/pipeline/<folder>` scope (the helper methods read the reactive folder, but the prompt string is only rebuilt on Global-mode reconfigure). Rebuild it on folder change so the next turn targets the new folder. - The pre-editor base Global prompt (seen before open_preview/get_instructions) still showed TS-only `// pipeline` / `// on`. Make it language-aware (`--` SQL, `#` Python/Bash, `//` TS) so the model can't draft invalid DuckDB/Postgres nodes before the pipeline tools are registered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): authoritative new-node probe + SQL-correct eval checklist Codex [P2] x2: - build_pipeline_node's collision check relied on the resolved graph, which can be empty while the session preview races open_preview (a build could shadow a deployed node before the graph loads) and only covered pipeline runnables, not a non-pipeline script at the same path. Add an authoritative backend probe (ScriptService.getScriptByPath): any deployed script at the path → reject with "use edit_pipeline_node". + regression test (empty graph, deployed script). - The DuckLake eval judgeChecklist required the exact `// pipeline` annotation, which would penalize the now-correct `-- pipeline` SQL output (or reward invalid DuckDB syntax). Make both cases syntax-aware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): rebuild Global prompt on session preview folder retarget Codex [P2]: open_preview(kind="pipeline", path="B") can retarget an existing pipeline preview from folder A to B without remounting. The retarget effect resets editor state and the helper methods read the new path, but the registration effect only depends on isActiveSession, so the Global system message stayed scoped to /pipeline/A. Mirror the route-page fix: rebuild the global system message on retarget (gated on isActiveSession — only the active session's helpers are registered; a hidden session reconfigures when it next becomes active). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): edit_pipeline_node preserves deployed script metadata Codex [P1]: editNode kept only the deployed script's language and staged a fresh makePipelineScript draft with empty hash/summary/description/tag/schema/settings. Deploying that edit from the pane (auto_parent) would update the script while wiping its metadata, and the route "Save all" path (no parent_hash) could hit the backend path-conflict branch on the occupied path. Base the draft on the existing draft's / deployed script object and replace ONLY content (+ inferred output assets), preserving hash and metadata. + regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-29 20:33:50 +00:00
Ruben FiszelandClaude Opus 4.8	003a262a4e	feat: column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) (#9814) ... * feat: column-level lineage for ducklake pipelines via // column annotation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: auto-derive column lineage from DuckDB SQL AST (annotation as override) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: clarify column-lineage inference is server-side; drafts use annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): surface inferred column lineage in live pipeline drafts Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through ScriptEditor -> details pane -> page -> resolveGraph, merged with // column annotations (annotation wins) so the live preview matches the deployed graph. Takes effect once windmill-parser-wasm-asset is republished with the inference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: column-lineage inference now runs live (WASM) too, merged with annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): transitive column-lineage trace (impact analysis) Stitches every producer's column_lineage into a pipeline-wide column graph (columnLineageGraph.ts) and replaces the single-hop diagram with an interactive ColumnLineageTrace: select an asset to see its columns' full upstream/downstream lineage across scripts; click any column to highlight its complete transitive impact set (forward + backward) and dim the rest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on column lineage (parse-fallback, node-id, perf, leak) - backend: DuckDB SQL parse failure now falls back to `// column` annotation lineage instead of dropping it (Codex P1) - columnLineageGraph: collision-proof JSON node ids; deterministic first-write output anchoring when a producer has multiple ducklake writes (cubic P2 ×2) - pipeline page: gate buildColumnGraph to a ducklake-asset selection so it doesn't rebuild on every editor keystroke (cubic P2) - ScriptEditor: clear inferredColumnLineage on parse error so it can't leak across a script switch (cubic P2) - AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: resolve JOIN inputs + anchor column lineage to // materialize target Addresses the second Codex review pass (two P1s): - SQL inference now walks JOINed tables: build_from_maps maps every FROM entry AND its joins into the alias map, and single-table attribution requires no joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped). - The column graph anchors a producer's lineage to its declared // materialize target (surfaced on the runnable node) instead of guessing a ducklake write-edge, which is unordered for deployed graphs and ambiguous for multi-output scripts. Falls back to a write-edge when no materialize target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate column-lineage badge to the // materialize target write-edge The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output producer showed the same column mapping on every ducklake write-edge. Use the same materialize-target anchor as buildColumnGraph: the badge lands only on the declared output's edge, falling back to the ducklake write-edge when there's no materialize annotation. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: build column trace from displayGraph so View hides draft lineage The transitive column trace was built from graphWithDraft regardless of mode, so in View with drafts hidden it could surface draft `// column` lineage the deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas actually renders) so the trace matches: draft overlays in edit / show-drafts, deployed-only in plain View. (Codex P2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: don't infer column lineage for local/temp staging CTAS A CTAS into a local/temp staging table isn't the materialized output, but its projection was inferred and (flat) column_lineage anchored to the script's // materialize target — so staging columns showed up as the final asset's. Gate inference to the actual output: a top-level managed-materialize SELECT, or a CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: scope inferred column lineage to one output asset Inference accumulated columns from every output-producing query into one flat list, all anchored (frontend) to the script's // materialize target — so an auxiliary CTAS into a different asset showed its columns on the materialized one. Tag each inferred entry with its output asset and, in parse_assets, scope the list to the // materialize target (keeping untagged top-level-SELECT entries); with no declared target, drop inference when entries span multiple output assets rather than attribute them to an arbitrary one. Parser-internal — no wire change. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp) before being registered local, so a final SELECT reading it invented `final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a phantom asset. track_table_definition now registers any temporary table/view as local up front, bypassing active-asset resolution; CreateTable/CreateView pass their `temporary` flag. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-27 21:08:00 +02:00
Ruben FiszelandClaude Opus 4.8	ade74b297f	feat: capture managed-materialize output schema as asset metadata (#2a) (#9812) ... * feat: capture managed-materialize output schema as asset metadata (#2a) After a managed `// materialize` run, capture the producer's output schema via a DESCRIBE folded into the existing one-row summary read (no extra round-trip) and persist it in a new versioned `materialized_asset_schema` sidecar table. This is the producer-side capture that pipeline parity gap #2b (save-time consumer-ref contract enforcement) will read back. - materialized_asset_schema sidecar (asset-level grain), versioned: a new version row is inserted only when the captured column set changes. - output_schema column added to the materialize summary codegen. - worker extracts + records the schema on a successful materialize. - /assets/asset_schemas read endpoint exposing the evolution history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on schema capture (partition col, order, status gate) - exclude the synthetic `_wm_partition` column from the captured schema for partitioned assets, so the recorded contract is the producer's logical output, not Windmill's storage detail (claude/cubic P1). - make the captured column list explicitly ordered (`row_number()` over the DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder columns and spuriously bump the schema version (cubic P2). - gate the API `record_materialization` schema upsert on a `Materialized` status, so a failed/running write (or a client attaching a schema to one) can't advance the schema history (cubic P2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Codex review (manual-mode schema gate + auth contract docs) - gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a `// materialize manual` run — whose result is the user's own query output — can't persist a caller-shaped `output_schema` into materialized_asset_schema (Codex P2). Verified e2e: a manual run returning a fabricated `output_schema:[{injected,EVIL}]` records the partition but writes no schema version, while the managed path still captures normally. - document the authorization contract on the new public `record_asset_schema` and `list_asset_schemas` helpers: they perform no access control (mirroring the materialized_partition siblings) and require callers to pass a workspace-authorized executor (Codex P1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): schema-history tab on the ducklake asset node (#2a) Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema versions persisted by the materialize run. Master-detail (mirrors the History tab): the version list (newest first, newest auto-selected) shows column count + snapshot + capture time; selecting a version renders its column/type table. Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling PartitionStatusGrid convention (these materialization endpoints are not in the generated client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: schema tab is strategy-aware (history vs fixed schema) Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema table, so their schema is pinned at first materialize and the "history" framing is degenerate (always one version). - backend: surface the managed `materialize_strategy` (`replace`/`append`/ `merge`) on the asset-graph runnable node, alongside the existing `partition_kind` (same parse-from-annotation path). - frontend: the pipeline page derives `schemaCanEvolve` for the selected asset from its write-producer (`replace` && not partitioned) and threads it to the Schema tab. Evolvable → master-detail version history; fixed → a single current-schema table with a short "schema is fixed" note. Unknown defaults to evolvable so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: schemaCanEvolve fails open on unknown producer strategy Previously a producer present but missing `materialize_strategy` (e.g. a draft-overlay runnable, synthesized without the field) fell through to canEvolve=false, hiding captured history behind the fixed-schema view — contradicting the "unknown defaults to evolvable" intent. Now the fixed view shows only when *every* producer is a known insert-style write (append/merge, or partitioned replace); any producer with unknown (missing) strategy is treated as evolvable, so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 19:46:48 +02:00
Ruben FiszelandClaude Opus 4.8	248540ac4d	feat: bounded-cascade selective execution for pipelines (UI + CLI) (#9695) ... * feat: bounded-cascade selective execution for pipelines (UI + CLI) Run a prefix of a pipeline cascade: from a schedule/manual root, fan downstream but stop at chosen end node(s) — the path-between set over the asset-graph lineage DAG. Exposed as a canvas 'Run downstream up to…' pick mode and a 'wmill pipeline run <folder> --to' CLI command. No backend or parser changes; reads the existing graph, tags, and triggers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: surface bounded-run on the run caret, trigger-node kebab, and Test button Move 'Run downstream up to…' from the runnable kebab onto the play-button caret popover (Edit mode, next to Run / Run + trigger N downstream); add it to the trigger-node kebab so schedule/data_upload entrypoints expose it on the View page; and to the ScriptEditor Test split caret for the open script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address CI review on bounded-cascade (cubic) - Port CLI engine test from Deno to bun:test under cli/test/ (won't run under bun test otherwise). - closure() now excludes the start node on a cycle back to it (descendants/ancestors contract); regression tests both engines. - CLI 'pipeline run --to' rejects unresolved/ambiguous end tokens instead of silently running a different subset. - Sort a copy in the runSelection order test so the launch-order assertions aren't invalidated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address standing review nits on bounded-cascade Resolves the four recurring P1/P2 findings from the codex/pi/claude reviews: - UI gate (P1): the canvas/trigger-node "Run downstream up to…" affordance was gated on the subscriber-only downstream map, so a valid start whose only downstream is a pure reader had a non-empty bounded set but no menu entry. Gate on the read-aware lineage downstream (buildLineageDownstreamMap), matching the bounded engine. - waitJob (CLI): a completed job without explicit success:true now counts as a failure, mirroring the frontend waitJobTerminal — the cascade only advances on a confirmed success. - Comment fix (CLI): the unbounded `run` path uses the read-aware lineage DAG (pure readers included); dropped the false "parity with the canvas cascade" (subscriber-only) claim. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: expose bounded-run caret for pure-reader-only starts (codex P1) The canvas wiring from the prior commit passed `onStartBoundedRun` from the read-aware lineage map, but the leaf components still hid the popover that holds the "Run downstream up to…" action behind a subscriber-only gate: - RunnableNode rendered the Run-button caret only when `hasCascade = downstreamCount > 0` (subscriber-only). A valid start whose only downstream is a pure reader got `onStartBoundedRun` but no visible action. Now the caret opens when there's a cascade OR a bounded-run start (`hasCaret`), and the "Run + trigger N downstream" item is gated on `hasCascade` so it never reads "trigger 0". - ScriptEditor's Test split button activated only when `downstreamSubscribers > 0`, falling through to a plain Test button (no caret) otherwise. Now it also activates when `onBoundedRun` is set, with the "Test + trigger N" item gated on the count. For a manual root (no trigger-node kebab fallback) with a pure-reader downstream this was the only UI entry point, so it was previously unreachable. Verified in-browser: a manual-root script writing an asset read-only downstream now exposes "Run downstream up to…" on the ScriptEditor Test caret with the cascade item hidden. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate ScriptEditor bounded-run on read-aware downstream; fix CLI asset-end warning (codex P2) - Details-pane (ScriptEditor) bounded-run entry was gated only on `validStartPaths`, broader than the canvas which also requires read-aware downstream (`hasLineageDownstream`). An isolated start could thus expose "Run downstream up to…" and enter pick mode with no selectable end. Now gated on `lineageDownstreamPaths` (script paths with a downstream in `buildLineageDownstreamMap`), matching the canvas. - CLI dropped-end warning called `scriptPathOf(d)` unconditionally, which slices `script:`-length chars off an asset id too — `datatable:main/raw` printed as `le:main/raw`. Now prefix-checks like the JSON output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: correct --from error to exclude only row-backed event triggers (codex P2) The bounded-start validation message listed `kafka/webhook/…` as event triggers that can't start a bounded run, but webhook/data_upload are rowless and read as manual roots (valid starts). Only the row-backed native kinds (kafka/mqtt/nats/postgres/sqs/gcp/email — EVENT_TRIGGER_KINDS) are excluded; the message now names those. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: surface dropped ends in CLI JSON; disambiguate shared-trigger bounded start (codex P2) - CLI `run --json` silenced the dropped-end warning, and the JSON payload echoed the originally-resolved `--to` list with no reachable/dropped split — a resolved-but-unreachable end looked like a clean plan that silently runs only the start. JSON now includes `reachableEnds` and `droppedEnds` (shared `idLabel` helper, asset-id safe). - Trigger nodes dedupe per (kind, ref), so a schedule shared across scripts collapses to one node, but `recordSourceTrigger` kept only the first target path — the bounded-run action then rooted at an arbitrary script (or hid when only that first script lacked downstream). Now all target paths are tracked and the action is offered only when exactly one is a valid start with downstream; multi-eligible nodes suppress it rather than guess. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't run hidden drafts in View-mode bounded cascade (codex P1) launchCascadeScript unconditionally preferred drafts.get(path) over the deployed script. In View mode with drafts hidden (displayGraph is deployed-only), a bounded run started from a trigger-node kebab would execute preview jobs from hidden local draft content instead of the deployed scripts the user is looking at. Gate draft execution on `mode === 'edit' || includeDrafts` — the exact condition under which displayGraph includes drafts — so execution always matches the displayed graph. No-op for scripts without a draft; the edit-mode "Run + trigger N downstream" cascade is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 12:08:05 +02:00