125 Commits

Include renames

Author	SHA1	Message	Date
Ruben FiszelandClaude Opus 5	0139467b01	feat: ingest dbt column lineage and real column schemas from the engine's parquet index (#10977) ... * feat: column-level lineage for dbt from the engine's parquet index `manifest.json` carries no column-to-column edges, which is why decision 14 recorded column lineage as unavailable. The edges live in a different artifact: `dbt compile --static-analysis strict --write-index` writes `target/index/`, whose `dbt.column_lineage.parquet` holds them and whose `dbt.node_columns.parquet` holds every column of every node, typed and ordered rather than only the ones an author documented. Strict analysis rejects SQL the default accepts, so this is a separate compile with its own `--target-path`, opt-in per project via `column_lineage: true`, and best-effort throughout: a project it cannot analyze keeps exactly the graph it had, with the engine's own diagnostics in the job log. Storage mirrors `dbt_edge`: `dbt_column_edge` keyed by (path, version, job) with the same composite FK to `script` and the same sweeps. The typed column list lands in `dbt_node.column_schema`, beside `columns` rather than merged into it, so `columns` stays what the author declared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: address review findings on the dbt column-lineage pass - The workspace fork copied every other dbt sidecar table and not this one, so a fork lost its column lineage silently and could not recover it: the cloned digest covers the column edges, so a dynamic run in the fork matched it and stored nothing. - The parquet was collected whole before the edge cap applied, which is exactly the input the cap exists for — a project whose `scan` lineage is quadratic in its widest model could take the worker process down. Decoded a row at a time with the bound enforced during the decode. - The pass swallowed every error from the runner, including the job poller's cancellation and deadline, so a run that blew its timeout inside an optional annotation could still publish a graph and report success. `run_captured` now carries the exit status in its value, so only a failed COMPILE is downgraded, and the pass may spend at most half the remaining wall clock so it cannot starve the build that follows it. - `scan` edges are stored but no longer served: they are most of a project's lineage, nothing renders them, and the graph endpoint is polled by the run page. They are also the first thing the storage cap gives up now, rather than evicting the direct edges the trace draws. - `column_schema` and the column edges take the same gate as the model's SQL. A column-level view is the shape of what the author wrote, one level finer than the `ref()` graph, which is ungated only because it draws relations the caller already sees. - `graph_digest` hashes the new section only when it has edges, so a project that never asked for the pass keeps the digest it has instead of re-snapshotting on every dynamic run until it is redeployed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: the editor buffer's column lineage, and three bounds that were wrong Round-2 review found four defects, all of them introduced by the round-1 fixes. - The `script_visible` gate on the column edges was copied from the node query without its `script_hash IS NULL` arm. `= NULL` is never true, so every version-less row was filtered out and an editor buffer's parse rendered its typed columns and none of their lineage — the one place the feature is meant to be used. Pinned by an assertion in `dbt_pinned_graph.rs`, which is where this class of bug already had a home. - The phase budget was handed to the poller, whose expiry is an `Err` indistinguishable from a cancellation or the job's own deadline, so a slow but valid analysis aborted the build it exists to annotate. The runner gets the full deadline again — those two must still fail the job — and the budget is a race around the whole pass, where expiring is this budget and nothing else. - The decode cap counted parquet ROWS, so `scan` and out-of-graph rows could spend it before a single drawn edge was read. It now counts what is kept, takes direct kinds in a first pass, and is handed the graph's own nodes so the budget cannot go on rows that could never be stored. - Hashing the new digest section conditionally did not preserve old digests, because an absent `column_schema` still serialized as `null` inside the nodes. It is skipped when absent instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: split the lineage pass by error contract, and read it in one query Round 3's findings were all consequences of round 1 and 2's fixes, clustered in the same two files, so this reshapes those two seams rather than patching again. The worker pass was one function being three things at once — a subprocess runner with job-lifecycle error semantics, a bounded decoder, and a best-effort degrader — which is why each fix to one perturbed another. It is now `compile_index`, which owns the JOB's semantics (only a cancellation or the job's deadline can `Err`; a non-zero exit, the output ceiling and the phase budget are outcomes), and `read_index`, which owns the ARTIFACT's and knows nothing about the job. The budget wraps the compile alone, so a decode can no longer outlive the timeout that reported the build would get the rest. The output ceiling likewise becomes a value rather than a job error, for the caller that can carry on without the tail of a compile's stdout. The column edges were read by a fourth hand-written copy of the `live`/`chosen` CTEs and the version/editor-buffer join conditions, and copying them is what dropped the `script_hash IS NULL` arm and hid every buffer parse's lineage. Both kinds of edge now come from ONE statement over a `UNION ALL`'d edge source, so those conditions exist once. The union is at the source rather than a join because column lineage can name a node pair `dbt_edge` has no row for: a model reading `{{ this }}` gets edges from itself to itself, and `parent_map` has no self-loop. The cap on the column half now sits after the scope filter, the visibility check and the graph joins — the scope moved into SQL via the existing `ScopePathFilter` — so a row the caller may not read can no longer spend it and leave an allowed project's trace short. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: serve dbt column lineage from its own endpoint The column edges rode on the folder-wide asset graph, which a run page polls, while the trace is drawn for one selected relation. That needed a cap, and a cap has to be applied after every filter that can drop a row. Keyed to the asset there is no cap: `assets/column_lineage` answers for one relation, and the caller's `scripts:read` scope and the project's visibility are decided once, for the script that owns it. Pinning to a run's snapshot or the editor's parse of its buffer costs the job-read gate, so that form is `jobs/dbt_column_lineage/{id}` — the same shape `jobs/dbt_graph/{id}` has. The worker's decode now bounds work and memory separately, and a compile stopped by the output ceiling reports as truncated rather than complete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve the owning dbt version the way the graph does The unpinned arm picked the newest live version at the path without narrowing to dbt, so a path since redeployed in another language answered with no lineage while the graph beside it still drew that project's stale nodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate pinned column lineage on reading the project, and answer the component Four things round 5 found, three of them in code this branch rewrote: - The pinned arm resolved the version from the job and stopped there, so a share-link viewer entitled to a run got the project's column names and edges while the graph beside it still redacted `raw_code` and `column_schema`. Resolving WHICH version answers is not deciding whether the caller may read it; the version-less editor buffer keeps its exemption, having no `script` row to ask. - The answer was the whole owning project's edges. The canvas lays out the connected component of the selected relation's columns, so the rest was unrenderable weight; a recursive walk over both directions returns exactly what is drawn, and the project key travels with it so a `unique_id` two projects share cannot walk from one graph into the other. - The decode had no exit but the 4M-row backstop once its buckets were full, spending wall clock the build below does not get. - An unreadable index was reported as a missing one, sending the reader to look at their engine rather than at the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stitch the two column graphs, and walk the component in Rust Round 6's two findings, both regressions this branch introduced: - The decode returned `Continue` on the edge that FILLED the direct-edge budget, so a `scan`-only tail after it decoded to the 4M-row backstop with nowhere to put anything. The read now ends on that edge. - Seam 3 made the pipeline page choose between the dbt graph and the producer one. They share node ids — `// column total <- dbt://wh/analytics/orders.amount` mints the same `(dbt, path, column)` node dbt's own lineage does — so choosing ended a trace at the boundary in both directions. They are merged again, and a ducklake selection asks about the dbt relation its producers name so the chain continues past it. The dbt editor gets the same merge. Also: the component is walked in Rust rather than by a recursive CTE. A CTE has no index, so the recursive term rescanned the doubled edge set once per level — 1243ms against 59ms for the query alone on a 3000-model project, 11.7M rows in the plan. Same answers, same tests; end to end 1.48s to 0.73s there and 1.60s to 0.26s on a 1000-deep chain. The client stops re-asking for a component it already holds, which is most clicks within one project. The four doc sites that described a whole-project answer are rewritten around what it now is, rather than edited where they disagreed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: expand every dbt boundary a selection reaches, and only skip what was asked Round 7's findings, all in the frontend seam this branch added: - A ducklake selection seeded the dbt fetch from the FIRST boundary relation it found, so a table derived from two unconnected dbt relations expanded one and left the other a leaf — the same "stops at the boundary" symptom the round-6 fix removed, one hop further along. Every distinct boundary is fetched now and the components merged. - The component cache skipped a relation merely PRESENT in the graph in hand. A relation two projects describe has an owner row in each, and a component fetched for one carries it as an endpoint without the other's half, so that skipped the request that would have resolved the second owner. Only a relation actually asked about under this pin is skipped. - A comment still called the producer graph gated to ducklake selections after it was widened to dbt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: land dbt column lineage as storage and ingest only The API surface that draws a column trace moves to a follow-up PR, on `dbt-column-lineage-surface`. It kept generating findings — a client cache whose premise was wrong for a two-owner relation, then staleness and a lost retry from tightening it, and a seed walk that stopped at the first boundary — and the fix for the last of them is a transitive owner expansion, which has to re-apply the caller's gate to every newly discovered project. That is the same shape as the leak four reviewers caught in the pinned arm, and it wants its own review rather than being the fourth fix at the end of this one. What lands here stands on its own: the analysis pass, `dbt_column_edge`, `dbt_node.column_schema`, the engine gating and the error-contract split — plus the one user-visible half, the typed and ordered column list, which rides the asset graph the details pane already fetches and replaces a panel that could only show the columns an author had documented. Also fixes a real bug in the pass, found in review: it compiled without the build's `--full-refresh`. `is_incremental()` branches on that flag, so an incremental model reading `{{ this }}` compiles its self-join — and any `ref()` inside that branch — only when the flag is absent, and the pass was storing lineage for SQL a full-refresh run never executed. The flag now comes from one place shared with the build, and a run that overrides it gets its own graph rather than standing as the version's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say why direct kinds get the budget without naming a view The bucketing comments explained the priority by what a trace draws, which is a forward reference now that the surface moved out. The reason stands on its own: `copy`/`mod` say the value travelled, `scan` says the column was read to produce the row and so reaches every output column of its model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: round-9 findings on the descoped PR - The `full_refresh` helper was inserted between `selection_is_overridden` and its doc comment, so thirteen lines about `select`/`exclude` echoes documented the wrong function and the one they were written for had none. Moved below it. - The parse path ran the analysis compile and the parquet decode BEFORE the guard that returns when there is no warehouse identity, paying for both and dropping the result. Moved after it. - Three sites still described a `/column_lineage` endpoint this branch no longer has, and two user-facing strings promised a column trace it no longer renders: the panel's hint and the descriptor template now say what the flag actually buys, which is the typed column schema. - Dropped test scaffolding the removed suite left behind: a `raw_orders` node and `dbt_edge` whose only assertion re-tested pre-existing graph behaviour, and a second editor-buffer node nothing asserts on. Documented rather than fixed: an incremental model has two shapes, and which one the index holds depends on whether the target existed when the pass ran. `is_incremental()` is false with no target as well as under `--full-refresh`, and dbt has no mode that emits both — so a version's graph describes the compile that produced it, and only a re-ingesting run describes its own run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep lineage_kind in the edge key, and one answer for --full-refresh - Both unique indexes omitted `lineage_kind`, so a column that is projected AND used as a predicate for the same output column — an ordinary shape — had its `copy` and `scan` edges collapse under `ON CONFLICT DO NOTHING`, while the digest counted both. The kind is part of the fact, so it is part of the key. Edited in the migration rather than added as a second one: it has not landed. - `full_refresh` was shared between the build and the analysis pass without the `command != "test"` condition that sat at the build's call site, so the two disagreed for exactly the runs that build nothing. The condition moved inside the function, which is the point of sharing it, and the command is threaded to the pass. - The "what a trace draws" rewrite missed the copy in `dbt_manifest.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop the unreachable full_refresh threading, test the uniqueness key `DBT_COMMANDS` is `["build", "retry", "show", "parse"]` and `default_command` returns `build` in every arm, so `command == "test"` cannot happen — the guard the last commit moved into `full_refresh` was already inert where it came from. Threading the command through five signatures to preserve it bought nothing, so it is gone; the build and the pass call one function of the descriptor and the invocation, which is what the sharing was for. The uniqueness-key fix now has a test: a column projected AND used as a predicate for the same output column stores both its `copy` and its `scan` row. Verified against the old key, where it returns 1 instead of 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: restore the dbt test --full-refresh guard I removed on a wrong premise The previous commit removed it after reading `DBT_COMMANDS` and concluding `"test"` was unreachable. That is only true of the command a CALLER can name: `run_dbt` is invoked with `"test"` directly for the `after_all` test phase, so an `after_all` project with `full_refresh: true` reached it — and dbt rejects `--full-refresh` on `test`, failing the phase. Both reviewers caught it. The guard is back inside the shared function, where the build and the pass get one answer, and its doc now records why reading the allowlist alone is misleading. The test covering the `test` case is restored with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: notice a job that ended during the decode, and name truncation as the cause - The parquet decode runs on a blocking thread with no poller watching it, so a cancellation or an expired deadline during it was invisible: `dbt_dep` went on to publish the graph and the job returned success. The job's state is checked once the decode returns, before the caller publishes anything, and an ended job `Err`s — which this module may always do for the job's own semantics. - A compile stopped by the output ceiling could leave no artifact, and the log then blamed the engine's capability, sending the reader to check their adapter rather than the ceiling. Truncation now names itself in the missing and unreadable branches too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read cancellation from the DB after the decode, not from a poller's field `ctx.canceled_by` is only ever written by a poller, and no poller runs during the blocking decode — which is the exact window the check was added for. So the guard caught only a cancellation already observed before it, and the comment beside it claimed more than it did. It now queries `v2_job_queue` directly, the same probe `worker_lockfiles` uses before it overwrites a flow. A failed probe answers "still running": this decides whether to discard work already done, so an unreachable database must not be the reason a healthy deploy loses its graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: reuse job_is_canceled rather than a second copy of it The probe added last round was `job_is_canceled` from the same file, retyped — same query, same `Connection::Http` behaviour. Reused instead. Its doc said a non-database connection was "a failed probe", which reads as an error path. It is not: it is the agent worker, and on one there is no database to ask, so only the deadline answers and a cancel issued during the decode is not observable. The retry path avoids that by refusing to run on an agent worker at all — which an optional annotation has no business doing — so the gap is recorded at both ends instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: close the agent-worker cancellation gap instead of documenting it The previous commit said a cancel issued during the decode is not observable on an agent worker. It is: `ping_job_status` returns `canceled_by` over both connection kinds, and is how the poller itself notices one there. So the check asks through the ping rather than querying `v2_job_queue` directly, and holds on an agent worker, where a direct query reaches no database at all. `job_is_canceled` goes back to private and its doc to what it said before — the retry that calls it still refuses to run on an agent worker for its own reasons. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decode the index under the job poller instead of checking after it Two findings with one cause: the decode was the only phase of this pass with no subprocess behind it, so nothing heartbeated while it ran. A large index left the worker silent for as long as it took, which the zombie sweep reads as a dead job and restarts — and the cancellation check bolted on afterwards could only ever report what had already happened, while dropping the ping's `already_completed`, so a force-cancelled deploy still published its graph. Running it under `run_future_with_polling_update_job_poller` answers all of it: the poller pings throughout, and ends the phase with an `Err` on cancellation, `AlreadyCompleted` or the phase timeout. The bespoke probe is gone with it. Verified on a live deploy: 32 edges and 4 typed schemas ingested through the polled decode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop a cancelled decode, and say what the read phase can now do Putting the decode under the poller heartbeats it and ends the phase when the job does, but dropping a `JoinHandle` detaches a blocking task rather than cancelling it — so a cancelled job left a thread decoding up to four million rows for a job that was over. The row loop reads an abandonment flag that a drop guard on the awaiting future sets, so the decode stops at its next row. That same change made the read phase able to `Err`, and three places still said it could not — decision 14 in as many words. The distinction that holds is narrower: nothing the ARTIFACT does or fails to do can fail a job, so absent, unreadable and partial are all values; the JOB can still end the phase the read runs in. Stated that way in the module doc, the `Artifact` doc, `MAX_INDEX_ROWS` and the decision. Verified on a live deploy: 32 edges and 4 typed schemas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: share AbortOnDrop, and stop citing a hazard that is now handled `Abandon` was `ansible_executor`'s `AbortOnDrop` retyped — same struct, same reason, same `spawn_blocking` shape. Moved to `common` and used from both. The paragraph explaining why the phase budget wraps the compile alone gave as its reason "a decode still running on a blocking thread", which is exactly what the abandonment flag now prevents. The reason that survives is the one that was always the point: the budget exists to leave the build its share of the clock, and only the compile can spend that share unboundedly. The decode's end is the job's, through the poller it runs under. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: put both doc comments back on the items they describe Moving AbortOnDrop orphaned a doc at each end: it landed between `raw_to_string`'s doc and `raw_to_string`, and the doc of the struct it replaced stayed behind to prefix `fetch_repo_archive`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the binding the row loop actually reads `Abandoned` was neither the type nor the binding; the flag is `abandoned`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-08 09:58:38 +02:00
Ruben FiszelandClaude Opus 5	54287102b2	fix: meter WAC compute per segment, not the whole sleep (#10985) ... * fix: clear started_at when a WAC parent suspends Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg * fix: restore started_at on the WAC dispatch rollback, fail loudly on a no-op suspend Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg * fix: restore the pulled segment start on the WAC dispatch rollback Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg * feat: meter WAC execution per segment instead of only the last one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg * fix: make the cloud feature self-sufficient per crate Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg * chore: name windmill-common/cloud directly in the worker cloud feature Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AzS5d5Dc3GL49kWZPQAQCg --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-05 11:11:29 +02:00
3 people	5099f405d4	feat: make the Git Repo Viewer work with GitHub App repositories (#10765) ... * fix: resolve the head commit of GitHub App repos in the git repo viewer `get_git_commit_hash` ran `git ls-remote` against the raw resource URL. A GitHub-App-backed repository stores a tokenless URL, so the probe failed with "could not read Username" and the viewer never got past its first step. Resolve the head over the GitHub REST API with a server-side installation token instead, reusing the lookup the auto-pull poller already uses for app repos. Non-app repositories keep the ls-remote path. Also picks up the EE-side allowlist fix that lets the clone hub script request an installation token. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 63c67e2a2db198af26a0334f5be14af7d9987eb1 This commit updates the EE repository reference after PR #732 was merged in windmill-ee-private. Previous ee-repo-ref: 2a260961fa0a9bb5631c17e2f718cb8efb4f9aa2 New ee-repo-ref: 63c67e2a2db198af26a0334f5be14af7d9987eb1 Automated by sync-ee-ref workflow. * fix: honour the app-repo head lookup's not-app-backed result `get_app_repo_head_for_autopull` documents `Ok(None)` as "this repo is not app-backed, use the ls-remote path", which is what the other two callers do. Fall through to `ls-remote` on `None` instead of turning it into a 500, and drop the handler's own `is_github_app` read now that the callee's answer is honoured. Also bumps ee-repo-ref to pick up route-safe ref handling in that lookup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: serve GitHub App repositories as an archive instead of a token The viewer's clone script asked the server for an installation token and put it in the clone URL. That token is installation-wide and carries the App's full permissions, so minting one requires a workspace admin, and the viewer was therefore admin-only for app-backed repositories. The server now streams a tarball of the commit instead, authorized by read access to the git_repository resource, so no GitHub credential reaches the job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: run delegate_to_git_repo playbooks from GitHub App repositories An Ansible job's runnable_path is the user's own script, which no entry in the git-sync script allowlist can match, so `delegate_to_git_repo` could never obtain a token for an app-backed repo. It also gave up entirely on agent workers, whose connection has no database to mint one from. A playbook run only reads a working tree: the clone is followed by one rev-parse for a log line, and nothing after that touches git. So take the same archive route the viewer uses, extracting the commit's tarball into the job's repository directory. No GitHub credential reaches the worker, and agent workers work because the route is HTTP. Archive entries are joined onto the target by hand so a crafted archive cannot write outside the job directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: drop the now-immutable secret_url binding Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: point the repo viewer at the archive-based clone script hub/28905 reads app-backed repositories through the server's archive route instead of minting an installation token, which the backend in this release no longer grants it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: stream repository archives to disk rather than into memory The archive download went through `AuthedClient::get`, whose client caps a request at 20 seconds and whose response was then buffered whole. A repository is arbitrarily large, so that cut off slow downloads and put every job on the worker at risk of running the process out of memory. Add `get_streaming`, the read counterpart to the streaming upload path, and write the response out chunk by chunk. Extraction now creates each entry's parent directory: a tar carries directory entries only by convention, and the traversal guard now has tests, one of which caught the missing parent. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: require admin to read an app-backed repository A `git_repository` resource names the repository rather than holding a credential for it, so read access to one authorizes nothing: anyone who can write a resource path can point one at any repository the GitHub App installation reaches, then read their own resource. The head lookup now requires admin for app-backed repos, matching the archive route and the repository picker, which already limits itself to workspaces where the caller is an admin. Repos that aren't app-backed are untouched and stay open to any reader. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: describe the repo viewer's hub script as it stands The file read as a patch waiting to be applied, against a hub version two releases stale. Describe what the published script does, including the archive route app-backed repositories now take. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: run the archive fetch under the job poller, off the job directory Three defects in the delegate path's fetch: The download and extraction ran outside the job poller that the git clone paths go through, so a cancelled or timed-out run kept streaming and extracting an arbitrarily large repository while holding the worker. There is no wall-clock bound on the download itself, by design, which is exactly why it needs the poller. The archive was written to a fixed name inside the job directory, where `create_file_resources` has already laid down the run's own files at paths the playbook chooses. A run naming a file `repo_archive.tar.gz` had it truncated and then deleted. It goes to a per-job temp path now. Link entries were unpacked with their target unchecked. `Entry::unpack` writes the link verbatim, so a link out of the tree plus a later entry descending through it writes wherever it points. Targets now face the same containment check as entry paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep repo symlinks, refuse only writes that go through them The link check rejected any target containing `..`, which is ordinary in a repository — `docs/x -> ../README.md` resolves inside the tree, and a git checkout keeps it. Rejecting it failed the whole extraction for repositories the clone path handles, and app-backed repos have no clone path to fall back to. Targets are preserved as git preserves them. What would let one escape is a later entry written at or underneath the link, so that is what is refused. Extraction also polls an abort flag now: a `spawn_blocking` task outlives the join handle its caller drops, so a cancelled job left it unpacking in the background. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: refuse hard links in a repository archive Leaving link targets verbatim is right for symlinks — git checks them out that way, and an escape needs a second entry descending through the link, which is refused. A hard link is not like that: unpacking one creates it against a target resolved there and then, so an escaping target is useful on its own. No git tree can express a hard link, so an archive carrying one did not come from a repository. Refuse it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 21f79bbbd39ae89665d1a89738630978616aa309 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore: update ee-repo-ref to 37695a769b25d16b34107eedc1076793a8b388c8 This commit updates the EE repository reference after PR #737 was merged in windmill-ee-private. Previous ee-repo-ref: 21f79bbbd39ae89665d1a89738630978616aa309 New ee-repo-ref: 37695a769b25d16b34107eedc1076793a8b388c8 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-08-20 22:05:24 +02:00
Ruben FiszelandClaude Opus 5	032300e28e	feat: run dbt projects as a first-class Windmill runtime (#10326) ... * fix: mount only the engine in the dbt jail, reject shadowed and malformed args Review round 42. The jail mounted the whole dbt cache directory, whose siblings of the engine are `repos/` and `packages/` — other workspaces' private checkouts and package trees, kept apart by cache key rather than by permissions. A jailed project could read them. It now mounts the engine's own directory, which the provisioner names; verified from inside the jail that `repos/`, `packages/` and `state/` are invisible while the engine stays usable. A `{{ placeholder }}` may no longer take the name of a run argument this runtime defines. It was silently dropped from the signature, so a descriptor like `value: "{{ select }}"` deployed and then could not be run at all: the built-in `select` is an array and the interpolation needs a scalar. Refused at parse, so the deploy says so. A `vars` override that is not an object is refused rather than ignored. Argument-schema validation is opt-in, so a string or an array silently ran the descriptor's own vars — against a different schema or alias than the caller asked for. `select` and `exclude` already refused theirs. * feat(dbt): the project is the script's module bundle, not a git checkout A dbt script now carries its whole dbt project as its module bundle. The descriptor is the script content; `<script>__dbt/` holds the project verbatim, so importing an existing project is `cp -r` plus `wmill sync push`, and the worker materialises the bundle into the job directory instead of cloning. Backend - `prepare_project` writes the script's modules and requires `dbt_project.yml` at the bundle root. `checkout`, the git-ssh command, the clone cache and the repository resource are gone, along with `repo`, `project`, `ref` and `git_ssh_identity` on the descriptor. - Run identity and the package cache key take a `project_digest` (sorted SHA256 over the bundle) where the commit used to sit, so an edited project cannot resume a previous run's `run_results.json` or reuse its `dbt_packages`. - The per-run graph re-ingest is now gated on `vars` placeholders and `$var:` env alone. - `capture_dependency_job` takes the script's modules so a dependency job, which has no generic module-writing step, materialises them itself. - `dbt deps` caching strips the git remote from every package it cached, not just the tree root: `packages.yml` can render a token into a `git:` URL. - `git_clone.rs` is dropped and `ansible_executor.rs` returns to its own copy of the clone helpers. CLI - `wmill sync pull` keeps a dbt script's lock beside its folder rather than inside it, so the folder holds nothing but the project. - Directories dbt generates (`target-path`, `packages-install-path`, `clean-targets` and the usual defaults, read from `dbt_project.yml`) are excluded from the bundle, from the sync diff and from staleness hashing. - A module-only edit now pushes its parent dbt script and is reported as a changed module rather than passing unnoticed. * fix: keep a script's modules in the worker's file-system cache The first fetch of a script version reads the database and carries its modules; every later fetch imports from the worker's cache directory, whose `RawScript::import` hard-coded `modules: None` and whose `export` never wrote them. A worker restart therefore started running the script without its own files, silently — for a dbt script, without its project, which fails with "carries no project"; for any other script with a module bundle, with the imports missing. `modules.json` is now written on every export and required on import, so an entry written by an older version fails to import and is refetched rather than serving a stripped script for as long as the directory lives. Also derives a dbt run's `project_digest` from the bundle the run actually carries: `handle_dbt_job` was passing `None`, which collapsed every project in a workspace onto one digest and let `dbt retry` resume a different project's `run_results.json`. * fix(dbt): give every phase the script's environment, bound the cache copies `dbt deps` ran without the script's environment variables on an unsandboxed worker, so a `packages.yml` resolving a private package URL through `env_var()` could not see them while the package cache key was still built on their digest. `with_invocation_env`, applied at three of the four call sites, is folded into `dbt_command` so no phase can be added without it, and `DBT_TARGET_PATH` is set after both environments rather than before. The package cache copies ran through a bare `Command::output()`: the tree is the project's, so a cancelled or timed-out job held its worker slot until `cp` finished. Both the restore and the publish now run under the job poller like every other phase. * fix(dbt): only offer commands whose writes match the graph, honour packages-install-path `dbt_command: run` is dropped from the allowed overrides. Asset dispatch fires a script's deploy-time writes on any successful job, and `dbt run` covers models only, so a project with seeds or snapshots notified consumers of relations the invocation left stale. That is the same reason `test` was already excluded. Narrowing what a run touches is `select`/`exclude`, which scope the graph too. `dbt deps` writes to the project's `packages-install-path`, so a project that moved it got no package cache at all: the publish found nothing at `dbt_packages` and every job resolved its dependencies over the network again. The path is read from `dbt_project.yml` and validated as project-relative, since both cache copies are rooted at it. Also states the sidecar's mutator contract at the module level: the dbt manifest tables carry no RLS and grant `windmill_user` full access, so a user-scoped transaction is not enforcement and every caller must have verified write access to the script itself. CLI: a module file is now grouped with its parent script for the push. Left in a group of its own it got its own `alreadySynced`, so a push touching several files of one bundle deployed the script once per file; the resulting versions raced, and the asset graph could end up describing none of them. * fix(dbt): seed a project for browser-created scripts, refuse a no-op retry A dbt script created in the browser only got a descriptor, and the runtime refuses a script whose bundle has no `dbt_project.yml`, so the advertised Create → Deploy → Run path always failed its dependency job. New dbt scripts now start with a project that builds: pointing `profile.resource` at a warehouse is the one edit, and growing it is `wmill sync pull` plus a local editor, which is where dbt development happens. `dbt retry` builds its graph from the previous run's error, fail and skipped nodes alone, so retrying an all-green run selected nothing and wrote nothing — and a job that succeeds having written nothing still dispatches every deploy-time write, waking every downstream consumer for relations no one touched. Refused, with the reason. CLI: a configured `target-path` or `packages-install-path` may be nested (`build/target`), and `clean-targets` has a block form as well as an inline one. Both are now parsed, and the exclusion compares the project-relative path rather than the top-level segment, so a nested generated tree no longer lands in the bundle and no longer makes a local `dbt run` look like a project change. * fix(dbt): lock a project once, find the parent on either path separator A dbt script's modules are its dbt project, not helper code with dependencies of its own, so the generic per-module lock loop is skipped for it: the parent lock already ran `dbt deps` and `dbt parse` over the whole project. Locking each file separately re-materialised the bundle and re-invoked dbt once per file, so a project of N files paid N project-sized passes and a large one timed the deploy out. The 13-file fixture went from 14 relock passes to 1. `pushParentScriptForModule` searched the raw path for `__dbt/`, so on Windows, where the folder is spelled `__dbt\`, a module-only edit returned without deploying its parent while the caller still recorded the file as synced. It now goes through `getScriptBasePathFromModulePath`, which normalizes separators. Also drops the last of the external-repository wording from the descriptor's module docs and from the `codebase` rejection a user can hit. * feat(dbt): infer the run form locally, keep test-only retries from cascading `windmill-parser-wasm-yaml` 1.770.0 carries `parse_dbt`, so the browser and the CLI derive a dbt script's run arguments from its descriptor instead of waiting for the deploy to hand back a schema. Pins bumped in both. `generate-metadata` was rewriting a dbt script's `lock` field on every run: a dbt lock comes from the dependency job on a worker, so nothing generates it locally and the resolved `!inline` reference was left inlined into the metadata or blanked. It is restored instead, and a push straight after `generate-metadata` is a no-op again. A retry now needs a failed node that materialises something. `dbt retry` builds its graph from error, fail and skipped nodes, and with `test_behavior: after_all` a failing test is what `run_results.json` ends up describing — so the retry reran tests, wrote nothing, succeeded, and still dispatched every deploy-time write. The dbt badge's destination is deterministic: writers outrank readers, and among several writers of one relation (which the backend permits) the smallest id wins, rather than whichever write edge arrived last. * feat(dbt): browse the project and read a run's per-node result Two views a dbt user expects and that the generic script surfaces do not give. **The project.** A dbt script's editor gains a Project tab beside its descriptor: the module bundle as the tree dbt itself expects, each file read-only with syntax highlighting. The existing module tab strip is a flat row built for a couple of helper files and does not survive a real project; a 13-file fixture already overflows it. Directories sort before files so it reads like the checkout on disk, and an empty bundle explains the `cp -r` instead of showing a blank pane. **The run.** `DisplayResult` renders a dbt invocation's per-node breakdown above the raw payload: totals, then a table of node, kind, target relation, rows and time, with failures and warnings sorted first and carrying their message. The data was already structured; it was being shown as JSON to scroll and PASS/WARN counts to find in the log. On a failed run the same JSON rides in the error message after the exit-status line, so it is parsed back out — that is the case worth rendering, since the failing node is what the user came for. * docs(dbt): say that profile.resource is what buys the asset graph The starter descriptor described `profile.resource` as the thing rendered into profiles.yml, with the project's own file as an equal alternative. It is not equal: the resource PATH is the warehouse's identity in the asset graph, so a project bringing its own profiles.yml runs fine and silently gets no assets, no lineage and no cascade. The deploy already says so in its log; now the descriptor a user starts from says it too, before they choose. * fix(dbt): authorize a resource used only for asset identity, clean up after failed installs A descriptor setting both `profile.profiles_yml` and `profile.resource` took its connection from the project's file but returned the resource path as the graph's warehouse identity without ever reading it. A script editor could therefore publish `table://<any resource>/...` writes, and wake that warehouse's subscribers, while connecting somewhere else. The resource is now read on that path too — reading is what authorizes it — so the combination keeps working for the case that wants it (keep your own profiles.yml, still get lineage) and fails closed otherwise. Provisioning cleaned up its staging directory only on the paths someone remembered, so a run of failed or cancelled first-use installs accumulated venvs, tarballs and installer scripts until the worker's disk was gone. All three engines now hold their scratch paths in a guard that removes them on drop, which is the one exit every path takes, cancellation included. Frontend: `partial success` is dbt's word for a node that built but whose tests failed, counted in `totals.error` and redone by a retry, so it ranks with the failures instead of rendering green with its message hidden. And the run panel now keys off the worker's engine discriminator rather than `{nodes, totals}`, which is a shape an ordinary script can return. Both pinned by unit tests on the extracted `parseDbtRun` helpers. * feat(dbt): show a run's models on the run page The run page is where you land on a running job, and until now it showed a dbt run as streaming text: the per-node table only renders once the job has produced a result, and the graph that moves per model lived on the pipeline page you had to navigate to. A Models section now sits above the result, scoped to the running script's own relations and its `ref()` lineage, polling while the job is in flight so nodes move as dbt walks the DAG. No `resolveGraph`: that merges drafts and live editor buffers into the persisted graph, and a run page has neither. * feat(dbt): retry failed nodes automatically, and from any worker **Node-level retry, in the job.** `retry_failed_nodes: {attempts, delay_seconds}` rebuilds only what a failed build left failed or skipped, before the job reports failure. dbt confines a failure to its own subtree and `dbt retry` resumes exactly that set, so a transient warehouse error costs those nodes rather than the project. Doing it in-job is what keeps the state question out of it: the previous attempt's `run_results.json` is still in the job directory, so there is nothing to persist and no worker to land back on. This is the granularity astronomer-cosmos gets from one Airflow task per model, without the ~6x that per-model tasks measured. A retry's `run_results.json` names only the nodes it redid, so it overlays the accumulated results rather than replacing them: the job's result has to be every node the job touched, or the nodes that succeeded before the retry settle no materializations. Pinned by a test. **Durable retry state.** `run_results.json` is now saved to `dbt_run_state` as well as the worker's local cache, so an explicit `dbt_command: retry` works from any worker of the group rather than only the one that failed. Only the results are stored: `dbt retry` also needs `manifest.json`, roughly sixty times larger and growing with the project (732 KB against 12 KB on the six-node fixture), but the manifest is a pure function of the project files, vars and env, all of which the stored identity already pins, so a worker restoring from the database re-derives it with a `dbt parse` of about a second. * fix(dbt): restore the sqlx cache, make retries cancellable and path-aware **SQLx cache.** A `cargo sqlx prepare` deleted 750 entries, including the enterprise queries CI needs under `SQLX_OFFLINE=true`, and the check that was supposed to catch it reported zero losses because it was run from `backend/` with a `backend/`-prefixed path, so its baseline was empty and it failed open. All 750 are restored; the branch now adds 19 and deletes none, and `SQLX_OFFLINE=true cargo check` passes. **Retry backoff observes cancellation.** `canceled_by` is only written by the job poller, which does not run between attempts, so re-reading it reported the state as of the failed attempt and missed every cancel issued during the wait — the whole window the check exists to cover. The wait now reads `v2_job_queue.canceled_by` each second, and the job's deadline is honoured before starting another dbt process. **Retry state follows its script.** `dbt_run_state` is path-keyed like the manifest sidecar but, unlike it, nothing regenerates it: a rename moves the row so a resumable failure survives, while archive and delete clear it, so a script later created at that path cannot inherit a stranger's failure and its arguments. **CLI.** `table` joins ducklake and s3object in the local graph's auto-trigger kinds, matching `is_auto_trigger_kind` and the frontend's set; without it a local graph and the generated docs omitted a cascade edge the deploy has. * fix(dbt): carry only the project files a bundle can hold, and say what it drops Exploring real and edge-case projects surfaced three frictions, all in the import path a user hits first. **A binary file broke the push, opaquely.** dbt projects carry images under `docs/`, stray `.DS_Store` files and occasionally a parquet seed. Read as text they become mojibake, and a NUL among them is rejected by Postgres with `unsupported Unicode escape sequence` — which `wmill sync push` then reported as success, exiting 0 with the script never created. Binary files are now detected the way `git` detects them, by a NUL in the first 8000 bytes rather than by extension, and skipped with the reason. **The size guard the docs promised did not exist.** Now it does: 5 MB per file, which only ever catches a committed dataset. Real dbt code is about 500 bytes median and 1.9 KB at p90. **Skipped files became a permanent phantom diff.** The push dropped them while the sync diff still offered them, so every push reported changes no push could resolve. One predicate now answers for the push, the staleness hash and the diff. Verified on a project with unicode filenames and content, CRLF endings, an empty model, an ephemeral model, a disabled model, a `.md` docs block, an extensionless README, six levels of nesting, a 7.6 MB seed and a PNG: it pushes, round-trips byte-for-byte through pull, deploys to 7 dbt nodes and 6 `table://` assets (ephemeral and disabled correctly absent), and runs green. * fix(dbt): resolve dbt-core against the adapter, settle partial success, unify status **Adapters could not be provisioned.** The 1.x engine pinned `dbt-core` to a fixed version independent of the adapter, but several adapters cap below it: `dbt-mysql` at `~=1.7`, `dbt-oracle` and `dbt-databricks` below 1.12, and `dbt-salesforce` has no package at all (it exists only inside Fusion). Those projects failed at provisioning with a uv resolver dump. The install now asks for a range and lets the adapter choose, and records what the resolver picked so the lock pins a version that adapter can take. The floor is the CLI this runtime invokes: resolving down to dbt-core 1.7 produced a working venv that then failed with `No such option '--target'`, which is worse than not resolving. An adapter with no release in range now fails naming itself and pointing at `dbt-core-2x` or `fusion`, instead of a resolver dump. Salesforce is refused up front with the reason. **`partial success` left a model stuck on `Running`.** It is dbt's word for a node that built and then failed its tests, and it was already treated as a failure when counting totals and deciding a retry — but the two sites that settle the RELATION fell through to "says nothing", so the tailer's `Running` was never replaced and a finished job showed a model still building. Six status comparisons had drifted apart, two folding case and four not, while dbt-core 1.x echoes the author's casing and 2.x uppercases; they are now one classifier. **Agent workers.** The durable retry state and the cancellation poll both need a database, which an agent worker reaches only through the API. The automatic node retry is refused there rather than running a wait it could not interrupt, and the docs say "any worker with a database connection" instead of overclaiming. Also clears `dbt_run_state` when a path stops being a dbt script, and moves `run_identity`'s contract onto `run_identity` from the digest helper below it. * feat(dbt): show the transform behind a model on the run graph The run page's graph carried a node for the script itself and drew every relation as a bare table. Both were wrong for that page: the graph there is already scoped to one script, so a node standing for it distinguishes nothing (on the pipeline page it separates one project from another, which is why it exists), and dbt's own DAG node is the model — the SQL and the relation it writes are one thing, so a graph of relations alone leaves out what a reader came to see. The script node is dropped, and selecting a model now shows its SQL underneath the canvas with its file path and materialization, read-only, the same view the pipeline details pane gives. * feat(dbt): move the graph with the run The worker has always recorded a state per relation as dbt walks the DAG — `running` when a model starts, `materialized` or `failed` when it ends — but nothing rendered it: the graph response carries what a relation IS, not what a particular run is doing to it, so the canvas had nothing to show and a running job looked identical to a finished one. `assets/run_progress/{job_id}` returns that state for one job, the run page polls it beside the graph, and the asset node carries a spinner or its outcome. Errors and retries need nothing extra: a failed node writes `failed`, and an in-job retry rewrites the same row, so the node returns to `running` and on to its new outcome by itself. `materialized_partition` holds a relation's CURRENT state keyed by relation, so filtering on `job_id` returns exactly what this run last touched — which is the question a run page asks, and why a superseded older run shows nothing. * feat(dbt): a dbt project is not a data pipeline Deploying a dbt script marked it `auto_kind = 'pipeline'`, which enrolled it in pipeline membership: the folder became a Pipeline entry on the home page, the script folded into it, and `/pipeline/<folder>` opened a canvas holding the project's whole model DAG next to the pipeline's own scripts. A folder holding both then read as two projects in one editor, and the pipeline editor offered to author transforms that are in fact authored in a local `dbt run` loop and pushed as the script's bundle. A dbt script is now never a pipeline member, and the pipeline canvas drops the dbt script node. Its models stay, with their `ref()` lineage: the relations are what a downstream pipeline script reads, and dropping them would break the cascade from a dbt run — the point of giving dbt models `table://` identity. Also drops a screenshot committed to this branch by accident. * fix(dbt): authorize run_progress through the job, drop dbt from the local graph `run_progress` read `materialized_partition` through `user_db` on the assumption that RLS would scope the rows. That table has RLS disabled and no policies, so any workspace member could pass a job id and read that run's relation paths, row counts and error text. It now joins `v2_job`, which does carry per-user policies, so a caller who cannot see the job sees nothing — the same pattern `v2_job_completed` reads need. Verified as a plain member: the old query returned 6 rows for another user's run, the new one returns 0, while the job's owner still sees all 6. The CLI's local graph still forced `in_pipeline` on every dbt script, so `pipeline docs --local` and `pipeline dev` kept presenting a dbt project as a pipeline the deploy no longer enrolls. It now skips them, matching the server. A dbt descriptor has no asset parser locally, so nothing is lost: its models come from the manifest the deploy derives. Declares `run_progress` in openapi.yaml so the frontend uses the generated client instead of a handwritten fetch; the generated `status` union also replaces a hand-rolled string mapping. * fix(dbt): drop the dbt node from the CLI's deployed pipeline views too `pipeline dev` and `pipeline docs` (without `--local`) read `/assets/graph` directly. That endpoint is asset-usage driven rather than membership driven, so it returns a dbt script like any producer — and both commands render every runnable, so a dbt project still showed up as a pipeline script there after the local builder stopped emitting one. `hideDbtRunnables` mirrors the frontend's projection of the same payload. It is generic over the graph shape so the bounded-cascade view (`BCGraph`, a narrower type over identical JSON) passes through without a cast. The relations stay: they are what a downstream pipeline script reads, and the node is what attributes them to a producer for every other consumer of the endpoint, so the filter belongs in the views rather than the query. * fix(dbt): narrow a selective run's cascade, settle the finished run graph Review-round fixes. A `select`/`exclude` run builds part of the project, but asset dispatch reads the deploy-time write set for the whole script, so a run selecting one model woke the subscribers of every other. Dispatch now intersects that set with the relations the run actually recorded as materialized, scoped to dbt because it is the only producer whose write set is decided per run. A run that recorded nothing still dispatches everything, so an agent worker whose reconciliation failed cascades as before. Verified both ways: `select: [extra_model]` no longer wakes the `fct_orders` subscriber, and a full run still does. `hideDbtRunnables` keyed its removal set on path alone while the graph keys runnables by `(usage_kind, path)`, so a flow sharing a path with a dbt script lost its node, edges and triggers too. Both copies now key on the pair. The run graph never took a final reading when a job finished, so the last state shown was whatever the tick before completion saw. Only `dbt-core-1x` streams node events; the other engines record every relation during end-of-run reconciliation, so their finished graph showed nothing until a reload. `DbtNodeOutcome::Inconclusive` collapsed statuses the tally has to tell apart, so two sites re-lowercased the status beside the classifier and `no-op` landed in `totals.error` — a clean run reporting an error in its own result. Split into Warn / Skipped / NoOp / Unknown so every site falls out of one match; `no-op` is kept out of the retry set, which dbt spells as error / fail / skipped. Also: reattach two doc comments to the items they describe, and correct the engine-distribution table — only dbt-core-2x is baked into the images, 1.x is a per-adapter venv provisioned on first use, and the default is compiled in rather than an instance setting. * fix(dbt): make the model chip inert where its project node is not on the graph The canvas passed `onDbtSelect` unconditionally, so the chip always rendered `cursor-pointer` and hover-highlighted — but the owner map is empty on both graphs this feature added, since the run page carries no runnables and the pipeline page hides the dbt node. The chip advertised a click that resolved to nothing. It now takes its handlers only when the relation has an owner on this graph, so it stays live on the surfaces that do show the project node. `classify_status` and `DbtNodeOutcome` were `pub` in a private module with no caller outside the file, unlike every neighbour. * fix(dbt): take the cascade's write set from the run's own result The previous narrowing read `materialized_partition`, which was wrong twice. That table keeps one row per relation and the newest writer takes `job_id`, so two overlapping runs over the same model erase each other's claim to it: the earlier job would dispatch a subset of what it built, or none of it. And an empty row set was read as "recording failed, dispatch everything" when it is also a real answer. A `select` matching no model, or one resolving to tests only, exits 0 having built nothing — and then woke every consumer of every model in the project, which is the opposite of what the narrowing exists to do and is reachable by a typo in a run argument. The run now reports the relations it materialized in its own result, which is immutable and per job. Absent means the producer said nothing (a job from before the field, a non-dbt producer) and the whole deploy-time set dispatches as before; present-and-empty means it built nothing and dispatches nothing. Verified on all three: an unmatched selector builds nothing and wakes nobody, a selector naming one unsubscribed model wakes nobody, and a full run wakes the subscriber. Also indexes `materialized_partition (workspace_id, job_id)` -- the run page polls that shape every 2s and no existing index leads with `job_id` -- corrects the selective-cascade section of the design doc, which still described the old deploy-time behavior, and reattaches `buildLocalPipelineGraph`'s doc comment. * docs(dbt): attach the CLI JSDoc to its function, correct the index rationale The `hideDbtRunnables` JSDoc ended up documenting the type declared beneath it — made while fixing the same mistake one function down. The migration's comment credited the cascade with a `job_id` lookup that the same commit replaced with a read of the job's own result. The run page's poll is the only reader keyed on that column. * fix(dbt): refuse graph publication for a removed script, allow test-only retries An archived or hard-deleted script could still republish its graph: the publication guard filtered `deleted` but not `archived`, and treated a missing row as "nothing newer exists" rather than "nothing left to publish for". A dependency job or dynamic run finishing after the removal put the asset, provenance and subscription rows back with nothing left to clear them. `dbt_command: retry` refused a run whose only failures were tests, which is precisely what `test_behavior: after_all` produces. That restriction existed because a successful job dispatched its whole deploy-time write set, so a test-only retry would have woken every consumer for relations no one touched — the cascade now dispatches what the run reports materializing, so it wakes nobody and the restriction only blocked a legitimate retry. * fix(dbt): gate run progress behind the job-read check, not RLS alone The endpoint joined `v2_job` so RLS would decide visibility, which it does — but `require_job_read_access` adds two things RLS does not: a scoped token's `if_jobs:filter_tags` restriction, and the app-embed cutoff that stops untrusted app JS from inheriting the viewer's broader job access. A scoped or embed token could therefore read relation names, statuses, row counts and errors for jobs the ordinary job endpoints deny it. That helper is private to `windmill-api`, which depends on `windmill-api-assets` rather than the reverse, so the endpoint moves to the job routes instead of the check being duplicated. It is job-scoped anyway: `/w/{ws}/assets/run_progress/{job_id}` becomes `/w/{ws}/jobs/run_progress/{id}`, and the frontend follows the generated client. * feat(dbt): a dbt run does not trigger downstream runs dbt orders its own DAG, so a cascade only ever adds one thing: waking a Windmill script that reads a mart. That edge is narrow, and only half of it can even be expressed — nothing outside dbt can declare a `table://` write, since `// materialize` accepts DuckLake targets only, so an ingestion script cannot wake a dbt project. Against that, dispatching correctly is not cheap. A run's `select` can build any subset of the project, so the deploy-time write set is not what ran; using it wakes consumers of relations the run never touched, and narrowing it needs a per-job record of what was built. The per-relation state table cannot supply one (it keeps a single row per relation stamped with the last writer), and the result field added for it made a run's own output carry the cascade's bookkeeping. So `asset_dispatch` returns early for `ScriptLang::Dbt`, before the producer gate. dbt still materializes, records per-model state and publishes its graph: models, `ref()` lineage and live run progress are unchanged, and a `# on table://<mart>` reader still renders beside the model it reads. It simply does not fire. Wiring it up later means deciding what a selective run should notify, which is the actual work. Verified: a full run of a 6-model project succeeds and starts nothing, where it previously triggered its subscriber; the run page still reports all 6 relations and the folder graph still carries 14 tables and 8 ref() edges. * fix(dbt): remove the cascade surface, settle stranded models, fix nested __mod Stopping dispatch left its surface behind. `table://` was still an auto-trigger kind, `persist_ingest` still derived subscriptions from a manifest's reads, and the deploy still accepted `# on table://` — so the canvas drew cascade arrows into scripts nothing could wake. All three are gone: the kind no longer derives, the ingest only deletes rows earlier versions wrote, and the deploy refuses the annotation with a message saying why rather than persisting a silent no-op. `DescriptorTriggers` went with them; every field it parsed was cascade config. A model marked `running` by the live tailer was never settled when the run did not finish: reconciliation only revisits nodes `run_results.json` names, and a cancelled or timed-out run has none for the model in flight, so the finished job showed a relation building forever. It is now settled on every exit path. Verified by cancelling a run mid-flight: 3 models `running` before, 3 `failed` after, none stranded. `getScriptBasePathFromModulePath` took the first matching suffix rather than the outermost boundary, so `proj__dbt/models/legacy__mod/a.sql` resolved to `proj__dbt/models/legacy`. dbt owns its directory names verbatim, so a folder ending `__mod` is legal inside a project, and a module-only sync would have looked for a descriptor that is not there and skipped the deploy. * fix(dbt): colour a finished run's models from its own result `materialized_partition` keeps one row per relation stamped with whichever job wrote it last, so reopening a run showed only the models no later run had touched since — down to none for an old run, which reads as a broken page rather than as stale data. Reproduced: a 6-model run reported 6 relations, then a second run rebuilt one shared model and the first reported 5. A finished run already carries the answer. Its result lists every node with a status, and the graph carries each asset's dbt `unique_id`, so the two join directly — no path derivation, nothing stored twice, and nothing a later run can overwrite. The endpoint stays for the live window, where the result does not exist yet, and as the fallback for a run that never produced one (cancelled or killed, whose relations the worker settles in the table instead). `relationOutcome` mirrors the worker's `classify_status` so the colour drawn over a record agrees with the record: `warn`, `skipped` and `no-op` leave the relation untouched and stay uncoloured, as do tests and analyses, which match no asset. Verified in the browser on the run whose model had been stolen: all six relations green again, both sources correctly uncoloured. * docs(dbt): record why only dbt-core 1.x has live per-model progress `emits_node_events()` reads as "the Rust engines produce no node events", which is false and would close off the option. They produce exactly the same events; they put them on the console and ignore `--log-format-file json`, which both accept. Measured on 2.0.0-alpha.5 and fusion 2.0.0-preview.202: 15 node events each on stdout, 0 in the file log, for a three-model project. Taking them means owning the job log's presentation to work around a flag that is documented and simply unimplemented, so the note records the measurement, the sample event, and that flipping the predicate is the whole change once either engine honours it. * fix(dbt): give HighlightCode a dialect-agnostic sql language `npm run check` had three errors the fast check does not reach: `"sql"` is not a value `HighlightCode` accepts. Every SQL dialect it knows maps to one grammar, but a dbt model is compiled by whichever adapter the project targets, so naming a dialect would be a guess — `sql` is now a value in its own right. `langOf` was typed `string` and returned `markdown`, `python` and `text`, none of which the component accepts either, so a dbt project's YAML and Python files rendered unhighlighted. It now returns the component's own prop type, which is what caught them, and `undefined` for what has no grammar rather than a name that silently means the same thing. Verified in the project panel: SQL 22 tokens, YAML 27, where YAML was plain. * fix(dbt): stop failing no-op models, drop table triggers client-side, keep cross-selection edges The sweep that settles a run's stranded relations was marking `no-op`, `warn` and `skipped` models FAILED on successful runs: reconciliation reports those nodes without settling their record, so they were indistinguishable from a model the run never reached. It now excludes every relation the run accounted for, so only the genuinely abandoned ones are settled. `table` was removed from the backend's auto-trigger kinds but left in both client mirrors, so the editor and `pipeline dev`/`docs` kept drawing cascade arrows the deploy will not create. `isModuleEntryPoint` scanned for the first `__mod/`, the same bug its sibling just had: a `legacy__mod/script.ts` nested in a dbt project — dbt owns those names verbatim — read as that script's entry point. Both now anchor on the outermost boundary. A script selecting a model whose parent another script builds dropped the parent entirely, so no `dbt_edge` could reach it and the two relations sat on the graph unconnected. The parent is now kept as an endpoint and recorded as a READ, since this script does not build it — splitting a project across selections only composes if the seam still draws. * fix(dbt): don't double-run after-all tests, count only models a script builds An `after_all` run whose test phase failed saves a `run_results.json` holding tests alone. Retrying it reran exactly those tests — and then the test phase ran the whole suite again, appending a second copy of every result: duplicate ids in the run table, doubled totals. A retry whose saved results are tests alone IS the test phase, so the suite is not run after it, and the two phases now merge by node id rather than concatenating. Keeping a selection's unselected parents as nodes made them count toward the `×N` badge, whose tooltip says "materializes N models" — a script selecting one mart claimed the staging models upstream of it, and the number grew with the seam. The count now comes from the relations the script writes. That change also made the cross-selection read block dead, with a comment asserting the inverse of what now happens; it is removed, and the test that covered it still passes on the new arm. The test I added landed between a neighbouring test's comment and its `#[test]`, orphaning the attribute so that test stopped running. Two display fixes: the run page no longer shows a relation's SQL when the provenance belongs to another project that materializes the same relation, and the editor no longer draws an explicit `# on table://` arrow the deploy refuses. The starter descriptor no longer promises the removed cascade. * fix(dbt): clear untouched models, reject unknown descriptor fields A `no-op` model was left `running` forever on a successful run. The previous attempt at this stopped the sweep marking such models FAILED but gave them no terminal state instead, so they simply never settled. Reconciliation now returns what it settled and what the run reported but did not build, and the two get opposite treatment: a relation the run left untouched has its row DELETED, which is what the finished run's own result says about it (`relationOutcome` colours a `no-op` nothing), so the live and settled views agree; only a relation the run never reached at all is failed. The descriptor accepted unknown fields, so `selcet:` was ignored and left an empty selection — building the whole project — and a misspelled `target` fell back to the profile's default. It rejects them now. That immediately caught two of our own test fixtures still passing `repo:`, a field removed with the git path, which is exactly the class of mistake it exists to stop. `isDbtModulePath` matched `__dbt/` anywhere in a path, the third site with that bug: `foo__mod/vendor/x__dbt/a.ts` read as a dbt project file, and the push then looked for `foo.script.yaml` and could skip the edit. A verbatim dbt bundle dropped any file named `*.lock` before it reached the module map, so an authored `uv.lock` never deployed and the unmodified-project round trip quietly lost it. The exclusion now applies only to `__mod` bundles, where `.lock` really is the script's own lockfile — in the walker that hashes modules too, or a change to such a file would not register as one. Also: `langOf` fell back to `undefined`, which HighlightCode resolves to TypeScript rather than to no highlighting, so seeds and Markdown were coloured as code; and four comments still gave the removed cascade as the reason for sharing an asset node, which is now lineage. * fix(dbt): retry failed tests too, anchor the last __dbt path check `retry_failed_nodes` only ran after the model phase, which fails before the `after_all` test phase exists — so a project whose models built and whose tests failed got no retry at all, exempting exactly the failure mode that separate phase produces. The loop is now a function, called after both phases. `isDbtGeneratedPath` matched `__dbt/` anywhere, the fourth site with that bug: `foo__mod/vendor/x__dbt/target/a.ts` counted as generated dbt output, so `ignoreF` excluded an ordinary module file and a module-only edit never deployed its parent script. `wmill sync push` still dropped an ADDED or DELETED `.lock` three branches before the module arm, so the earlier fix only covered a first push: adding a `uv.lock` to a deployed project was reported as a change forever and never applied, and deleting one left it deployed. Editing worked, which is what made the round trip look whole. Also removes a duplicate `#[test]` that was double-registering a test and detaching its neighbour's comment, and rewrites seven comments that still gave the cascade as the reason for behaviour that now serves lineage only. * fix(dbt): bound the excluded-file read, keep the retry budget job-wide `isBundledModuleFile` read a file in full before deciding it was too big or binary, so a project sitting next to a multi-gigabyte parquet seed loaded the whole thing only to reject it. It now takes the size from `stat` and reads at most the 8 KB the NUL check needs: a 191 MB file is rejected in 0.0ms at 82 MB RSS. Calling the retry helper after both phases gave each its own `attempts` budget, so a job could spend double what the descriptor asked for — the bound exists because every attempt is a real dbt invocation holding a worker slot. The budget is now the job's, spent across whichever phases fail, and the field says so. Extracting that helper had also placed it between `#[allow(clippy:: too_many_arguments)]` and `run_dbt`, taking the attribute off the 12-argument function it was written for. * fix(dbt): actually spend the retry budget `retry_failed_nodes` looped on `while *remaining > 0` and never decremented it, so a failing job reissued `dbt retry` — logging "attempt 1 of 3" each time — until the job's deadline instead of `attempts` times. The decrement existed briefly and was lost when the function was re-extracted by hand. Claiming and counting are now one operation, `claim_attempt`, because keeping them apart is exactly how the bound goes missing: the loop cannot iterate without spending the budget. Its test is bounded by its own `for` rather than by the function under test. An earlier version collected `std::iter::from_fn(|| claim_attempt(..))`, which against a non-spending `claim_attempt` is an infinite iterator — it allocated until the machine died. A test for a loop bound must fail an assertion when the bound regresses, not consume the host: it now reports `[1, 1, 1, …]` against `[1, 2, 3]` in 0.00s. * fix(dbt): ask before reading, not after Bounding `isBundledModuleFile` did nothing for the bundle builder, which read the whole file into memory and only then asked whether to keep it — so a multi-gigabyte seed beside a project was still loaded in full just to be skipped. The predicate is now consulted first, and the read happens only for files the bundle actually carries. * feat(dbt): animate the ref() edges feeding the model being built The nodes moved during a run but the edges did not, so the graph showed where dbt had got to without showing it flowing there. Reuses the canvas's existing rule rather than adding a second one: an edge animates when it touches what is happening. For a pipeline that is the running script; for dbt the unit of work is the model, so a `ref()` edge animates while its target builds. Same `animated` field, same visual language, no new styling. Verified mid-run on a 7-model project: of six `ref()` edges only the two feeding the model then building were animated, and none once the job finished. * feat(dbt): show what each model wrote, and say when its SQL is another project's Three things a reader wanted from the run graph and could not get. Row counts: the worker already records one per relation and `run_progress` already returned it, but the graph used only `status` and dropped the number. A model that built green having emitted zero rows is the failure that looks like a success, so the count is on the node. The relation's fully-qualified name, copyable: there is no table browser to open, so the next best affordance is the exact identifier to paste into a SQL client. It is parsed with `splitRelation`, which honours quoting the way the worker's `split_relation` does — splitting on every period renders `"wh"."analytics.v2"."orders"` as a relation `orders` in a schema `v2`, which does not exist. And when two projects materialize one relation, the graph keeps a single provenance winner, so the losing project's node carries the other's model. The SQL was already suppressed there — correctly, it is not this run's code — but silently, which reads as a dead click. It now says so. * fix(dbt): a finished run's graph is the models it built, not today's project `/assets/graph` is the current deploy, so an old run's graph drifted with the project: a model added after it appeared as though the run had built it, and the older the run the wronger the picture. A finished run's node set now comes from its own result, which named exactly what it touched. Sources survive the filter regardless — dbt never lists them in `run_results.json` because it does not build them, but they are the upstream the run read, and dropping them would leave the models hanging. The graph is still the current deploy's, so a model renamed or deleted since cannot be drawn at all. Rather than a silently shorter graph, the count is stated above it. Verified by adding a model after a run: the old run renders 7 models without it, a fresh run renders 8 with it. * feat(dbt): preview a model's rows with `dbt show` There was no way to see the data behind a node — only its SQL and its row count. `dbt show` selects from a model and returns rows, and every engine ships it, so the preview needs no adapter code of ours: no connection path, no dialect-correct quoting, no type coercion for ten warehouses. It runs against the profile the run already renders. It is a `dbt_command` rather than a new endpoint, so it inherits the whole job path — authorization, isolation, cancellation, logs, engine provisioning — and `limit` joins the run form beside it. That the allowlist can admit it at all is a consequence of dropping the cascade: while a successful job dispatched its deploy-time write set, a command that wrote nothing woke every consumer for relations nothing had touched. Read-only, and treated as such: no graph republish, no materialization records, no retry state, no test phase. Captured rather than streamed, like `dbt ls` — these rows are the result, not commentary, and the job-log writer is what `NO_LOGS_AT_ALL` discards. Verified: `{"dbt_command":"show","select":["stg_customers"],"limit":3}` returns three rows; a preview leaves `materialized_partition` untouched (62 → 62, 0 rows for the job); `clean` is still refused by the allowlist. * feat(dbt): preview a model's rows from the graph, and keep our locks out of dbt projects The run page could show a model's SQL and how many rows it wrote, but not the data. Selecting a model now offers "Preview rows", which runs the script with `dbt_command: show` and renders the result as a table. Explicit rather than on-select: a preview is a job, so it costs a worker slot and the engine's start-up, and previewing on every click would spend both on mere navigation. Sources are excluded — dbt shows what a model SELECTs, and a source is not one. Also: `updateModuleLocks` was the one module helper that never learned about verbatim bundles, so it walked a dbt project writing `foo.lock` beside `foo.sql`. None of those files is a Windmill script needing a lockfile, and the bundle promises to round-trip the project byte-for-byte — our artifacts have no business in it. Verified in the browser: selecting `stg_customers` and previewing returns the columns `id`/`src` and five rows from the warehouse. * fix(dbt): keep a run's models when another project owns their provenance Scoping a finished run's graph to the ids it named dropped relations whose provenance winner belongs to a different project — so a run of a project sharing a schema showed 3 of the 6 models it had built. An id that was never this run's package cannot be judged against its result, so it is kept: the relation IS one the run wrote, and hiding it understates the run. The same rule applies to the "no longer in the project" count, which otherwise reported deletions that were only provenance collisions. Previews are now cached per model and survive the selection moving. One was thrown away whenever the reader clicked elsewhere, which for a job costing a worker slot and an engine start-up meant re-running it to see it again — and the run continues in the background, so leaving and returning finds the rows there. The spinner also never span: `startIcon` takes the icon and its classes separately, so the animation has to be passed alongside. How long it took is shown with the rows. A preview is a job, and its cost should not be something the reader has to guess at. * fix(dbt): resolve argument references, clamp the show limit, flag renamed relations `handle_dbt_job` cloned `job.args` where every other executor calls `build_args_map`, so a `$var:` / `$res:` / `$encrypted:` argument reached dbt as the literal string. A placeholder holding a schema or an `enabled` flag would then build a different slice of the project than the caller asked for. `--limit` took any positive i64, and the worker buffers the whole of dbt's stdout to read the rows out of it — so a caller with only run permission could make it hold an unbounded allocation. It is clamped to a ceiling now, extracted as `show_limit` so the bound is pinned by a test rather than inline in an async function nothing can reach. And a model keeps its id when its alias or schema changes, so an old run's node showed today's relation while the run wrote another — the page asserting it had materialized a table that did not exist yet. The run's result carries the relation each node actually wrote, so the drift is detectable without a graph snapshot, and the count is stated above the graph. Rendering the run's own lineage still needs a per-job snapshot; this stops the page claiming otherwise. * fix(dbt): stop persisting resolved secrets, bound the preview by bytes Resolving `$var:` / `$res:` / `$encrypted:` for dbt — added in the previous commit — meant `save_run_state` wrote the resolved PLAINTEXT into `dbt_run_state.args` and the worker's `state.json`. The row outlives the job, so a secret stayed in the database and a later `dbt_command: retry` replayed it after the grant was revoked or the value rotated. The invocation now carries the args as submitted alongside the resolved ones, run state persists those, and the restore path resolves them again under whoever is retrying. Clamping `--limit` bounded the row COUNT, not the size: one column can hold a megabyte, so a thousand rows is a thousand megabytes, and `run_capturing` buffers all of it. The captured output has a byte ceiling now. `limit` became a built-in argument without joining `RESERVED_ARG_NAMES`, so a descriptor writing `{{ limit }}` was silently handed the preview control's default instead of being told the name is taken. Two display fixes: the relation-drift banner compared a canonicalized (lower case) asset path against the warehouse's own spelling, so it fired on every model of every finished Snowflake run; and caching a preview's failure left `Preview rows` dead for that model until reload. * feat(dbt): key the graph by script version so a run renders its own project The dbt graph was keyed by path alone, so a deploy overwrote the only copy and a run page could only ever show today's project — an older run rendered today's models, SQL and `ref()` lineage no matter what it had run. My previous attempt filtered that view to the ids the run named, which stopped it lying but could not show what was gone: the data no longer existed. `dbt_node` / `dbt_edge` now carry `script_hash` in their primary key, so each deployed version keeps its own graph, and the run page passes the version its job recorded. Per DEPLOY, not per run — ten thousand runs of one version share one graph — and a composite FK to `script (workspace_id, hash)` with ON DELETE CASCADE means a version's graph dies with it. Nothing pruned these before, because there was one copy per path; they would otherwise have accumulated with no sweep. Two deploys of one path now write disjoint rows, so the graph can no longer be lost to a race. `claim_graph_publication` remains only for what is still path-keyed — the `asset` usage rows — and an older deploy finishing late records its own graph before declining to touch those, where before it published nothing at all. A pinned request is scoped by the version's own nodes rather than by `asset`: that table describes the current deploy, so scoping through it would filter a model out of the very run that built it. Verified end to end: deployed v1 (8 models), ran it, deployed v2 with four models removed and one rewritten. The old run renders 8 models, 6 ref() edges and v1's SQL; a new run renders 4 and the v2 rewrite. * fix(dbt): scope graph cleanup to one version, bound the preview capture Archive and delete both act on a single `hash`, but the graph cleanup they called deleted every row for the path. Now that the graph is keyed per version, archiving an old version erased the live one's models, SQL and lineage, and nothing repaired it. Both callers have the path in hand, so the by-hash wrapper is gone and they use the version-scoped clear directly. `dbt show` checked its 8 MB ceiling after `wait_with_output` had already buffered everything, so the ceiling could not bound what the worker held. `run_capturing` now reads both pipes incrementally against a caller-supplied limit and kills the child on overflow. The read buffers are heap-allocated: as arrays they were baked into the future, which the job poller boxes several layers deep, and that overflowed the worker thread's stack — a `dbt show` run aborted the whole worker process. A retry's `dbt parse` ran on the arguments as submitted while the build ran on resolved ones, so a `$var:` shaping the graph parsed verbatim. The parse moves to the caller, after resolution. A run that names its own `select`/`exclude` now drops the descriptor's `selector`: dbt resolves `--selector` instead of `--select`, so passing both made a preview of one model return another's rows. Also: log instead of silently swallowing a `modules` column that fails to deserialize (pre-existing, but for dbt it means running with no project at all); keep the model SQL reachable once a preview has landed; render which node the rows came from; stringify object-valued cells; document `dbt_script_hash` in the OpenAPI spec. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: record the per-worktree dev environment and the backend-run check Three mistakes this guidance would have prevented, each of which cost a cycle: A worktree has its own database and ports, but AGENTS.md stated the single-checkout defaults as facts. Pointing `DATABASE_URL` at another worktree's database makes `cargo sqlx prepare` fail on every query touching a table your migrations added — and it deletes `.sqlx/` before it fails, so the cache is gutted rather than merely stale. Starting a backend on the wrong port leaves the UI up with every call 502ing, which reads as an application bug. Both values are now discoverable with commands that work as written. `prepare` is also documented as the wrong tool for a removal-only change: the cache is already complete for CI, and the only residue is orphaned entries that can be found by text-matching against the sources without a database. Nothing told a reader that `cargo check` does not exercise a worker path. A read buffer declared as an array inside an async block is baked into the future, and once boxed by the job poller it overflows the worker thread's stack — compiling and unit-testing clean while aborting the whole worker process at runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): scope the remaining path-wide reads and clears to one version Four places still spoke for a whole path after the graph became per-version: The relation-root drift check read `dbt_node` by path with an unordered `LIMIT 1`, so with v1 at root A and v2 at root B it could answer with v1's row, suppress the refresh v2 needed, and leave v2's graph naming relations the run does not build. It now reads this job's version. `dbt_dep`'s no-resource branch cleared the path, so a descriptor edited to bring its own `profiles.yml` emptied every earlier version's graph and with it every finished run's page. The ownership being given up is the path-keyed `asset` usages cleared beside it; the graph clear is now this version's. The graph was inserted before the publication claim checked the version was still live. Archive and delete only soft-update `script`, so the foreign key still accepted an in-flight dependency job's rows and the failed claim committed them — and because pinned queries deliberately serve archived versions, deleted model SQL became readable again. The write is now gated on a `FOR UPDATE` liveness check. `clear_dbt_run_state_by_script_hash` resolved a hash to a path and deleted the path's saved run. `dbt_run_state` is keyed by path by design — one saved run per script — so archiving one version discarded the live version's resumable failure. It clears only once no live version of the path is left; `identity` already refuses a resume whose project, warehouse or engine moved. "Preview rows" ran `runScriptByPath` while the SQL beside it was pinned to a hash, so an old run showed its own SQL over today's rows. Verified end to end: with v3 deployed, the v2 run's preview runs v2's hash and returns v2's rows. Also: keep the TAIL of a captured stderr, since dbt prints its summary last; one `$derived` for the parsed result rather than five; collapse three near-identical argument accessors onto one generic; fold the single-use `copy_dir_command` into its caller; and give `parseDbtRun.ts` one status classifier instead of spelling dbt's failure vocabulary twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(dbt): one JobCtx down the executor, one table per adapter Two changes aimed at the operations this code will keep having: adding a phase, and adding a warehouse. `JobCtx` already bundled the five values every phase needs, and nine functions took it — but the top of the executor threaded the fields apart and rebuilt the struct at each call, so the same literal appeared eight times and each new phase meant five more parameters. It is now built once per entry point and reborrowed. `prepare_project` goes from 21 parameters to 17, `retry_failed_nodes` from 15 to 11, and `run_dbt` drops below the lint threshold. The two remaining constructions are the worker boundary, where the pieces genuinely arrive apart. `DbtAdapter` answered five questions with five parallel matches over the same eleven variants, plus a sixth list of adapters kept by hand in a test. The facts now live in one `AdapterSpec` per adapter, reached through one exhaustive match, so adding a warehouse states its name, driver, package, port, database key and licensing together and the compiler demands the arm. Each arm spreads from a Postgres base, which makes the inheritance visible per adapter instead of hidden in the `_ =>` defaults `default_port` and `database_key` used to carry. `DbtAdapter::ALL` replaces the list the test kept separately. Verified by dumping all seven facts for all eleven adapters before and after: byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): job-keyed run progress, and stop path-wide reads and clears Six findings from the last round, in the order they bite. The publication liveness gate refused on `archived`, but `create_script` archives the parent on every redeploy — so deploying v2 while v1's dependency job was still parsing left v1 without a graph, permanently, which is the exact case the unconditional write existed to serve. It gates on `deleted` alone now; an explicit archive is still covered by the `FOR UPDATE` ordering. A project-owned `profiles.yml` trusted `profile.type` instead of reading the file. The Rust engines carry every adapter, so a CE script could declare `postgres` over a target that is `sqlserver` and have dbt connect with the enterprise adapter. The file is read whichever way, and a descriptor that disagrees with it is refused. Renaming a dbt script, or editing one so its newest version is no longer dbt, cleared the graph for the whole path — every older version's models, SQL and lineage, which their own finished runs still render. Neither needs it: graph queries join on `(path, hash)` through a `language = 'dbt'` CTE, so an old version's rows cannot attach to whatever lives at that path next. Live progress read `materialized_partition`, whose key is the relation and whose `job_id` is only the last writer. Two runs of one project took rows from each other. Progress now has its own job-keyed table; the relation table is untouched, because one row per relation is right for the pipeline canvas and fork defer. Verified with two overlapping builds: both keep 6 rows in the new table, while the old one attributes 6 to one run and 0 to the other. `Scratch::drop` removed a half-installed virtualenv synchronously from inside the job future, blocking a runtime thread; it goes to `spawn_blocking`, with a direct call when there is no runtime to hand it to. The E2E list asked for a `# on table://` subscription the deploy now refuses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(dbt): snapshot a dynamic descriptor's graph per run A `{{ }}` placeholder in `vars` can enable a different set of models per run, so those runs re-ingest the graph. Keyed by version alone, each re-ingest overwrote the last: reopening an older run showed the newer run's project, and a model only the older run built was gone entirely — no SQL, no lineage, and nothing the saved result could colour, since it can only tint nodes that are there. `dbt_node` / `dbt_edge` gain `job_id`. A run of a dynamic descriptor writes its own snapshot under its job id and its page reads it back; a static descriptor writes the version's graph once, under a zero-UUID sentinel, and every run of it reads that. The sentinel is a value rather than NULL because `job_id` is part of the primary key and Postgres does not treat two NULLs as the same key, so each re-ingest would add a row set instead of replacing one. `/assets/graph` takes `dbt_job_id` and prefers a snapshot when one exists, falling back to the version's graph otherwise — so a run page passes it unconditionally and static descriptors are unaffected. Snapshots age out after 30 days, pruned by the runs that write them, so no background sweep has to learn about these tables. Verified end to end: one deploy, two runs of it with `extra=yes` and `extra=no` gating a model's `enabled`. The version's graph holds 6 models, run 1's snapshot 7 including `opt_extra`, run 2's 6 without it; the endpoint returns each run's own and falls back to the version's when the parameter is omitted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(dbt): only snapshot a run whose graph differs, and prune from every run Two costs the per-run snapshot carried, both found by measuring it rather than by reading it. A snapshot was written for every run of a dynamic descriptor, but marking one dynamic is conservative: `graph_is_per_run` is true whenever `vars` holds a `{{ }}` placeholder or `env` holds a `$var:`, which says the arguments reach dbt and not that they change which models exist. The usual case is a date var, whose graph is identical run after run, so the table filled with copies of an unchanging picture — around 1 KB per model per run, which is a gigabyte or so a month for a 200-model project on an hourly schedule. A row set now carries a digest of its nodes, edges and relation root, and a run whose digest matches the version's writes nothing; the read already falls back to the version's graph, so those pages are unchanged. Only a run whose model set really differs pays. The prune was hung off the progress reporter, which exists only for engines that emit node events — so a Fusion or dbt-core-2x instance accumulated snapshots and never deleted any. Retention that stops working because of an engine choice is not retention; it runs detached from every dbt run instead. Verified against a descriptor with a var-gated model: the version's graph holds 8 rows, a run that resolves to that same graph stores none at all, and a run that enables the extra model stores its own 9. Both pages still render their own project — 6 assets without the extra model, 7 with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): scope every dbt_node join to the chosen snapshot `job_id` joined the key, but only the scoping CTE and `dbt_edge` were taught to filter on it. The outer node SELECT and the parent/child joins in the edge query were not, so each model came back once per retained snapshot plus once for the version's graph, and each edge matched every combination of the two — the model count multiplied and the edge join fanned out quadratically. Measured against one stored snapshot: 17 node rows where 8 are wanted, and 28 edge pairs where 7 are. The response dedup hid the edge blow-up from the payload, not from the plan, and the run page refetches the graph every two seconds. The progress table gained writers it was missing. `terminalize_running_relations` settled only the relation-keyed table, so a cancelled or killed run — the case that function exists for, since it leaves no `run_results.json` — showed every in-flight model still spinning on the run page for as long as the row lived. An agent worker cannot write the new table at all, having no database of its own, so the read falls back to the relation-keyed one when a job has no rows there. Also: a wrapped string literal missing its backslash put eighteen spaces in the middle of the profile-disagreement error; a comment still described concurrent runs of one dynamic version overwriting each other's graph, which is what keying by job removed; and the `materialized_partition` index justified itself by a run-page poll that has since moved to another table, though the closing sweep still earns it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): give a graph snapshot a marker row, and scope what reads it Five findings, four of which are the same mistake in different places: a snapshot's identity was inferred from its contents. Existence was inferred from a `dbt_node` row, so a dynamic run that disabled every model — a legitimately empty graph — read as "no snapshot" and its page showed the deployed models instead. The digest was a column repeated on every node and read back with a `LIMIT 1` carrying no `job_id`, so a run could compare itself against another run's digest and suppress a snapshot it needed. The relation-root drift check read the same rows unscoped, so after a drift it could find a previous run's root and conclude nothing had moved. `dbt_graph_snapshot` holds one row per stored graph — path, version, job, digest, timestamp. Existence is that row, the digest lives there once, the drift check reads the deployed row explicitly, and the retention sweep deletes markers first and then the rows no marker stands for. The digest is SHA-256 rather than `DefaultHasher`, whose output is documented as unstable across Rust releases: this value outlives the process that computed it, so a toolchain bump would have silently stopped every comparison matching and quietly reinstated the duplicate snapshots the digest exists to prevent. `/run_progress` ignored the view token, so a share-link viewer got the graph and was refused the progress that colours it. A preview sent only its own three arguments, so a descriptor with a required `{{ }}` var could not be previewed at all and an overridden one previewed a different relation than the page was showing. The run's arguments go first now, with the preview's three overriding. Verified on a project whose only model is var-gated: the deploy stores a marker with zero nodes, a run with the var set stores a marker with one, and the endpoint answers 0 and 1 respectively rather than showing the deployed models for both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor(dbt): squash the runtime's migrations into one Ten migrations reshaping the same three tables is a history no installation ever had. `dbt_node` gained `script_hash`, then `job_id`, then `ingested_at`, with its primary key rebuilt twice; `graph_digest` was added by one migration and dropped by the next after the digest moved to its own table. On a fresh database all of that replays to arrive at a shape the schema can simply state, and this feature has never shipped, so there is no upgrade path to preserve. One migration now creates `dbt_node`, `dbt_edge`, `dbt_graph_snapshot`, `dbt_run_state` and `dbt_run_progress` in their final shape, carrying forward the rationale each of the replaced migrations recorded. The enum additions stay in `add_dbt_lang`, since a value cannot be added and used in one transaction, and the `materialized_partition` index stays separate because it belongs to a table this feature did not introduce. Verified by rebuilding: dropped the five tables, replayed from the single migration, and confirmed the result is identical — same primary keys, the same two composite `script` foreign keys, the same seven indexes. Every `sqlx::query!` in the workspace then compiled against it, which checks each column's name, type and nullability, and a deploy plus run on the rebuilt schema produced 8 nodes, 7 edges, a snapshot marker and 6 progress rows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): authorize snapshot reads, bound the prune, give the marker a lifecycle `dbt_job_id` is caller-supplied and selected straight from `dbt_graph_snapshot`, which carries no RLS — so a caller who could see the script could read any run's model set and relation paths, which a dynamic alias or schema can encode. Both graph queries now require the job itself to be visible, in the authed transaction, the same gate `raw_code` already applies to the script that produced it. The drift check compared against the deployed graph alone, which misses the way back: a run at root B republishes the path-keyed `asset` usages at B, and returning the profile to A then matches the deploy and skips the refresh, leaving those usages at B while dbt builds A. It reads the most recent ingest for the version instead — the one that last wrote them — ordered rather than an arbitrary `LIMIT 1`. The prune anti-joined every non-deployed node and edge with no age predicate, so each run scanned the whole retained sidecar and concurrent runs duplicated it. All three deletes share one age bound again, with the sentinel spelled as a literal so the partial indexes apply — a bound parameter cannot be proven to match the index predicate. `dbt_graph_snapshot` was the one dbt table nothing in the script lifecycle deleted: no `script` foreign key and absent from both `clear_dbt_manifest*` sites. A marker outliving its rows is read as a snapshot with no nodes, and its digest still answers the suppression check, so an identical run would write nothing and then render an empty graph. It cascades like the rows now and both clears take it. Also: the preview cleared `exclude` rather than inheriting it, since previewing a model the run excluded reached dbt as `--select m --exclude m`; and `terminalize_running_relations` no longer claims to cover a killed worker, which never reaches it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): quote profile names, record where usages were published, stop polling the graph A profile name comes from the project's own `dbt_project.yml` and a target from the descriptor, and both were interpolated into `profiles.yml` as bare YAML — including as mapping keys. A name like `prod # hidden` truncates the mapping and a newline opens a sibling key of the author's choosing. Both are rendered as quoted scalars now, as are the BigQuery keyfile's keys, with a test that asserts the document still parses to exactly the keys we wrote. The drift check read the most recent ingest, which latches: a run that returns to the deployed root re-ingests but stores no snapshot (its digest matches the version's), so the moved run's rows stay newest and every later run pays an extra parse and ingest. The publisher now records the root it published the path-keyed usages at, which is the only thing that answers "where do the current usages point" — the deploy's own root goes stale as soon as a run republishes. The run page polled `/assets/graph` every two seconds alongside progress, so it re-sent every node's SQL for the length of a run — hundreds of KB a tick on a real project, for a graph that a dynamic descriptor re-ingests exactly once before the build. It fetches once more shortly after mount and then polls progress alone. Node results carry `outcome` beside `status`. `status` stays dbt's own word, but dbt owns that vocabulary — 1.x and 2.x differ on casing and `no-op` arrived in a minor release — so publishing only it would force a break or a lie the first time it moves. `outcome` is the stable half a downstream script branches on. Also: the worker's dbt entry points are `pub(crate)`, since nothing outside the crate calls them and they resolve secrets and launch processes; and the snapshot gate records that it is RLS-only where `/jobs/run_progress` also honours a share-link token, which is a gap in what a shared page shows rather than in what it protects. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(dbt): pin the graph storage invariants against a real database Every defect review found in this area was DB-shaped — which row set a read resolves to, which rows a clear takes, whether a snapshot exists at all — and none of it is reachable from a unit test on a pure function. Four rounds established these answers and nothing guarded them, which is why each round kept finding another. Six cases, on the harness the repo already uses for schema-shaped behaviour: an identical run stores no snapshot and leaves no marker; a differing run keeps its own while the version's is untouched; an empty run graph is still a snapshot rather than an absent one; clearing one version leaves the others whole; the path-wide clear takes the markers with it; and the sweep ages out run snapshots while never touching a version's own graph. `IngestedNode` gains `Default` so a test can state the two fields a case is about rather than the eighteen it is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * perf(dbt): bound a script's stored graphs by deploy count Run snapshots expire on a clock, but a VERSION's graph could not: its reader is every finished run of that version, and a run page is as old as its job. So nothing reclaimed them — a deploy graph went only when its `script` row was hard deleted, which Windmill does not routinely do. A CI deploying on every commit added a full model set with SQL bodies per commit, forever: roughly 200 KB a deploy for a 200-model project, which is gigabytes a year across an instance. Bounded by COUNT instead of age, since age is the thing that cannot be right here. The newest 50 deploys per path keep their graph and older ones are reclaimed, making growth `versions x models` rather than unbounded in time. Generous on purpose: reaching the bound empties that version's run pages, so it exists to stop unbounded growth rather than to be hit in normal use. Ordered by the script's own `created_at`, so a late-finishing job re-ingesting an old version cannot promote it. Pinned by a test that deploys past the bound and asserts both halves: the count holds, and the newest version is always among the survivors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): let the run page know when its snapshot has landed The one-shot graph refetch was wrong: a dynamic descriptor's ingest happens before the build but after cloning, dependency install and parse, so a fixed delay either fires too early — and the run page then shows the deployed models for the whole run, never that run's own — or keeps re-sending the whole graph for the length of it. Neither is a timing problem to tune; the page had no way to tell "the snapshot is not written yet" from "this run has none". `/assets/graph` answers that directly: `dbt_snapshot_job` is the job the dbt half resolved from, when one was asked for and found. The page polls the graph until that is its own job, and stops. A static descriptor never snapshots, so an attempt cap ends it there rather than polling for the run's duration. `dbt_node.relation_root` is gone. The drift check moved to the marker's `published_relation_root`, which left the column written on every node and read by nothing. `outcome` was published as the stable half of the result contract, but the in-tree consumer still ranked and coloured from dbt's own word — so the field existed and nothing used it. `statusRank` takes it, `DbtRunResult` passes it, and `classifyStatus` is documented as the fallback for results that predate it and for the live event stream, which carries dbt's word alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(dbt): record what a share-link viewer actually sees The comment at the snapshot gate said the graph "falls back to the deployed set", which is only the rarer half of it. A share link is an extra grant for a logged-in user who lacks access to the job, so the usual case is no read on the script either — and then the `live` CTE matches nothing and the whole dbt half comes back empty. A blank Models panel over working progress rows, not a fallback. `docs/dbt-runtime.md` now carries the analysis a follow-up needs: that relaxing this leaks nothing, because `v2_job_completed.result` already gives that viewer every node's `unique_id` and `relation_name` — the graph's only incremental exposure is `raw_code`, which is gated separately on seeing the script. And the shape of the fix: `OptViewToken` and `validate_view_token` are self-contained enough to move into `windmill-api-auth`, which `windmill-api-assets` already depends on, after which the gate can honour a token for that job's snapshot alone while `raw_code` stays where it is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): keep model SQL behind the scripts:read scope, and unbreak CI `/assets/graph` is authorized as `assets:read`, and RLS decides whether the caller can see the script that produced a node — but RLS is not a scoped token's grants. A token deliberately narrowed to `assets:read` could therefore read model source and repository paths for scripts outside its `scripts:read` paths. The same `build_scope_path_predicate` the macro endpoint already applies now gates `raw_code` and `original_file_path`; the relation's shape is unaffected, only its body is withheld. `DbtAdapter::ALL` exists for the tests that must cover every adapter, so it is dead in a release build and `-D warnings` failed all four backend checks on it. It is `#[cfg(test)]` now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): stop the graph poll at the ingest, and type `limit` in the schema The poll's stop condition was a snapshot appearing, with a 40-attempt cap behind it — so a STATIC descriptor, which never snapshots, took the cap every time and re-fetched the whole graph forty times. That is most of what removing the poll was meant to save, and static is the common case. The ingest runs BEFORE the build, so the first model to report progress proves it has already happened: a snapshot absent by then is one this run never writes. Progress arriving is now the second exit, and the cap is only a backstop for a run that reports none at all. `limit` is declared `Typ::Int` but `dbt_arg_schema` had no integer arm, so the run form and the generated clients saw an untyped default and offered no numeric control for a value the worker clamps. Covered by the schema test. `relationOutcome` still re-derived from dbt's word while `statusRank` had moved to `outcome`; both read it now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): make a retry prove its arguments still resolve the same The saved arguments are the ones SUBMITTED, so a `$var:` in them is re-resolved on retry. The identity did not cover the resolved values, so a variable that changed between the failed run and the retry was accepted — and which graph the retry then used depended on WHERE it landed: a worker holding the local snapshot replays the saved manifest, while a database restore reparses with the new value. Placement decided whether the resumed failures described the relations being built. The identity gains a digest of the resolved arguments, and is compared in two halves because resolution happens between them. Project, warehouse, engine and env are checkable up front; the arguments are not, because a retry request carries only `dbt_command` and the ones to compare are the SAVED arguments after this caller has re-resolved them. Comparing the whole string up front would have refused every retry — which is what the obvious version of this fix does. A row written before the digest existed has no last segment, and still restores rather than being refused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): keep pre-upgrade retries working, and stop losing a late snapshot Splitting the identity on its last `|` read a pre-upgrade row's env digest as an arguments digest and left only `<run_identity>` as the prefix, so every saved failure on an upgraded instance became unretryable — a regression the previous commit's own test missed by using an identity with no `|` in it at all, which is not what an old one looks like. The digest is tagged (`|args=`) rather than positional, and the test now uses a real pre-upgrade identity. The graph poll gave up after a bounded number of tries, but provisioning and `dbt deps` precede the ingest and can outlast that on a cold worker — and the engines that emit no node events never produce the progress that ends it early. A finished run now reloads the graph unconditionally, and the poll's own exit issues one last load: progress proves the ingest happened, not that the previous tick saw it, and dbt's compile window is wider than one tick. A `dbt retry` restores the failed run's arguments inside the worker and they are never written back to the retry job, whose own args are just `{"dbt_command": "retry"}` — so previewing a row on a retry's page ran without the vars the run used. The result now carries the invocation's arguments as SUBMITTED, so a `$var:` stays a reference and no resolved value is published. The deploy-count sweep ran instance-wide on every dbt run: `FROM script WHERE language = 'dbt'` has no index to stand on, and both orphan deletes are the complement of every partial index here. It is scoped to the running script's `(workspace_id, path)` — which `index_script_on_path_created_at` serves — and the orphan deletes only run when a marker actually went. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): hide dbt from module-less pickers, and stabilise the retry digests `processLangs` feeds every language picker, including flow steps and app inline scripts. Those are raw bodies with nowhere to carry a module bundle, and a dbt script IS its bundle — so choosing dbt there produced a job that could only fail once the worker looked for `dbt_project.yml`. Those two surfaces use `processInlineLangs`, which drops the languages that need modules; a flow still reaches dbt the way it reaches any script, by path to a deployed one. `graph_digest` moved to SHA-256 because it is persisted and compared by a later worker, and `DefaultHasher` is documented as unstable across Rust releases — but the retry identity's own digests were left on it, and they are persisted in `dbt_run_state.identity` for exactly the same comparison. A toolchain bump would have refused every saved failure as a different project. All three go through one `stable_digest`, length-prefixed so no split of the same bytes collides. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): enforce the tag scope on snapshot reads, reset state between runs A tag scope is an orthogonal hard restriction: a token limited to some tags must not read a job outside them however else it is authorized. The snapshot lookup went through `v2_job` RLS alone, which knows nothing about tags, so such a token could still retrieve a run's model set and its dynamic relation paths. The same predicate `require_job_read_access` applies for the progress half of the page is applied here — `get_scope_tags` is already public in `windmill-api-auth`, and it is `None` for an unscoped caller, so a normal session pays nothing. SvelteKit reuses the run graph between run ids, and `graphTries`, `polled` and `raw` all describe the previous job: a spent retry count stopped the next run's snapshot poll before it began, and stale progress coloured its models with another run's statuses. All three reset when the graph key changes. Also a wrapped string literal missing its backslashes, which put two ~22-space runs in the middle of the retry-refusal message. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(dbt): put each digest helper's rationale on its own function Inserting `stable_digest` above `split_identity` split that function's doc, so five lines describing where the identity divides ended up introducing the hasher. Each is back on the function it describes, stated once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(dbt): read the run-pinned graph through the job, not the asset graph Pinning the asset graph to one run is job-scoped data, but `dbt_job_id` sat on `/assets/graph`, authorized as `assets:read`. The job-read contract — `require_job_read_access` — is five parts that pull in opposite directions (tag scope restrictive, `created_by` permissive, app-embed restrictive-overriding, view token permissive, RLS underneath), so plain RLS is neither a stricter nor a looser approximation of it. Restating the parts near the graph query kept leaving one out: first the job check entirely, then the share-link asymmetry, then the tag scope, and the app-embed restriction was still missing and failing open. The helper cannot be called from `windmill-api-assets`, because `windmill-api` depends on that crate. So move the read instead of the check: the run-pinned graph is now `GET /w/{w_id}/jobs/dbt_graph/{id}` in `windmill-api`, on the same gate as the `run_progress` it colours, and `/assets/graph` has no `dbt_job_id` parameter at all. - `asset_graph_for` takes the job as an argument from an already-authorized caller; the route handler passes `None`. - Extract the graph response into a `AssetGraph` component schema, now that two paths return it. - The run page fetches the job route when it has a job id. * fix(dbt): charge assets:read on the run-graph route, trust the job gate in SQL Round 13 findings on the route moved last commit. The scope domain comes from the URL segment, so putting the read under `/jobs` asked a scoped token for `jobs:read` alone while returning asset-graph data that `/assets/graph` charges `assets:read` for. A token narrowed to polling run status could read workspace topology, and the missing-job fallback made it cheaper still — any random UUID skipped the job gate. Both scopes are now required: the job gate reaches this run, `assets:read` reaches asset data at all. The `chosen` CTE re-decided job visibility under plain RLS after the caller had already passed `require_job_read_access`. It could only disagree, and did so silently by falling back to the deployed graph — a share-link viewer entitled to the run was shown a different run's model set. Dropped; the contract is that a job reaching `asset_graph_for` is already authorized. Also: the flow editor's `+` insert menu still offered dbt (the third `processLangs` caller, missed when the other two moved to `processInlineLangs`), the docs still described the deleted `dbt_job_id` parameter, and the new handler had again been inserted between `get_run_progress`'s doc comment and its function. * fix(dbt): resolve a pinned run's version from the job row, not script RLS Local codex review of the branch. A share-link viewer is entitled to the run and usually has no grant on the project — that is what the link works around. The graph's `live` CTE resolved the version by selecting `script` inside the viewer's RLS transaction, so it answered for their access to the project rather than for the run they were given: the Models panel came back blank beneath working progress rows. A pinned run now takes its path and hash from the job row the handler already read after authorizing the job, so `live` does not consult `script` at all. `raw_code` keeps its own `EXISTS` against `script`, so the model bodies stay behind access to the project. Verified under RLS as an unprivileged role: the shape query goes 0 rows -> 1, the `raw_code` gate stays 0. Taking the version from the job also means a caller can no longer pin one project's version while naming another's run, since `dbt_script_hash` is ignored when a job is given. The run page's graph fetch is a raw `fetch`, which bypasses the interceptor that adds `X-View-Token` to generated-client calls, so a shared page was refused before any of this mattered; it goes through `appendViewToken` now. Also trims three comments to the AGENTS.md limit, dropping drafting-history rationale that belongs in docs/dbt-runtime.md. * fix(dbt): snapshot vars-overridden runs, carry the pinned version everywhere Second local codex pass. A `vars` run argument overrides the descriptor's, and vars drive `enabled`, alias, schema, database and materialization — so such a run builds relations the deployed graph does not describe. It now snapshots under its own job id, which per-job keying makes safe: the version's graph stays for runs that did not override. The old comment claimed gating on it would strand the override's graph for the next default run, which was true only when the write went to the deployed slot. Two sites still read the caller's `dbt_script_hash` instead of the version resolved from the job, so `/jobs/dbt_graph/{id}` without that redundant parameter dropped models the run's version had and a later deploy removed. The `dbt_snapshot_job` marker re-checked `v2_job` under RLS — the recheck the graph query itself drops. A share-link viewer got the right graph and a null marker, so the run page refetched it 40 times before giving up. `wmill script preview` read the bundle with the generic `__mod` suffix and script-module parsing, so previewing a `.dbt.yaml` omitted the project and failed on the missing `dbt_project.yml`. It uses the same suffix and verbatim read as deploy. * fix(dbt): key retry state by principal, not by script path alone `dbt_run_state` held one row per (workspace, script path), and a retry replaces the caller's arguments with the saved ones. Anyone able to run the script could therefore retry whoever ran it last, replaying that run's literal `select` and `vars` against the warehouse and publishing them as their own job's `invocation_args`. Running the script was already theirs to do; seeing another principal's arguments was not. `permissioned_as` joins the key, so a retry resumes only state written under the same authority. Two runs sharing an authority can already act for each other, so this is the boundary that matches the rest of the job model. * test(dbt): pin what a caller without access to the project sees of its run The share-link case had no regression guard, and every fix in this area touched one of its two halves: the graph's SHAPE has to survive a caller who cannot read the script, and the model SQL must not. Two cases against a real database, calling `asset_graph_for` as a member with no grant on the project's folder: pinned to a run, the models render and `raw_code` is withheld; unpinned, the same caller sees nothing of it, so making the first work did not relax the second. Both assertions were checked by mutation — reverting the `live` bypass empties the graph, and dropping the `raw_code` script gate leaks `select 1` — so neither passes on the code it is meant to catch. * fix(dbt): key the worker-local retry cache by principal too Keying `dbt_run_state` by `permissioned_as` left its worker-local twin keyed by workspace and script path alone, so the boundary held only where the database row was consulted. An agent worker never reads that table — `Connection::Http` leaves `latest_job` as `None` — so there the local cache was the whole boundary and it had none: the next principal to retry the script on that worker restored the previous one's `select` and `vars`. Also records the sqlx `--all-targets` trap in the update-sqlx skill: it is needed for queries inside tests, and in a CE checkout it aborts on `tests/otel.rs` (EE-only `otel_ee`) after having already emptied the cache. * fix(dbt): log a dropped retry-state save, correct the run-progress contract Saving retry state is best-effort — losing it costs a retry, not the run that just finished — but `.ok()` dropped the reason too. The only symptom was `dbt retry` reporting nothing to resume, which reads as a bug in retry rather than a failed write. Found by running a real failing build against a worker whose binary predated the `permissioned_as` column: the insert violated NOT NULL and said nothing. The run-progress endpoint's OpenAPI description promised an empty list for a caller who cannot see the job. It is refused instead; an empty list means the job recorded nothing yet or is unknown here. * fix(dbt): return the retry-state write failure the warning was added to report `save_run_state` discarded the insert result, so the caller's warning could never fire and a lost retry row stayed silent — the symptom being `dbt retry` finding nothing on another worker. The error is held rather than returned at once: the worker-local copy is what an agent worker resumes from, so a failed insert must not cost that too. Every exit after it surfaces it, including the ones that give up on the local save. * fix(dbt): decide a retry's graph from its restored args, keep local state in step Three from the seventh local review. A retry submits only `dbt_command`, so the vars-override check ran against an empty argument set and left `graph_is_per_run` false. The failed run's arguments are restored afterwards, and those are what the retry builds with — an overridden one wrote no snapshot for its own job and its page fell back to the deployed graph, showing the wrong enabled models, aliases and schemas. The decision is re-asked once the restore has happened. A failed durable write no longer publishes the worker-local generation either. `restore` accepts a local generation only when the database row names it, so publishing one the database never recorded made this worker reject its own newest state and resume the previous run's — its selection and vars, or "nothing to retry" if that one had succeeded. An agent worker attempts no durable write, so it keeps its local copy as before. A rename that also converts away from dbt moved the old path's retry state onto the new one, reinstating what the conversion had just cleared and leaving one user's arguments and results under a path no dbt script occupies. It moves only while the destination stays dbt, and clears the source otherwise. * fix(dbt): drop retry state when a run produced none, let module pushes fail loudly Three from the eighth local review. A run that never wrote `run_results.json` — cancelled, timed out, or dead before dbt got there — left the PREVIOUS run's state authoritative in both the database and the local pointer, so a later `dbt retry` resumed that older invocation's failed nodes. Producing nothing resumable now clears both copies, so neither can answer for the other. `wmill sync push` wrapped the descriptor lookup and its deployment in one try/catch meant for a missing parent. Any API failure or invalid descriptor was reported as "no parent found" and swallowed, so a module-only push exited zero with the remote project unchanged. Only the lookup is tolerated now. Also condenses a comment that narrated how earlier status comparisons behaved. * fix(dbt): forget retry state on pre-build exits too, drop cascade claims A dynamic run whose pre-build `dbt parse` or graph ingest fails returns before the save that clears stale state, so the previous run stayed authoritative in both the database and the local pointer and `dbt retry` resumed ITS failed nodes — writing relations the run that just failed never touched. Both exits now invalidate, through one helper shared with the no-artifact case. Two frontend comments described dbt producer rows as driving cascade dispatch. The executor returns before dispatch for every dbt job and deployment rejects `table://` subscriptions, so they promised behaviour that cannot occur; they describe the lineage and ownership that is actually retained. * fix(dbt): let the database decide retry state where it is reachable A SQL worker treated "no `dbt_run_state` row" as no opinion and accepted any worker-local generation. But no row is the authoritative answer that the last invocation left nothing resumable, so a local pointer that outlived it — an unlink that failed, a process killed between the delete and the removal, a stale cache — resurrected a replaced run and let `dbt retry` write relations it never touched. An agent worker keeps accepting its local copy: it has no authority to consult. Invalidation failures are logged rather than dropped, since a silent one is exactly what leaves the pointer behind. * docs(dbt): record how to run an agent worker locally, keep archived graphs Every step of standing one up fails as something else: a normal build cannot start one at all, the server's routes need a separate feature, and all three token mistakes surface as a bare 401 on the agent with the reason only in the server log. Written down with the error each produces. Also keeps a dbt script's graph when it is ARCHIVED rather than deleted. The pinned read resolves versions through a CTE that already skips archived rows, so clearing bought nothing and emptied the Models panel of every completed run of the project. Deletion still clears it. * feat(dbt): let an agent worker publish its graph, through one endpoint An agent worker was refused any dbt script whose profile comes from a Windmill resource — the common case — because it could neither read the stored relation root to check for drift nor re-ingest a corrected one. Those look like two needs but collapse into one: verification exists only to decide whether the stored graph still describes reality, so a worker that can PUBLISH never has to ask. It stores what it just parsed. `POST /api/agent_workers/dbt_graph/{workspace_id}` is the whole addition. It wraps the same `replace_dbt_manifest` the SQL path calls, so digest suppression, the marker write and retention cannot drift between the two transports, and it refuses a job the token's tags do not cover. `IngestedManifest`/`IngestedNode` gain Deserialize to cross the wire. Two guards go, both now false: the pre-build refusal, and the `Connection::Sql` gate added earlier to stop a `vars` override grounding an agent run. Live progress stays SQL-only — that is a per-model event stream, and routing it through the API would mean a round trip per node. * docs(dbt): warn that a differing cargo feature set swaps the shared binary * docs(dbt): record the verified agent-worker behaviour and the tmpfs quota trap An agent worker now runs a dbt job end to end, retries, and publishes its graph — confirmed with a dynamic descriptor whose per-run snapshot came back through the new endpoint. The doc said it was refused; that was true before the endpoint existed. Also `WINDMILL_DIR`: on a dev box the job dies with `Disk quota exceeded (os error 122)` writing the project's files while `df` shows free space AND free inodes, because /tmp is a tmpfs carrying a per-USER quota. Point the worker at a real disk rather than trying to clean up beneath it. * chore(dbt): pin the EE revision carrying the agent graph endpoint * fix(dbt): bind the published graph to the job, break the completed-page poll loop Five from the thirteenth local review. The EE endpoint took `script_path` and `script_hash` from the payload and checked only that the supplied job carried one of the agent's tags, so an agent holding any matching-tag job could name another script and replace its graph. Both are read from the verified queue row now and the request carries only the job id. A raw preview has no version, so it no-ops rather than 422ing before dbt runs. `IngestedManifest`/`IngestedNode` take `#[serde(default)]`: they were serialize-only, and a field the serializer skips made the whole manifest unparseable on the receiving side. A completed run page fetched the graph forever — `load()` assigns `raw`, which recomputes `settled`, which re-entered the same effect. The final fetch is keyed to the job by a plain (non-reactive) variable, and `settled` is read untracked. The pin now names a revision that compiles: the previous one still called `authed.tags()`, a method that does not exist, because both that fix and the JSON response landed after it was committed. * fix(dbt): keep a run snapshot out of the script's deployed ownership Everything `persist_ingest` writes after the manifest is keyed by PATH — one row set per script, describing what is deployed there. A run snapshot was still reaching it, so a one-off `vars` override republished that invocation's relations as the script's ownership and the workspace graph stayed on the override's schemas and aliases: an ordinary run of a static descriptor never ingests again to correct it, so only a redeploy would. A snapshot now stops after recording its own rows. The row preview also selected a bare model name, which dbt resolves across every installed package while `show` takes a single node — a project model sharing its name with a package's was previewed wrongly or refused. It selects the package-qualified FQN. * fix(dbt): forget stale retry state when preparation itself fails `prepare_project` runs before every path that could clear it, and it fails for reasons unrelated to the saved run — a profile that stopped resolving, a provision cancelled, packages that will not install. The invocation still left nothing resumable, so the previous one must not stay authoritative: a repaired project would otherwise let `dbt retry` rebuild an older run's selection and write relations the latest invocation never reached. A retry is exempt, since it is trying to use that state and failing to prepare says nothing about it. Also corrects the runtime doc, which still described agent workers as unable to run dynamic descriptors or Windmill-resolved profiles. They publish their graph through the API now; what they do not get is live progress and a durable retry row, and the doc says so. * fix(dbt): spell the whole FQN for preview, bound retained retry generations The FQN selector added last commit was `<package>.<name>`, but a dbt FQN is the resource's path within its package and the matcher must consume the selector and end on equal lengths — so it matched nothing for a model under `models/marts/`, which is the layout most projects use and the one this repo's own complex fixture has. The middle segments come from `original_file_path`, whose first element is the resource root the FQN excludes. Without a path it falls back to the bare name: ambiguous across packages, but a selector dbt resolves rather than rejects. Tested on a nested model, which is the input that separates the three spellings. Superseded retry generations were removed only when a later run published one, and never inside the hour-long grace period — so a burst left a manifest and a results copy per run with nothing afterwards to collect them. At most four now sit in the grace window, oldest evicted first. * fix(dbt): scope preview state to the run, seed the project on a language switch Previews are keyed by `unique_id`, which is the same string for the same model in every run, and the run-change effect reset only the graph and progress. Opening a second run of one project therefore showed the previous run's rows immediately, and `runPreview` treated them as cached and refused to fetch. A generation counter also drops a preview that resolves after navigation, which the reset alone cannot catch. The dbt project was seeded only by the empty-script bootstrap, but dbt is in the ordinary language picker: reaching it by switching a draft produced a script with no `dbt_project.yml`, which the runtime refuses to deploy or run. Both entry points seed now, and neither touches modules that already exist. * chore(dbt): cache the agent graph endpoint's query for the EE offline build * fix(dbt): publish the graph a moved profile built A run snapshot stopped before everything `persist_ingest` keys by PATH, which is right for a one-off `vars` override and wrong for the other two reasons a run re-ingests. `graph_is_per_run` was one bool for all of them, and the profile drift check both sets it and reads back what the publisher recorded: a profile moved A->B was detected by every run forever, each paying a `dbt parse` for a snapshot nobody reads while the asset rows went on naming schema A. The reason is carried now (`GraphRefresh`), and it decides both writes. Drift is the version's own move, so it rewrites the VERSION's graph and republishes the ownership that ends the drift; a dynamic descriptor snapshots under its job id and still publishes; anything the CALLER scoped — an overridden `vars`, a narrowed `select` — snapshots and publishes nothing, so one invocation's subset can neither stand as what the script owns nor drop the models it left out from the version's graph. Where they meet the caller wins, and the next ordinary run settles the drift. A restore also rebuilt `run_results.json` by copying the generation directory a second time, so a burst of saves pruning it mid-restore left `dbt retry` with nothing to resume and a job that reported success. It is written from the bytes the restore already read; a manifest that went the same way falls back to the parse a database restore pays anyway, and a generation that vanished before either read falls back to the database's row for that same run instead of reporting there is nothing to retry. * fix(dbt): select a row preview by package, not by file path The preview built dbt's FQN by dropping one segment of `original_file_path`, which assumes the model root is `models/`. A project setting `model-paths: ["src/models"]` turned `src/models/marts/orders.sql` into `pkg.models.marts.orders`, and dbt's matcher — equal lengths, compared from the front — resolves that to nothing: the preview came back empty for every model in the project. It selects `<name>,package:<pkg>` instead. The comma is dbt's intersection operator, so this names the node by its own name and the package it belongs to, which is what the FQN was reaching for and needs no knowledge of the resource root. Verified on dbt-core 1.12, dbt-core 2.0.0-alpha.5 and fusion 2.0.0-preview.202, including a package shipping a model whose name the root project also uses. * fix(dbt): refuse a lockfile version that is not one, keep a named selector Two things a preview reaches that a deploy does not vouch for. A raw preview submits its own `lock`, so `engine_version` arrives from the caller and was interpolated straight into the engine cache path — `../..` in it made the download, extraction and rename land anywhere the worker can write, and provisioning runs on the host rather than inside the dbt jail. Both it and `adapter_version` (a pip requirement) are now accepted only as a plain version token. `effective_selector` also read any submitted `select`/`exclude` as an override of the descriptor's named selector. The generated run form posts a default back for every field the caller left untouched, and a selector descriptor's `select` default is `[]` — so pressing Test, saving a schedule or firing a webhook built the WHOLE project instead of `--selector nightly`. An override is now one that DIFFERS from the descriptor's own value; a run that wants the whole project despite the selector asks with `["*"]`. * fix(dbt): let a moved profile settle, from the runs that actually happen Two ways the drift check could never come to rest, both verified against a real run of a real project on a normal worker. `add_caller_args` read any submitted `select`/`exclude` as a caller's narrowing. The generated run form posts a default back for every field left untouched, so every run from the UI, a schedule, a webhook or a flow step carried them and was marked caller-scoped: with the profile moved A->B, each one stored its models under its own job id and left the workspace graph — and the root the check reads back — at A. Since no UI run omits the field, the "an ordinary run settles it" escape hatch was unreachable. Both this and `effective_selector` now ask one question, `selection_is_overridden`: DIFFERENT from the descriptor's, not merely submitted. The root was also recorded beside the path-keyed publication rather than beside the graph it describes, so a version that cannot claim the path — an older one run by hash, a deploy overtaken by a newer one — rewrote its graph at the moved root and recorded nothing. Its next run then compared against a root that was absent or two moves stale and skipped the refresh its own run page needed. It is written wherever the deployed row's graph is. Verified end to end: same UI-shaped arguments before and after, the moved profile now republishes (asset rows and version graph both move to the new schema), a second run detects nothing and re-parses nothing, and moving the profile back settles it again. `prune_dbt_run_graphs` also ran from runs alone, while a deploy writes a whole node set of its own, `raw_code` per model included: a project redeployed on every push by CI and run nightly kept one full graph per push until the next run, and one deployed but never run kept them for good. * fix(dbt): drop a self-dependent effect in the run graph `previewGen` was `$state` written by the effect that also reads it, three lines under a `finalLoadFor` that is a plain `let` for exactly that reason. Nothing reactive reads it — the only reads are inside `runPreview`, a plain async function — so it becomes a plain `let` too. * fix(dbt): pin a retry to the engine versions it resolved `run_identity` carried the engine KIND but not the version it resolved, nor the dbt-core 1.x adapter's. Redeploy an unchanged project after a release and it locks a newer dbt or adapter while the saved `run_results.json` still passes the check, so `dbt retry` feeds one version's artifacts to another — the exact reproducibility the lockfile exists to hold. Both resolved versions are in the identity now; a real failure and retry still resumes. Also drops three comments that outlived what they describe: two said `[]` clears a descriptor's selector, which `selection_is_overridden` reversed, and one pointed at an agent-worker guard that no longer exists — the agent path reaches the ingest deliberately and publishes through the API. * fix(dbt): discard a run graph the page has already navigated away from The component is reused across runs, so a slow `/jobs/dbt_graph` or progress response could land after the reset and put the previous run's models, statuses and failure state on the current run's page, where nothing would fetch again to correct it. Every response is now checked against the generation it was requested under — the counter the preview path already used, renamed for what it means. The graph poll also backs off. Neither of its stops is reachable for a whole class of runs — `dbt_snapshot_job` never matches a static descriptor, and `polled` stays empty for the engines that emit no node events — so an ordinary run walked to the cap, re-sending every model's SQL 40 times in two minutes. * fix(dbt): forget the previous run when the durable save fails, keep quoting `save_run_state` returns the database error when its upsert fails, which leaves run N-1's row and local generation in place: same project, same arguments, so a `dbt retry` matches them and resumes an older attempt's failed nodes against this checkout — the outcome the no-results branch twelve lines above calls `invalidate_run_state` to prevent, reached by another door. It now goes through the same call. Best effort, since the delete goes to the database that just refused a write, but the local pointer is what a retry landing back here reads. The run page also rejoined a relation's parts after `splitRelation` stripped their quotes, so the one name the button exists to paste — `"wh"."analytics.v2"."Order Items"` — was copied as something no client resolves. It copies `relation_name` verbatim. And a source on a finished run was called another project's: the check that guards against two projects claiming one relation asks whether this run executed the node, and a run executes no sources — they appear in no `run_results.json`. Nothing materializes a source, so that warning could never be true of one. Docs: the `vars`-override paragraph still said such a run does not refresh the graph, which the table above it contradicts — it refreshes under its job id and publishes nothing. * fix(dbt): keep the asset rows and the version's models describing one graph The workspace graph takes an asset's relations from the path-keyed `asset` rows and its models, SQL, tests and lineage from the version's `dbt_node`/`dbt_edge`. A dynamic descriptor published the former while storing the latter under its own job id, so a placeholder that moved an alias or a schema left the current graph with assets no model stands behind — nothing dbt contributes to them survives. Ownership is published exactly when the VERSION's graph was written now, which is the only state in which the two agree. Two cases are settled elsewhere by design: an override's relations are a one-off, and a dynamic descriptor at a moved profile keeps the deploy's ownership until a redeploy — its runs each show their own models and it re-parses regardless, so the undetected drift costs it nothing it was not already paying. An agent worker has no durable row, so its local `current` pointer is the whole of what a retry reads — and every local publication failure returned success with the PREVIOUS run's pointer still in place. Where a row exists that is harmless (`restore` takes a local generation only when the row names it), so the abandonment is scoped to the agent case. A failed `dbt show` also cleared the retry state: the preparation-failure exempts `retry` but not a read-only command, and the run page's row preview is exactly that, run as the principal the state is keyed by — so a preview that could not provision took the retry away from the run being looked at. Frontend: the run-change reset left `loading` and `failed` behind, so the gap before the next run's answer rendered "no models in the asset graph" — a claim about the descriptor — over a project that is fine. And three derivations argued from "the graph is the current deploy", which the pinned endpoint made untrue; each is still needed, for the version-graph rewrite and retention reasons now written down. * fix(cli): let --skip-scripts cover a script's module files The module shortcut in `elementsToMap` maps the file and `continue`s before every skip filter, and a module is deployed as part of its parent script — so `wmill sync push --skip-scripts` still pushed the script whenever one of its modules changed, and pull still overwrote them locally. Harmless while a module was a rare helper file; every file of a dbt project is one of these now. * docs(dbt): a dynamic descriptor's ownership stays the deploy's * fix(dbt): read the run out of a failure whose message has braces of its own `parseDbtRun` anchored on the FIRST `{` in the error message and parsed everything after it. The worker appends the structured result after the error text, and dbt's errors carry braces — a Jinja template, the compiled SQL, an adapter's own JSON — so the failures most worth reading were the ones whose summary and per-node outcomes the run page dropped. Every brace is tried now, bounded, and the first that parses as a run wins. Pins the EE revision that gives the agent publish endpoint the deleted-version guard the SQL path takes: deletion is soft, the foreign key still accepts graph rows, and the pinned graph query serves non-live versions, so an agent finishing during a delete put a deleted project's model SQL back on screen. The query is byte-identical to `persist_ingest`'s, so the offline cache already covers it — verified with a full-EE `SQLX_OFFLINE=true` check. * fix(dbt): seed a project when a modular draft switches to dbt `seedDbtProject` returned whenever the draft carried any module at all, so a modular script holding a `helper.ts` reached dbt with none of what dbt needs: the project view is read-only, and the worker refuses a bundle without `dbt_project.yml`, so that draft could neither run nor deploy. Keyed on the project file now, and the seed goes in under whatever is already there — the previous language's helpers are inert to dbt and the user's to remove. Also records this runtime's schema in `backend/summarized_schema.txt`: the `table` asset kind, the `dbt` script language, the five dbt tables and the `materialization_status` enum the progress table uses. * docs(dbt): move the pipeline-membership rationale out of the deploy path * fix(dbt): gate a pinned run's model SQL on the version it belongs to The `EXISTS` against `script` is the only thing standing between a share-link viewer and the project's source, and it matched the workspace and path alone. `extra_perms` is a grant on a ROW: archive a version that granted someone access, recreate the path with narrower permissions, and that stale grant satisfied the probe while the query returned the NEW version's `raw_code`. Both probes name the hash now. The regression test drives exactly that shape and fails without it, returning `select 2` to a caller granted only on the archived version. * fix(dbt): keep a delimiter an identifier escaped by doubling Every dialect these relations come from escapes its own delimiter by doubling it, and both split functions closed the quoted section on the first half and reopened on the second: `"schema"."a""b"` came out as `a.b`. The manifest keeps the real spelling, so the run wrote its per-model status and row counts under an asset path no graph node has — the node simply never moves, which is the failure mode this splitter exists to prevent. Fixed in the worker and in its frontend mirror, which have to agree, with a case per delimiter on both sides. * fix(dbt): key retry state by the caller, not only by the principal it runs as An `on_behalf_of` script executes every caller's job as its owner, so `permissioned_as` names one principal for all of them and the retry state — the durable row and the worker-local generation both — collapsed onto a single entry. After one caller's run failed, the next could submit `dbt_command: retry` and resume it: their arguments replayed against the warehouse, and handed back through `invocation_args`. Nothing else separated them, and on an agent worker the local directory is the whole boundary. `created_by` joins the key in both places. For an ordinary script it changes nothing — `permissioned_as` is already that caller — and a run that was itself superseded was never resumable anyway. Includes the offline cache for the four changed queries and the three the pinned-graph regression test added last commit, which had none: `prepare` without `--all-targets` does not compile test targets, so CI's `SQLX_OFFLINE=true ... --all-targets` would have failed on them. * fix(dbt): compare the schema too when reporting a relation that moved `relationDrift` compared the leaf name alone, and the move it exists to report — a profile repointed at another schema, which a later run then writes into the version's graph — leaves every model's name exactly where it was. So the one case that reliably produces a graph naming relations this run did not write was the one case the notice stayed silent for. The schema segment joins the comparison, qualified against qualified: an unqualified one means the target's own database, which the relation names anyway, so comparing that would report a move on every node. * fix(dbt): bound the retry state now that it is keyed per caller Keying by `created_by` fixed one caller resuming another's run and created a growth problem doing it: a shared `on_behalf_of` script kept one row and one worker directory for everyone who had ever run it, and the generation prune only bounds files INSIDE a directory. Three bounds, none of them new machinery. A run with nothing failed or skipped saves nothing — `dbt retry` builds from those nodes alone, so that state could only ever be refused — while still clearing what the previous run left, since its failures are no longer what last happened here. The rows expire on the same 30-day clock as a run snapshot, swept per path by the prune every dbt job already spawns. And the worker-local directories are swept there too, by the age of the pointer a save rewrites, because their digest names neither the script nor the caller. * docs(dbt): the retry state is worker-affine only on an agent worker * fix(auth): only the server may set a token label that names a user `create_token_internal` wrote `NewToken.label` verbatim, and the auth layer reads some labels as an IDENTITY: `username_override_from_label` maps `ephemeral-script-end-user-<name>` to exactly `<name>`, which then becomes `created_by` on every job that token pushes. The label is free-form request input, so any member could mint a token that speaks as somebody else — the shape `require_job_read_access` already works around when it refuses to trust `username_override` and falls back to an RLS probe, and the one that made dbt's retry-state key (`created_by`) forgeable for an `on_behalf_of` script. The labels are refused where request input enters: the member-facing `tokens/create`, and `impersonate`, which names its subject in `impersonate_email` and has no business renaming the caller too. The legitimate producers are unaffected — a job's own token comes from `create_token_for_owner` in the worker, and native triggers and app-embed tokens build their labels themselves rather than accepting one. `Ephemeral lsp token` stays allowed: its override is the fixed sentinel `lsp`, not a name the caller chose, and the editor mints exactly that label through this endpoint for its language server. The test pins the two lists together, so an arm added to `username_override_from_label` that lets a label choose a name fails until it is reserved too. * Revert "fix(auth): only the server may set a token label that names a user" This reverts commit efaa498d82. * fix(auth): only the server may set a token label that becomes a bare username * revert(dbt): key retry state by the execution principal again Reverts the per-caller key and the retention it needed. `created_by` cannot carry an isolation boundary: it is `display_username()`, which a token LABEL supplies, so two callers can share one value and — before the guard two commits back — one could name a third person. GHSA-8x8x-88qc-qp4r settled that class by refusing to trust the name for authorization, and keying on it here was the same mistake in a new place. It also cost a migration, a PK column, two sqlx cycles and a retention sweep to defend. Back to `(workspace, script_path, permissioned_as)`, which is derived from the authenticated username and cannot be chosen by a request. What that boundary is, and the one case it does not cover, is now written down where the retry is specified rather than left to be re-derived: anyone entitled to run the script as that principal may resume its last failure — the same capability as re-running that job, since running it requires the read access that already shows them the run and its arguments — except for a run pushed `invisible_to_owner`, whose arguments a retry still returns. Closing that means having a retry NAME the job it resumes and authorizing it as a job read, which is a change to the run argument, not to the key. * fix(dbt): give a caller's own selection a graph, and load a finished run once A run that overrides `select`/`exclude` was marked caller-scoped but not per-run, so it ingested nothing and its page fell back to the deployed graph. That is only right when the override NARROWS the descriptor's selection — `["*"]`, or any model outside it, builds relations the deployed graph never had, and those are exactly the ones whose progress, SQL and lineage had nothing to draw. It ingests its own graph now, still caller-scoped, so the subset stays out of what the script owns. `relationDrift` also read a schema whose own name contains a period as a move: a segment holding one is ambiguous — an overridden database, or a schema really called `a.b` — so either spelling now counts as unmoved. And a finished run fetched the whole graph twice on mount, every model's SQL included: the reset effect and the finished-run effect both fired in the same tick. The reset only loads while the run is in flight; the other owns the finished case, because a snapshot can land after the run ends. * fix(dbt): classify a restored retry by its resolved arguments `add_caller_args` was handed `raw_args` on the retry path — the arguments as SUBMITTED, where a `select` spelled `$res:` is still a string. `arg_list` then refuses it as "must be a list of strings" and the retry dies before parsing, for a reference that resolves to the very list the failed run built with. The resolved map decides it now, matching the selection resolver and the build; `raw_args` stays what is persisted and published, so no resolved secret outlives the job. Verified against the shape it breaks on: a `select` from a resource, failed, then resumed. Two docs that outlived their code: `dbt_script_hash` is described as a fallback for a job naming no deployed script rather than as the pinning mechanism, since a script job's version comes from the job row and the query value is ignored (spec plus the generated client, which carries the same sentence); and `GraphRefresh`'s fields no longer claim a caller's selection only ever narrows, which is the premise the previous commit disproved. * fix(dbt): a hidden run keeps no retry state The state is keyed by the execution principal, which every caller of an `on_behalf_of` script shares, and a retry publishes the arguments it restored — so for a run the other callers cannot read, that retry was the one way to see them. The equivalence the key rests on ("resuming it is the same capability as re-running it") holds only while the run IS readable, and exactly one run is not: one pushed `invisible_to_owner`. Those now save nothing, and clear whatever the previous run left, since this invocation happened. A hidden run therefore cannot be resumed by anyone, its author included — the cheaper half of the trade, and the reason having a retry NAME its source job is the design that would give it back. Verified: a visible failure still saves and resumes; a hidden one leaves no row, and the retry after it refuses without returning the hidden `vars`. Also collapses `GraphRefresh::caller_scoped`, which stopped distinguishing anything once a selection override became per-run: both writers set both flags, so `snapshot_job` and `publishes_ownership` are now one field and its negation — which is what lets the agent payload carry a single `per_run` bit, recorded there. `is_reserved_token_label` says why `ephemeral-` is reserved (a client-chosen system label is a token its owner cannot list or revoke) rather than restating a username rule its second prefix does not follow. And the module-cache import comment states the invariant once, without the shape of the cache that preceded it. * fix(dbt): a pre-build failure leaves the saved run alone Three exits cleared the retry state before `dbt build` ever ran: a preparation failure, a failed `dbt parse`, a failed ingest. None of them touches a relation, so the warehouse is exactly what the previous run left and its failures are still the accurate description of it — clearing there just costs a resumable failure, and a resume that no longer fits is refused by `run_identity` and the arguments digest regardless. The reachable one is a cancellation during provisioning. Only an interrupted BUILD invalidates, which the save at the end of the job already decides. Verified: a failure saves state, a run whose selection matches no node fails in the ingest and leaves it, and the retry after that still resumes the original failure. Agent-published graphs also had no sweep — `Connection::Http` snapshots every run and spawns none — so the EE endpoint prunes too (pinned at f58bf06), and `prune_dbt_run_graphs` now names every writer rather than only the runs. Drops `ephemeral-webhook-` from the reserved token labels: it yields the label verbatim, not a bare username, so the check now matches the rule its doc and its test state, and the reason given for it was wrong about what `is_user_token` costs an owner. And the result contract lists `invocation_args` — the field most needing it, being another invocation's arguments on a retry. * fix(assets): decode a doubled delimiter when canonicalizing a table key `split_relation` decodes it on the worker side, so the canonicalizer had to as well: a relation whose identifier escapes its own delimiter — `"sales""east"` — was filed by the ingest under `sales"east` and by a hand-written `table://` annotation under `sales""east`. Two nodes for one table, no edge, which is the exact split this key exists to prevent (decision 11). Fixed for both the name and each half of a database-qualified schema segment, with a case per dialect. `invalidate_run_state`'s contract also still promised the pre-build behaviour the previous commit removed; it now names the cases that do invalidate — an interrupted build, and a run hidden from the script's owners. * fix(dbt): reclaim package trees no project asks for any more The cache key covers the whole project digest — a `local:` dependency's content is in no manifest, so nothing narrower is safe — which means every edit of a project that declares packages publishes another full dependency tree, and nothing ever removed one. A worker that lives through a hundred deploys held a hundred trees until an operator cleared the entire cache by hand, and the disk that fills fails every job on that worker, not only dbt's. Swept by last USE, not by publication: a hit dates the tree, so a project unchanged for months is not evicted from under the jobs still running it. The marker is a sibling rather than a file inside the tree, which the restore copies into the project. Staging directories go after a day — one belongs to a single job and is removed when it ends, so an older one is from a worker that died mid-publish. Verified by planting a 30-day-old tree and a stale staging dir: one run reclaimed both and left a fresh tree alone. Narrowing the key to the declared `local:` paths would fix the churn as well, and is deliberately not done here: getting it wrong runs a project against another revision's packages, which is worse than the disk it saves. * docs(api): declare the dbt half of the asset-graph response `AssetGraph` is the response of both `/assets/graph` and the new `/jobs/dbt_graph/{id}`, and the spec described neither `assets[].dbt`, `runnables[].dbt`, `dbt_edges` nor `dbt_snapshot_job` — so every client generated from it saw a graph with no dbt metadata, no `ref()` lineage and no snapshot marker, which is why the run page reaches the endpoint with a raw `fetch` and a hand-written cast. `frontend/src/lib/gen` is gitignored, so the spec is the only tracked description of this surface. Declared, and the client regenerated from it now carries all four (verified: the regen is otherwise a byte-for-byte no-op). * fix(dbt): stop the package sweep from racing the restore it protects Three faults in the sweep the previous commit added. The `.last_used` marker was written AFTER the copy, and the tree it matters for is the one at the retention edge — where the first use in a fortnight is also when the sweep fires. So the source looked stale for the length of the restore and a concurrent job's sweep could remove it mid-copy, failing a run that should merely have refetched. Marked before the copy, and a restore that loses the race anyway is treated as a cache MISS: `dbt deps` resolves the tree again, costing a fetch rather than the run. The sweep also sat inside `if let Connection::Sql`, though it is a walk of the worker's own disk that needs no database — so an agent worker, which has none, published a tree per project edit and reclaimed nothing. That is the state the doc claimed was fixed, and the same argument the graph sweep makes two lines above: retention that depends on the connection is not retention. And the canonicalizer's two halves disagreed after `5890436`. `unquote_identifier` requires the quote at both ends, so a lone `"` in an already-decoded name survives; the schema walk treated one as opening a quote and dropped it, filing the ingest's `sa"les` under `sales` while an annotation's `"sa""les"` decoded to `sa"les` — the split this canonicalization exists to prevent, in the function I had just touched. The halves share one rule now, and the test drives the decoded spelling against the quoted one rather than asserting only the annotation's. * fix(cli): track a dbt project's parent descriptor for every authored file `buildTracker` decides whose top hash `wmill-lock.yaml` refreshes, and it reached the module branch only for files matching a Windmill script extension. A dbt project is mostly files that are not: `dbt_project.yml`, `packages.yml`, schema YAML, seed CSVs. Editing any of them left the descriptor untracked and its module-inclusive hash stale, so the lock disagreed with the bundle that was pushed. Module paths are now handled ahead of that gate. The parent was also derived by searching the RAW path for `__dbt/`, which finds nothing on Windows, where the folder is spelled `__dbt\` — so even a model edit was skipped there. It goes through `getScriptBasePathFromModulePath`, which normalizes separators, and which the sibling helpers already used for this exact reason. The regression test drives all three shapes. With the old gate reinstated it fails on the non-script files and on the backslash path, and passes only for `.sql`. * fix(dbt): re-resolve unlocked dependencies, and reclaim retry state nobody writes A project with no checked-in `package-lock.yml` asked dbt to RESOLVE its ranges and mutable git revisions, and dbt does that on every run. A cache hit skipped `dbt deps` entirely and every hit refreshed the marker, so the first resolution was pinned for as long as the project stayed in use — a version range that moved was never picked up. An unlocked tree is now a miss once it is a day old; a locked project keeps its tree, since the lock is in the key and pinning is what it asked for. Retry state: `invalidate_run_state` removed the pointer and left the generations, and the whole state root was swept by nothing after `a236972` — so a script that stops running, or a principal who stops running it, kept its last generations (a `manifest.json` each) for good. Forgetting the state now prunes its generations, and a 30-day sweep of the state root runs beside the package sweep, for the reason that one already gives: retention that runs only for the script being run is not retention for the ones that are not. This restores one half of what `a236972` reverted — the local sweep, not the per-caller key or the row retention — because it never depended on that key. The graph orphan sweep also committed its marker delete separately from the two deletes it gates, so an error or a restart in the gap left graph rows whose marker was gone — and since the sweep only runs when a marker went, every later call skipped it and those rows were unreachable for good. One transaction now. `parseDbtRun` scans from the LAST brace: the payload is appended, so backwards finds it immediately where forwards parsed the whole message once per brace in the error text. And `reject_reserved_label` moved into `create_token_internal`, the one path every caller-supplied label reaches, so the invariant sits where a new route would break it. * fix: undo two of my own regressions, and agree with the backend everywhere `2e34dd48` hoisted the module check above the extension gate, which is right, but did not bring the entry-point rule with it: a folder-layout script's METADATA is `<base>__mod/script.yaml`, an entry-point path, and pushing it as a content file makes the metadata pass ask for the language of `.yaml` and abort the whole command. Reached by editing the summary of ANY modular script — pre-existing, nothing to do with dbt. Metadata resolves to its content file now, with a test. `d3716a09`'s unlocked-dependency refresh is reverted. It measured staleness against the last-USE marker, which every hit rewrites, so an actively used project never reached the threshold — and when it did fire, the re-publish `rename` cannot land on a populated directory, so the freshly resolved tree was discarded and the stale one re-stamped: a network resolve thrown away per idle day. Doing it properly needs a publication timestamp and a publish that can replace a tree; the limitation is recorded where the cache is keyed instead of half-implemented. `parseDbtRun` no longer counts braces in either direction. The payload is appended pretty-printed, so its `{` is the only one at column zero — forwards blew the cap on an error full of braces, backwards on one `{` per node, which a few hundred nodes reaches. Test drives a 400-node failure. And `parsePipelineAnnotations` was the third copy of the quote rule, still splitting a doubled delimiter: the live canvas keyed `"sales""east"` as `saleseast` where the deploy keys `sales"east`, so an annotation pointed at a node the deployed graph does not have. All three agree now, including on a lone delimiter in an already-decoded name. * chore(dbt): leave cache retention and the migration's comment out of this PR Two deliberate subtractions, so what ships is the set that has been verified rather than the set that was written. The package-tree and state-directory sweeps are gone (recoverable on `dbt-cache-retention-followup`). They delete directories on a worker, they were the newest code here, and one of them already needed a second pass for racing an active restore — while what they buy is disk hygiene, not correctness: today an operator's `cache_clear` reclaims these caches, exactly as it does for every other language. Both limitations they addressed are now recorded where the cache is keyed. `invalidate_run_state` still prunes the generations it orphans, since that reuses the existing pruner and its grace window rather than sweeping a root by age. And the migration is back to its pre-PR bytes. The only change left in it was a corrected comment, which every database that already applied this migration would have paid for with a checksum mismatch at startup — the explanation lives in `prepare_project` and in the design doc, which is where it is read. * fix(dbt): pin the resolved package tree to the deployed version A cache hit skipped `dbt deps`, so an unlocked range or mutable git revision stayed on whatever the first worker resolved, and a retry could feed one resolution's run_results.json to another. The deploy now records the digest of the generated package-lock.yml, the package cache is keyed by it, and it joins the run identity; a worker that resolves anything else is refused rather than run. Also carries the agent-wire round trip test for an ingested manifest, and the EE pin for the agent publish endpoint. * docs(dbt): state the dependency contract the deploy pins The design doc still described the package cache as keyed by the project digest alone, and carried an open question about refreshing unlocked dependencies that the deploy-time pin answers: resolution happens once, at deploy, and every run of that version installs it or is refused. Records what that costs and buys — committing package-lock.yml makes deploys cache-hit, deploying again is what picks up a newer range — and that these worker-local caches are reclaimed by cache_clear, as every other language's are. * fix(dbt): four findings from the review round - An interrupted retry republished the results it had only restored. A retry starts with the previous attempt's run_results.json in place; cancelled before dbt rewrites it, the save dated those failures to this job, so the next retry rebuilt nodes this one had already redone. Verified by A/B: without the check the row survives, carrying the cancelled retry's id. - The deploy refused a committed package-lock.yml that dbt itself updates, which it does whenever the sha1_hash it recorded for packages.yml no longer matches. Only a run has a resolution to be held to; the deploy establishes one. - A second graph load for the SAME run could land out of order, leaving a finished run showing the deployed fallback for good. - Directory nodes in the project tree carried no path, so two of one name at the same depth shared a collapse key and folded together. * fix(dbt): name every adapter, and stop guessing which project owns a model The invalid-adapter message listed four of the eleven accepted spellings. And a relation may have several script producers, none of which the provenance record identifies — prefixing the first one names a __dbt folder that does not exist, so an ambiguous relation now shows the path inside the project alone. * refactor(dbt): name the graph-snapshot column for when it is written `published_relation_root` described a design persist_ingest does not use: the root is recorded by every ingest that is not a run's own snapshot, including one that publishes no ownership — deliberately, since a version that cannot claim the path would otherwise record nothing and compare against a stale root forever. The column comment said the opposite and the name followed it. Safe to edit the migration in place because this PR introduces it: no database outside a checkout of this branch has ever applied it. One that has needs UPDATE _sqlx_migrations SET checksum = decode('<sha384 of the file>','hex') WHERE version = 20260725084314; alongside the column rename, or a fresh database. * style(dbt): bring comment blocks within the four-line guidance AGENTS.md asks for each invariant at the place someone would break it, in at most four lines. The new dbt files carried 60 inline blocks past that, several of them three rationales deep at one site. Nothing durable is dropped: what covered several constraints at once is split to the lines it constrains, and eight comments that had drifted above the wrong test — six stacked over one profile test, two over the wrong executor test — are reattached to the tests they describe. * docs(dbt): carry the snapshot-column rename into the summarized schema The rename reached the migration, both queries, the sqlx cache and the design doc, but not the compact schema summary — which is the file agents read instead of the migrations, so it was the one copy that could mislead silently. * docs(dbt): bound the retry-state and package-pin residuals precisely Two claims in the design doc were true but not precise enough to act on. Retry state: the equivalence between resuming a failure and re-running the job holds because folder read grants job read, which is also what grants execution. It does NOT hold for a u/<owner> script shared through extra_perms, where no folder policy applies and a grantee cannot read even their own on-behalf run. The package pin: the refusal is per worker. A ranged dependency with no committed lock reproduces its resolution only from a cache hit, so it keeps running where the tree is warm and fails on the first cold worker. * fix(dbt): refuse a retry of a run its caller cannot read Retry state is keyed by execution principal, so every caller of an on_behalf_of script shares one saved run. Under a folder that matches job visibility exactly, but a u/<owner> script shared through extra_perms has no folder policy: a grantee can read neither the run nor their own, while a retry published its arguments. The restore now applies require_job_read_access's rule — you can always read a job you launched, otherwise the row must be visible under your own RLS. Deliberately NOT job_perms, which carries the identity the job runs AS: for an on_behalf_of script that is the owner, and probing with it authorizes everyone. Verified with two members on one shared principal: bob is refused alice's run on the u/ path and her marker never reaches his result, alice resumes her own, and bob still resumes alice's run of the same script under a folder. * fix(dbt): close the retry check's own gaps The check landed with three holes and four rough edges, all found by review: - A pruned source run was allowed. dbt_run_state outlives job retention, so that handed the last failure's arguments to whoever asked next. It now fails closed, and says so rather than claiming the run was someone else's. - An agent worker skipped the check entirely: it reaches no database. The saved generation now records who launched it, and an agent authorizes the half it can prove — the launcher may resume. A generation written before this field is not resumable there. - The row was authorized on one read and restored on another, so a run completing in between was restored unauthorized. restore_from_db now takes the job it was authorized against. Also: restore_run_state's doc comment had drifted onto the new helper, the not-a-member comment described the saved run rather than the caller, and the refusal message carried a run of spaces from its line continuation. * fix(dbt): keep the descriptor in --json sync, and fail closed on a stateless row A dbt script's content is `<name>.dbt.yaml`, and `elementsToMap` drops every .yaml when metadata is JSON — so a --json workspace tracked the metadata, the lock and the whole project bundle but not the descriptor: a descriptor-only edit pushed nothing and a fresh pull wrote a script with no source. Pinned by a test that fails without the exemption. And the retry check read job_id flattened, which merged "no row" with "a row naming no job" and skipped authorization for the second. Unflattened, the row that cannot be authorized is refused. Also graph_digest: serde rather than {:?} for edges, and delimited parts. * docs(dbt): state that retry authorization is by identity, not token scope The section claimed the read-access equivalence was enforced. It is, for identity — but the worker never sees the submitting token, and neither v2_job nor job_perms records a scope, so a token denied jobs:read can still resume its own principal's last failure. What it gets is the arguments as submitted, so a reference is re-resolved under whoever retries rather than disclosed. Also names the cases the check refuses outright, which were spread across three commit messages and nowhere a reader would look. * chore(dbt): repin EE after rebasing the agent-graph branch onto EE main * fix(dbt): say why a test-only run has no models, instead of blaming the profile A selection of tests alone builds nothing, and a test is an assertion rather than a relation, so the ingest keeps nodes with no asset to hang on and the graph comes back empty. The run is correct and its results render — but the empty state named the one cause it is not, a project with no warehouse identity, and sent the reader to their profile. The graph itself is unchanged: retaining a selected test's attached model would have a test-only script claim a read on every model it asserts against, which is an asset-graph semantics change and wants its own review. * fix(dbt): authorize a retry by RLS alone, and eight review findings Retry authorization drops the created_by grant that mirrored require_job_read_access's "you launched it". That name is a display name derived from a token label, so a worker cannot tell a launcher from a collision — the objection Codex raised on four heads. Visibility under the caller's own RLS is the whole rule now. An on_behalf_of script under u/<owner> shared by extra_perms is resumable by the owner alone; under a folder every caller keeps the resume. Also, and each verified against the code first: - The local generation is revalidated after the file work, so a newer run publishing state mid-restore no longer lets the superseded one resume. - Both dbt zip lookups in sync normalize the OS separator, as the resource lookup beside them already did: on Windows a pulled dbt project was laid out as __mod, the one layout dbt cannot run from. - A resource storing port as a string no longer silently connects to the adapter default; a value that is not a port is refused. - An oversized but readable project file fails the deploy instead of shipping an incomplete project that compiles and fails at run time. - cache_clear does NOT reach the dbt caches: it clears cache/ while these live under cache_nomount/, as bun's do. The doc said the opposite, and that claim was the justification for dropping the retention sweeps. - A dbt unit test counts as a test for the empty-state message, as it already does for the worker's test-phase detection. - Per-node results keep the catalog segment when rows disagree on it. - The project panel copies through the app's clipboard helper, .dbt.yaml resolves back to dbt in EXTENSION_TO_LANGUAGE, and two test blocks clean up their temp directories. A comment describing a race against a sweep this PR does not ship is gone. * fix(cli): keep the oversized-file check to a bounded read The refusal statted the file and then read it whole to tell text from binary, which is the cost the two comments above it and isBundledModuleFile exist to avoid: a multi-gigabyte seed would be loaded just to be refused. Same 8 KB head read as that predicate. * revert(dbt): accept the retry residual instead of gating it Resuming grants no capability a caller lacks: they may already run the script as that principal, and a plain run builds a superset of what a retry rebuilds. The one thing a retry adds is information -- the result echoes the resumed run's submitted arguments, for the row preview -- and in almost every shape the caller could already read that run: a folder grants job read alongside execution, and an ordinary script's runs are keyed under the caller's own principal. It takes a user-path script AND extra_perms sharing AND on_behalf_of for the two to diverge. Gating it needed an identity the worker does not have. created_by is display_username(), which a token label supplies: trusting it authorizes a collision, and resolving it as a username denied every labelled token -- a CI token is `label-<name>`, no workspace member, so its own retry was refused. That regression cost more than the exposure. Kept, because none of it is about the caller: - a cancelled retry no longer republishes the results it only restored, - the restore is pinned to the row it chose rather than re-reading it, - a newer run publishing mid-restore no longer lets the superseded generation resume. The three sqlx entries the gate needed are gone with it. * fix(dbt): select unit tests, and say why the state key is the boundary `dbt ls` enumerated five resource types and omitted `unit_test`, which dbt treats as its own type rather than a flavour of `test`. A descriptor selecting `test_type:unit` or a unit test by name therefore resolved to the empty set, which the deploy refuses outright with "matched no dbt nodes". Verified both ways on a fresh path: the deploy fails without this and succeeds with it. All three engines list `unit_test` among the accepted values. The restore's state lookup now carries the reason it keys on the principal and not the caller. That decision was in the design doc, the commit messages and a PR thread — everywhere except the line someone would change to "fix" it, which is where AGENTS.md asks for it and where I twice went wrong myself. Also: the `GraphRefresh` doc claimed a dynamic descriptor publishes its graph as the script's ownership, while `per_run_models` snapshots it under the job id and `publishes_ownership()` returns false — only a moved profile republishes. And `fn n` in dbt_profiles.rs had no callers left after `port_of` replaced it. * feat(dbt): offer the resume on a failed run, where the choice is relevant "Run again" prefills the arguments the run just used, so the obvious action after a failure rebuilds the whole project while the cheap resume stays invisible unless the reader knows dbt_command has a retry value. A finished run with failed or skipped nodes now says how many, and links to its own run form with the command already switched. A link rather than a submission: the caller still presses Run, so nothing about permissions or arguments changes. Deliberately not automatic. A resume is not a re-run — models are usually rebuilt because upstream data moved, not because they failed — so a schedule that resumed after a failure would leave every model that succeeded carrying yesterday's data while reporting success. * feat(dbt): put the command first, and offer the resume where the rebuild is Three things made rebuilding the whole project the path of least resistance after a failure, when resuming its failed and skipped nodes is what dbt offers: - `dbt_command` was LAST in the generated signature, under `limit`. The schema's `order` drives the run form, so the argument that decides what the run does sat below the ones that only narrow it. It leads now, and the signature test asserts the order rather than the set, with the reason it is pinned. - "Run again" prefills the arguments of the run it came from, so for a failed run it reproduces the rebuild. It now carries `dbt_retry_hint`, and the form shows what resuming would do instead -- without choosing it, so the caller decides. - Its dropdown offers "dbt retry with same args" above the plain re-run, for a failed dbt job only: a run that succeeded has no saved failure to resume. Verified in the browser: both dropdown labels render unclipped, the alert appears only on that path and disappears once the command is switched, and the schema of a redeployed script lists dbt_command first. * feat(dbt): describe the run arguments, and hide the ones a command ignores The form offered six fields at once with no indication that most of them apply to one command each. `retry` reuses the arguments of the run it resumes, so every override is ignored; `limit` belongs to `show`; `full_refresh` to `build`. Each argument now carries what it is for, and a `showExpr` for when it applies, which SchemaForm honours by hiding the field AND dropping its value from the payload. Selecting `retry` leaves the command alone on the form. Not modelled as a `oneOf`, which is the more precise shape: Windmill renders that as a variant selector whose value is a nested object keyed by a discriminator, so the arguments would stop being flat -- and the worker reads them by name, as do schedules, webhooks, the CLI, saved inputs and the retry state's own copy of them. * fix(dbt): a retry may name the run it means to resume The retry actions I added an hour ago appear on any failed run, but the saved failure is keyed by script and principal rather than by job — so from an older run's page they resumed whatever failed most recently, under a label promising "same args". Both now pass `dbt_retry_job`, and a retry naming a run the state no longer holds is refused with the id it does hold. Verified: with failures A then B saved, a retry naming A is refused and names B in the message, while a retry naming B resumes. Also from the same review: the `select` help advertised `state:modified`, which needs a comparison manifest this runtime supplies no `--state` for, and the openflow note still described a dbt script as a descriptor naming a git repo rather than the module bundle this ships. * fix(dbt): make both retry actions actually name their run `dbt_retry_job` is not a form field, so the banner's link to the run form dropped it and the retry resumed whatever failed last — the bug the argument exists to prevent, reintroduced by the affordance meant to use it. The banner submits the run itself now, as the "Run again" dropdown item already did. And an agent worker never applied the check at all: its `latest_job` is always None, so the local generation was accepted unconditionally. Both reviewers found this independently. The decision moved into `chosen_generation`, which the agent path shares, with the refusal naming the run the worker does hold. Pinned by a test that fails without the check, on the connection kind where it was missing rather than on the one that already worked. * fix(dbt): let an unchanged push be a no-op, and stop advertising a retry the form cannot aim Two of a dbt script's fields are DERIVED by the server after the no-op check runs: the lock, which only a dependency job can produce, and the schema, which comes from the descriptor and which no client can derive (windmill-parser-wasm has no dbt arm). The check compared what arrived instead of what would be stored, so every unchanged `wmill sync push` created another version and another dependency job -- the sync churn skip_if_noop exists to prevent. Both are now compared as stored, with the parent required to hold a lock so a deploy whose dependency job failed can still be retried by pushing again. Verified by A/B on a faithful payload: without this the same unchanged push creates a version, with it the existing hash comes back and the count holds. The run form's retry hint is gone. `dbt_retry_job` is not a form field, so a retry started there resumes the last failure of the script rather than the run the message pointed at -- the two buttons name their run, that path could not. Better to not offer it than to offer it wrong. Also reunites a comment with the derivation it describes, 40 lines below where an earlier edit of mine left it. * docs(dbt): an unchanged push no longer re-resolves dependencies The section told the reader that a byte-identical deploy re-pins a ranged dependency, which was true only because no-op detection was broken for dbt. Now that an unchanged push is skipped, moving a pinned resolution takes an actual change. * feat(dbt): offer the resume only on the run it would actually reach * fix(dbt): a refusal names both the run asked for and the one held * docs(dbt): say that show previews one node when several are selected * feat(dbt): a retry names the run it resumes, and the form can fill it in * refactor(dbt): make the run's command a oneOf carrying its own overrides * fix(dbt): refuse a command block that names no command, and keep state off retired paths * fix(assets): let copilot and the assets filter see warehouse tables * fix(dbt): drop the unused db handle from the retry-principal lookup * fix(dbt): stop a preview poll when the run page moves on * fix(dbt): refuse an escaping packages-install-path, re-arm the graph poll on navigation * docs(dbt): concurrent runs of one script are the script's to serialize * fix(dbt): bound what the log tailer holds in the worker process * fix(dbt): a stale prefill answer must not aim another script's retry * fix(dbt): a rename leaves the old path archived, so state must not be rewritten there * fix(dbt): keep a retry's routing inputs, and stop an app-embed token probing resumability * docs(assets): state the authorization the graph helper expects of its callers * fix(dbt): merge the retry into the fetched arguments, not the too-big placeholder * fix(dbt): the command block's label decides the command, not map order * fix(cli): an oversized dbt project file must not read as generated output * fix(dbt): the prefill names a run only when the caller may read it * fix(cli): refuse an oversized dbt file before anything reads its body * fix(dbt): fold table paths as the deploy does, and preview with fetched args * fix(dbt): stream an engine archive to disk instead of holding it in the worker * refactor(assets): the dbt asset kind is dbt://, keyed on the relation * chore: regenerate the auto-generated prompts for the dbt asset kind * fix(assets): rename the kind at the callers that request it, and refuse an unknown one * feat(dbt): show the project's models under the run form on the script page * fix(dbt): the rename missed the dbt-edge membership keys, so the DAG lost every edge * fix(dbt): a preview must run the dbt writer, with the form's arguments * feat(dbt): the models graph takes the bottom of the script page, as a flow's does * fix(dbt): an unknown outcome is not a pass, and a preview keys on its arguments * feat(dbt): a warehouse is configured on the workspace, named like the lake * feat(dbt): asset identity keys on the warehouse name, resolved for agents too * feat(dbt): warehouses are configured in workspace settings * feat(dbt): the descriptor lives in the project and is optional * docs(dbt): the warehouse is a workspace setting and the descriptor is optional * fix(dbt): the warehouse route names its capture, and the editor stores the map * test(dbt): pin the warehouse setting round-trip and the job-scoped route * fix(dbt): a project-owned profile must authorize the warehouse it names * fix(dbt): the settings tab imports TextInput from where it lives * fix(dbt): a warehouse name is validated wherever one enters, and an absent descriptor is not a diff * fix(dbt): the settings inputs pass their placeholder the way TextInput takes it * fix(dbt): pushing a project that has no descriptor is not an error * chore(dbt): keep the module suffix private to its module * fix(dbt): warehouses survive a fork or rename, and an empty descriptor is never a file * fix(dbt): a workspace rename carries its dbt graph with it * fix(dbt): a rename takes the run state with it, not only the graph * chore(dbt): drop the query-cache entries the settings edits orphaned * docs(dbt): state the three rules that keep an absent descriptor absent * fix(dbt): an unreadable descriptor is an error, and an emptied one is a deletion * feat(dbt): the warehouse is unpermissioned, like the workspace bucket * feat(dbt): an agent worker's run reports its per-model state too * docs(dbt): how an agent worker reports what it ran * feat(dbt): a project-owned profile reports its database, so its models share nodes * style(dbt): rustfmt the files this branch touched * fix(dbt): drop the job id write_profiles no longer reads a resource with * fix(dbt): a preview reads the graph's edges and sends the model, not the cache key * fix(dbt): a deleted producer explains itself instead of a bare 404 * fix(dbt): one warehouse resolution path, interpolated against the job * fix(dbt): a project with no descriptor pulls and opens in dev mode * feat(dbt): one file tree, editable, and Test builds the model you have open * fix(dbt): a preview asks the writer's own graph, and lfs keeps its omission * fix(dbt): metadata stays beside the project, and only a model narrows a build * fix(dbt): a descriptor-less project previews, and dev reloads the whole bundle * fix(dbt): forks carry the graph, and a vanished project stops the push * fix(dbt): no-auth resolves warehouses, and a preview runs the version it read * fix(dbt): a retry recognizes its own run when the profile carries a job token * fix(dbt): the project marker is not deletable and the descriptor path is reserved * fix(dbt): a templated adapter type is refused, and a removed project archives * perf(dbt): the manifest lands in batches, and .py models are creatable * fix(dbt): only a 404 excuses a failed archive, and edges are covered * fix(dbt): a python model narrows the build, and a failed removal is not silence * fix(dbt): a settings name may be any word, and the reserved path is stated once * fix(dbt): an absent descriptor needs its project, and the form matches the run * fix(dbt): a bundle is dbt by language, and both job routes agree who may call * docs(dbt): every identity doc names the warehouse, as the code does * perf(dbt): an agent posts its run's progress once, and show cannot touch a seed * fix(dbt): bound what a warehouse may be named, and say who may ask * fix(dbt): a run's graph withholds what the project's author wrote, and dbt_project.yml is rendered before it is read * feat(dbt): withhold dbt from the language picker until it has its own editor * fix(dbt): refuse a path claimed by both a dbt project and an ordinary script * test(dbt): the bundle assertion is a set, not an order * fix(dbt): a project's generated dirs are re-read when its config changes * test(dbt): assert the synthesized descriptor without assuming a separator * fix(dbt): both push paths refuse a shared path, and every writer sweeps its progress rows * fix(dbt): preview only what dbt show can select, and a retry keeps its command block * fix(dbt): an identifier carrying a path separator gets no asset key * fix(dbt): no engine ships in an image, and show may name only one node --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>	2026-08-01 11:58:05 +00:00
Ruben Fiszel	a8455acd7d	feat: make bigquery and snowflake script languages available in CE (#10324) ... * feat: make bigquery and snowflake script languages available in CE * docs: add snowflake to backend cargo feature map * fix: stop logging the snowflake bearer token at debug level	2026-07-25 12:07:28 +02:00
Ruben FiszelandClaude Opus 4.8	7e2f1afffb	fix(python,windows): cross-platform cross-process wheel-install lock (#10264) ... * fix(python,windows): cross-platform cross-process wheel-install lock The advisory lock that serializes concurrent uv installs into a shared Python wheel-cache dir was gated `#[cfg(unix)]` and used `nix::fcntl::flock` directly, so on Windows there was no cross-process serialization at all. Multiple agents running as services on one Windows host share a single per-user cache dir (`.../Temp/windmill/cache/python_<v>/`); when several jobs install the same package at once their uv processes clobber each other's atomic renames, surfacing as "no .dist-info directory", "RECORD ... cannot find the file specified", and "failed to rename ... os error 2" install failures. Replace the unix-only flock with `fs4`'s cross-platform advisory lock (flock on unix, LockFileEx on windows). Unix behavior is unchanged (same flock syscall, whole-file, released on handle close/process death); Windows now gets a real per-package cross-process lock so co-located agents serialize their installs instead of corrupting the shared cache. Fixes WIN-2225 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(worker): drop now-unused nix `fs` feature The `fs` feature was only pulled in for `nix::fcntl::flock`, which the previous commit replaced with `fs4`. Remaining nix usages need only `user` (plus the workspace-inherited `process`/`signal`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-22 22:42:35 +02:00
3 people	396fb1c475	feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs (#10139) ... * [ee] feat(otel-tracing-proxy): trust internal endpoints with untrusted CAs Add `insecure_upstream_hosts` and `upstream_ca_certs` to the HTTP Request Tracing settings so the OTEL tracing proxy can reach internal endpoints with untrusted or private-CA certificates while keeping them traced. Wires the two settings through the worker config and live reload, adds the inputs to the instance settings UI, and pulls in the rustls upstream-client deps (hyper-rustls/tokio-rustls/rustls/ rustls-native-certs/rustls-pemfile; hyper-http-proxy switched to its rustls feature). The proxy-side implementation lives in the companion EE PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: bump ee-repo-ref to otel_ca companion commit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(otel-tracing-proxy): expose new fields in declarative config; bump ee-ref Addresses code-review findings: - Add `insecure_upstream_hosts` and `upstream_ca_certs` to the declarative `OtelTracingProxySettings` in instance_config.rs so operator/GitOps-managed installs can set them and reconciliation no longer drops values saved via the UI. - Restore the trailing newline on ee-repo-ref.txt and bump it to the companion EE commit carrying the strict host-matching / port-ordering fixes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 This commit updates the EE repository reference after PR #665 was merged in windmill-ee-private. Previous ee-repo-ref: 49f458e4446395e98915c220baa757ab3b2ed2d8 New ee-repo-ref: 51e50629f48dbc4f5520a787b4bdfb76f4cd38d3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-07-17 15:57:00 +02:00
Ruben FiszelandClaude Opus 4.8	be57dd91e4	fix(ansible): keep persistent-connection socket path under the AF_UNIX limit (#10167) ... * fix(ansible): keep persistent-connection socket path under the AF_UNIX limit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): root socket dir at WINDMILL_DIR and verify we own it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): fail closed on an untrusted socket root and honor commented section headers Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): validate the socket root after creating it, not before Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): root socket dir at sticky /tmp so the image's 0777 windmill dir cannot disable it Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): only sweep socket dirs the worker could have created Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): drop the configurable socket root, fixing its ancestor and sweep hazards Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): require the socket root be usable, not just safe Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ansible): do not override a control_path_dir the job set itself Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(ansible): drop the dead parent-creation step for the fixed /tmp root Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-17 15:09:24 +02:00
Ruben FiszelandClaude Opus 4.8	0dbd9c1231	perf: eliminate dual-connection DB pool contention across worker, queue, and api (#9798) ... * perf: eliminate dual-connection DB pool contention across worker, queue, and api Reuse the held transaction (or move pool reads before begin()) instead of checking out a second pool connection while a tx is open, extending the fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus several server-pool API handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pass owned pool to get_email_from_permissioned_as in http trigger handler The generified signature takes impl PgExecutor; the http trigger handler passed &db where db is already &DB, yielding &&Pool which does not impl PgExecutor (only surfaced under the full feature set in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint Addresses review of the dual-connection sweep: - worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The caller swallows the error and keeps using tx, so a DB read failure must not leave the outer transaction aborted (it would fail the later commit). Matches the previous pool-read semantics. - Revert reads that were moved onto an RLS (user_db) transaction back to the non-RLS pool, since RLS row-visibility/role context can change results: push_scheduled_job (email/tag/settings lookups; reachable with a user_db tx from api-schedule/api-flows), push_inner native-retry dedicated_worker routing (RLS isolation variants), resources.rs app-namespace folder auto-create (non-admins must not be blocked), and the script archive/delete UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read Adds a worker-crate failpoints feature and a data-driven hook: when the stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still completes (iteration marked failed) — which only holds if the savepoint keeps the outer status-update transaction committable. Without the savepoint the abort would poison the outer tx and the job would never complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 21:33:17 +00:00
Ruben FiszelandClaude Opus 4.8	11d83ab1ec	fix(python): serialize concurrent installs into shared wheel cache dir (#9787) ... Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 14:08:40 +00:00
Ruben FiszelandClaude Opus 4.8	12f92e3ab7	[ee] feat(backend): native script retry without one-step-flow wrapping (#9688) ... * feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-25 13:22:26 +00:00
Ruben FiszelandClaude Opus 4.8	61f3291b24	fix: actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death (#9563) ... * fix: actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * decouple LIMIT_WINDOWS_TO_1CU memory cap from DISABLE_PROCESS_GROUP escape hatch Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * create windows job children suspended, assign job, then resume (close post-spawn race) Addresses Codex P1: AssignProcessToJobObject post-spawn could miss a grandchild forked before assignment. Creating the child suspended and resuming after assignment guarantees it is in the job before running any code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-14 09:57:52 +02:00
Ruben FiszelandClaude Opus 4.7	aa12c66c25	feat(snowflake): derive public key from private key when omitted (WIN-1959) (#9251) ... * feat(snowflake): derive public key from private key when omitted (WIN-1959) Snowflake key-pair auth needs a SHA256 fingerprint of the public key for the JWT iss claim, but the public key is mathematically derivable from the RSA private key. Other tools (e.g. Power BI) only require the private key, so requiring users to supply both is redundant. When public_key is missing, fall back to deriving it from private_key (PKCS#8 or PKCS#1 PEM) instead of erroring out. Fixes WIN-1959 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(snowflake): treat empty public_key/private_key as missing --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-20 06:48:10 +00:00
centdix	17cf538a2d	refactor: move ai providers to windmill-ai (#9120)	2026-05-12 14:09:31 +00:00
Ruben Fiszel	e1a7c75e19	perf(flows): cache resolved flow_env per flow execution (#9079) ... * perf(flows): cache resolved flow_env per flow execution * perf(flows): tighten flow_env cache cap to 1024 and clarify memory note * perf(flows): don't cache transient flow_env resolution failures	2026-05-08 07:43:46 +02:00
centdixandClaude Opus 4.6	26a6d1e4ce	refactor: create windmill-ai crate (part 1 — types, traits, base modules) (#8530) ... * refactor: create windmill-ai crate and move base AI types from windmill-common Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move worker AI types to windmill-ai crate Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move QueryBuilder trait and StreamEventSink abstraction to windmill-ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add base64 dependency to windmill-ai for bedrock PDF support Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add windmill-ai refactor plan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — remove dead bedrock feature, add boxed_sink helper, move plan to docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-22 11:20:27 +00:00
hugocasaandClaude Opus 4.5	172a7d16db	fix: fix otel tracing on nativets ... Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-04-17 14:05:58 +00:00
4 people	c57c769dea	feat: add CI test scripts with auto-trigger on deploy (#8736) ... * feat: add CI test scripts with auto-trigger on deploy Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fix annotation parser early return and handle renames correctly Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move CI test results to top of script/flow detail pages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: improve CI test results spacing, icon, and remove pass label Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: support one-line annotation and use script/path format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: move CI test trigger logic to EE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: move CI badge next to New badge and add deduplicated CI summary Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add CI test e2e tests and fix nullable column annotations Add integration tests for CI test annotation parsing (creates/removes ci_test_reference rows) and the CI test results API (single + batch endpoints). Add backend test for auto-trigger on deploy (private+python). Fix sqlx LEFT JOIN LATERAL nullable column annotations in get_ci_test_results and get_ci_test_results_batch queries — sqlx cannot infer nullability from LATERAL subqueries, causing runtime decode errors when no matching job exists. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix build/sqlx * fix * feat: CI test improvements and templates - Fix windmill-dep-map/private feature propagation in worker, api-scripts, and api-flows Cargo.toml so CI test triggers actually fire in EE mode - Clone ci_test_reference rows during workspace fork - Add polling to CiTestResults component (refetch every 3s while running) - Add running state and auto-refresh to ForkWorkspaceBanner CI summary - Add yellow "CI test" badge on script list rows and detail page - Fix Library badge border color (remove indigo border override) - Add CI Test TypeScript and CI Test Python templates in ScriptBuilder - Update sqlx offline cache - Add debug tracing for CI test trigger in worker_lockfiles Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing children prop to WorkspaceDeployLayout Fixes svelte-fast-check type error when passing named snippets as children content inside the component tag. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback - Remove empty wrapper divs around CiTestResults, move mb-4 into component - Add batch endpoint size cap (max 200 items) - Add ON DELETE CASCADE to ci_test_reference workspace FK (new migration) - Downgrade CI test trigger logs from info to debug - Fix false-positive polling: only treat status='running' as running, not null status (CiTestResults, CompareWorkspaces, ForkWorkspaceBanner) - Fix test numbering in integration tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to latest EE commit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to d9d68c2406df0b59f413ea0b2cb24780a9817d04 This commit updates the EE repository reference after PR #516 was merged in windmill-ee-private. Previous ee-repo-ref: d7ccd9b86da99ec056a0e8708e3637d64290387a New ee-repo-ref: d9d68c2406df0b59f413ea0b2cb24780a9817d04 Automated by sync-ee-ref workflow. * fix: treat queued jobs (job_id set, null status) as running Jobs that have been pushed but not yet picked up by a worker have a job_id but null status. Treat these as 'running' to avoid showing misleading 'pass' badges or '0 passing'. Tests that were never triggered (no job_id, null status) remain neutral/hidden. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: hugocasa <hugo@casademont.ch> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-04-09 17:21:36 +00:00
Ruben FiszelandClaude Opus 4.6	d2d6810db9	feat: add LIMIT_WINDOWS_TO_1CU env var for Windows worker memory limits (#8681) ... * feat: add LIMIT_WINDOWS_TO_1CU env var for Windows worker memory limits Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address CI review — stricter env var parsing and SAFETY comment Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 19:31:07 +00:00
3 people	a46aa641f9	feat: add R language support (#8263) ... * feat: add R language support Add R as a new supported scripting language in Windmill, following the same pattern used for Ruby. Includes: - Backend: ScriptLang::Rlang enum variant, DB migration, tree-sitter-r parser crate with tests, WASM parser binding, R executor with NSJail sandboxing, job dispatch and signature parsing - Frontend: language picker, R icon, syntax highlighting, editor bar insertions (Sys.getenv, get_variable, get_resource), schema inference, init code template, BETA badge - CLI: .r extension mapping, sync support, bootstrap template R scripts use `main <- function(...)` syntax, jsonlite for JSON serialization, and system curl for the Windmill client helper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add R package resolution and installation Parse library()/require() calls from R scripts to extract dependencies. Resolve versions from CRAN, cache lockfiles in pip_resolution_cache, and install packages to a shared R library cache. The run step sets R_LIBS_USER so installed packages are available to the script. - Parser: parse_r_requirements() extracts package names from AST - Executor: resolve() generates lockfile, install() installs from CRAN - Worker lockfiles: wire up R resolve for dependency jobs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add nsjail sandboxing for R resolve and install phases Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: fix R get_variable/get_resource and add sandbox annotation + e2e tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: fix R arg inference with JS fallback parser and get_variable/get_resource Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix flake * nsjail * nits * fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits - Suppress renv verbose output during resolve and install (controlled by #verbose annotation) - Filter renv from install list (already loaded, causes noisy restart message) - Log compact "resolved N packages" instead of full renv.lock JSON - Add R (r-base, r-cran-renv) to DockerfileFull and DockerfileFullEe - Use disable_rl for nsjail install config (R compiles from source) - Reduce default concurrency from 20 to 5 - Add rlang to openflow.openapi.yaml - Fix MainArgSignature (no_main_func -> auto_kind) after main merge Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * final * fix: remove accidental R install from multiplayer Dockerfile Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: remove R from Windows build and DockerfileExtra Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: rename R migration to avoid timestamp collision with trigger_filter_logic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * all * fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: add clear error when Rscript binary is missing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: fix type errors in R fallback parser, use format! in wrap(), add R system prompts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: pyranota <pyra@duck.com>	2026-04-01 06:11:37 +00:00
3 people	ce2e6c8c01	fix: add Authority Key Identifier to MITM proxy leaf certs (#8576) ... * test: add x509-parser dev-dep for MITM proxy cert tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref.txt for ssl-verify-fix branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref.txt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to a90b083660b372bf1da1c18769cbd50936ea8040 This commit updates the EE repository reference after PR #494 was merged in windmill-ee-private. Previous ee-repo-ref: db665a09d5b9a485977d73c22908629e3dda6200 New ee-repo-ref: a90b083660b372bf1da1c18769cbd50936ea8040 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-03-27 23:49:40 +00:00
3 people	56253c04cb	feat: IAM RDS auth for PostgreSQL worker resources (#8573) ... * feat: add IAM RDS auth support for PostgreSQL worker resources Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: use Config builder for IAM RDS connections Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR review feedback for IAM RDS auth Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: update ee-repo-ref to ebea6ef1e5bfcfc3f0151da9687dac6c61bbfab6 This commit updates the EE repository reference after PR #493 was merged in windmill-ee-private. Previous ee-repo-ref: 1228561a98c5195bb97a81d4a57ce2bb2ecfca79 New ee-repo-ref: ebea6ef1e5bfcfc3f0151da9687dac6c61bbfab6 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>	2026-03-27 21:50:29 +00:00
Ruben FiszelandClaude Opus 4.6	d578e40101	feat: add selfApproval option to WAC + inline approval buttons (#8440) ... * feat: add selfApproval option to WAC waitForApproval + inline approval buttons Add self-approval configuration to WAC workflows and inline approve/reject buttons in WorkflowTimeline. - TS SDK: add selfApproval option to waitForApproval() - Python SDK: add self_approval param to wait_for_approval() - Backend: store approval_conditions in flow_status for WAC, enforce self-approval checks on resume endpoints - Frontend: show Approve/Reject buttons in timeline with form support (EE), gated by user permissions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert sqlx query change + regenerate system prompts - Revert get_suspended_flow_info to use original sqlx::query_as! with COALESCE to avoid sqlx offline cache mismatch in CI - Detect WAC by checking if FlowStatus parsing fails + suspend > 0 - Re-fetch flow_status column separately for WAC approval conditions - Regenerate auto-generated system prompt files for SDK changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: use resume URLs for WAC inline approval buttons - Backend generates HMAC-signed resume/cancel URLs when creating WAC approval, stores them in timeline entry and approval meta - Frontend uses anonymous resume endpoint (like classic flows) with fallback to resumeSuspendedFlowAsOwner for admins - Buttons show for everyone when URLs are present; server-side self_approval_disabled check enforces restrictions - Show warning for admins/owners when self-approval is disabled - selfApproval: false requires EE (errors at dispatch on CE) - self_approval_disabled check moved outside user_auth_required gate so it works independently - WAC detection no longer requires task import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add resume_suspended and approval_info endpoints - New approval_token DB table for token-based approval access - New POST /jobs_u/flow/resume_suspended/{job_id} endpoint: - OptAuthed: works with login or approval_token - Checks approval_conditions (self_approval, groups, auth) - Admins/owners bypass rules - New GET /jobs_u/flow/approval_info/{job_id} endpoint: - Returns form, rules, can_approve status - HMAC anonymous endpoint now bypasses all approval_conditions (secret = full capability) - getResumeUrls approvalPage URL now uses token format - WAC approval dispatch generates and stores approval tokens - Mark resumeSuspendedFlowAsOwner as legacy Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: simplify frontend to use resume_suspended endpoint - OpenAPI spec updated with resume_suspended and approval_info endpoints - WorkflowTimeline: removed URL parsing, now calls single resumeSuspended endpoint for both approve and reject - Buttons show for any logged-in user viewing the job (backend enforces authorization rules) - Kept self-approval warning for admins Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: stateless approval tokens, new approval page, FlowStatusWaitingForEvents update - Replace DB-stored approval tokens with stateless HMAC derivation: token = HMAC(workspace_key, job_id + "approval_token") Verifiable without DB lookup, not reversible to resume secret - Drop approval_token migration (no DB table needed) - FlowStatusWaitingForEvents: use resumeSuspended endpoint instead of URL parsing + resumeSuspendedFlowAsOwner - New approval page route /approve/{ws}/{job}?token= that uses approval_info and resume_suspended endpoints - Old approval page route kept for back-compat Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: match old approval page content in new approval page - Add FlowMetadata, JobArgs, FlowGraphV2, DisplayResult - Add approvers with tooltips, flow arguments section - Add admin self-approval bypass warning - Add "Open run details" link - Fetch full job alongside approval_info for all UI data Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: filter _MODULES from args, show 'workflow' for WAC approvals Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove deno template from approval/prompt SuspendDrawer Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: approval page form display + hide deno from approval script picker - Fix form schema rendering on new approval page by wrapping flat WAC form schemas in { properties, order } for SchemaForm - Hide deno from the approval step language picker in flow editor Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove deno from canHaveApproval in script_helpers.ts The insert menu uses canHaveApproval() from script_helpers.ts via FlowInputsQuick, not the displayLang function in FlowInputs.svelte. Revert the unnecessary FlowInputs.svelte change. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: return form schema and description in approval_info for classic flows The approval_info endpoint was returning None for form_schema on classic flows. Now fetches raw_flow to get suspend.resume_form schema, hide_cancel, and the step's completed result for description. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: inline Login component on approval page instead of redirect Show the Login component directly on the approval page when authentication is required. On successful login, reloads user and approval info without navigating away. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: show resume buttons for all users, not just owners The resume_suspended endpoint handles authorization server-side, so the frontend should always show the buttons. Remove isOwner gate and the "cannot resume" message. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: prevent layout shift on resume by removing spinner from cancel button Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: prevent resume button expansion by using disabled instead of loading The loading prop adds a Loader2 spinner that expands the button width. Use disabled={loading} instead to prevent layout shift. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: approval page login redirects back with full page reload Set rd to the full URL (starts with http) so Login.redirectUser() uses window.location.href instead of goto(), triggering a full page reload after login. This ensures the approval page re-fetches data as an authenticated user. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fetch flow definition from flow_version when raw_flow is null Deployed flows don't store raw_flow on the job. Fall back to flow_version table using runnable_id to get suspend settings (form schema, hide_cancel) for the approval_info endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: show specific reasons when user cannot approve Display whether denial is due to self-approval being disabled, required group membership, or both. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: support both nested and flat form schema in waitForApproval Users can now pass either: waitForApproval({ form: { schema: { name: { type: "string" } } } }) or: waitForApproval({ form: { name: { type: "string" } } }) Both WorkflowTimeline and approval page handle both formats. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: convert sqlx query macros to non-macro for CI offline cache Replace sqlx::query! and sqlx::query_scalar! with sqlx::query and sqlx::query_as to avoid SQLX_OFFLINE cache misses in CI. Also remove unused LogIn import from approval page. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: suppress dead code warning + unused isOwner variable - Add #[allow(dead_code)] to without_flow method (CI -D warnings) - Rename isOwner to _isOwner in FlowStatusWaitingForEvents (unused) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: security and robustness fixes from PR review - Add workspace_id verification in resume_suspended to prevent cross-workspace approval (#3) - Fix token leakage: use relative path for login redirect instead of full URL with token (#4) - Handle getJob failure independently from approval_info so the page works for unauthenticated users (#7) - Clear error state on successful data load (#13) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address review feedback — shared token gen, rand resume_id, UX - Move generate_approval_token to windmill-common::variables (shared between windmill-api and windmill-worker, eliminates duplicate HMAC) - Use rand::random::<u32>() for resume_id instead of DefaultHasher - Stop polling after approve/reject on approval page - Add cancelLoading state to WorkflowTimeline Reject button Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-24 21:22:35 +00:00
Ruben FiszelandClaude Opus 4.6	5296adeddf	test: add powershell module detection and execution tests (#8373) ... Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-15 04:11:43 +00:00
Ruben Fiszel	5f0ef936d1	feat: add sandbox annotations, volume mounts, for AI sandbox starting with claude (#8058)	2026-03-05 06:19:51 +00:00
Ruben FiszelandClaude Opus 4.6	6bf544f507	refactor: extract object store into dedicated crate with filesystem backend (#7996) ... * refactor: extract object store code into windmill-object-store crate with filesystem backend Consolidate all object_store-dependent code from windmill-common into a new windmill-object-store crate. Add a filesystem-backed object store implementation using LocalFileSystem for dev/testing without cloud credentials. Includes 30 comprehensive tests covering render_endpoint, lfs_to_object_store_resource, duckdb_connection_settings, error mapping, and filesystem-backed integration tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * all * all * all * fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-18 12:24:45 +00:00
Ruben FiszelandClaude Opus 4.6	623407de6a	chore: make rquickjs, kube, otel proxy deps optional behind feature flags ... - rquickjs: gated behind `quickjs` feature in windmill-jseval, propagated through windmill-worker/windmill-api, added to oss_core - windmill-autoscaling: made optional in windmill-api (was unconditional), enabled via enterprise feature - opentelemetry-proto, prost, hudsucker, rcgen, hyper-http-proxy, hyper-tls, hyper-util: made optional in windmill-worker, enabled via enterprise feature This significantly reduces compilation time for vanilla `cargo check` without features. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-08 16:13:16 +00:00
Ruben Fiszel	bb57f8cd29	remove unecessary deps for vanilla cargo run	2026-02-08 14:08:36 +00:00
Ruben FiszelandClaude Opus 4.6	3f68e3a58e	refactor: isolate deno_core into windmill-runtime-nativets subcrate (#7848) ... * refactor: isolate deno_core into windmill-runtime-nativets subcrate Remove deno_core from flow eval and isolate nativets V8 runtime into a dedicated subcrate so deno_core compilation no longer blocks windmill-worker or windmill-api. - Create windmill-jseval crate: QuickJS-based JS eval for flow expressions and batch rerun, extracted from windmill-worker - Create windmill-runtime-nativets crate: all deno_core/V8 deps and nativets script execution, with build.rs snapshot generation - Simplify windmill-worker: remove all deno_* direct deps, empty build.rs, gate nativets behind optional dep - Update windmill-api: use windmill-jseval for batch rerun instead of deno_core, remove deno_core feature entirely - Add nativets integration tests (nativets_jobs.rs) and parallel stress test (nativets_stress.rs, 8 workers x 200 jobs) - Remove dead code: deno flow eval path, USE_QUICKJS env var, parity tests (replaced with 63 standalone expected-value tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR review feedback for deno_core isolation - Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source) - Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/ - Fix operator precedence in windmill-jseval with explicit parentheses - Remove unnecessary return keyword in heap limit callback - Remove redundant as usize casts - Remove ~150 lines of commented-out code from runtime.js - Remove commented-out #[cfg] in build.rs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * otel ee --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-08 09:16:37 +00:00
Ruben FiszelandClaude Opus 4.6	e1a815f6a0	refactor: extract windmill-dep-map crate for parallel api/worker compilation (#7846) ... * refactor: extract windmill-dep-map crate for parallel api/worker compilation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve WebhookShared type mismatch and missing enterprise propagation - Make windmill-api webhook_util re-export from windmill-common instead of duplicating types, fixing Extension<WebhookShared> mismatch between windmill-store and windmill-api - Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature so check_license_key_valid is available when trigger subcrates enable enterprise on windmill-trigger Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: stop trigger features from unconditionally enabling enterprise Move enterprise propagation for all trigger subcrates from individual trigger feature definitions to the enterprise feature itself, so enterprise is only enabled when explicitly requested. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: remove unused pub use re-exports and disable CI cargo cache - Remove unused re-exports from windmill-worker/src/lib.rs: trigger_dependents_to_recompute_dependencies, handle_job_error, and unused bun/otel items - Fix callers to use direct module paths instead - Add windmill-dep-map as dev-dependency for tests - Disable cargo cache in backend-check CI (faster from-scratch builds) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: restore bun re-exports used by tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * all * chore: re-enable cargo cache for check_ee_full CI job Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-08 00:39:56 +00:00
Alexander PetricandRuben Fiszel	afa6e7ab5b	feat: mssql integrated auth (gssapi) (#7760) ... * feat: mssql integrated auth (kerberos/ntlm) * install krb5 headers * also make it work for windows --------- Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-02-05 18:35:31 +00:00
Ruben FiszelandClaude Opus 4.5	5c20b37a53	feat: add QuickJS as alternative JS engine for flow expression evaluation (#7664) ... * feat: add QuickJS as alternative JS engine for flow expression evaluation Add rquickjs as an optional alternative to deno_core for evaluating JavaScript expressions in flow transformations. QuickJS offers ~8-16x faster startup times for simple expressions, making it ideal for evaluating many small expressions in flows. Key changes: - Add new `quickjs` feature flag for windmill-worker - Implement js_eval_quickjs.rs with true async Rust callbacks for variable(), resource(), and results.xxx access (no pre-fetching) - Share expression transformation logic (replace_with_await, replace_with_await_result) between both implementations - Add USE_QUICKJS_FOR_FLOW_EVAL env var to switch engines at runtime - When only quickjs feature is enabled (no deno_core), QuickJS is automatically used - Add comprehensive parity tests comparing QuickJS and deno_core output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * all * quickjs * quickjs * all * all * all --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-29 01:44:40 +00:00
3 people	c5963957b6	feat: add otlp/http internal collector (#7690) ... * feat: add otlp/http internal collector [merge-ee-first] Signed-off-by: pyranota <pyra@duck.com> * remove unused import Signed-off-by: pyranota <pyra@duck.com> * chore: update ee-repo-ref to b2e51eceaf00719b7b6794b516d42f20a1d96ff7 This commit updates the EE repository reference after PR #390 was merged in windmill-ee-private. Previous ee-repo-ref: b576da804a0dd3fadf1ad2fbcc0a0df12684d795 New ee-repo-ref: b2e51eceaf00719b7b6794b516d42f20a1d96ff7 Automated by sync-ee-ref workflow. * fix panic Signed-off-by: pyranota <pyra@duck.com> * finish Signed-off-by: pyranota <pyra@duck.com> * chore: update ee-repo-ref to c23a9fb439a71bf574139422b8a43770167ed233 This commit updates the EE repository reference after PR #392 was merged in windmill-ee-private. Previous ee-repo-ref: 657a61696699d7d9751dc8e30935372bb5af2db7 New ee-repo-ref: c23a9fb439a71bf574139422b8a43770167ed233 Automated by sync-ee-ref workflow. * defensive programming Signed-off-by: pyranota <pyra@duck.com> * ee repo ref Signed-off-by: pyranota <pyra@duck.com> * Apply suggestions from code review Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * fix silly claude Signed-off-by: pyranota <pyra@duck.com> * chore: update ee-repo-ref to e9f7a4485b2d21746c6a4a4c38fa1cbac2b942af This commit updates the EE repository reference after PR #393 was merged in windmill-ee-private. Previous ee-repo-ref: bc6c149c01d7063171488b7930f40ffce73a65d3 New ee-repo-ref: e9f7a4485b2d21746c6a4a4c38fa1cbac2b942af Automated by sync-ee-ref workflow. --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>	2026-01-26 19:29:09 +00:00
3 people	2553d987d9	feat(ai): native bedrock compatibility (#7668) ... * check endpoint * use flag for cred check on worker * use sdk for auth chat side * cleaning * cleaning * also handle GET * Add session token support in BedrockClient::from_credentials Users with temporary STS credentials need session tokens for AWS authentication. This adds the optional session_token parameter to support these use cases. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Cache AWS SDK config loading to avoid repeated I/O aws_config::load_defaults() performs environment variable lookups and potentially file I/O on each call. Cache the result in a static OnceCell to improve performance on high-frequency Bedrock requests. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add debouncing to Bedrock credential check button Prevent rapid clicks from spawning multiple concurrent flow preview jobs for the worker credential check. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Remove orphaned SigV4 signing comment The manual SigV4 signing code was removed in favor of the AWS SDK, but this documentation comment was left behind. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Remove verbose tracing info and change-related comments - Remove per-request auth method info logs (too verbose for production) - Simplify from_env log to single debug-level message - Remove comments describing future changes rather than current behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * cleaning * Fix Bedrock feature flag compilation warnings Gate AWS-specific imports, struct fields, and methods behind the bedrock feature flag to eliminate dead code warnings when building without the bedrock feature enabled. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * bedrock flag * cleaning * Move BedrockCredentialsCheck from edit drawer to add drawer The component was incorrectly shown when editing resources instead of when adding new ones. Moved it from ResourceEditor.svelte to ApiConnectForm.svelte. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * cleaning * cleaning * make aws-config optional --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>	2026-01-26 11:04:14 +00:00
Ruben Fiszel	0b6100c8b2	clean up cargo + update rmcp	2026-01-26 08:04:17 +00:00
Pyra	95df7b9a6a	feat: otel REST tracing (#7571)	2026-01-20 13:38:02 +00:00
Ruben Fiszel	d432cdbec1	split out windmill-mcp + pin tokio-utils	2026-01-09 06:16:26 +00:00
centdix	a8e34de58d	upgrade rmcp crate (#7518)	2026-01-07 16:02:51 +00:00
centdixandClaude	79ac6312e8	feat(ai): handle aws bedrock as provider (#7155) ... * backend draft * fix for tool and streaming * do frontend side * working * working tools * rm * handle list endpoint * handle for ai agents * fix for models requiring inference id * cleaning * fix desc issue * fix tool usage * fix structured output * cleaning * fix for api * rm * fix input images * cleaning * chore: use aws sdk (#7156) * feat(ai): Add AWS SDK dependencies for Bedrock integration - Add aws-sdk-bedrockruntime v1.113.0 - Add aws-credential-types for bearer token authentication - Update rustls to v0.23.35 for compatibility - Dependencies added to windmill-common for AI features * feat(ai): Add bearer token provider for Bedrock authentication - Implement BearerTokenProvider using aws_credential_types - Simple token-based auth using API keys from Windmill resources - Add basic unit tests for provider creation - Export bedrock_auth module in lib.rs * feat(ai): Add Bedrock client wrapper with region extraction - Implement BedrockClient wrapper around AWS SDK client - Bearer token authentication integration - Extract AWS region from Bedrock base URL automatically - Comprehensive unit tests for region extraction - Make aws-config non-optional dependency for AI features - Update feature flags to reflect new dependency structure * cargo * feat(ai): Implement non-streaming Bedrock via AWS SDK Use official AWS SDK instead of manual HTTP requests for better type safety and maintainability. Implements the Bedrock converse() API for non-streaming requests with proper bearer token authentication and message format conversion between OpenAI and Bedrock formats. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor(ai): Eliminate Simple* conversion types for Bedrock SDK - Move AI types to windmill-common/src/ai_types.rs for shared access - Update bedrock_converters to work directly with OpenAI types - Remove ~200 lines of conversion boilerplate from ai_executor.rs and bedrock.rs - Remove unused imports to clean compilation warnings - Benefits: 50% fewer conversion steps, no information loss, easier maintenance 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat(ai): Add streaming support for AWS Bedrock SDK - Implement converse_stream() for Bedrock streaming responses - Use EventReceiver.recv() to process stream events - Extract text deltas using bedrock_stream_event_to_text() - Send TokenDelta events to StreamEventProcessor for real-time updates - Refactor request building to eliminate duplication between streaming and non-streaming - Clean, minimal implementation following AWS SDK patterns 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * revert flake change * fix * feat(ai): Add tool calls and image support for Bedrock streaming **Phase 1: Streaming Tool Call Support** - Add stream event processing functions in bedrock_converters.rs: - bedrock_stream_event_to_tool_start() - Extract tool use start from ContentBlockStart - bedrock_stream_event_to_tool_delta() - Extract tool input deltas from ContentBlockDelta - bedrock_stream_event_is_block_stop() - Detect ContentBlockStop events - streaming_tool_calls_to_openai() - Convert accumulated tool calls to OpenAI format - Update ai_executor.rs streaming loop with tool call accumulator (HashMap) - Track current tool use ID during streaming - Send ToolCallArguments events to StreamEventProcessor - Return accumulated tool calls instead of empty vector **Phase 2: Image Input Support** - Add parse_image_data_url() to extract format and base64 data from data URLs - Add content_part_to_block() to convert ContentPart to Bedrock ContentBlock - Refactor convert_message() to handle multi-part content with images - Support ImageUrl conversion to Bedrock ImageBlock with proper format (png/jpeg/gif/webp) - Import AWS SDK image types: ImageBlock, ImageSource, ImageFormat - Keep content_to_text() helper for system message text extraction **Benefits**: - ✅ Tool calling now works in both streaming and non-streaming modes - ✅ Images are properly converted instead of being silently dropped - ✅ Structured output works in streaming (uses tool calling) - ✅ Full feature parity with manual HTTP implementation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * cleaning * fix(ai): Add S3 image support and structured output for Bedrock **Fixes:** 1. **S3 Image Support**: Call prepare_messages_for_api() before Bedrock SDK path to convert S3Objects to ImageUrls - Downloads images from S3 and encodes as base64 data URLs - Ensures images are properly handled in both streaming and non-streaming modes 2. **Structured Output**: Add ToolChoice::Any when structured output tool is present - Forces Bedrock to call the structured_output tool - Ensures JSON schema compliance for structured output - Works in both streaming and non-streaming modes **Changes:** - ai_executor.rs: Call prepare_messages_for_api() for Bedrock SDK path - ai_executor.rs: Set tool_choice to Any when structured_output_tool_name is present - aws_bedrock.rs: Remove unused ToolChoice imports (used via full path in worker) **Testing:** - ✅ S3 images are now downloaded and converted before API call - ✅ Structured output now forces tool usage with ToolChoice::Any - ✅ Both work in streaming and non-streaming modes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * cleaning * cleaning * cleaning * better error * cleaning * cleaning * rm * rename * apply region --------- Co-authored-by: Claude <noreply@anthropic.com> * fix default * no panic * no print * use utils file * cleaning --------- Co-authored-by: Claude <noreply@anthropic.com>	2025-11-17 18:57:59 +00:00
hugocasa	2e245accc5	fix(backend): improve sse parsing (#7100)	2025-11-10 14:32:16 +00:00
centdixandClaude	97ac1be036	feat(aiagent): allow mcp as tools (#6790) ... * draft mcp client * testing * fix * cleaning * mcp resource in inputtransforms * cleaning * big cleaning * cleaning * no arc * add utils file * refactor tools * add mcp actions * draft frontend * send arguments from backend * better frontend * cleaning * use token for auth * add logo * rm * fix * fix * chore: refactor mcp for ai agents (#6829) * Add Tool enum for AIAgent with backward compatibility - Created Tool enum that can be either Windmill (FlowModule) or Mcp (resource reference) - Created McpToolRef struct to hold MCP resource path - Implemented custom Deserialize for Tool with backward compatibility: - New format: {type: 'windmill'|'mcp', ...} - Old format: FlowModule objects (automatically wrapped in Tool::Windmill) - Updated AIAgent to use Vec<Tool> instead of Vec<FlowModule> - Updated FlowValue::traverse_leafs to handle Tool enum - Backward compatible: old flows with Vec<FlowModule> will deserialize correctly * Refactor AI executor to process Tool enum instead of extracting MCP from input_transforms - Separate Windmill tools and MCP resource paths from tools list - Process Windmill FlowModules into Tool definitions - Load MCP tools from resource paths in Tool::Mcp variants - Remove old logic that extracted mcp_resources from input_transforms - Import FlowModule, remove unused InputTransform - Fix type issues: use .as_str() for path and handle Option<bool> properly * handle in args * mcp as flowmodule * frontend * config for mcp * simplify logic * fix ai executor logic * cleaning * clean frontend * fix * better resource picker * fix and styling * add endpoint to fetch tools * apply tool filtering * fix name validation * better ui * use cache * fix * fix merge * refactor: Separate MCP tools from FlowModule in AIAgent - Add new AgentTool, ToolValue, and McpToolValue types - Update AIAgent to use Vec<AgentTool> instead of Vec<FlowModule> - Implement From traits for clean conversion between AgentTool and FlowModule - Add backward compatibility via custom deserializer for AgentTool - Simplify resolve_module logic by reusing existing resolve_modules function - Update traverse_leafs to handle AgentTool structure This refactoring separates MCP tools from FlowModule tools, making the type system clearer and eliminating the need to treat MCP servers as a special case of FlowModule. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Update ai_executor and worker_lockfiles for AgentTool - Update ai_executor.rs to handle new AgentTool structure - Separate MCP tools from FlowModule tools using ToolValue enum - Convert AgentTool to FlowModule for backward compatibility - Add imports for AgentTool and ToolValue types - Update worker_lockfiles.rs for lazy loading optimization - Convert AgentTool <-> FlowModule in insert_flow_modules - Preserve lazy loading for FlowModule tools via modules_node - Keep MCP tools inline (lightweight, no need for lazy loading) - Maintain backward compatibility with existing flows This enables the lazy loading optimization for FlowModule tools while keeping MCP tools inline, balancing performance and simplicity. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * cleaning * adapt frontend * cleaning * cleaning * type fix * cleaning * fix back comp * move mcp button position * nit * cleaning * fix nested removal * cleaning * opti * fix chat markdown display * fix chat messages layout * fix back comp frontend * fix deserializer * nit * simpler serializer * use if else --------- Co-authored-by: Claude <noreply@anthropic.com>	2025-10-21 15:28:52 +00:00
hugocasa	d365014882	build on macos with deno_core (#6642)	2025-09-19 17:33:05 +00:00
centdix	3199f9fffd	feat(ai agent): allow multiple images input for ai agent + code cleaning (#6591) ... * add in frontend * draft openai handling * upload to s3 * simpler output * return s3 directly if any * low quality * implement for gemini * handle imagen model * handle image input * cleaning * remove base64 from output * cleaning * fix timeout * handle openrouter * remove log * allow image input when creating image * cleaning * increase stack size * inline everything * revert stack size * refactor: move AI executor types to separate module - Created ai module structure with types.rs - Moved all type definitions from ai_executor.rs to ai/types.rs - No functional changes, just code organization * refactor: add QueryBuilder trait and provider detection utilities - Created QueryBuilder trait for abstracting provider-specific logic - Added helper functions for provider detection (is_anthropic_provider) - Implemented placeholder QueryBuilder for all providers - Updated OpenAIRequest to use slices instead of Vec references - All providers now have QueryBuilder implementations (using default for now) * feat: implement OpenAI query builder with image support foundation - Created proper OpenAI query builder implementation - Added image_handler module for S3 upload/download utilities - Separated text and image request building logic - Added prepare_messages_for_api to handle S3Object conversion - Foundation laid for supporting tools with image output * refactor(ai): complete AI executor refactoring with query builder pattern - Created modular structure under ai/ module - Moved all types to ai/types.rs - Created QueryBuilder trait for provider abstraction - Implemented OpenAI query builder with image+tools support - Added unified agent runner supporting both text and image outputs with tools - Refactored run_agent to delegate to new unified implementation - Added image handler utilities for S3 operations - Improved code organization and maintainability * cleaning * feat(ai): implement remaining provider query builders - Added Anthropic query builder with proper message conversion - Added Google AI query builder with Gemini API support - Added OpenRouter query builder delegating to OpenAI for compatibility - Added missing Anthropic and Gemini types to types.rs - Fixed type references and compilation errors - All providers now support the unified query builder interface * fixes * fixes * mime type + cleaning * image to images * handle mutlitple images * fix * remove agent_runner file * clean query builder logic * cleaning * cleaning * hide structured_output based on output type * fix * user images and not nested * better descriptions	2025-09-17 16:55:54 +00:00
centdix	20f48e6ded	feat(ai agent): handle images in ai agent (#6572) ... * add in frontend * draft openai handling * upload to s3 * simpler output * return s3 directly if any * low quality * implement for gemini * handle imagen model * handle image input * cleaning * remove base64 from output * cleaning * fix timeout * handle openrouter * remove log * allow image input when creating image * cleaning * increase stack size * inline everything * revert stack size * cleaning * fix for openai * better mime type * add descriptions	2025-09-15 07:51:09 +00:00
Diego Imbert	304f953710	Separate duckdb crate to fix c++ build/link issues (#6551) ... * call ffi * remove duckdb dep * rename windmill_duckdb_ffi_internal * static lib * ci * back to dylib, bug isn't fixed in static * feature flag and copy dynamic lib * fix dynlib in docker * load libwindmill_duckdb_ffi_internal at runtime on usage * lazy static deadlocks * Cache dynamic library handles * update auto s3 path insert from editor bar * Fix duckdb S3 freezing worker because of blocking task in tokio async context * build dll windows GH workflow * try fix windows build * revert build.rs * nit fixes CI * Dockerfile update (not tested yet * build dev sh for duckdb lib * mistake * attach windmill_duckdb_ffi_internal.so artefact * rhel9 * docker fixes * fix dockerfile * better err msg * forgot lib prefix .so * add column_order * fix column_order	2025-09-10 14:43:46 +00:00
dieriba	f10cac1c4b	feat: conditional retry workflow (#6461) ... * feat retry * fix * fix * fix and nits * fix query * fix test * nits and perf * update sqlx * once * remove * update sqlx	2025-09-03 20:24:38 +00:00
hugocasa	958e8af782	feat: ai agent steps (#6393) ... * feat: ai agent steps base * better backend and graph * feat: anthropic, log viewer * nit * fix(frontend): hide tool nodes from timeline * move ai agent actions from flow status to flow status module * nits and workspace/hub scripts support * tmp ref * fix merge * feat: display agent tools status in the graph * fix reactivity * fix flow status * nit	2025-08-20 22:40:57 +00:00
pyranotaandellipsis-dev[bot]	11dd4118ce	feat: add ruby support (#5939) ... * implement single line pin * make panic-safe * use pin even if multiple modules imported withing single statement * add repins and make imports respect pins * keep all pins * Allow multiple pins * add comments + handle stuff more safely * fix fully qualified imports * remove ignore * sort nested * apply unique to output requirements list * fix typo * remove mut * update sqlx * sort imports * sort imports * fix formatter and format * feat: add ruby support * fix * update sqlx * add ruby to flake.nix Signed-off-by: pyranota <pyra@duck.com> * upgrade par_install_language_dependencies helper rename fields + allow passing custom payload Signed-off-by: pyranota <pyra@duck.com> * change migrations Signed-off-by: pyranota <pyra@duck.com> * make full pipeline work! Signed-off-by: pyranota <pyra@duck.com> * do a bit fixing * stage Signed-off-by: pyranota <pyra@duck.com> * add ruby in dockerfiles Signed-off-by: pyranota <pyra@duck.com> * implement auto-require + 'windmill/inline' Signed-off-by: pyranota <pyra@duck.com> * create 'windmill/inline' only if does not exist Signed-off-by: pyranota <pyra@duck.com> * add nsjail Signed-off-by: pyranota <pyra@duck.com> * update global settings Signed-off-by: pyranota <pyra@duck.com> * add ruby icon * generate lockfile on deploy (but broken) Signed-off-by: pyranota <pyra@duck.com> * fix(frontend): display deployed script lockfile and other info Signed-off-by: pyranota <pyra@duck.com> * feat: add mini-wmill client Methods: - get_resource - get_variable Additionally add shortcuts on frontend Signed-off-by: pyranota <pyra@duck.com> * add nit warning Signed-off-by: pyranota <pyra@duck.com> * support private repos credentials Signed-off-by: pyranota <pyra@duck.com> * fix compilation error Signed-off-by: pyranota <pyra@duck.com> * found rustc bug + refactor universal installer Signed-off-by: pyranota <pyra@duck.com> * more refactor + fixes Signed-off-by: pyranota <pyra@duck.com> * bug fixing + polishing Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * change 'windmill/inline' Signed-off-by: pyranota <pyra@duck.com> * delegate env_clear to universal installer Signed-off-by: pyranota <pyra@duck.com> * update init script Signed-off-by: pyranota <pyra@duck.com> * add cli support Signed-off-by: pyranota <pyra@duck.com> * unhardcode frontend Signed-off-by: pyranota <pyra@duck.com> * fix cli Signed-off-by: pyranota <pyra@duck.com> * fix init script Signed-off-by: pyranota <pyra@duck.com> * make it work on ms windows Signed-off-by: pyranota <pyra@duck.com> * do not clean env in the end Signed-off-by: pyranota <pyra@duck.com> * fix windows (again) Signed-off-by: pyranota <pyra@duck.com> * include PROXY_ENVS (untested) Signed-off-by: pyranota <pyra@duck.com> * feat: self-signed certs + proxy support (tested) Signed-off-by: pyranota <pyra@duck.com> * flake: pin ruby to 3.4 * merge follow-ups Signed-off-by: pyranota <pyra@duck.com> * remove obsolete libc crate Signed-off-by: pyranota <pyra@duck.com> * cleanup parser Signed-off-by: pyranota <pyra@duck.com> * fix java ci Signed-off-by: pyranota <pyra@duck.com> * cleanup Signed-off-by: pyranota <pyra@duck.com> * fix nits Signed-off-by: pyranota <pyra@duck.com> * Update backend/windmill-worker/src/universal_pkg_installer.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Update backend/windmill-worker/src/ruby_executor.rs Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add nsjail for lock stage Signed-off-by: pyranota <pyra@duck.com> * fix merge Signed-off-by: pyranota <pyra@duck.com> * Update common.rs * regen windmill-api-client Signed-off-by: pyranota <pyra@duck.com> * fix tests Signed-off-by: pyranota <pyra@duck.com> * fix tests again Signed-off-by: pyranota <pyra@duck.com> * reference published ruby parser Signed-off-by: pyranota <pyra@duck.com> --------- Signed-off-by: pyranota <pyra@duck.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>	2025-08-15 13:03:32 +00:00
Ruben Fiszel	b83aca30d2	feat: use process groups to improve zombie job handling (#6157) ... * useProcessGroups * nit	2025-07-10 08:49:56 +00:00
Diego Imbert	0e316239dd	EE Refactor (#5844) ... * app compiles with every ee substituted * Replace all oss files content * Revert "Replace all oss files content" This reverts commit ea4017d59f. * delete all ee * hide all _ee files under private flag * hide every oss stuff when private flag set * pub use * * gitignore and substitute script * pub mod for ee needed for ee repo * small mistakes * remove oidc_oss impl * ee ref (temp) * ee ref * fix --all-features selecting private in OSS CI * ee repo ref * allow unused	2025-06-02 22:12:33 +02:00