mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-14 08:02:58 +00:00
datatable-external-instance-2
44
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
9f2685d788 |
perf: retire the legacy pre-partitioning audit table (#11240)
* perf: index audit logs by operation and limit each audit table on its own Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: key the audit operation index on id with a trailing timestamp Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: index the legacy audit table by operation too Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: drop the legacy audit index and per-table limit from this PR Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: build the audit operation index in a background migration Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: retire the legacy pre-partitioning audit table Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the legacy audit retirement's lock waits and keep the backfill guard Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: bound the audit index parent's lock wait and retry background steps Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 5370ae3a95a3dc10171f72da74286a654996eee6 This commit updates the EE repository reference after PR #818 was merged in windmill-ee-private. Previous ee-repo-ref: aa367619a970de231f5c10cad470d9d42169fff8 New ee-repo-ref: 5370ae3a95a3dc10171f72da74286a654996eee6 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
 hugocasaandClaude Opus 5
|
9d335de87a |
feat: add a workspace toggle that adds its admins and developers to new forks (#11215)
* feat: add a workspace toggle that adds its admins and developers to new forks Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: add members copied into a fork as manual members, not instance-group ones Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style: keep the fork members copy comment at the query and drop the raw spacer Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
 GuilhemandClaude Opus 5
|
a571117f3f |
feat: store mcp tool call, result and reasoning on flow conversation rows (#11176)
* feat: store mcp tool call, result and reasoning on flow conversation rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep the failure reason and web search citations on tool rows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: persist a structured answer as its own row and keep reasoning-only rows from closing a turn Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep a failed Windmill tool's error on its conversation row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: let a structured answer row claim its streamed thinking Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: scope a structured answer's claim to its own turn and document the row fields as stored Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: claim a textless assistant row only within the newest turn Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor: store the thinking that led to a tool call on the tool row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep tool calls in stream order and scope a structured answer to its turn Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: store the files a user message carried as object-storage references on its row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(chat-sdk): read a message's attachments and build their download URL Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(chat-sdk): carry a loaded user message's attachments in AI SDK metadata Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: store the model's call and what it got back on every tool row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: pass no extras in the orphaned-conversation test's message insert Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(chat-sdk): keep a stored JSON null tool result instead of the row text Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: label a tool call the turn finished without as not finished Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a structured answer's streamed call out of the did-not-finish label Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: write a turn's conversation rows in order and describe stored tool calls Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: await the image answer row like the agent loop's other rows Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(chat-sdk): place a row nothing streamed by its sequence, not at the end Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(chat-sdk): place only tool rows by sequence, keep a closing row last Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(flow chat): show a failed tool's error, and no Retry on a running or stopped turn A failed tool card showed the row label in place of the error the row now stores. The live tool result now reports failures, so a failed call briefly flagged a running turn as failed; a stopped turn, whose last row is the failed tool or the cancelled flow's failure, offered Retry too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
GuilhemandClaude Fable 5.1
|
4eab995cf7 |
feat: tell test flow conversations from deployed ones and rename a chat (#11179)
* feat: mark test flow conversations apart from deployed ones and allow renaming a chat Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep the conversation kind across refreshes and reject NUL titles Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: ignore conversation lists for a kind no longer selected Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: start a fresh conversation listing when the kind changes on a later page Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse sending into a conversation of the other kind Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse cross-kind conversation continuations on the server Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the conversation filter unavailable while an answer runs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
hugocasa
|
57a99f66a8 |
feat: rename saved agents from the agent editor and flag broken links (#11147)
* feat: list flows that link a saved agent and flag broken agent links * feat: rename saved agents from the agent editor and repoint the flow * fix: show an unreadable linked agent as not accessible, not missing * fix: address review nits on agent rename and missing-agent state * fix: open content search above modals and keep Escape for it * fix: register content search on the opener's overlay stack * docs: scope the global search z-index comment to the bases it clears * refactor: show linked agents' rename warning as for scripts and flows * fix: keep the failed-lookup rename warning to resources |
||
|
|
796b6e5297 |
feat: back AI sessions up to the workspace object storage (#11116)
* feat: back AI sessions up to the workspace object storage Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SG5qEPM6Fmf7VerXS5nnWp * fix: bind the backup key to the user and pack pushes within the server caps Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SG5qEPM6Fmf7VerXS5nnWp * fix: keep refused and unavailable marks, one mark per key, stream the flush Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SG5qEPM6Fmf7VerXS5nnWp * fix: settle only fully sent sessions, keep removals while backups are off, cap pull bodies Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SG5qEPM6Fmf7VerXS5nnWp * fix: bound removal marks while backups are off and stale the sync rows instead of dropping them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: retry a lost lock, cap nested push lists and oversized pieces, drop a stale copy of a chat that outgrew the backup Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: cap pieces per push, size requests in UTF-8, keep a move's removal for an off workspace, disclose the restore counter Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: file a move's removal only once the new copy landed, retire marks through the sync row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep a session marked while deletes are carried over, drop only gone sessions' marks Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: record what a refused flush already stored, stop early when every mark is retired Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: restore past another workspace's removal mark, file a move's removal before its row, bound the first pulled session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: re-key the backups on workspace key rotation, accept only base64 images, carry a delete on the sync row when its mark cannot be written Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: durable conditional re-key of session backups, re-push on a storage switch Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: fail a push the key rotated under, settle no session split across storages, narrow the re-key module Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep a delete filed during a push, bound the pull and re-key listings Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: bound the session listing, mark the store's own user on a write that lands after a user switch Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: list sessions through per-session index markers, hold a session's parts back after a failed one Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: record a rotation on every build, list a session only on the part that completes its push Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: leave an object larger than any push writes unread Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: read each object against its listed size, carry a dirty mark that cannot be written on the sync row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: note the storages the re-key walk completed on, reach another user's rows on a failed mark, read a head at its cap Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: read the replaced key under its row lock, carry a refused dirty bump on the sync row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: pull a session that outgrew one answer in pages, imported only whole Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: build a pull page from the smallest keys of the whole listing, stage each page as it lands Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: re-record a key rotated back to, admit earlier-page images, restage over a cut-short restore Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor: delete the backups on key rotation instead of re-keying them Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: end a pull page before an object that grew since the listing, prune what a cut-short restore staged Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: delete the backups before the key commits, skip a planted object whatever its listing says, lock a restore across tabs, prune stale artifact versions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep the backups under a prefix named by the key, delete the previous key's prefix after the commit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: name the backup prefix by a generation the rotation bumps, never write an older record over a newer one on restore Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: retire a removal only against the storage holding the backup, restart a paged pull whose listing moved Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: fingerprint a pull page before reading it, answer the backup generation apart from the storage identity Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: answer needs_head for a headless session push, prune restaged pieces by id Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: serialize a session's push and removal, open whole pushes with the head, prune only own restores Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: require a head on a whole push, prune before the record lands, restore only under Web Locks Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: incremental pushes ride on a listed session, removals wait for every storage holding a copy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: a whole push replaces the backup under a per-push token, a pull page is checked after its reads Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: fingerprint a pull page by entity tag and version too Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: a moved session's removal mark names the storages holding the old copy Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: restore a workspace family together, the newest copy of a moved session winning Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: name the marker by the session's move count, abort a family restore a listing failed in Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: list the family again before a restored record lands, require the pull fingerprint Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: list the whole family once per restored workspace, off members included Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: a push split over parts, incremental too, unlists the session until its last part Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: refuse a partial push part that names no push Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: keep a refused bump for a session with no row yet, probe an off workspace again Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: backfill row-carried bumps after a reload, ask an off workspace again on a timer Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: backfill a row for its bumps only when it carries some Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix: a backfilled mark that cannot be written counts from the page's counter Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: say an off workspace is asked again, in the mirror's comments Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * chore: update ee-repo-ref to 1c1dab33563c4907aff8b0da825fb66db60af82a This commit updates the EE repository reference after PR #796 was merged in windmill-ee-private. Previous ee-repo-ref: 289b477ca3fc993da06ec09b11c8f55d5e4e39c1 New ee-repo-ref: 1c1dab33563c4907aff8b0da825fb66db60af82a Automated by sync-ee-ref workflow. * fix: unlist a session while an incremental push changes more than one object Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 5
|
42f489685b |
feat: store resource type display names and label hub integrations (#11113)
* feat: label resource types and integrations with hub display names Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: load hub integration names in the app and flow pickers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: load hub resource type names where drawers title a type Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: store resource type display names and drop the hardcoded list Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: leave display_name out of the fork comparison Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: ignore over-long synced display names, move name loaders Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: share the hub integration list cache, backfill admins only Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a name over a nameless duplicate, retry failed hub reads Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
57a134e2de |
feat(ai-sessions): share session artifacts with the workspace by link (#11115)
* feat(ai-sessions): share session artifacts with the workspace by link Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pyjp67oR269QAx3b4yf4oH * chore: cache the shared artifact queries for offline sqlx Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pyjp67oR269QAx3b4yf4oH * fix: replace a literal NUL byte in the shared artifact body limit comment Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pyjp67oR269QAx3b4yf4oH * test: pin that a shared artifact is confined to its workspace's path Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pyjp67oR269QAx3b4yf4oH * fix: sanitize shared artifact markdown and validate the artifact id on every route The shared page renders another member's markdown, so ArtifactBody now runs the repo's rehype-raw + rehype-sanitize chain with the chat's link renderer on top; only the session viewer opts into the chat code block (mermaid, apply button). The link renderer keeps a link's text when its href is empty or unsafe, and the scheme check moves to a tested helper. The status route checks artifact_id like share does, so a NUL is a 400 rather than a 500. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pyjp67oR269QAx3b4yf4oH * fix(ai-sessions): say which way re-sharing moves an artifact link The popover offered "Update to v1" when a v2 link was open on a pinned v1, which reads as if v1 were newer. Each direction now has its own sentence and action: a newer version on screen updates the link, an older one shares that version instead, a rename updates the name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
80eba80d6e |
feat(git-sync): gate GitHub PRs on Windmill CI test results (WIN-2051) (#10096)
* docs: add design doc for automatic git-to-windmill pull sync
* docs: add migration plan and implementation phases to git-sync pull design
* feat(git-sync): add auto_pull settings schema and pull enqueue primitive
Adds AutoPullSettings/AutoPullMode/AutoPullStatus on GitRepositorySettings
(workspace_settings.git_sync JSONB), the GIT_SYNC_PULL_SCRIPT_PATH constant,
and should_pull/effective_poll_interval_s helpers with unit tests. Exports the
EE enqueue_git_pull_job primitive. Foundation for repo→Windmill auto-pull.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): poll repos and auto-pull new commits into the workspace
Phase 1 of automatic repo → Windmill sync. A monitor task (EE-licensed,
single-replica via advisory lock) git ls-remotes each auto-pull-enabled
repository ~every minute and enqueues a pull when the tracked branch moves,
reusing the {workspace_id}:git_sync concurrency key so pulls serialize with
in-flight push commits.
- windmill-store: background (no-authed) resolver get_git_repo_head_for_autopull
that resolves the repo resource (incl. $var: refs) and ls-remotes; GitHub-App
repos are skipped here and will sync via webhooks (phase 2).
- monitor.rs: poll/reconcile/persist with optimistic sha advance and failure
status; targeted jsonb update so concurrent settings edits aren't clobbered.
- edit_git_sync_repository: preserve server-owned auto_pull state on UI save.
- openapi: AutoPullSettings/AutoPullMode/AutoPullStatus + auto_pull field.
- frontend: per-repo "Automatically deploy changes from Git" toggle with last
sync status; demote the GitHub Actions link to an advanced CI option.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): wire webhook lifecycle + receiver; share reconcile logic
OSS side of phase 2 auto-pull webhooks:
- edit_git_sync_repository creates/removes the repo webhook on save (EE-gated,
best-effort → falls back to polling).
- monitor poller now delegates to the shared windmill_git_sync reconcile/persist
helpers (also used by the webhook receiver), removing duplicated logic.
- export the shared reconcile/persist/failure helpers; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref for phase 3 in-app PR creation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): show webhook vs polling status on the auto-pull toggle
When a repo has an active webhook (auto_pull.webhook_id set), the status line
reads "instant via webhook"; otherwise it reads the ~1-minute polling cadence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(git-sync): post PR diff check on dry-run completion (phase 4)
Worker completion hook in process_completed_job: when a DeploymentCallback job
carrying the __git_sync_pr_check marker finishes, parse the dry-run SyncResponse
and patch the GitHub check run with the diff summary (success/neutral/failure).
Export enqueue_git_pull_dry_run; bump EE ref.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(git-sync): bump EE ref (drop unused GHES webhook_secret)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* revert(git-sync): defer phase 4 PR diff checks (OSS side)
Remove the worker completion hook that posted the PR check run, drop the
enqueue_git_pull_dry_run re-export and the orphaned sqlx cache, bump EE ref.
Phases 1-3 (polling, webhooks, in-app PR creation) are unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert(git-sync): defer phase 4 PR diff checks (OSS side)"
This reverts commit
|
||
|
hugocasaandClaude Opus 5
|
d8d7332eb6 |
feat: add per-route CORS origin allowlist for HTTP triggers (#10833)
* feat: add per-route CORS origin allowlist for HTTP triggers Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: fail closed on cold router cache and invalid origin input Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: resolve CORS route from the decoded path like the request handler Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: add instance-wide default allowed origins for HTTP routes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: let non-superadmins read the default allowed origins setting Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: badge the advanced section when a route's origins are restricted Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: state inherited origins on the control and use one hint row Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: trim the origins tooltip and relabel the toggle when a default exists Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: keep the origins format hint visible until an entry is wrong Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: state the at-least-one requirement in the origins hint Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: import the origins validator in the trigger-http tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: make an empty allowlist deny rather than fall back to the default Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: address review nits on origin validation and the CORS editor * fix: derive the origins error from the stored list and tighten host validation * fix: parse real IPv6 hosts and refuse a newly emptied allowlist * refactor: make origin validation advisory except for null and non-ascii * feat: let an empty allowlist be saved as deny every origin * docs: document the empty allowlist as deny every origin * fix: bound allowlists, reject commas, and decide cors after the handler * chore: revert unrelated rustfmt churn in windmill-common tests * chore: revert unrelated rustfmt churn in windmill-common * chore: drop the route types the cors restructure replaced * fix: take the stricter cors decision from before and after the handler * fix: strip runnable cors headers when the routers are unavailable * docs: document the allowlist bounds in the openapi schema * fix: let an unavailable cors read defer to one that resolved * refactor: carry the resolved cors policy from the handler to the middleware * docs: describe why an unavailable read fails closed on the paths that reach it * fix: validate the default origins on the declarative settings path * test: keep the webhook doc comment with the test it describes * fix: warn on impossible schemes and ports, and validate the instance setting * feat: treat an empty allowlist as unset at both levels * perf: decode the cors path only when the fallback needs it * docs: document the empty allowlist as unset in the api schema * docs: describe an empty allowlist as unset in the frontend comments * docs: say what a null allowlist resolves to, not what it meant before the default existed * docs: state what the validator refuses and why methods stay broad * feat: exempt static asset routes from the origin allowlist * fix: hide the origin control for every static target, not just websites * fix: exempt only static websites, not single-file static assets * fix: warn on an unclosed ipv6 host in the origins advisory * fix: require assets present, not just the static website flag --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
e62bfdcd8c |
fix: give every table a primary key so the db can be logically replicated (#11036)
* fix: give every table a primary key so the db can be logically replicated Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WS9iNfYQiLJNuBnxysBzi * fix: tighten replicability guard and trim migration comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WS9iNfYQiLJNuBnxysBzi * fix: split deployment_metadata into its own primary-key migration Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WS9iNfYQiLJNuBnxysBzi * docs: correct the partial-index predicate note after the deployment_metadata split Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017WS9iNfYQiLJNuBnxysBzi --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 5
|
448fce93f7 |
fix: make the native trigger disable/enable toggle actually save (#11024)
* feat: let a native trigger be disabled without deleting it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: show and control the native trigger pause outside the flow editor Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: create a native trigger already paused instead of pausing it after Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the native trigger enabled comments for create-time init Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
0139467b01 |
feat: ingest dbt column lineage and real column schemas from the engine's parquet index (#10977)
* feat: column-level lineage for dbt from the engine's parquet index `manifest.json` carries no column-to-column edges, which is why decision 14 recorded column lineage as unavailable. The edges live in a different artifact: `dbt compile --static-analysis strict --write-index` writes `target/index/`, whose `dbt.column_lineage.parquet` holds them and whose `dbt.node_columns.parquet` holds every column of every node, typed and ordered rather than only the ones an author documented. Strict analysis rejects SQL the default accepts, so this is a separate compile with its own `--target-path`, opt-in per project via `column_lineage: true`, and best-effort throughout: a project it cannot analyze keeps exactly the graph it had, with the engine's own diagnostics in the job log. Storage mirrors `dbt_edge`: `dbt_column_edge` keyed by (path, version, job) with the same composite FK to `script` and the same sweeps. The typed column list lands in `dbt_node.column_schema`, beside `columns` rather than merged into it, so `columns` stays what the author declared. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: address review findings on the dbt column-lineage pass - The workspace fork copied every other dbt sidecar table and not this one, so a fork lost its column lineage silently and could not recover it: the cloned digest covers the column edges, so a dynamic run in the fork matched it and stored nothing. - The parquet was collected whole before the edge cap applied, which is exactly the input the cap exists for — a project whose `scan` lineage is quadratic in its widest model could take the worker process down. Decoded a row at a time with the bound enforced during the decode. - The pass swallowed every error from the runner, including the job poller's cancellation and deadline, so a run that blew its timeout inside an optional annotation could still publish a graph and report success. `run_captured` now carries the exit status in its value, so only a failed COMPILE is downgraded, and the pass may spend at most half the remaining wall clock so it cannot starve the build that follows it. - `scan` edges are stored but no longer served: they are most of a project's lineage, nothing renders them, and the graph endpoint is polled by the run page. They are also the first thing the storage cap gives up now, rather than evicting the direct edges the trace draws. - `column_schema` and the column edges take the same gate as the model's SQL. A column-level view is the shape of what the author wrote, one level finer than the `ref()` graph, which is ungated only because it draws relations the caller already sees. - `graph_digest` hashes the new section only when it has edges, so a project that never asked for the pass keeps the digest it has instead of re-snapshotting on every dynamic run until it is redeployed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * fix: the editor buffer's column lineage, and three bounds that were wrong Round-2 review found four defects, all of them introduced by the round-1 fixes. - The `script_visible` gate on the column edges was copied from the node query without its `script_hash IS NULL` arm. `= NULL` is never true, so every version-less row was filtered out and an editor buffer's parse rendered its typed columns and none of their lineage — the one place the feature is meant to be used. Pinned by an assertion in `dbt_pinned_graph.rs`, which is where this class of bug already had a home. - The phase budget was handed to the poller, whose expiry is an `Err` indistinguishable from a cancellation or the job's own deadline, so a slow but valid analysis aborted the build it exists to annotate. The runner gets the full deadline again — those two must still fail the job — and the budget is a race around the whole pass, where expiring is this budget and nothing else. - The decode cap counted parquet ROWS, so `scan` and out-of-graph rows could spend it before a single drawn edge was read. It now counts what is kept, takes direct kinds in a first pass, and is handed the graph's own nodes so the budget cannot go on rows that could never be stored. - Hashing the new digest section conditionally did not preserve old digests, because an absent `column_schema` still serialized as `null` inside the nodes. It is skipped when absent instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: split the lineage pass by error contract, and read it in one query Round 3's findings were all consequences of round 1 and 2's fixes, clustered in the same two files, so this reshapes those two seams rather than patching again. The worker pass was one function being three things at once — a subprocess runner with job-lifecycle error semantics, a bounded decoder, and a best-effort degrader — which is why each fix to one perturbed another. It is now `compile_index`, which owns the JOB's semantics (only a cancellation or the job's deadline can `Err`; a non-zero exit, the output ceiling and the phase budget are outcomes), and `read_index`, which owns the ARTIFACT's and knows nothing about the job. The budget wraps the compile alone, so a decode can no longer outlive the timeout that reported the build would get the rest. The output ceiling likewise becomes a value rather than a job error, for the caller that can carry on without the tail of a compile's stdout. The column edges were read by a fourth hand-written copy of the `live`/`chosen` CTEs and the version/editor-buffer join conditions, and copying them is what dropped the `script_hash IS NULL` arm and hid every buffer parse's lineage. Both kinds of edge now come from ONE statement over a `UNION ALL`'d edge source, so those conditions exist once. The union is at the source rather than a join because column lineage can name a node pair `dbt_edge` has no row for: a model reading `{{ this }}` gets edges from itself to itself, and `parent_map` has no self-loop. The cap on the column half now sits after the scope filter, the visibility check and the graph joins — the scope moved into SQL via the existing `ScopePathFilter` — so a row the caller may not read can no longer spend it and leave an allowed project's trace short. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ * refactor: serve dbt column lineage from its own endpoint The column edges rode on the folder-wide asset graph, which a run page polls, while the trace is drawn for one selected relation. That needed a cap, and a cap has to be applied after every filter that can drop a row. Keyed to the asset there is no cap: `assets/column_lineage` answers for one relation, and the caller's `scripts:read` scope and the project's visibility are decided once, for the script that owns it. Pinning to a run's snapshot or the editor's parse of its buffer costs the job-read gate, so that form is `jobs/dbt_column_lineage/{id}` — the same shape `jobs/dbt_graph/{id}` has. The worker's decode now bounds work and memory separately, and a compile stopped by the output ceiling reports as truncated rather than complete. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve the owning dbt version the way the graph does The unpinned arm picked the newest live version at the path without narrowing to dbt, so a path since redeployed in another language answered with no lineage while the graph beside it still drew that project's stale nodes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate pinned column lineage on reading the project, and answer the component Four things round 5 found, three of them in code this branch rewrote: - The pinned arm resolved the version from the job and stopped there, so a share-link viewer entitled to a run got the project's column names and edges while the graph beside it still redacted `raw_code` and `column_schema`. Resolving WHICH version answers is not deciding whether the caller may read it; the version-less editor buffer keeps its exemption, having no `script` row to ask. - The answer was the whole owning project's edges. The canvas lays out the connected component of the selected relation's columns, so the rest was unrenderable weight; a recursive walk over both directions returns exactly what is drawn, and the project key travels with it so a `unique_id` two projects share cannot walk from one graph into the other. - The decode had no exit but the 4M-row backstop once its buckets were full, spending wall clock the build below does not get. - An unreadable index was reported as a missing one, sending the reader to look at their engine rather than at the file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stitch the two column graphs, and walk the component in Rust Round 6's two findings, both regressions this branch introduced: - The decode returned `Continue` on the edge that FILLED the direct-edge budget, so a `scan`-only tail after it decoded to the 4M-row backstop with nowhere to put anything. The read now ends on that edge. - Seam 3 made the pipeline page choose between the dbt graph and the producer one. They share node ids — `// column total <- dbt://wh/analytics/orders.amount` mints the same `(dbt, path, column)` node dbt's own lineage does — so choosing ended a trace at the boundary in both directions. They are merged again, and a ducklake selection asks about the dbt relation its producers name so the chain continues past it. The dbt editor gets the same merge. Also: the component is walked in Rust rather than by a recursive CTE. A CTE has no index, so the recursive term rescanned the doubled edge set once per level — 1243ms against 59ms for the query alone on a 3000-model project, 11.7M rows in the plan. Same answers, same tests; end to end 1.48s to 0.73s there and 1.60s to 0.26s on a 1000-deep chain. The client stops re-asking for a component it already holds, which is most clicks within one project. The four doc sites that described a whole-project answer are rewritten around what it now is, rather than edited where they disagreed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: expand every dbt boundary a selection reaches, and only skip what was asked Round 7's findings, all in the frontend seam this branch added: - A ducklake selection seeded the dbt fetch from the FIRST boundary relation it found, so a table derived from two unconnected dbt relations expanded one and left the other a leaf — the same "stops at the boundary" symptom the round-6 fix removed, one hop further along. Every distinct boundary is fetched now and the components merged. - The component cache skipped a relation merely PRESENT in the graph in hand. A relation two projects describe has an owner row in each, and a component fetched for one carries it as an endpoint without the other's half, so that skipped the request that would have resolved the second owner. Only a relation actually asked about under this pin is skipped. - A comment still called the producer graph gated to ducklake selections after it was widened to dbt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: land dbt column lineage as storage and ingest only The API surface that draws a column trace moves to a follow-up PR, on `dbt-column-lineage-surface`. It kept generating findings — a client cache whose premise was wrong for a two-owner relation, then staleness and a lost retry from tightening it, and a seed walk that stopped at the first boundary — and the fix for the last of them is a transitive owner expansion, which has to re-apply the caller's gate to every newly discovered project. That is the same shape as the leak four reviewers caught in the pinned arm, and it wants its own review rather than being the fourth fix at the end of this one. What lands here stands on its own: the analysis pass, `dbt_column_edge`, `dbt_node.column_schema`, the engine gating and the error-contract split — plus the one user-visible half, the typed and ordered column list, which rides the asset graph the details pane already fetches and replaces a panel that could only show the columns an author had documented. Also fixes a real bug in the pass, found in review: it compiled without the build's `--full-refresh`. `is_incremental()` branches on that flag, so an incremental model reading `{{ this }}` compiles its self-join — and any `ref()` inside that branch — only when the flag is absent, and the pass was storing lineage for SQL a full-refresh run never executed. The flag now comes from one place shared with the build, and a run that overrides it gets its own graph rather than standing as the version's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: say why direct kinds get the budget without naming a view The bucketing comments explained the priority by what a trace draws, which is a forward reference now that the surface moved out. The reason stands on its own: `copy`/`mod` say the value travelled, `scan` says the column was read to produce the row and so reaches every output column of its model. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: round-9 findings on the descoped PR - The `full_refresh` helper was inserted between `selection_is_overridden` and its doc comment, so thirteen lines about `select`/`exclude` echoes documented the wrong function and the one they were written for had none. Moved below it. - The parse path ran the analysis compile and the parquet decode BEFORE the guard that returns when there is no warehouse identity, paying for both and dropping the result. Moved after it. - Three sites still described a `/column_lineage` endpoint this branch no longer has, and two user-facing strings promised a column trace it no longer renders: the panel's hint and the descriptor template now say what the flag actually buys, which is the typed column schema. - Dropped test scaffolding the removed suite left behind: a `raw_orders` node and `dbt_edge` whose only assertion re-tested pre-existing graph behaviour, and a second editor-buffer node nothing asserts on. Documented rather than fixed: an incremental model has two shapes, and which one the index holds depends on whether the target existed when the pass ran. `is_incremental()` is false with no target as well as under `--full-refresh`, and dbt has no mode that emits both — so a version's graph describes the compile that produced it, and only a re-ingesting run describes its own run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep lineage_kind in the edge key, and one answer for --full-refresh - Both unique indexes omitted `lineage_kind`, so a column that is projected AND used as a predicate for the same output column — an ordinary shape — had its `copy` and `scan` edges collapse under `ON CONFLICT DO NOTHING`, while the digest counted both. The kind is part of the fact, so it is part of the key. Edited in the migration rather than added as a second one: it has not landed. - `full_refresh` was shared between the build and the analysis pass without the `command != "test"` condition that sat at the build's call site, so the two disagreed for exactly the runs that build nothing. The condition moved inside the function, which is the point of sharing it, and the command is threaded to the pass. - The "what a trace draws" rewrite missed the copy in `dbt_manifest.rs`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: drop the unreachable full_refresh threading, test the uniqueness key `DBT_COMMANDS` is `["build", "retry", "show", "parse"]` and `default_command` returns `build` in every arm, so `command == "test"` cannot happen — the guard the last commit moved into `full_refresh` was already inert where it came from. Threading the command through five signatures to preserve it bought nothing, so it is gone; the build and the pass call one function of the descriptor and the invocation, which is what the sharing was for. The uniqueness-key fix now has a test: a column projected AND used as a predicate for the same output column stores both its `copy` and its `scan` row. Verified against the old key, where it returns 1 instead of 2. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: restore the dbt test --full-refresh guard I removed on a wrong premise The previous commit removed it after reading `DBT_COMMANDS` and concluding `"test"` was unreachable. That is only true of the command a CALLER can name: `run_dbt` is invoked with `"test"` directly for the `after_all` test phase, so an `after_all` project with `full_refresh: true` reached it — and dbt rejects `--full-refresh` on `test`, failing the phase. Both reviewers caught it. The guard is back inside the shared function, where the build and the pass get one answer, and its doc now records why reading the allowlist alone is misleading. The test covering the `test` case is restored with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: notice a job that ended during the decode, and name truncation as the cause - The parquet decode runs on a blocking thread with no poller watching it, so a cancellation or an expired deadline during it was invisible: `dbt_dep` went on to publish the graph and the job returned success. The job's state is checked once the decode returns, before the caller publishes anything, and an ended job `Err`s — which this module may always do for the job's own semantics. - A compile stopped by the output ceiling could leave no artifact, and the log then blamed the engine's capability, sending the reader to check their adapter rather than the ceiling. Truncation now names itself in the missing and unreadable branches too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read cancellation from the DB after the decode, not from a poller's field `ctx.canceled_by` is only ever written by a poller, and no poller runs during the blocking decode — which is the exact window the check was added for. So the guard caught only a cancellation already observed before it, and the comment beside it claimed more than it did. It now queries `v2_job_queue` directly, the same probe `worker_lockfiles` uses before it overwrites a flow. A failed probe answers "still running": this decides whether to discard work already done, so an unreachable database must not be the reason a healthy deploy loses its graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: reuse job_is_canceled rather than a second copy of it The probe added last round was `job_is_canceled` from the same file, retyped — same query, same `Connection::Http` behaviour. Reused instead. Its doc said a non-database connection was "a failed probe", which reads as an error path. It is not: it is the agent worker, and on one there is no database to ask, so only the deadline answers and a cancel issued during the decode is not observable. The retry path avoids that by refusing to run on an agent worker at all — which an optional annotation has no business doing — so the gap is recorded at both ends instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: close the agent-worker cancellation gap instead of documenting it The previous commit said a cancel issued during the decode is not observable on an agent worker. It is: `ping_job_status` returns `canceled_by` over both connection kinds, and is how the poller itself notices one there. So the check asks through the ping rather than querying `v2_job_queue` directly, and holds on an agent worker, where a direct query reaches no database at all. `job_is_canceled` goes back to private and its doc to what it said before — the retry that calls it still refuses to run on an agent worker for its own reasons. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: decode the index under the job poller instead of checking after it Two findings with one cause: the decode was the only phase of this pass with no subprocess behind it, so nothing heartbeated while it ran. A large index left the worker silent for as long as it took, which the zombie sweep reads as a dead job and restarts — and the cancellation check bolted on afterwards could only ever report what had already happened, while dropping the ping's `already_completed`, so a force-cancelled deploy still published its graph. Running it under `run_future_with_polling_update_job_poller` answers all of it: the poller pings throughout, and ends the phase with an `Err` on cancellation, `AlreadyCompleted` or the phase timeout. The bespoke probe is gone with it. Verified on a live deploy: 32 edges and 4 typed schemas ingested through the polled decode. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop a cancelled decode, and say what the read phase can now do Putting the decode under the poller heartbeats it and ends the phase when the job does, but dropping a `JoinHandle` detaches a blocking task rather than cancelling it — so a cancelled job left a thread decoding up to four million rows for a job that was over. The row loop reads an abandonment flag that a drop guard on the awaiting future sets, so the decode stops at its next row. That same change made the read phase able to `Err`, and three places still said it could not — decision 14 in as many words. The distinction that holds is narrower: nothing the ARTIFACT does or fails to do can fail a job, so absent, unreadable and partial are all values; the JOB can still end the phase the read runs in. Stated that way in the module doc, the `Artifact` doc, `MAX_INDEX_ROWS` and the decision. Verified on a live deploy: 32 edges and 4 typed schemas. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: share AbortOnDrop, and stop citing a hazard that is now handled `Abandon` was `ansible_executor`'s `AbortOnDrop` retyped — same struct, same reason, same `spawn_blocking` shape. Moved to `common` and used from both. The paragraph explaining why the phase budget wraps the compile alone gave as its reason "a decode still running on a blocking thread", which is exactly what the abandonment flag now prevents. The reason that survives is the one that was always the point: the budget exists to leave the build its share of the clock, and only the compile can spend that share unboundedly. The decode's end is the job's, through the poller it runs under. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: put both doc comments back on the items they describe Moving AbortOnDrop orphaned a doc at each end: it landed between `raw_to_string`'s doc and `raw_to_string`, and the doc of the struct it replaced stayed behind to prefix `fetch_repo_archive`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the binding the row loop actually reads `Abandoned` was neither the type nor the binding; the flag is `abandoned`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
621fac55ab |
feat: durable dbt state per environment, and --defer onto it (#10975)
* feat: durable dbt state per environment, and `--defer` onto it `dbt retry` worked off two artifacts and only one was durable: `dbt_run_state` holds `run_results.json` keyed by principal, and the manifest lived on worker-local disk under a four-generation cache. That is enough to resume the last run and nothing else — the next run of a project usually lands on a worker holding neither artifact — so deferral had nothing to read. Adds `dbt_environment_state`: one row per (workspace, script path, environment), holding `manifest.json` and `run_results.json` from the last successful run, with the blob inline under `DBT_STATE_INLINE_MAX_BYTES` and in the workspace's object storage above it. Environment is the warehouse, the target, and the database and schema they resolve to, so a repointed warehouse or a moved schema reads as an environment nothing has published rather than as state whose relation names no longer fit. A run publishes it when its graph becomes what the script owns and it succeeded — the same condition, and the same reason: an invocation that scoped its own model set describes where the caller put those relations, not where the project's models live. `defer` is a `build` command-block field defaulting to the descriptor's own, and the state is materialised into the job directory for `--defer --state`. The retry path already did that materialisation for `dbt retry`; both go through one `write_state_dir` now. `--state` is also where `dbt retry` reads the run it resumes, so a retry on dbt-core 1.x takes `--defer-state` instead, and one on an engine without that flag is refused before the build rather than rebuilding its nodes with every unbuilt `ref()` resolving into the schema this run writes into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ned2pmRJwB3GpenEcrA9TF * fix: address the local review of the dbt environment state The oversized-artifact home moves from the workspace's object storage to the instance's, where every other internal worker artifact already lives. The workspace bucket is the one members read and write through `job_helpers/*` with a caller-supplied key and only `volumes/` is reserved there, so a manifest under it is one any member could replace — and the next deferring run would hand dbt an attacker-chosen `defer_relation` for every unbuilt `ref()` while holding the script's warehouse credentials. The environment key takes the target dbt actually runs rather than the descriptor's `profile.target`, which is absent whenever the target is inherited from the workspace warehouse or the project's own `profiles.yml` — filing every inherited target under one empty name, while a `target.name` macro decides where a model is built. `write_profiles` returns a named struct now that it resolves one more thing. Publishing takes the row's lock before uploading, so two publishers of one environment cannot interleave their uploads and leave one run's manifest beside another's results, and carries the live-dbt-script guard the retry state already had, so a job finishing after its script was renamed, archived or deleted cannot recreate state at a path for whatever is created there next. A rename now clears the environment state instead of moving it: an oversized artifact's key is derived from the path, so a moved row would keep pointing at a key a script created at the old path publishes over. A build recovered by the automatic in-job node retry publishes its manifest without results — `run_results.json` is then the retry's, naming only the nodes it redid — and the refusal for an environment with nothing published names the runs that cannot publish rather than suggesting a run that would not help. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: serialize dbt state publishers on an advisory lock The row lock only serializes publishers once a row exists, and the first publish of an environment — two runs of a newly deployed script — is exactly when two of them are most likely to race and interleave their uploads. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: make dbt state publication atomic and bind it to the version that ran Every publication now writes its own object keys and the row switches to them in one statement, so an upload never overwrites an artifact the committed row still names: a run failing between its two uploads, or between them and its row, leaves the state pointing at the pair it already had. The objects a commit displaces are dropped afterwards — never before, since a reader that has already read the row is about to fetch them — and a reader that loses that race re-reads the row once rather than reporting a state that is there. What a publication uploaded and then could not commit is dropped on the way out. The write's guard names the VERSION rather than the path: the live dbt script there must be the one this job ran, or a later version of it. "Some live dbt script is here" is also satisfied by a script created at a path this one was renamed away from, and this job's manifest would then become that project's deferral state. A preview names no version and so publishes nothing. A `show` defers too. It compiles the model it previews, so a model whose upstream this environment built and this run did not is exactly the case a deferral exists for, and every engine takes the flags on it. Three comments said "the workspace's object storage" where the code deliberately uses the instance's, which is the whole security argument; `mib()` labelled MiB values MB. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hold the script row across a dbt state publication, and let a rename move it The version guard read `script` without a lock, so lifecycle cleanup could find no environment row to clear, finish, and leave this transaction to commit state at a path a new script goes on to occupy. It now holds that row (`FOR SHARE`) for the rest of the publication — taken before the sidecar, the order every other dbt writer takes — and the artifacts are uploaded before the transaction, so the lock covers the row work rather than a network round trip. A commit that reports an error may still have committed: what was lost can be the acknowledgement. Dropping this run's objects then leaves the committed row naming objects that are gone, so an orphan is the cheaper side to take. A failed second upload left the manifest it had already written behind; it is dropped now. Per-publication keys retired the reason a rename cleared the environment state rather than moving it: the path is only a prefix, and the row is what names an artifact, so a script created at the old path can no longer publish over a moved row. The rename moves both halves again. `dbt ls` gets the deferral flags too, without which a `result:` selector — which reads `run_results.json` out of the state directory, and which `select` passes to dbt verbatim — fails before the build that would have honoured it. Also: the migration was the last site describing the workspace's object storage rather than the instance's, `publication_lock` folded 32 bits where it claimed 64, and `ResolvedProfile` had taken `write_profiles`'s doc block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: a deferring dbt run never publishes the state it read `publishes_ownership` reads the CALLER's overrides, so a descriptor that already narrows `select` needs none and a run of it with `defer: true` published. A deferring run built some of the relations its manifest names and resolved the rest out of the state it read, so recording that manifest claims relations nothing built — and a model renamed since is recorded under a name only a full build creates, breaking every later deferral until one repairs it. Also: `publication_lock` parsed 16 hex digits as `i64`, which overflows for every digest with the top bit set — half of them — collapsing those environments onto one advisory key; and a failure to open the transaction returned without dropping the objects already uploaded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only a deployed dbt run publishes state, and key its objects per execution A preview carries a caller-supplied `script_hash` into `runnable_id` (`run_preview_script`), so the version guard alone let anyone who may run a job publish arbitrary content as a deployed script's deferral state. The job's KIND is checked beside it now. Verified: a preview submitted with the deployed path and hash builds and leaves the row untouched. Object keys carry a per-execution nonce. Zombie recovery re-runs a job under its own id, so keyed on that alone a second attempt overwrote the objects the first attempt's committed row still named, then read those same keys back as displaced and dropped them — leaving the row unreadable. The displaced set is also filtered against this publication's own keys, so the invariant is stated rather than re-derived from the key format. A project-owned `profiles.yml` that templates its schema or database is refused a deferral: dbt renders those and Windmill does not, so two renderings resolve to one `relation_root` and would share one environment key. Plainly absent is left alone — that is the adapter's default, which does not move. The deferral log line now says the run publishes no state of its own, which was otherwise invisible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: a templated profile location publishes no dbt state either, on every path A `dbt_profile` resource is one block of the user's own `profiles.yml` copied through unchanged, and `profile.schema` is written as given, so either can carry a template dbt renders and this runtime does not — exactly as a project-owned file can. Only the project-owned path detected it. And the refusal now covers publication as well as deferral: a published template would sit under a key a literal profile shares, so de-templating later would make that stale manifest readable as the new location's. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: recognise Jinja statement blocks as a rendered dbt profile location dbt renders a profile through Jinja, so `{% if env_var('ENV') == 'prod' %}…{% endif %}` moves a schema exactly as an `env_var()` substitution does — and only `{{` was detected, so such a profile published and deferred under one environment key for every rendering. One predicate now serves both profile paths, with a test for each delimiter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: a dbt state read outruns successive publications rather than one The loader re-read once, which answers a single publication overtaking it: a reader takes no lock and the advisory lock is released before the displaced objects are dropped, so back-to-back publications could each overtake the same read and the second was reported as a missing object. It now re-reads for as long as the row keeps MOVING, bounded, and reports only when an unmoved row's objects are genuinely gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: a dbt state read outruns successive publications, not one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name both ways a dbt state read can fail Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: length-prefix the dbt environment key's components A dbt target name and a schema are both the user's own strings, so joining them on `|` let one component spell another tuple's key: `prod|analytics` + `scratch` and `prod` + `analytics|scratch` were one environment, and a profile moving between them read as the same one rather than as one nothing has published — the collision the key exists to prevent. The schema and database are also taken apart now rather than through `relation_root`'s own join, so neither can absorb the other's delimiter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: name the dbt environment in words where a message shows it The key is length-prefixed for storage, which is not something to put in front of a caller: the "nothing published yet" refusal now reads "warehouse `main`, target `prod`, relations in `dbt_wh_defer.analytics`". The worked example of the encoding also miscounted a component. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: delete a script version in the transaction that cleans up after it `delete_script_by_hash` soft-deleted through the pool, committing before the cleanup that follows it in `tx`. In that window the path has no live version, so a concurrent deploy can take it — and `clear_dbt_script_state_if_path_retired` then finds that new script live, keeps the deleted project's dbt state, and leaves the replacement able to defer through its manifest. The update moves into the same transaction, which is what `archive_script_by_hash` beside it already does. The retirement guard itself was pinned by nothing: the existing test moved the only row away before calling the conditional clear, so it could not fail. `state_goes_only_once_no_live_version_is_left` covers both directions — a second live version keeps the state, the last one leaving takes it — and fails if the predicate is inverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: archive a script by path in the transaction that cleans up after it The last of the four routes still writing outside its own cleanup transaction. Archived on its own, a cleanup that then fails leaves dbt state at a path no live version occupies, and whatever is created there next can defer through it. The by-hash archive and both deletes already take their write in `tx`; this makes the set uniform. Two comments beside those clears still called the state the RETRY state alone, which the rename made false — they cover both halves now — and the merged verification list had two `11.`, main's #10978 having inserted an item above it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: refuse a dbt state selector the engines resolve inconsistently `state:`, `result:` and `source_status:` selectors resolve against the artifacts in `--state`, which only a deferring run is handed. The engines disagree about what happens without one, and two of the three disagree silently: dbt-core 1.x raises, but dbt-sa-cli 2.x and fusion read a missing state as an empty one and exit 0, so `state:modified` builds nothing and `state:new` builds the whole project, each reporting success. Refuse them up front instead, naming `defer`. From the descriptor they are refused outright, since that selection also decides which nodes the script owns and the deploy resolves it with no state at all. `source_status:` is refused under any setting: it compares `sources.json`, which no run publishes here. A caller's selection is now allowed to match nothing, which is what `state:modified+` returns when nothing changed since the published state. It is stored as that run's own snapshot and never becomes what the script owns, so the ownership-wipe the refusal guarded against cannot happen. The descriptor's selection still may not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: refuse a dbt result selector the published state cannot answer Round 18 findings. Codex P1: `defer` alone was enough to allow a `result:` selector, but a build recovered by node retry publishes a manifest with no `run_results.json` — the only file such a selector reads. dbt-core then raises an internal error and the Rust engines match nothing and exit 0. The deferral now reports whether the state carries results, and a `result:` selection against one that does not is refused, naming the run that published it. Claude P2: a `parse` returns before `defer` is read, so its deferral is always absent and "turn `defer` on" was advice that led nowhere. The check now distinguishes a run that could defer from a command that never does, and the parse path says so. Codex P2 / Claude P2: the roadmap still listed `state:modified` as out of scope while the same file documented it as working. Narrowed both that line and the scope list to the slim-CI work that genuinely remains. Also pins the invariant the relaxed empty-selection guard rests on: an overridden selection must not publish ownership, or an empty caller selection would wipe the script's graph. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exempt an empty dbt selection by method, not by who chose it Round 19 findings. Codex P1: the empty-selection exemption keyed on whether the caller overrode the selection, so a misspelled model name resolved to nothing, passed the guard and reported a build that did its work. Key it on the selector instead: only a `state:` or `result:` method may match nothing, its empty answer being a real one. Every other selection matching nothing is refused again, from a run as from the descriptor, each with the message that applies to it. Claude P2: the spec still described a node-retry-recovered publication as one where `result:` selectors merely lose their input, which the previous commit stopped being true, and the section stating the selector rules recorded neither the `result:`-without-results refusal nor the `parse` one. Both written down. Also drops the refusal's claim that the publishing run WAS recovered by node retry: an unreadable file reaches the same absent-results state, and the remedy is the same either way. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: record why an exempted empty dbt selection cannot wipe the graph The safety argument left with the origin-based condition it justified. Under the method-based one it is a consequence of the descriptor refusal in check_state_selectors, two hops from this site, so state it here: relaxing that refusal would let a descriptor-narrowed `state:modified+` reach the exemption and be ingested as owning nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
8aab5034a6 |
feat: guest JWT entry for embedded apps (#10954)
* feat: guest JWT entry for embedded apps (jwt_guest_) A second way in for a guest, alongside the signed-in guest session: a JWT the embedding customer's backend mints and signs, verified per request against a per-workspace key (a PEM public key or a JWKS URL), resolving to the same seatless guest identity confined to the one app its app_path claim names. Bearer prefix jwt_guest_, stateless (no token row). See PR #10954. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: surface guest JWT as the embed method in the app deploy drawer The deploy drawer explained the secret-URL embed but not the guest JWT path, so the primary way to embed an app for a customer's own authenticated users was undiscoverable. For a guest-mode app with guests enabled, show how to mint a `jwt_guest_` token and append `guest.<jwt>` to the app URL, with a copyable iframe template pre-filled with this app's workspace_id and app_path, and a note that new guest emails are refused past the instance's free allowance (the live count is shown just above). Also log a guest JWT allowance refusal at warn, not info: the caller gets a bare 401 (the reason must not leak to an unauthenticated caller), so the log is the admin's signal that the instance hit its guest cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: correct the guest JWT minting instructions in the embed block The block said "sign it with the workspace's guest JWT key", but that setting holds the public verification key. Clarify the keypair relationship (configure the public key or a JWKS URL in the workspace; sign with the matching private key), name the accepted algorithms (RS/PS/ES; HS* refused), and keep the required claims, so an embedder knows how to actually mint the token. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: fall back to the instance JWT issuer for guest verification (off on cloud) A workspace with no guest key of its own now verifies guest JWTs against the instance issuer (JWT_EXT_JWKS_URL, already used by jwt_ext_), so an operator running one issuer configures it once. Verification and the guest grant are CE; granting a full login from that issuer stays EE (jwt_ext_, unchanged). Disabled under CLOUD_HOSTED, where one instance issuer must not be trusted to mint guests in every tenant's workspace — there the per-workspace key is the only source, which also stays the override everywhere. The workspace settings note (hidden on cloud) explains the fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: embed instructions cover both the workspace key and instance issuer The embed block said to set the workspace's guest JWT key; now it says Windmill verifies against the workspace key or, off cloud, the instance issuer (JWT_EXT_JWKS_URL) when no workspace key is set. The instance clause is hidden under isCloudHosted(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: show the guest JWT embed block only when Embed is toggled It belongs with the iframe snippet, not the plain-URL view, so gate it on embedMode alongside the guest-mode / guests-enabled checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: trust the instance issuer in the guest fallback; refresh stale docs P1 (CI review): the fallback wrapped JWT_EXT_JWKS_URL as a workspace JwksUrl, so it hit validate_guest_jwks_url and was refused for http/private issuers unless ALLOW_PRIVATE_GUEST_JWKS_URLS was also set — a self-hosted internal issuer that works for jwt_ext_ failed for guests, though the UI says setting the env var is enough. fetch_jwks now fetches the instance issuer without the https/private restriction (matching the jwt_ext_ loader; it stays operator-trusted), while a workspace-admin URL is validated and pinned as before. All the size/key/URL bounds still apply to both. P2 (CI review): refresh the stale docs that said a missing workspace key always refuses a guest JWT — the module, bearer, key-source, and EditGuestJwtKey field docs now describe the workspace key with the off-cloud instance-issuer fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: fetch the trusted instance issuer like the jwt_ext_ loader P1 (CI review): the instance-issuer fetch skipped SSRF validation but still disabled redirects and default cert validation, so an instance issuer that works for jwt_ext_ through a redirect or an operator-approved self-signed cert failed the guest fallback. Fetch it with HTTP_CLIENT_PERMISSIVE (follows redirects, honors ACCEPT_INVALID_CERTS) — the same behavior jwt_ext_ has — while a workspace-admin URL stays validated, DNS-pinned and redirect-free. The body size cap still bounds both. P2 (CI review): the WorkspaceSettings field doc still said None/None means no JWT guests; it now names the off-cloud instance-issuer fallback. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: schema summary + OpenAPI cover the guest JWT columns and fallback P2 (CI review): summarized_schema.txt was missing guest_activity.jwt_entry and the two workspace_settings guest-JWT key columns (required by docs/validation.md after a schema change). The edit_guest_jwt_key OpenAPI description now notes that clearing the workspace key falls back to the instance issuer (JWT_EXT_JWKS_URL) off cloud rather than necessarily stopping guest JWTs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: keep JWKS single-flight locks in a self-cleaning map, not a bounded cache P1 (CI review): JWKS_FETCH_LOCKS was a 200-entry quick_cache. Past 200 cold URLs it can evict a lock whose fetch is still in flight; the next request for that URL then mints a fresh lock and starts a second fetch, so cycling configured workspaces defeats single-flight and can storm the issuers. Replace it with a plain map guarded by a JwksFetchLock RAII handle that removes each entry once its last holder drops, so the map only ever holds the fetches in flight and never evicts an in-flight lock. Add a unit test pinning the shared-lock and self-cleaning invariants. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: update ee-repo-ref to c2270eb5fe2d9f0968253e6b460c33186363f4e7 This commit updates the EE repository reference after PR #773 was merged in windmill-ee-private. Previous ee-repo-ref: 5a1d9dee34159512c0823fddcd3d096490edbcce New ee-repo-ref: c2270eb5fe2d9f0968253e6b460c33186363f4e7 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
fce635d3c4 |
feat: guest app execution mode, a role that takes no seat (#10929)
* feat: guest app execution mode, a fourth role that takes no seat Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: make the guest grant a server-minted label, not a declarable scope Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * chore: pin ee-repo-ref to the guest session companion branch Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: close the relabel hole, guest embed tokens, read-path switch, custom-path entry Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest tokens are not rescopable and guest embed tokens keep the sentinel Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest-derived tokens share one constraint set; gate sign-in on guest discovery Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: the label alone governs a guest; refuse guests with accounts; unserialize discovery Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest discovery fails closed; SAML aborts if the guest cookie write fails Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * refactor: enforce the guest switch once at the auth door; sign-in for a guest of another app Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest app-mode decided once at the on-behalf resolver; clear a stale guest session before offering another app's sign-in Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: a guest may use anonymous apps; await the stale-session logout; trim comments Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: a guest's path confinement waits for the app's mode, so anonymous apps stay open to it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest target survives http (Lax cookie), rides SAML RelayState; tell account holders on arrival * fix: a guest uses an anonymous app as itself; S3 uploads confined by app mode * fix: a guest upload needs an app policy; a missing app does not skip the confinement * fix: guests are gated on the Enterprise plan server-side; pin ee-repo-ref * fix: the guest plan gate fails closed on non-enterprise builds; settings report the effective switch * fix: guest controls read the plan, not the key; gate the guest tests on the features they need * docs: tighten the guest session invariant comments * feat: 100 free guests per 30 days, then a quarter seat each on Enterprise and a hard cap elsewhere; superadmin guest list; refusals reach the page Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: the cap is exact, an account ends a guest session at the door, popups close, and guest mode survives the CLI round trip Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * feat: a superadmin switch over guests for the whole instance; the pre-existing-user flag keeps its meaning Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: drop the dead guest-access helper, name the instance setting once, guests tab states, CE save order Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: a guest app path is refused at the mint if it could widen the scope; the instance toggle waits for its reload Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guests stop at the launched-by-me job grant; canonical app paths at the mint and discovery; the toggle ends on the stored value Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: only the scope grammar's own characters bar an app path from guests, refused at deploy as well as at the mint Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: the deploy-time guest path guard checks the destination of a rename and refuses a leading slash Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: a workspace rename keeps the guest switch; the rename guard reads the deployed mode under the row lock Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * fix: guest_activity follows a workspace rename and goes with a workspace delete Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * chore: pin ee-repo-ref to the state-bound guest target Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * chore: pin ee-repo-ref; the guest cookie is never cleared by a callback Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * docs: the workspace-scoped guest_activity delete moves an instance-wide count; assert the mint records the guest Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * test: the seeded allowance is a day old, so only the mint can write today's guest_activity row Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BayTppRCstWX6qTf3LMco5 * chore: update ee-repo-ref to 1a10132e4f3cb442c7d0c2cf6e5d92d150bf6e07 This commit updates the EE repository reference after PR #769 was merged in windmill-ee-private. Previous ee-repo-ref: 32841072aa396bff91d30bd91854fa348cb3c439 New ee-repo-ref: 1a10132e4f3cb442c7d0c2cf6e5d92d150bf6e07 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
716ce2ece0 |
feat: free AI tokens + home search/filter revamp (#10020)
* feat: add free Claude Opus tier with per-user token limit Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit move alert * Home AI Chat * wire home ai chat * auto send prompt * refactor: remove keyboard arrow-navigation from home list Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: replace home search bar with unified FilterSearchbar Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: replace home quick tags with FilterSearchbar presets Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: add content filter to home FilterSearchbar with EE-gated content view - Clear the kind filter by deleting the key (was showing a 'kind: null' tag on All) - Remove the standalone Content button - Add a 'content' filter; when set, render the Ctrl-K content-search view (ContentSearchInner) which shows text-match snippets and its own EE warning Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: disable home AI chat and prompt to configure AI when no model Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * track cost instead of tokens * nit * fix: load copilot config on home so AI chat isn't wrongly gated Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Home page update * nits * example prompts * nit * feat: switch free AI tier to DeepSeek with daily cost budgets Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * Move bottom buttons to HomeAIChat * [ee] feat: surface free AI tier state and make its metering abort-proof Makes the free Windmill AI tier legible to the user and closes an abuse hole. Backend: - AIConfig gains a response-only free_tier marker (skip_deserializing so a client can't store a forged one via edit_copilot_config). get_copilot_info keeps returning it once the grant is spent, so the client knows AI is off because the grant ran out, not because nothing was configured. - Per-user grant becomes one-time (migration drops the day key from ai_free_token_usage); the daily table stays as the instance kill-switch. - Reserve-then-reconcile metering (see EE commit) so a mid-stream disconnect can no longer dodge the usage report and get metered zero. Frontend: - copilotInfo carries freeTier; model settings show a "Free" pill and a usage meter that warns past 80%. - The home chat and the session chat show a dedicated "you've used your free Windmill AI, add your own API key" state instead of the generic "no provider configured" one. - A failed send re-fetches copilot_info so the exhausted state (and its banner) appears live, without a page reload. Bumps ee-repo-ref.txt to the matching EE commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: free AI usage meter reusing the context-usage gauge Show free-tier spend with the same gauge as context usage instead of a bespoke block: - Extract the meter+tooltip into a shared UsageMeter; ContextUsageIndicator uses it, and a new FreeTierUsageIndicator renders it from copilotInfo.freeTier. Placed in the session-chat toolbar and next to the home-chat model settings; the old meter block in the model-settings dropdown is removed (the "Free" pill stays). - Hide the context-usage bar while on the free tier so the free meter takes that slot. - Refresh copilotInfo after every free-tier turn (AIChatManager finally) so the meter advances live and the turn that exhausts the grant flips to the exhausted state, instead of both only updating on reload. Gated to active free-tier users, so it costs nothing for configured-key users. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: fix stale free-tier comments after DeepSeek/cost rework Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: always show context bar, replace free-tier meter with usage banner Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit * fix: atomic free-tier budget reservation (ee ref + sqlx) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep CLI/MCP and Hub buttons unblurred on AI chat hover Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add back arrow nav * nit * nit * fix: three review P1s in the home AI chat & search - AIChatManager: refreshFreeTierUsage now bails unless the global copilot state still belongs to the completing manager's workspace, so a warm session finishing after a workspace switch can't reload its (background) workspace over the active one's models/client/copilotWorkspace. - HomeAIChat: block submission until the copilot config is loaded AND enabled (new `canSend`), so a prompt submitted during the unknown-config window isn't handed to a session that never sends it and silently lost. The disabled overlay still gates on config-loaded to avoid a flash. - ItemsList: the content-search reload effect now depends on $workspaceStore so content results follow the active workspace instead of showing the previous one's. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [ee] fix: harden the three home-AI-chat/search P1s after deeper review Follow-up to the previous P1 commit; sharper review found the earlier guards insufficient: - refreshFreeTierUsage now compares against the most-recently-*requested* workspace (new copilotWorkspaceRequested in aiStore, set synchronously in loadCopilot), not the last-*resolved* one — otherwise a warm session finishing while a newer workspace's load is still in flight could win the monotonic token and restore its stale workspace over the one being loaded. - The content-search view is keyed by workspace ({#key $workspaceStore}) so a switch remounts ContentSearchInner; late in-flight responses from the previous workspace can no longer land in the new one's component. Backend (EE, via ee-repo-ref bump to 03ef0eb): the free-tier reservation now also prices the worst-case input cap (at the cache-miss rate), and enforce_free_tier_body rejects oversized prompts and pins n=1 — so an aborted large-prompt request can no longer dodge the input bill that reconciliation would otherwise charge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: exclude service accounts from the free AI tier Free-tier eligibility was keyed solely on authed.email. Workspace admins can create and impersonate arbitrary service accounts (synthetic *.sa.wm.dev identities), each of which would receive its own one-time grant — letting one tenant mint many grants and drain the instance-wide daily allowance. Skip the free-tier fallback for *.sa.wm.dev identities. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: activate free AI tier when clearing a workspace provider edit_copilot_config returned AIConfig::default() when the saved workspace config had no providers and no instance config existed; the frontend applies that response immediately, disabling AI even though the free-tier key is available. A later get_copilot_info (on reload) returns the synthetic free-tier config, so clearing a provider behaved inconsistently until reload. Give this response path the same free-tier fallback as get_copilot_info. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: gate the home AI composer behind the global-AI dev flag The "Build with AI" composer starts a session and navigates to /sessions, which lives behind the same wm_dev_global_ai dev gate as the global AI chat. With the gate off (the default), /sessions renders only its gate message, SessionWrapper never mounts, and the queued prompt is silently dropped. Hide the home entry point behind isGlobalAiEnabled() so it isn't exposed before the sessions gate opens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [ee] chore: bump ee-repo-ref for deepseek-v4-flash price/model fix Points at the EE commit that pins deepseek-v4-flash and its real prices (pico-precision accounting). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [ee] fix: provable byte bound for the free-tier input cap (ee-repo-ref) Bumps ee-repo-ref to the EE commit that caps the raw request body byte length directly (token_count <= byte_count is provable), replacing the unsafe body.len()/2 token estimate that high-entropy prompts could beat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * nit isGlobalAiEnabled * empty commit * fix(frontend): address Codex review on free-tier / home filters - P1: home filters now sync from the URL reactively, so browser Back/Forward updates the chips, kind toggle and results (and clears keys dropped from the URL) instead of leaving them stale until the next filter edit. - Free-tier banner buttons drop deprecated Button props (size/color/border variant) for unifiedSize + a supported variant. - Condense refreshFreeTierUsage comments to a single race-condition constraint beside the guard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): hide empty kind badge on draft-only scripts A draft-only script can carry an empty `kind`, which still isn't 'script' so the row rendered a blue badge whose only content was capitalize('') — an empty pill left of the "Draft only" badge. Guard the badge on a non-empty kind. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): animate home tree-view group expand/collapse Wrap each owner group's children in ResizeTransitionWrapper so height changes animate. A slide transition only animates the initial mount, but a freshly-opened owner fetches its rows and passes through a transient empty state before they land — the ResizeObserver animates that second growth too. Nested TreeViews inherit the wrapper's context and skip their own, so one observer per top-level owner animates the whole subtree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): FilterSearchbar boolean auto-set and string-filter presets - A default-false boolean filter has only one useful value, so selecting it sets true immediately instead of opening a true/false picker. A default-true boolean (e.g. "Include library scripts") still shows the picker, where false is the meaningful choice — expressed via a new optional `default` on the schema. - A plain string filter now surfaces any presets targeting it (`<tag>:<value>`) as suggestions once selected, integrated into menuItems so keyboard nav works — previously selecting e.g. "Owner" showed nothing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): home page toolbar and content-filter revamp - "New" create-menu button (scripts/flows/apps/…) replaces the old Content button; the search bar moves to the right of the toggle group. - Restore the content filter dropped in a merge: a `content` searchbar filter swaps the list for the full-text ContentSearchInner view (EE), aligned flush with -mx-2. - Move the owner/group and label chips off the page into FilterSearchbar presets; ownerFilter/labelFilter now derive from the searchbar keys (data layer unchanged). - Move the list controls (select / tree view / expand-all / sort) inline into the top row between the toggle group and search bar; add margin above the list. - Beta tag on the home AI chat; a bit more bottom margin under it; tighten the gap between the admin/tutorial banners and the list. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ai): pass the request body to the free-tier reservation Thread the prompt body into resolve_free_tier_credentials so the free tier can size its upfront reservation from the actual request length instead of a fixed worst case (EE c2e248b), fixing normal chats being rejected as "too large". Updates the OSS stub signature and bumps ee-repo-ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): gate home Create/Import menu on edit permissions The relocated CreateActionsMenu rendered unconditionally, so operators and users in workspaces protected from direct deployment saw create/import actions they can't use. Restore the original gate (!operator && showEditButtons, the latter from NoDirectDeployAlert). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): address Codex review on filter searchbar - P1: the boolean shortcut now goes through the same tag-insertion path as the normal branch, so it removes the typed search segment instead of leaving it as a stray free-text (_default_) term. - Mark the Runs `show_future_jobs` filter default: true so selecting it opens the picker (false is the meaningful choice) rather than being a no-op. - Home owner/label presets now emit the canonical `key:\ value` form so the applied-preset check matches after a reparse and can't re-offer a duplicate; update the suggestion extraction to strip the leading separator. - Replace deprecated Button props (size/spacingSize/color) on the relocated list controls with unifiedSize. - Fix stale comments: UsageMeter no longer claims a free-tier consumer; the home filter schema comment describes presets, not the removed ListFilters/label badges. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): boolean filter shortcut sets value canonically The round-1 shortcut baked `true` into the tag text, which merged into a following tag (e.g. `archived:\ truekind:\ flow`). Instead remove the typed segment, set the value, and reparse so the text is rebuilt canonically — no lingering free-text and no merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(ai): restate free-tier caller identity contract in the OSS stub; bump ee-repo-ref Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): keep flanking tags separate when boolean shortcut drops a segment Joining `before`/`after` directly fused the tags a removed mid-segment sat between (e.g. `kind:\ flowsummary:\ bar`). Join with a space; reparse then canonicalizes. Also trims the comment to the essential constraint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ai): update sqlx cache for free-tier daily-day queries; bump ee-repo-ref The reserve/reconcile daily-usage queries now bind the reservation day (EE change); refresh their offline query cache and point ee-repo-ref at the EE commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ai): activate free tier when instance ai_config has no provider An instance ai_config row won precedence just by existing, so an empty {} (valid via global settings / declarative config) suppressed the free-tier fallback and left AI disabled — even though build_copilot_settings_state already treats it as unconfigured. Apply the same has_providers() check to the instance config in the proxy and edit_copilot_config paths. Also refresh the sqlx cache for the reservation ceiling change and bump ee-repo-ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(frontend): migrate legacy Home filter URLs to the searchbar keys The old Home UI stored free-text in `search`, owner scope in `filter`, and could write `kind=all`; the generic searchbar sync uses `_default_`, `owner`, and a kind enum without `all`. Rewrite those params once before the sync reads the URL so shared/bookmarked links restore, and drop `kind=all` which would otherwise wedge later filter edits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(ai): empty instance config in get_copilot_info; label user-disabled Home AI - get_copilot_info returned any existing instance ai_config row before the free-tier fallback, so an empty {} disabled AI in the copilot-info UI even though the proxy now serves the free tier. Apply the same has_providers() gate here. - The Home chat overlay said "No AI provider is configured" when the user had disabled AI in account settings (providers still present). Distinguish that state ("Windmill AI is disabled in your account settings") as the docked chat does, and drop the misleading workspace-config button in that case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ai): drop redundant proxy service-account check; trim TreeView comment The service-account exclusion now lives in the free-tier helper, so the proxy calls it directly. Also condense the tree-view resize-transition comment to the essential reason. Bumps ee-repo-ref. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(frontend): the Home content filter is not EE-gated ContentSearchInner loads the workspace's scripts/flows/apps/resources and matches their contents client-side, so it works on any instance. Drop the misleading "(EE)" from the filter label and the "EE indexer / off-EE fallback" comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(ee): bump ee-repo-ref for free-tier pricing + exhaustion fixes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): show disabled Home AI overlay statically, not on hover The disabled-state overlay (reason + configure/add-key action) was opacity-0 and pointer-events-none until group-hover, so keyboard and touch users saw an inert composer with no visible remedy. Render it and the composer blur statically when disabled instead. Also bumps ee-repo-ref for the trimmed free-tier comments. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): give account-disabled Home AI overlay a recovery action The account-disabled branch showed a reason but hid every action, on the mistaken premise that account settings has no linkable route. It opens from the #user-settings hash (the same one the sidebar Account menu uses), so link there. Bumps ee-repo-ref for the free-tier fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): gate Home AI composer for operators; a11y and filter-sync fixes - Home composer now uses prefersSessionHandoff($userStore?.operator) instead of isGlobalAiEnabled(): operators reached this route and could submit a prompt into a /sessions page that refuses them, silently dropping it. Also drops the leftover empty header spacer div above the chat. - HomeAIChat: mark the blurred/disabled subtrees inert so keyboard users can't tab into the unreadable textarea (pointer-events-none didn't stop Tab). - ItemsList: keep the role-dependent searchbar keys (include_library, only_user_folders) in the schema unconditionally and toggle `hidden` instead, so useUrlSyncedFilterInstance (which snapshots the key set once) still URL-syncs a key that first appears after a workspace switch. - Bumps ee-repo-ref for the indexer non-parquet build fix. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): keep CLI/MCP connect row for operators; trim filter comment The previous commit gated all of HomeAIChat behind the operator/session check, which also removed the AI-independent CLI/MCP "Connect workspace" drawer that operators (and the sessions-beta opt-out) had on main. Render HomeAIChat for the same audience as before (isGlobalAiEnabled) and gate only the composer (title, input, examples, overlay) on operator status inside the component; the connect row always shows. Also trims the role-dependent filter-schema comment to the <=4 line rule. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): reconnect Home keyboard navigation to the unified searchbar The searchbar migration replaced the <input id="home-search-input"> the ItemsList keyboard handler keys off, so Arrow/Enter no longer drove the results list. Thread an `id` down to the searchbar's contenteditable (via TaggedTextInput/FilterSearchbar `inputId`) so the handler and the workspace-switch focus restoration find it again; read the caret through the Selection API instead of an <input>'s selectionStart/End; and stand the list's arrows down while the searchbar's suggestion dropdown is open (tracked via onDropdownVisibleChange). In free-text mode the searchbar no longer opens its dropdown on a bare arrow key, so an empty box passes Arrow/Enter to the list as before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * fix(frontend): stop searchbar Enter inserting a newline; idle typewriter for operators - TaggedTextInput is a single-line filter input, so Enter now preventDefaults the contenteditable's newline insertion (surrounding suggestion-select / list-open handlers still run on bubble). Previously Enter with no row highlighted dropped a literal \n into the query. - HomeAIChat's placeholder typewriter effect now runs only while the composer is shown, so it no longer loops forever driving an unrendered input for operators. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BSqa1iRxn9GUE9fegT7bDS * chore: update ee-repo-ref to f2a31156ac08ecb02d89dbc66d72be58e9c877ff This commit updates the EE repository reference after PR #652 was merged in windmill-ee-private. Previous ee-repo-ref: e59b96a2eea5d1110b40c842f17b337ab051bdd3 New ee-repo-ref: f2a31156ac08ecb02d89dbc66d72be58e9c877ff Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
aa4a6ffd66 |
fix: track outstanding service log files on the rows themselves (#10894)
* fix: track outstanding service log files on the rows themselves Adds `log_file.indexed_at` so the service log ingest can read outstanding rows instead of walking a cursor over `log_ts`. A row registered after the pass had gone by its minute was skipped for good, and no ordering fixes that — an arrival sequence fails the same way, since a row can take a lower value and commit after a higher one has moved the cursor past it. The migration marks existing rows with a sentinel; the first pass returns the ones the old cursor had not reached to the queue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPAP96jJNYpPQ8bpxZcU1C * [ee] refactor: drop the claim/confirm phase from the service log ingest queue Two states are enough: a row is outstanding or it is marked. The migration no longer creates the index for the claim sentinel, and the sqlx cache loses the two queries the event-time cursor used. * [ee] fix: make re-indexing a service log file idempotent Corrects the `init_last_log_file_sent` note: a rewritten row keeps the `indexed_at` it had, so one the indexers already took is not offered again. * [ee] fix: let a rebuild take the rows it covered out of the ingest queue Adds the query that releases them; the index layout stays v4. * [ee] fix: index the lookup a rebuild releases rows by A rebuild takes rows out of the queue by the file it read out of the store, which is the one lookup that arrives without a `log_ts`. The primary key is `(hostname, log_ts)`, so nothing covered it and each batch scanned every outstanding row — worst in exactly the state a rebuild follows. Verified at 50k outstanding rows: sequential scan becomes an index scan. Also records `log_file.indexed_at` in the schema reference. * [ee] fix: treat a state handed back without its line count as behind * [ee] fix: give the converted state a line count * [ee] fix: keep the converted cursor from being rewound by the rebuild * [ee] fix: inherit the legacy cursor from one source, not field by field * [ee] fix: count a file's lines against the buffer before reading it * [ee] fix: bound the row buffer on what it holds, not on reported counts * [ee] fix: settle the upgrade from the store rather than from event time * [ee] docs: describe the conversion's second half as it now works * [ee] refactor: settle the upgrade with one rebuild instead of reconciling The migration records existing rows as done rather than marking them with a sentinel: the indexer puts back what the old cursor had not reached on its first pass, which is the only place that cursor's position is known. * [ee] fix: repair the rows the old cursor skipped instead of recording them as done The migration marks pre-existing rows with a sentinel again, so the indexer can tell them from rows registered since and put the window's worth back on the queue. * [ee] fix: keep a source file whole in one partition * [ee] revert the file-atomic partition change * [ee] fix: dedupe the public reads, and repair an index without a cursor * [ee] fix: repair an index whose cursor is gone, and keep what the repair found * [ee] fix: seed a pass from both axes of what a rebuild recovered * [ee] fix: settle the cursor on what the store holds, not on what was read * [ee] fix: an empty rebuild must not claim ground it has not covered * [ee] test: pin the cursor a rebuild settles on * chore: update ee-repo-ref to bc0c7051585194474078b6c1941a3fb73893d9e5 This commit updates the EE repository reference after PR #755 was merged in windmill-ee-private. Previous ee-repo-ref: 328f5a90afeae9c683bf3294f0d9eb293a3e1a92 New ee-repo-ref: bc0c7051585194474078b6c1941a3fb73893d9e5 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 5
|
b6e059116a |
feat: track token cost in AI sessions and chats (#10688)
* feat: track token cost in AI sessions and chats * fix: address review findings on AI cost tracking * fix: price inherited and overridden models at their real rates * fix: stop newer model revisions inheriting an older price * fix: stop a sub-model inheriting its family's price * fix: keep alias suffixes resolving to their model's price * fix: count OpenRouter cache writes and drop unverifiable rates * refactor: move AI spend out of the chat into workspace and user settings * fix: pin the usage workspace per turn and stop inventing cache rates * fix: leave Sonnet 5 unpriced while its promotional rate runs * docs: record the new table in the schema summary and tighten comments * fix: mark estimated AI costs with ~ and drop session grouping Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: name the workspace in the self-scoped AI usage title Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state that overrides never replace a provider-returned cost Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let a cleared cache rate inherit again and flag partial totals Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: clear a refused rate's error when the input snaps back Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop a revision variant inheriting its base family's rate Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: report AI usage before tools run and price self usage consistently Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: key pricing rows on the model id usage is reported under Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: surface Bedrock and Gemini usage the chat proxy was dropping Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count Gemini tool-use prompt tokens as input Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: price flat-rate Gemini Flash models Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the tool-use token invariant once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 5
|
633d7bcb2e |
feat: add trigger_history table with source tracking (#10696)
* feat: add trigger_history table with source tracking Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: gate trigger history reads on scopes and harden its writers Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: filter trigger history scopes in SQL and match the cleared-handler diff Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record a trigger restore from the trashbin in its history Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: record bulk http trigger creates and document the recording boundary Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: lock the trigger row when capturing its history preimage Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only record an auto-disable that actually flipped the schedule Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: state the auto-disable invariant once instead of at four call sites Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: render trigger history changes as a structured field diff Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: make a server-initiated disable atomic with its history row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: note that the auto-disable savepoint takes no pool connection Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: note the flow fallback is the last chance to disable Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never leave a trigger enabled because its history row failed Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: retry the disable history row instead of dropping it on first failure Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: use the design-system Button for the change-value expander Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hold the trigger row lock across its disable history row Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the history-loss alert out of the listener cancellation race Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: read the history workspace through the trigger-workspace seam Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
baefa1345b |
feat: give dbt its own editor with an explicitly refreshed model graph (#10448)
* feat: give dbt its own editor with an explicitly refreshed model graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: bump ee ref for the agent-worker dbt editor graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope editor graph retention by principal, carry parse context, honor nlang Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: bump ee ref Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the dbt editor's model graph and log panel mounted across tabs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the dbt_edge to dbt_node joins on an index-usable equality Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: poll a parse until the job ends, resolve the project key, correct the docs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: surface a slow parse's job, bound poll failures, drop banned bindable defaults Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide the dbt Generated UI content, not only its tab Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: honor disabled Triggers in the dbt tab fallback, record permissioned_as Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one dbt pane with the run drawn on the models, and a full-height script graph Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: move the dbt build arguments behind the Build button Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: trim the dbt editor toolbar and stop the graph asserting a cause it lacks Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: mark dbt as alpha in the language picker and announce it once Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: trim the dbt alpha notice Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: give a selected dbt model the whole detail section, with a close that deselects Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: close the dbt detail panel by clicking away, and make its close obvious Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cache the agent-worker dbt query, which needs the private feature to compile Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never fall back to a settings tab the embedder disabled Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preview dbt rows from the same project the graph was parsed from * feat: hide the script-kind selector for dbt projects * fix: pin a dbt row preview to the project its graph was parsed from * fix: pin a dbt row preview to the arguments its graph was parsed under * fix: keep dbt preview placeholders live while its vars stay pinned * fix: report a warehouse-less dbt parse's counts and flag stale preview args * fix: tell the pinned-vars case apart from a stale placeholder * chore: update ee-repo-ref to 59044635769f18f8ff5073236cfc7b5f41e917cc This commit updates the EE repository reference after PR #707 was merged in windmill-ee-private. Previous ee-repo-ref: 7e424384cdd4cef8653b55b04f17ad3f801bc50c New ee-repo-ref: 59044635769f18f8ff5073236cfc7b5f41e917cc Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
fb82748296 |
fix: make on_behalf_of control permissions for scripts and flows (#10438)
* fix: make on_behalf_of control permissions for scripts and flows Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: inherit the recorded on-behalf-of identity when a preserving deploy omits it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep an omitted permissioned_as from re-versioning an unchanged script Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: derive the on-behalf-of principal from the email and reject mismatched pairs Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop workspace deploys from carrying a source-workspace principal Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the onBehalfOfPermissionedAs param doc Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test: pin that workspace deploys never carry a source-workspace principal Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the omitted-principal contract and refresh generated prompts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep external-superadmin principals on email-only redeploys Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: scope the recorded principal to its workspace and prefer real accounts Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: carry the recorded principal correctly through drafts and set-permissioned-as Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: sweep draft identity pairs on email change and offboarding Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: leave group identities alone when sweeping a user's email Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: treat only g/ without an email as a group, and match the offboard preview Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: stop the group guard from skipping rows with no recorded principal Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: state the group guard once instead of restating it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: make the permissioned_as the only stored on-behalf-of identity Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * perf: skip resolving the on-behalf-of address for sync clients that discard it Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: address the local review of the identity refactor Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve the on-behalf-of identity coherently across clones, offboarding and no-op deploys * test: pin that a fork keeps only the on-behalf-of identities that resolve in it * fix: decide a principal prefix-first everywhere and canonicalize bare addresses * fix: prefix a slash-containing address so a reader cannot take it for a group * fix: read an address as a username before the group- convention * fix: rewrite the canonical principal when an account's address moves * fix: keep the address form of a principal to accounts without a usr row * fix: reject an identity a job row cannot carry and read it uncached at dispatch * fix: count characters against the job identity width and cap the backfill * refactor: name the script/flow principal on_behalf_of, as apps do * docs: state the caller-must-authorize contract on the identity resolvers * fix: keep writing on_behalf_of_email until every worker reads the principal * fix: err high on the compatibility version and document the last resolver * fix: keep the compatibility address current through identity mutations * fix: carry the compatibility address with the principal on every copy path * chore: re-pin the EE ref to the companion branch merged with EE main * fix: key the dbt retry lookup on the stored principal * fix: keep a mixed-version address recoverable through a fork * fix: read a round-tripped address uncached so a redeploy is not rejected * fix: refuse an email change that would make a principal unenqueueable * chore: update ee-repo-ref to ac3d7d015296f041ae44ab6bc4953485f44d36e4 This commit updates the EE repository reference after PR #704 was merged in windmill-ee-private. Previous ee-repo-ref: 219b0b03905a1a0028054b3a4985724e77d09036 New ee-repo-ref: ac3d7d015296f041ae44ab6bc4953485f44d36e4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 5
|
032300e28e |
feat: run dbt projects as a first-class Windmill runtime (#10326)
* fix: mount only the engine in the dbt jail, reject shadowed and malformed args
Review round 42.
The jail mounted the whole dbt cache directory, whose siblings of the
engine are `repos/` and `packages/` — other workspaces' private checkouts
and package trees, kept apart by cache key rather than by permissions. A
jailed project could read them. It now mounts the engine's own directory,
which the provisioner names; verified from inside the jail that `repos/`,
`packages/` and `state/` are invisible while the engine stays usable.
A `{{ placeholder }}` may no longer take the name of a run argument this
runtime defines. It was silently dropped from the signature, so a
descriptor like `value: "{{ select }}"` deployed and then could not be
run at all: the built-in `select` is an array and the interpolation needs
a scalar. Refused at parse, so the deploy says so.
A `vars` override that is not an object is refused rather than ignored.
Argument-schema validation is opt-in, so a string or an array silently
ran the descriptor's own vars — against a different schema or alias than
the caller asked for. `select` and `exclude` already refused theirs.
* feat(dbt): the project is the script's module bundle, not a git checkout
A dbt script now carries its whole dbt project as its module bundle. The
descriptor is the script content; `<script>__dbt/` holds the project verbatim,
so importing an existing project is `cp -r` plus `wmill sync push`, and the
worker materialises the bundle into the job directory instead of cloning.
Backend
- `prepare_project` writes the script's modules and requires `dbt_project.yml`
at the bundle root. `checkout`, the git-ssh command, the clone cache and the
repository resource are gone, along with `repo`, `project`, `ref` and
`git_ssh_identity` on the descriptor.
- Run identity and the package cache key take a `project_digest` (sorted SHA256
over the bundle) where the commit used to sit, so an edited project cannot
resume a previous run's `run_results.json` or reuse its `dbt_packages`.
- The per-run graph re-ingest is now gated on `vars` placeholders and `$var:`
env alone.
- `capture_dependency_job` takes the script's modules so a dependency job, which
has no generic module-writing step, materialises them itself.
- `dbt deps` caching strips the git remote from every package it cached, not
just the tree root: `packages.yml` can render a token into a `git:` URL.
- `git_clone.rs` is dropped and `ansible_executor.rs` returns to its own copy of
the clone helpers.
CLI
- `wmill sync pull` keeps a dbt script's lock beside its folder rather than
inside it, so the folder holds nothing but the project.
- Directories dbt generates (`target-path`, `packages-install-path`,
`clean-targets` and the usual defaults, read from `dbt_project.yml`) are
excluded from the bundle, from the sync diff and from staleness hashing.
- A module-only edit now pushes its parent dbt script and is reported as a
changed module rather than passing unnoticed.
* fix: keep a script's modules in the worker's file-system cache
The first fetch of a script version reads the database and carries its
modules; every later fetch imports from the worker's cache directory, whose
`RawScript::import` hard-coded `modules: None` and whose `export` never wrote
them. A worker restart therefore started running the script without its own
files, silently — for a dbt script, without its project, which fails with
"carries no project"; for any other script with a module bundle, with the
imports missing.
`modules.json` is now written on every export and required on import, so an
entry written by an older version fails to import and is refetched rather than
serving a stripped script for as long as the directory lives.
Also derives a dbt run's `project_digest` from the bundle the run actually
carries: `handle_dbt_job` was passing `None`, which collapsed every project in
a workspace onto one digest and let `dbt retry` resume a different project's
`run_results.json`.
* fix(dbt): give every phase the script's environment, bound the cache copies
`dbt deps` ran without the script's environment variables on an unsandboxed
worker, so a `packages.yml` resolving a private package URL through
`env_var()` could not see them while the package cache key was still built on
their digest. `with_invocation_env`, applied at three of the four call sites,
is folded into `dbt_command` so no phase can be added without it, and
`DBT_TARGET_PATH` is set after both environments rather than before.
The package cache copies ran through a bare `Command::output()`: the tree is
the project's, so a cancelled or timed-out job held its worker slot until `cp`
finished. Both the restore and the publish now run under the job poller like
every other phase.
* fix(dbt): only offer commands whose writes match the graph, honour packages-install-path
`dbt_command: run` is dropped from the allowed overrides. Asset dispatch fires a
script's deploy-time writes on any successful job, and `dbt run` covers models
only, so a project with seeds or snapshots notified consumers of relations the
invocation left stale. That is the same reason `test` was already excluded.
Narrowing what a run touches is `select`/`exclude`, which scope the graph too.
`dbt deps` writes to the project's `packages-install-path`, so a project that
moved it got no package cache at all: the publish found nothing at
`dbt_packages` and every job resolved its dependencies over the network again.
The path is read from `dbt_project.yml` and validated as project-relative,
since both cache copies are rooted at it.
Also states the sidecar's mutator contract at the module level: the dbt manifest
tables carry no RLS and grant `windmill_user` full access, so a user-scoped
transaction is not enforcement and every caller must have verified write access
to the script itself.
CLI: a module file is now grouped with its parent script for the push. Left in
a group of its own it got its own `alreadySynced`, so a push touching several
files of one bundle deployed the script once per file; the resulting versions
raced, and the asset graph could end up describing none of them.
* fix(dbt): seed a project for browser-created scripts, refuse a no-op retry
A dbt script created in the browser only got a descriptor, and the runtime
refuses a script whose bundle has no `dbt_project.yml`, so the advertised
Create → Deploy → Run path always failed its dependency job. New dbt scripts
now start with a project that builds: pointing `profile.resource` at a
warehouse is the one edit, and growing it is `wmill sync pull` plus a local
editor, which is where dbt development happens.
`dbt retry` builds its graph from the previous run's error, fail and skipped
nodes alone, so retrying an all-green run selected nothing and wrote nothing —
and a job that succeeds having written nothing still dispatches every
deploy-time write, waking every downstream consumer for relations no one
touched. Refused, with the reason.
CLI: a configured `target-path` or `packages-install-path` may be nested
(`build/target`), and `clean-targets` has a block form as well as an inline
one. Both are now parsed, and the exclusion compares the project-relative path
rather than the top-level segment, so a nested generated tree no longer lands
in the bundle and no longer makes a local `dbt run` look like a project change.
* fix(dbt): lock a project once, find the parent on either path separator
A dbt script's modules are its dbt project, not helper code with dependencies
of its own, so the generic per-module lock loop is skipped for it: the parent
lock already ran `dbt deps` and `dbt parse` over the whole project. Locking
each file separately re-materialised the bundle and re-invoked dbt once per
file, so a project of N files paid N project-sized passes and a large one timed
the deploy out. The 13-file fixture went from 14 relock passes to 1.
`pushParentScriptForModule` searched the raw path for `__dbt/`, so on Windows,
where the folder is spelled `__dbt\`, a module-only edit returned without
deploying its parent while the caller still recorded the file as synced. It now
goes through `getScriptBasePathFromModulePath`, which normalizes separators.
Also drops the last of the external-repository wording from the descriptor's
module docs and from the `codebase` rejection a user can hit.
* feat(dbt): infer the run form locally, keep test-only retries from cascading
`windmill-parser-wasm-yaml` 1.770.0 carries `parse_dbt`, so the browser and the
CLI derive a dbt script's run arguments from its descriptor instead of waiting
for the deploy to hand back a schema. Pins bumped in both.
`generate-metadata` was rewriting a dbt script's `lock` field on every run: a
dbt lock comes from the dependency job on a worker, so nothing generates it
locally and the resolved `!inline` reference was left inlined into the metadata
or blanked. It is restored instead, and a push straight after
`generate-metadata` is a no-op again.
A retry now needs a failed node that materialises something. `dbt retry` builds
its graph from error, fail and skipped nodes, and with `test_behavior:
after_all` a failing test is what `run_results.json` ends up describing — so the
retry reran tests, wrote nothing, succeeded, and still dispatched every
deploy-time write.
The dbt badge's destination is deterministic: writers outrank readers, and among
several writers of one relation (which the backend permits) the smallest id
wins, rather than whichever write edge arrived last.
* feat(dbt): browse the project and read a run's per-node result
Two views a dbt user expects and that the generic script surfaces do not give.
**The project.** A dbt script's editor gains a Project tab beside its
descriptor: the module bundle as the tree dbt itself expects, each file
read-only with syntax highlighting. The existing module tab strip is a flat row
built for a couple of helper files and does not survive a real project; a
13-file fixture already overflows it. Directories sort before files so it reads
like the checkout on disk, and an empty bundle explains the `cp -r` instead of
showing a blank pane.
**The run.** `DisplayResult` renders a dbt invocation's per-node breakdown above
the raw payload: totals, then a table of node, kind, target relation, rows and
time, with failures and warnings sorted first and carrying their message. The
data was already structured; it was being shown as JSON to scroll and PASS/WARN
counts to find in the log. On a failed run the same JSON rides in the error
message after the exit-status line, so it is parsed back out — that is the case
worth rendering, since the failing node is what the user came for.
* docs(dbt): say that profile.resource is what buys the asset graph
The starter descriptor described `profile.resource` as the thing rendered into
profiles.yml, with the project's own file as an equal alternative. It is not
equal: the resource PATH is the warehouse's identity in the asset graph, so a
project bringing its own profiles.yml runs fine and silently gets no assets, no
lineage and no cascade. The deploy already says so in its log; now the
descriptor a user starts from says it too, before they choose.
* fix(dbt): authorize a resource used only for asset identity, clean up after failed installs
A descriptor setting both `profile.profiles_yml` and `profile.resource` took
its connection from the project's file but returned the resource path as the
graph's warehouse identity without ever reading it. A script editor could
therefore publish `table://<any resource>/...` writes, and wake that
warehouse's subscribers, while connecting somewhere else. The resource is now
read on that path too — reading is what authorizes it — so the combination
keeps working for the case that wants it (keep your own profiles.yml, still get
lineage) and fails closed otherwise.
Provisioning cleaned up its staging directory only on the paths someone
remembered, so a run of failed or cancelled first-use installs accumulated
venvs, tarballs and installer scripts until the worker's disk was gone. All
three engines now hold their scratch paths in a guard that removes them on
drop, which is the one exit every path takes, cancellation included.
Frontend: `partial success` is dbt's word for a node that built but whose tests
failed, counted in `totals.error` and redone by a retry, so it ranks with the
failures instead of rendering green with its message hidden. And the run panel
now keys off the worker's engine discriminator rather than `{nodes, totals}`,
which is a shape an ordinary script can return. Both pinned by unit tests on
the extracted `parseDbtRun` helpers.
* feat(dbt): show a run's models on the run page
The run page is where you land on a running job, and until now it showed a dbt
run as streaming text: the per-node table only renders once the job has
produced a result, and the graph that moves per model lived on the pipeline
page you had to navigate to. A Models section now sits above the result,
scoped to the running script's own relations and its `ref()` lineage, polling
while the job is in flight so nodes move as dbt walks the DAG.
No `resolveGraph`: that merges drafts and live editor buffers into the
persisted graph, and a run page has neither.
* feat(dbt): retry failed nodes automatically, and from any worker
**Node-level retry, in the job.** `retry_failed_nodes: {attempts, delay_seconds}`
rebuilds only what a failed build left failed or skipped, before the job reports
failure. dbt confines a failure to its own subtree and `dbt retry` resumes
exactly that set, so a transient warehouse error costs those nodes rather than
the project. Doing it in-job is what keeps the state question out of it: the
previous attempt's `run_results.json` is still in the job directory, so there is
nothing to persist and no worker to land back on. This is the granularity
astronomer-cosmos gets from one Airflow task per model, without the ~6x that
per-model tasks measured.
A retry's `run_results.json` names only the nodes it redid, so it overlays the
accumulated results rather than replacing them: the job's result has to be every
node the job touched, or the nodes that succeeded before the retry settle no
materializations. Pinned by a test.
**Durable retry state.** `run_results.json` is now saved to `dbt_run_state` as
well as the worker's local cache, so an explicit `dbt_command: retry` works from
any worker of the group rather than only the one that failed. Only the results
are stored: `dbt retry` also needs `manifest.json`, roughly sixty times larger
and growing with the project (732 KB against 12 KB on the six-node fixture), but
the manifest is a pure function of the project files, vars and env, all of which
the stored identity already pins, so a worker restoring from the database
re-derives it with a `dbt parse` of about a second.
* fix(dbt): restore the sqlx cache, make retries cancellable and path-aware
**SQLx cache.** A `cargo sqlx prepare` deleted 750 entries, including the
enterprise queries CI needs under `SQLX_OFFLINE=true`, and the check that was
supposed to catch it reported zero losses because it was run from `backend/`
with a `backend/`-prefixed path, so its baseline was empty and it failed open.
All 750 are restored; the branch now adds 19 and deletes none, and
`SQLX_OFFLINE=true cargo check` passes.
**Retry backoff observes cancellation.** `canceled_by` is only written by the
job poller, which does not run between attempts, so re-reading it reported the
state as of the failed attempt and missed every cancel issued during the wait
— the whole window the check exists to cover. The wait now reads
`v2_job_queue.canceled_by` each second, and the job's deadline is honoured
before starting another dbt process.
**Retry state follows its script.** `dbt_run_state` is path-keyed like the
manifest sidecar but, unlike it, nothing regenerates it: a rename moves the row
so a resumable failure survives, while archive and delete clear it, so a script
later created at that path cannot inherit a stranger's failure and its
arguments.
**CLI.** `table` joins ducklake and s3object in the local graph's auto-trigger
kinds, matching `is_auto_trigger_kind` and the frontend's set; without it a
local graph and the generated docs omitted a cascade edge the deploy has.
* fix(dbt): carry only the project files a bundle can hold, and say what it drops
Exploring real and edge-case projects surfaced three frictions, all in the
import path a user hits first.
**A binary file broke the push, opaquely.** dbt projects carry images under
`docs/`, stray `.DS_Store` files and occasionally a parquet seed. Read as text
they become mojibake, and a NUL among them is rejected by Postgres with
`unsupported Unicode escape sequence` — which `wmill sync push` then reported as
success, exiting 0 with the script never created. Binary files are now detected
the way `git` detects them, by a NUL in the first 8000 bytes rather than by
extension, and skipped with the reason.
**The size guard the docs promised did not exist.** Now it does: 5 MB per file,
which only ever catches a committed dataset. Real dbt code is about 500 bytes
median and 1.9 KB at p90.
**Skipped files became a permanent phantom diff.** The push dropped them while
the sync diff still offered them, so every push reported changes no push could
resolve. One predicate now answers for the push, the staleness hash and the
diff.
Verified on a project with unicode filenames and content, CRLF endings, an
empty model, an ephemeral model, a disabled model, a `.md` docs block, an
extensionless README, six levels of nesting, a 7.6 MB seed and a PNG: it
pushes, round-trips byte-for-byte through pull, deploys to 7 dbt nodes and 6
`table://` assets (ephemeral and disabled correctly absent), and runs green.
* fix(dbt): resolve dbt-core against the adapter, settle partial success, unify status
**Adapters could not be provisioned.** The 1.x engine pinned `dbt-core` to a
fixed version independent of the adapter, but several adapters cap below it:
`dbt-mysql` at `~=1.7`, `dbt-oracle` and `dbt-databricks` below 1.12, and
`dbt-salesforce` has no package at all (it exists only inside Fusion). Those
projects failed at provisioning with a uv resolver dump. The install now asks
for a range and lets the adapter choose, and records what the resolver picked so
the lock pins a version that adapter can take.
The floor is the CLI this runtime invokes: resolving down to dbt-core 1.7
produced a working venv that then failed with `No such option '--target'`, which
is worse than not resolving. An adapter with no release in range now fails
naming itself and pointing at `dbt-core-2x` or `fusion`, instead of a resolver
dump. Salesforce is refused up front with the reason.
**`partial success` left a model stuck on `Running`.** It is dbt's word for a
node that built and then failed its tests, and it was already treated as a
failure when counting totals and deciding a retry — but the two sites that
settle the RELATION fell through to "says nothing", so the tailer's `Running`
was never replaced and a finished job showed a model still building. Six status
comparisons had drifted apart, two folding case and four not, while dbt-core 1.x
echoes the author's casing and 2.x uppercases; they are now one classifier.
**Agent workers.** The durable retry state and the cancellation poll both need a
database, which an agent worker reaches only through the API. The automatic node
retry is refused there rather than running a wait it could not interrupt, and
the docs say "any worker with a database connection" instead of overclaiming.
Also clears `dbt_run_state` when a path stops being a dbt script, and moves
`run_identity`'s contract onto `run_identity` from the digest helper below it.
* feat(dbt): show the transform behind a model on the run graph
The run page's graph carried a node for the script itself and drew every
relation as a bare table. Both were wrong for that page: the graph there is
already scoped to one script, so a node standing for it distinguishes nothing
(on the pipeline page it separates one project from another, which is why it
exists), and dbt's own DAG node is the model — the SQL and the relation it
writes are one thing, so a graph of relations alone leaves out what a reader
came to see.
The script node is dropped, and selecting a model now shows its SQL underneath
the canvas with its file path and materialization, read-only, the same view the
pipeline details pane gives.
* feat(dbt): move the graph with the run
The worker has always recorded a state per relation as dbt walks the DAG —
`running` when a model starts, `materialized` or `failed` when it ends — but
nothing rendered it: the graph response carries what a relation IS, not what a
particular run is doing to it, so the canvas had nothing to show and a running
job looked identical to a finished one.
`assets/run_progress/{job_id}` returns that state for one job, the run page
polls it beside the graph, and the asset node carries a spinner or its outcome.
Errors and retries need nothing extra: a failed node writes `failed`, and an
in-job retry rewrites the same row, so the node returns to `running` and on to
its new outcome by itself.
`materialized_partition` holds a relation's CURRENT state keyed by relation, so
filtering on `job_id` returns exactly what this run last touched — which is the
question a run page asks, and why a superseded older run shows nothing.
* feat(dbt): a dbt project is not a data pipeline
Deploying a dbt script marked it `auto_kind = 'pipeline'`, which enrolled it
in pipeline membership: the folder became a Pipeline entry on the home page,
the script folded into it, and `/pipeline/<folder>` opened a canvas holding
the project's whole model DAG next to the pipeline's own scripts. A folder
holding both then read as two projects in one editor, and the pipeline editor
offered to author transforms that are in fact authored in a local `dbt run`
loop and pushed as the script's bundle.
A dbt script is now never a pipeline member, and the pipeline canvas drops the
dbt script node. Its models stay, with their `ref()` lineage: the relations are
what a downstream pipeline script reads, and dropping them would break the
cascade from a dbt run — the point of giving dbt models `table://` identity.
Also drops a screenshot committed to this branch by accident.
* fix(dbt): authorize run_progress through the job, drop dbt from the local graph
`run_progress` read `materialized_partition` through `user_db` on the
assumption that RLS would scope the rows. That table has RLS disabled and no
policies, so any workspace member could pass a job id and read that run's
relation paths, row counts and error text. It now joins `v2_job`, which does
carry per-user policies, so a caller who cannot see the job sees nothing —
the same pattern `v2_job_completed` reads need. Verified as a plain member:
the old query returned 6 rows for another user's run, the new one returns 0,
while the job's owner still sees all 6.
The CLI's local graph still forced `in_pipeline` on every dbt script, so
`pipeline docs --local` and `pipeline dev` kept presenting a dbt project as a
pipeline the deploy no longer enrolls. It now skips them, matching the server.
A dbt descriptor has no asset parser locally, so nothing is lost: its models
come from the manifest the deploy derives.
Declares `run_progress` in openapi.yaml so the frontend uses the generated
client instead of a handwritten fetch; the generated `status` union also
replaces a hand-rolled string mapping.
* fix(dbt): drop the dbt node from the CLI's deployed pipeline views too
`pipeline dev` and `pipeline docs` (without `--local`) read `/assets/graph`
directly. That endpoint is asset-usage driven rather than membership driven, so
it returns a dbt script like any producer — and both commands render every
runnable, so a dbt project still showed up as a pipeline script there after the
local builder stopped emitting one.
`hideDbtRunnables` mirrors the frontend's projection of the same payload. It is
generic over the graph shape so the bounded-cascade view (`BCGraph`, a narrower
type over identical JSON) passes through without a cast.
The relations stay: they are what a downstream pipeline script reads, and the
node is what attributes them to a producer for every other consumer of the
endpoint, so the filter belongs in the views rather than the query.
* fix(dbt): narrow a selective run's cascade, settle the finished run graph
Review-round fixes.
A `select`/`exclude` run builds part of the project, but asset dispatch reads
the deploy-time write set for the whole script, so a run selecting one model
woke the subscribers of every other. Dispatch now intersects that set with the
relations the run actually recorded as materialized, scoped to dbt because it is
the only producer whose write set is decided per run. A run that recorded
nothing still dispatches everything, so an agent worker whose reconciliation
failed cascades as before. Verified both ways: `select: [extra_model]` no longer
wakes the `fct_orders` subscriber, and a full run still does.
`hideDbtRunnables` keyed its removal set on path alone while the graph keys
runnables by `(usage_kind, path)`, so a flow sharing a path with a dbt script
lost its node, edges and triggers too. Both copies now key on the pair.
The run graph never took a final reading when a job finished, so the last state
shown was whatever the tick before completion saw. Only `dbt-core-1x` streams
node events; the other engines record every relation during end-of-run
reconciliation, so their finished graph showed nothing until a reload.
`DbtNodeOutcome::Inconclusive` collapsed statuses the tally has to tell apart,
so two sites re-lowercased the status beside the classifier and `no-op` landed
in `totals.error` — a clean run reporting an error in its own result. Split into
Warn / Skipped / NoOp / Unknown so every site falls out of one match; `no-op` is
kept out of the retry set, which dbt spells as error / fail / skipped.
Also: reattach two doc comments to the items they describe, and correct the
engine-distribution table — only dbt-core-2x is baked into the images, 1.x is a
per-adapter venv provisioned on first use, and the default is compiled in rather
than an instance setting.
* fix(dbt): make the model chip inert where its project node is not on the graph
The canvas passed `onDbtSelect` unconditionally, so the chip always rendered
`cursor-pointer` and hover-highlighted — but the owner map is empty on both
graphs this feature added, since the run page carries no runnables and the
pipeline page hides the dbt node. The chip advertised a click that resolved to
nothing. It now takes its handlers only when the relation has an owner on this
graph, so it stays live on the surfaces that do show the project node.
`classify_status` and `DbtNodeOutcome` were `pub` in a private module with no
caller outside the file, unlike every neighbour.
* fix(dbt): take the cascade's write set from the run's own result
The previous narrowing read `materialized_partition`, which was wrong twice.
That table keeps one row per relation and the newest writer takes `job_id`, so
two overlapping runs over the same model erase each other's claim to it: the
earlier job would dispatch a subset of what it built, or none of it.
And an empty row set was read as "recording failed, dispatch everything" when it
is also a real answer. A `select` matching no model, or one resolving to tests
only, exits 0 having built nothing — and then woke every consumer of every model
in the project, which is the opposite of what the narrowing exists to do and is
reachable by a typo in a run argument.
The run now reports the relations it materialized in its own result, which is
immutable and per job. Absent means the producer said nothing (a job from before
the field, a non-dbt producer) and the whole deploy-time set dispatches as
before; present-and-empty means it built nothing and dispatches nothing.
Verified on all three: an unmatched selector builds nothing and wakes nobody, a
selector naming one unsubscribed model wakes nobody, and a full run wakes the
subscriber.
Also indexes `materialized_partition (workspace_id, job_id)` -- the run page
polls that shape every 2s and no existing index leads with `job_id` -- corrects
the selective-cascade section of the design doc, which still described the old
deploy-time behavior, and reattaches `buildLocalPipelineGraph`'s doc comment.
* docs(dbt): attach the CLI JSDoc to its function, correct the index rationale
The `hideDbtRunnables` JSDoc ended up documenting the type declared beneath it —
made while fixing the same mistake one function down.
The migration's comment credited the cascade with a `job_id` lookup that the
same commit replaced with a read of the job's own result. The run page's poll is
the only reader keyed on that column.
* fix(dbt): refuse graph publication for a removed script, allow test-only retries
An archived or hard-deleted script could still republish its graph: the
publication guard filtered `deleted` but not `archived`, and treated a missing
row as "nothing newer exists" rather than "nothing left to publish for". A
dependency job or dynamic run finishing after the removal put the asset,
provenance and subscription rows back with nothing left to clear them.
`dbt_command: retry` refused a run whose only failures were tests, which is
precisely what `test_behavior: after_all` produces. That restriction existed
because a successful job dispatched its whole deploy-time write set, so a
test-only retry would have woken every consumer for relations no one touched —
the cascade now dispatches what the run reports materializing, so it wakes
nobody and the restriction only blocked a legitimate retry.
* fix(dbt): gate run progress behind the job-read check, not RLS alone
The endpoint joined `v2_job` so RLS would decide visibility, which it does — but
`require_job_read_access` adds two things RLS does not: a scoped token's
`if_jobs:filter_tags` restriction, and the app-embed cutoff that stops untrusted
app JS from inheriting the viewer's broader job access. A scoped or embed token
could therefore read relation names, statuses, row counts and errors for jobs
the ordinary job endpoints deny it.
That helper is private to `windmill-api`, which depends on `windmill-api-assets`
rather than the reverse, so the endpoint moves to the job routes instead of the
check being duplicated. It is job-scoped anyway:
`/w/{ws}/assets/run_progress/{job_id}` becomes
`/w/{ws}/jobs/run_progress/{id}`, and the frontend follows the generated client.
* feat(dbt): a dbt run does not trigger downstream runs
dbt orders its own DAG, so a cascade only ever adds one thing: waking a Windmill
script that reads a mart. That edge is narrow, and only half of it can even be
expressed — nothing outside dbt can declare a `table://` write, since
`// materialize` accepts DuckLake targets only, so an ingestion script cannot
wake a dbt project.
Against that, dispatching correctly is not cheap. A run's `select` can build any
subset of the project, so the deploy-time write set is not what ran; using it
wakes consumers of relations the run never touched, and narrowing it needs a
per-job record of what was built. The per-relation state table cannot supply one
(it keeps a single row per relation stamped with the last writer), and the
result field added for it made a run's own output carry the cascade's bookkeeping.
So `asset_dispatch` returns early for `ScriptLang::Dbt`, before the producer
gate. dbt still materializes, records per-model state and publishes its graph:
models, `ref()` lineage and live run progress are unchanged, and a
`# on table://<mart>` reader still renders beside the model it reads. It simply
does not fire. Wiring it up later means deciding what a selective run should
notify, which is the actual work.
Verified: a full run of a 6-model project succeeds and starts nothing, where it
previously triggered its subscriber; the run page still reports all 6 relations
and the folder graph still carries 14 tables and 8 ref() edges.
* fix(dbt): remove the cascade surface, settle stranded models, fix nested __mod
Stopping dispatch left its surface behind. `table://` was still an auto-trigger
kind, `persist_ingest` still derived subscriptions from a manifest's reads, and
the deploy still accepted `# on table://` — so the canvas drew cascade arrows
into scripts nothing could wake. All three are gone: the kind no longer derives,
the ingest only deletes rows earlier versions wrote, and the deploy refuses the
annotation with a message saying why rather than persisting a silent no-op.
`DescriptorTriggers` went with them; every field it parsed was cascade config.
A model marked `running` by the live tailer was never settled when the run did
not finish: reconciliation only revisits nodes `run_results.json` names, and a
cancelled or timed-out run has none for the model in flight, so the finished job
showed a relation building forever. It is now settled on every exit path.
Verified by cancelling a run mid-flight: 3 models `running` before, 3 `failed`
after, none stranded.
`getScriptBasePathFromModulePath` took the first matching suffix rather than the
outermost boundary, so `proj__dbt/models/legacy__mod/a.sql` resolved to
`proj__dbt/models/legacy`. dbt owns its directory names verbatim, so a folder
ending `__mod` is legal inside a project, and a module-only sync would have
looked for a descriptor that is not there and skipped the deploy.
* fix(dbt): colour a finished run's models from its own result
`materialized_partition` keeps one row per relation stamped with whichever job
wrote it last, so reopening a run showed only the models no later run had
touched since — down to none for an old run, which reads as a broken page rather
than as stale data. Reproduced: a 6-model run reported 6 relations, then a second
run rebuilt one shared model and the first reported 5.
A finished run already carries the answer. Its result lists every node with a
status, and the graph carries each asset's dbt `unique_id`, so the two join
directly — no path derivation, nothing stored twice, and nothing a later run can
overwrite. The endpoint stays for the live window, where the result does not
exist yet, and as the fallback for a run that never produced one (cancelled or
killed, whose relations the worker settles in the table instead).
`relationOutcome` mirrors the worker's `classify_status` so the colour drawn over
a record agrees with the record: `warn`, `skipped` and `no-op` leave the relation
untouched and stay uncoloured, as do tests and analyses, which match no asset.
Verified in the browser on the run whose model had been stolen: all six
relations green again, both sources correctly uncoloured.
* docs(dbt): record why only dbt-core 1.x has live per-model progress
`emits_node_events()` reads as "the Rust engines produce no node events", which
is false and would close off the option. They produce exactly the same events;
they put them on the console and ignore `--log-format-file json`, which both
accept. Measured on 2.0.0-alpha.5 and fusion 2.0.0-preview.202: 15 node events
each on stdout, 0 in the file log, for a three-model project.
Taking them means owning the job log's presentation to work around a flag that
is documented and simply unimplemented, so the note records the measurement, the
sample event, and that flipping the predicate is the whole change once either
engine honours it.
* fix(dbt): give HighlightCode a dialect-agnostic sql language
`npm run check` had three errors the fast check does not reach: `"sql"` is not a
value `HighlightCode` accepts. Every SQL dialect it knows maps to one grammar,
but a dbt model is compiled by whichever adapter the project targets, so naming
a dialect would be a guess — `sql` is now a value in its own right.
`langOf` was typed `string` and returned `markdown`, `python` and `text`, none
of which the component accepts either, so a dbt project's YAML and Python files
rendered unhighlighted. It now returns the component's own prop type, which is
what caught them, and `undefined` for what has no grammar rather than a name
that silently means the same thing.
Verified in the project panel: SQL 22 tokens, YAML 27, where YAML was plain.
* fix(dbt): stop failing no-op models, drop table triggers client-side, keep cross-selection edges
The sweep that settles a run's stranded relations was marking `no-op`, `warn`
and `skipped` models FAILED on successful runs: reconciliation reports those
nodes without settling their record, so they were indistinguishable from a model
the run never reached. It now excludes every relation the run accounted for, so
only the genuinely abandoned ones are settled.
`table` was removed from the backend's auto-trigger kinds but left in both
client mirrors, so the editor and `pipeline dev`/`docs` kept drawing cascade
arrows the deploy will not create.
`isModuleEntryPoint` scanned for the first `__mod/`, the same bug its sibling
just had: a `legacy__mod/script.ts` nested in a dbt project — dbt owns those
names verbatim — read as that script's entry point. Both now anchor on the
outermost boundary.
A script selecting a model whose parent another script builds dropped the parent
entirely, so no `dbt_edge` could reach it and the two relations sat on the graph
unconnected. The parent is now kept as an endpoint and recorded as a READ, since
this script does not build it — splitting a project across selections only
composes if the seam still draws.
* fix(dbt): don't double-run after-all tests, count only models a script builds
An `after_all` run whose test phase failed saves a `run_results.json` holding
tests alone. Retrying it reran exactly those tests — and then the test phase ran
the whole suite again, appending a second copy of every result: duplicate ids in
the run table, doubled totals. A retry whose saved results are tests alone IS
the test phase, so the suite is not run after it, and the two phases now merge
by node id rather than concatenating.
Keeping a selection's unselected parents as nodes made them count toward the
`×N` badge, whose tooltip says "materializes N models" — a script selecting one
mart claimed the staging models upstream of it, and the number grew with the
seam. The count now comes from the relations the script writes.
That change also made the cross-selection read block dead, with a comment
asserting the inverse of what now happens; it is removed, and the test that
covered it still passes on the new arm. The test I added landed between a
neighbouring test's comment and its `#[test]`, orphaning the attribute so that
test stopped running.
Two display fixes: the run page no longer shows a relation's SQL when the
provenance belongs to another project that materializes the same relation, and
the editor no longer draws an explicit `# on table://` arrow the deploy refuses.
The starter descriptor no longer promises the removed cascade.
* fix(dbt): clear untouched models, reject unknown descriptor fields
A `no-op` model was left `running` forever on a successful run. The previous
attempt at this stopped the sweep marking such models FAILED but gave them no
terminal state instead, so they simply never settled. Reconciliation now returns
what it settled and what the run reported but did not build, and the two get
opposite treatment: a relation the run left untouched has its row DELETED, which
is what the finished run's own result says about it (`relationOutcome` colours a
`no-op` nothing), so the live and settled views agree; only a relation the run
never reached at all is failed.
The descriptor accepted unknown fields, so `selcet:` was ignored and left an
empty selection — building the whole project — and a misspelled `target` fell
back to the profile's default. It rejects them now. That immediately caught two
of our own test fixtures still passing `repo:`, a field removed with the git
path, which is exactly the class of mistake it exists to stop.
`isDbtModulePath` matched `__dbt/` anywhere in a path, the third site with that
bug: `foo__mod/vendor/x__dbt/a.ts` read as a dbt project file, and the push then
looked for `foo.script.yaml` and could skip the edit.
A verbatim dbt bundle dropped any file named `*.lock` before it reached the
module map, so an authored `uv.lock` never deployed and the unmodified-project
round trip quietly lost it. The exclusion now applies only to `__mod` bundles,
where `.lock` really is the script's own lockfile — in the walker that hashes
modules too, or a change to such a file would not register as one.
Also: `langOf` fell back to `undefined`, which HighlightCode resolves to
TypeScript rather than to no highlighting, so seeds and Markdown were coloured
as code; and four comments still gave the removed cascade as the reason for
sharing an asset node, which is now lineage.
* fix(dbt): retry failed tests too, anchor the last __dbt path check
`retry_failed_nodes` only ran after the model phase, which fails before the
`after_all` test phase exists — so a project whose models built and whose tests
failed got no retry at all, exempting exactly the failure mode that separate
phase produces. The loop is now a function, called after both phases.
`isDbtGeneratedPath` matched `__dbt/` anywhere, the fourth site with that bug:
`foo__mod/vendor/x__dbt/target/a.ts` counted as generated dbt output, so
`ignoreF` excluded an ordinary module file and a module-only edit never deployed
its parent script.
`wmill sync push` still dropped an ADDED or DELETED `.lock` three branches
before the module arm, so the earlier fix only covered a first push: adding a
`uv.lock` to a deployed project was reported as a change forever and never
applied, and deleting one left it deployed. Editing worked, which is what made
the round trip look whole.
Also removes a duplicate `#[test]` that was double-registering a test and
detaching its neighbour's comment, and rewrites seven comments that still gave
the cascade as the reason for behaviour that now serves lineage only.
* fix(dbt): bound the excluded-file read, keep the retry budget job-wide
`isBundledModuleFile` read a file in full before deciding it was too big or
binary, so a project sitting next to a multi-gigabyte parquet seed loaded the
whole thing only to reject it. It now takes the size from `stat` and reads at
most the 8 KB the NUL check needs: a 191 MB file is rejected in 0.0ms at 82 MB
RSS.
Calling the retry helper after both phases gave each its own `attempts` budget,
so a job could spend double what the descriptor asked for — the bound exists
because every attempt is a real dbt invocation holding a worker slot. The budget
is now the job's, spent across whichever phases fail, and the field says so.
Extracting that helper had also placed it between `#[allow(clippy::
too_many_arguments)]` and `run_dbt`, taking the attribute off the 12-argument
function it was written for.
* fix(dbt): actually spend the retry budget
`retry_failed_nodes` looped on `while *remaining > 0` and never decremented it,
so a failing job reissued `dbt retry` — logging "attempt 1 of 3" each time —
until the job's deadline instead of `attempts` times. The decrement existed
briefly and was lost when the function was re-extracted by hand.
Claiming and counting are now one operation, `claim_attempt`, because keeping
them apart is exactly how the bound goes missing: the loop cannot iterate
without spending the budget.
Its test is bounded by its own `for` rather than by the function under test. An
earlier version collected `std::iter::from_fn(|| claim_attempt(..))`, which
against a non-spending `claim_attempt` is an infinite iterator — it allocated
until the machine died. A test for a loop bound must fail an assertion when the
bound regresses, not consume the host: it now reports `[1, 1, 1, …]` against
`[1, 2, 3]` in 0.00s.
* fix(dbt): ask before reading, not after
Bounding `isBundledModuleFile` did nothing for the bundle builder, which read
the whole file into memory and only then asked whether to keep it — so a
multi-gigabyte seed beside a project was still loaded in full just to be
skipped. The predicate is now consulted first, and the read happens only for
files the bundle actually carries.
* feat(dbt): animate the ref() edges feeding the model being built
The nodes moved during a run but the edges did not, so the graph showed where
dbt had got to without showing it flowing there.
Reuses the canvas's existing rule rather than adding a second one: an edge
animates when it touches what is happening. For a pipeline that is the running
script; for dbt the unit of work is the model, so a `ref()` edge animates while
its target builds. Same `animated` field, same visual language, no new styling.
Verified mid-run on a 7-model project: of six `ref()` edges only the two feeding
the model then building were animated, and none once the job finished.
* feat(dbt): show what each model wrote, and say when its SQL is another project's
Three things a reader wanted from the run graph and could not get.
Row counts: the worker already records one per relation and `run_progress`
already returned it, but the graph used only `status` and dropped the number. A
model that built green having emitted zero rows is the failure that looks like a
success, so the count is on the node.
The relation's fully-qualified name, copyable: there is no table browser to open,
so the next best affordance is the exact identifier to paste into a SQL client.
It is parsed with `splitRelation`, which honours quoting the way the worker's
`split_relation` does — splitting on every period renders
`"wh"."analytics.v2"."orders"` as a relation `orders` in a schema `v2`, which
does not exist.
And when two projects materialize one relation, the graph keeps a single
provenance winner, so the losing project's node carries the other's model. The
SQL was already suppressed there — correctly, it is not this run's code — but
silently, which reads as a dead click. It now says so.
* fix(dbt): a finished run's graph is the models it built, not today's project
`/assets/graph` is the current deploy, so an old run's graph drifted with the
project: a model added after it appeared as though the run had built it, and the
older the run the wronger the picture. A finished run's node set now comes from
its own result, which named exactly what it touched.
Sources survive the filter regardless — dbt never lists them in
`run_results.json` because it does not build them, but they are the upstream the
run read, and dropping them would leave the models hanging.
The graph is still the current deploy's, so a model renamed or deleted since
cannot be drawn at all. Rather than a silently shorter graph, the count is
stated above it.
Verified by adding a model after a run: the old run renders 7 models without it,
a fresh run renders 8 with it.
* feat(dbt): preview a model's rows with `dbt show`
There was no way to see the data behind a node — only its SQL and its row count.
`dbt show` selects from a model and returns rows, and every engine ships it, so
the preview needs no adapter code of ours: no connection path, no dialect-correct
quoting, no type coercion for ten warehouses. It runs against the profile the
run already renders.
It is a `dbt_command` rather than a new endpoint, so it inherits the whole job
path — authorization, isolation, cancellation, logs, engine provisioning — and
`limit` joins the run form beside it. That the allowlist can admit it at all is a
consequence of dropping the cascade: while a successful job dispatched its
deploy-time write set, a command that wrote nothing woke every consumer for
relations nothing had touched.
Read-only, and treated as such: no graph republish, no materialization records,
no retry state, no test phase. Captured rather than streamed, like `dbt ls` —
these rows are the result, not commentary, and the job-log writer is what
`NO_LOGS_AT_ALL` discards.
Verified: `{"dbt_command":"show","select":["stg_customers"],"limit":3}` returns
three rows; a preview leaves `materialized_partition` untouched (62 → 62, 0 rows
for the job); `clean` is still refused by the allowlist.
* feat(dbt): preview a model's rows from the graph, and keep our locks out of dbt projects
The run page could show a model's SQL and how many rows it wrote, but not the
data. Selecting a model now offers "Preview rows", which runs the script with
`dbt_command: show` and renders the result as a table.
Explicit rather than on-select: a preview is a job, so it costs a worker slot
and the engine's start-up, and previewing on every click would spend both on
mere navigation. Sources are excluded — dbt shows what a model SELECTs, and a
source is not one.
Also: `updateModuleLocks` was the one module helper that never learned about
verbatim bundles, so it walked a dbt project writing `foo.lock` beside `foo.sql`.
None of those files is a Windmill script needing a lockfile, and the bundle
promises to round-trip the project byte-for-byte — our artifacts have no business
in it.
Verified in the browser: selecting `stg_customers` and previewing returns the
columns `id`/`src` and five rows from the warehouse.
* fix(dbt): keep a run's models when another project owns their provenance
Scoping a finished run's graph to the ids it named dropped relations whose
provenance winner belongs to a different project — so a run of a project sharing
a schema showed 3 of the 6 models it had built. An id that was never this run's
package cannot be judged against its result, so it is kept: the relation IS one
the run wrote, and hiding it understates the run. The same rule applies to the
"no longer in the project" count, which otherwise reported deletions that were
only provenance collisions.
Previews are now cached per model and survive the selection moving. One was
thrown away whenever the reader clicked elsewhere, which for a job costing a
worker slot and an engine start-up meant re-running it to see it again — and the
run continues in the background, so leaving and returning finds the rows there.
The spinner also never span: `startIcon` takes the icon and its classes
separately, so the animation has to be passed alongside.
How long it took is shown with the rows. A preview is a job, and its cost should
not be something the reader has to guess at.
* fix(dbt): resolve argument references, clamp the show limit, flag renamed relations
`handle_dbt_job` cloned `job.args` where every other executor calls
`build_args_map`, so a `$var:` / `$res:` / `$encrypted:` argument reached dbt as
the literal string. A placeholder holding a schema or an `enabled` flag would
then build a different slice of the project than the caller asked for.
`--limit` took any positive i64, and the worker buffers the whole of dbt's
stdout to read the rows out of it — so a caller with only run permission could
make it hold an unbounded allocation. It is clamped to a ceiling now, extracted
as `show_limit` so the bound is pinned by a test rather than inline in an async
function nothing can reach.
And a model keeps its id when its alias or schema changes, so an old run's node
showed today's relation while the run wrote another — the page asserting it had
materialized a table that did not exist yet. The run's result carries the
relation each node actually wrote, so the drift is detectable without a graph
snapshot, and the count is stated above the graph. Rendering the run's own
lineage still needs a per-job snapshot; this stops the page claiming otherwise.
* fix(dbt): stop persisting resolved secrets, bound the preview by bytes
Resolving `$var:` / `$res:` / `$encrypted:` for dbt — added in the previous
commit — meant `save_run_state` wrote the resolved PLAINTEXT into
`dbt_run_state.args` and the worker's `state.json`. The row outlives the job, so
a secret stayed in the database and a later `dbt_command: retry` replayed it
after the grant was revoked or the value rotated. The invocation now carries the
args as submitted alongside the resolved ones, run state persists those, and the
restore path resolves them again under whoever is retrying.
Clamping `--limit` bounded the row COUNT, not the size: one column can hold a
megabyte, so a thousand rows is a thousand megabytes, and `run_capturing`
buffers all of it. The captured output has a byte ceiling now.
`limit` became a built-in argument without joining `RESERVED_ARG_NAMES`, so a
descriptor writing `{{ limit }}` was silently handed the preview control's
default instead of being told the name is taken.
Two display fixes: the relation-drift banner compared a canonicalized (lower
case) asset path against the warehouse's own spelling, so it fired on every
model of every finished Snowflake run; and caching a preview's failure left
`Preview rows` dead for that model until reload.
* feat(dbt): key the graph by script version so a run renders its own project
The dbt graph was keyed by path alone, so a deploy overwrote the only copy and a
run page could only ever show today's project — an older run rendered today's
models, SQL and `ref()` lineage no matter what it had run. My previous attempt
filtered that view to the ids the run named, which stopped it lying but could not
show what was gone: the data no longer existed.
`dbt_node` / `dbt_edge` now carry `script_hash` in their primary key, so each
deployed version keeps its own graph, and the run page passes the version its job
recorded. Per DEPLOY, not per run — ten thousand runs of one version share one
graph — and a composite FK to `script (workspace_id, hash)` with ON DELETE
CASCADE means a version's graph dies with it. Nothing pruned these before,
because there was one copy per path; they would otherwise have accumulated with
no sweep.
Two deploys of one path now write disjoint rows, so the graph can no longer be
lost to a race. `claim_graph_publication` remains only for what is still
path-keyed — the `asset` usage rows — and an older deploy finishing late records
its own graph before declining to touch those, where before it published nothing
at all.
A pinned request is scoped by the version's own nodes rather than by `asset`:
that table describes the current deploy, so scoping through it would filter a
model out of the very run that built it.
Verified end to end: deployed v1 (8 models), ran it, deployed v2 with four models
removed and one rewritten. The old run renders 8 models, 6 ref() edges and v1's
SQL; a new run renders 4 and the v2 rewrite.
* fix(dbt): scope graph cleanup to one version, bound the preview capture
Archive and delete both act on a single `hash`, but the graph cleanup they
called deleted every row for the path. Now that the graph is keyed per
version, archiving an old version erased the live one's models, SQL and
lineage, and nothing repaired it. Both callers have the path in hand, so the
by-hash wrapper is gone and they use the version-scoped clear directly.
`dbt show` checked its 8 MB ceiling after `wait_with_output` had already
buffered everything, so the ceiling could not bound what the worker held.
`run_capturing` now reads both pipes incrementally against a caller-supplied
limit and kills the child on overflow. The read buffers are heap-allocated:
as arrays they were baked into the future, which the job poller boxes several
layers deep, and that overflowed the worker thread's stack — a `dbt show` run
aborted the whole worker process.
A retry's `dbt parse` ran on the arguments as submitted while the build ran on
resolved ones, so a `$var:` shaping the graph parsed verbatim. The parse moves
to the caller, after resolution.
A run that names its own `select`/`exclude` now drops the descriptor's
`selector`: dbt resolves `--selector` instead of `--select`, so passing both
made a preview of one model return another's rows.
Also: log instead of silently swallowing a `modules` column that fails to
deserialize (pre-existing, but for dbt it means running with no project at
all); keep the model SQL reachable once a preview has landed; render which
node the rows came from; stringify object-valued cells; document
`dbt_script_hash` in the OpenAPI spec.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs: record the per-worktree dev environment and the backend-run check
Three mistakes this guidance would have prevented, each of which cost a cycle:
A worktree has its own database and ports, but AGENTS.md stated the
single-checkout defaults as facts. Pointing `DATABASE_URL` at another
worktree's database makes `cargo sqlx prepare` fail on every query touching a
table your migrations added — and it deletes `.sqlx/` before it fails, so the
cache is gutted rather than merely stale. Starting a backend on the wrong port
leaves the UI up with every call 502ing, which reads as an application bug.
Both values are now discoverable with commands that work as written.
`prepare` is also documented as the wrong tool for a removal-only change: the
cache is already complete for CI, and the only residue is orphaned entries that
can be found by text-matching against the sources without a database.
Nothing told a reader that `cargo check` does not exercise a worker path. A
read buffer declared as an array inside an async block is baked into the
future, and once boxed by the job poller it overflows the worker thread's
stack — compiling and unit-testing clean while aborting the whole worker
process at runtime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): scope the remaining path-wide reads and clears to one version
Four places still spoke for a whole path after the graph became per-version:
The relation-root drift check read `dbt_node` by path with an unordered
`LIMIT 1`, so with v1 at root A and v2 at root B it could answer with v1's
row, suppress the refresh v2 needed, and leave v2's graph naming relations the
run does not build. It now reads this job's version.
`dbt_dep`'s no-resource branch cleared the path, so a descriptor edited to
bring its own `profiles.yml` emptied every earlier version's graph and with it
every finished run's page. The ownership being given up is the path-keyed
`asset` usages cleared beside it; the graph clear is now this version's.
The graph was inserted before the publication claim checked the version was
still live. Archive and delete only soft-update `script`, so the foreign key
still accepted an in-flight dependency job's rows and the failed claim
committed them — and because pinned queries deliberately serve archived
versions, deleted model SQL became readable again. The write is now gated on a
`FOR UPDATE` liveness check.
`clear_dbt_run_state_by_script_hash` resolved a hash to a path and deleted the
path's saved run. `dbt_run_state` is keyed by path by design — one saved run
per script — so archiving one version discarded the live version's resumable
failure. It clears only once no live version of the path is left; `identity`
already refuses a resume whose project, warehouse or engine moved.
"Preview rows" ran `runScriptByPath` while the SQL beside it was pinned to a
hash, so an old run showed its own SQL over today's rows. Verified end to end:
with v3 deployed, the v2 run's preview runs v2's hash and returns v2's rows.
Also: keep the TAIL of a captured stderr, since dbt prints its summary last;
one `$derived` for the parsed result rather than five; collapse three
near-identical argument accessors onto one generic; fold the single-use
`copy_dir_command` into its caller; and give `parseDbtRun.ts` one status
classifier instead of spelling dbt's failure vocabulary twice.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(dbt): one JobCtx down the executor, one table per adapter
Two changes aimed at the operations this code will keep having: adding a
phase, and adding a warehouse.
`JobCtx` already bundled the five values every phase needs, and nine functions
took it — but the top of the executor threaded the fields apart and rebuilt the
struct at each call, so the same literal appeared eight times and each new
phase meant five more parameters. It is now built once per entry point and
reborrowed. `prepare_project` goes from 21 parameters to 17, `retry_failed_nodes`
from 15 to 11, and `run_dbt` drops below the lint threshold. The two remaining
constructions are the worker boundary, where the pieces genuinely arrive apart.
`DbtAdapter` answered five questions with five parallel matches over the same
eleven variants, plus a sixth list of adapters kept by hand in a test. The
facts now live in one `AdapterSpec` per adapter, reached through one exhaustive
match, so adding a warehouse states its name, driver, package, port, database
key and licensing together and the compiler demands the arm. Each arm spreads
from a Postgres base, which makes the inheritance visible per adapter instead
of hidden in the `_ =>` defaults `default_port` and `database_key` used to
carry. `DbtAdapter::ALL` replaces the list the test kept separately.
Verified by dumping all seven facts for all eleven adapters before and after:
byte-identical.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): job-keyed run progress, and stop path-wide reads and clears
Six findings from the last round, in the order they bite.
The publication liveness gate refused on `archived`, but `create_script`
archives the parent on every redeploy — so deploying v2 while v1's dependency
job was still parsing left v1 without a graph, permanently, which is the exact
case the unconditional write existed to serve. It gates on `deleted` alone now;
an explicit archive is still covered by the `FOR UPDATE` ordering.
A project-owned `profiles.yml` trusted `profile.type` instead of reading the
file. The Rust engines carry every adapter, so a CE script could declare
`postgres` over a target that is `sqlserver` and have dbt connect with the
enterprise adapter. The file is read whichever way, and a descriptor that
disagrees with it is refused.
Renaming a dbt script, or editing one so its newest version is no longer dbt,
cleared the graph for the whole path — every older version's models, SQL and
lineage, which their own finished runs still render. Neither needs it: graph
queries join on `(path, hash)` through a `language = 'dbt'` CTE, so an old
version's rows cannot attach to whatever lives at that path next.
Live progress read `materialized_partition`, whose key is the relation and
whose `job_id` is only the last writer. Two runs of one project took rows from
each other. Progress now has its own job-keyed table; the relation table is
untouched, because one row per relation is right for the pipeline canvas and
fork defer. Verified with two overlapping builds: both keep 6 rows in the new
table, while the old one attributes 6 to one run and 0 to the other.
`Scratch::drop` removed a half-installed virtualenv synchronously from inside
the job future, blocking a runtime thread; it goes to `spawn_blocking`, with a
direct call when there is no runtime to hand it to.
The E2E list asked for a `# on table://` subscription the deploy now refuses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(dbt): snapshot a dynamic descriptor's graph per run
A `{{ }}` placeholder in `vars` can enable a different set of models per run, so
those runs re-ingest the graph. Keyed by version alone, each re-ingest
overwrote the last: reopening an older run showed the newer run's project, and
a model only the older run built was gone entirely — no SQL, no lineage, and
nothing the saved result could colour, since it can only tint nodes that are
there.
`dbt_node` / `dbt_edge` gain `job_id`. A run of a dynamic descriptor writes its
own snapshot under its job id and its page reads it back; a static descriptor
writes the version's graph once, under a zero-UUID sentinel, and every run of it
reads that. The sentinel is a value rather than NULL because `job_id` is part of
the primary key and Postgres does not treat two NULLs as the same key, so each
re-ingest would add a row set instead of replacing one.
`/assets/graph` takes `dbt_job_id` and prefers a snapshot when one exists,
falling back to the version's graph otherwise — so a run page passes it
unconditionally and static descriptors are unaffected. Snapshots age out after
30 days, pruned by the runs that write them, so no background sweep has to learn
about these tables.
Verified end to end: one deploy, two runs of it with `extra=yes` and `extra=no`
gating a model's `enabled`. The version's graph holds 6 models, run 1's snapshot
7 including `opt_extra`, run 2's 6 without it; the endpoint returns each run's
own and falls back to the version's when the parameter is omitted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(dbt): only snapshot a run whose graph differs, and prune from every run
Two costs the per-run snapshot carried, both found by measuring it rather than
by reading it.
A snapshot was written for every run of a dynamic descriptor, but marking one
dynamic is conservative: `graph_is_per_run` is true whenever `vars` holds a
`{{ }}` placeholder or `env` holds a `$var:`, which says the arguments reach dbt
and not that they change which models exist. The usual case is a date var, whose
graph is identical run after run, so the table filled with copies of an
unchanging picture — around 1 KB per model per run, which is a gigabyte or so a
month for a 200-model project on an hourly schedule. A row set now carries a
digest of its nodes, edges and relation root, and a run whose digest matches the
version's writes nothing; the read already falls back to the version's graph, so
those pages are unchanged. Only a run whose model set really differs pays.
The prune was hung off the progress reporter, which exists only for engines that
emit node events — so a Fusion or dbt-core-2x instance accumulated snapshots and
never deleted any. Retention that stops working because of an engine choice is
not retention; it runs detached from every dbt run instead.
Verified against a descriptor with a var-gated model: the version's graph holds
8 rows, a run that resolves to that same graph stores none at all, and a run
that enables the extra model stores its own 9. Both pages still render their own
project — 6 assets without the extra model, 7 with it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): scope every dbt_node join to the chosen snapshot
`job_id` joined the key, but only the scoping CTE and `dbt_edge` were taught to
filter on it. The outer node SELECT and the parent/child joins in the edge query
were not, so each model came back once per retained snapshot plus once for the
version's graph, and each edge matched every combination of the two — the model
count multiplied and the edge join fanned out quadratically. Measured against
one stored snapshot: 17 node rows where 8 are wanted, and 28 edge pairs where 7
are. The response dedup hid the edge blow-up from the payload, not from the
plan, and the run page refetches the graph every two seconds.
The progress table gained writers it was missing. `terminalize_running_relations`
settled only the relation-keyed table, so a cancelled or killed run — the case
that function exists for, since it leaves no `run_results.json` — showed every
in-flight model still spinning on the run page for as long as the row lived. An
agent worker cannot write the new table at all, having no database of its own,
so the read falls back to the relation-keyed one when a job has no rows there.
Also: a wrapped string literal missing its backslash put eighteen spaces in the
middle of the profile-disagreement error; a comment still described concurrent
runs of one dynamic version overwriting each other's graph, which is what
keying by job removed; and the `materialized_partition` index justified itself
by a run-page poll that has since moved to another table, though the closing
sweep still earns it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): give a graph snapshot a marker row, and scope what reads it
Five findings, four of which are the same mistake in different places: a
snapshot's identity was inferred from its contents.
Existence was inferred from a `dbt_node` row, so a dynamic run that disabled
every model — a legitimately empty graph — read as "no snapshot" and its page
showed the deployed models instead. The digest was a column repeated on every
node and read back with a `LIMIT 1` carrying no `job_id`, so a run could compare
itself against another run's digest and suppress a snapshot it needed. The
relation-root drift check read the same rows unscoped, so after a drift it could
find a previous run's root and conclude nothing had moved.
`dbt_graph_snapshot` holds one row per stored graph — path, version, job,
digest, timestamp. Existence is that row, the digest lives there once, the drift
check reads the deployed row explicitly, and the retention sweep deletes markers
first and then the rows no marker stands for. The digest is SHA-256 rather than
`DefaultHasher`, whose output is documented as unstable across Rust releases:
this value outlives the process that computed it, so a toolchain bump would have
silently stopped every comparison matching and quietly reinstated the duplicate
snapshots the digest exists to prevent.
`/run_progress` ignored the view token, so a share-link viewer got the graph and
was refused the progress that colours it.
A preview sent only its own three arguments, so a descriptor with a required
`{{ }}` var could not be previewed at all and an overridden one previewed a
different relation than the page was showing. The run's arguments go first now,
with the preview's three overriding.
Verified on a project whose only model is var-gated: the deploy stores a marker
with zero nodes, a run with the var set stores a marker with one, and the
endpoint answers 0 and 1 respectively rather than showing the deployed models
for both.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(dbt): squash the runtime's migrations into one
Ten migrations reshaping the same three tables is a history no installation
ever had. `dbt_node` gained `script_hash`, then `job_id`, then `ingested_at`,
with its primary key rebuilt twice; `graph_digest` was added by one migration
and dropped by the next after the digest moved to its own table. On a fresh
database all of that replays to arrive at a shape the schema can simply state,
and this feature has never shipped, so there is no upgrade path to preserve.
One migration now creates `dbt_node`, `dbt_edge`, `dbt_graph_snapshot`,
`dbt_run_state` and `dbt_run_progress` in their final shape, carrying forward
the rationale each of the replaced migrations recorded. The enum additions stay
in `add_dbt_lang`, since a value cannot be added and used in one transaction,
and the `materialized_partition` index stays separate because it belongs to a
table this feature did not introduce.
Verified by rebuilding: dropped the five tables, replayed from the single
migration, and confirmed the result is identical — same primary keys, the same
two composite `script` foreign keys, the same seven indexes. Every `sqlx::query!`
in the workspace then compiled against it, which checks each column's name, type
and nullability, and a deploy plus run on the rebuilt schema produced 8 nodes,
7 edges, a snapshot marker and 6 progress rows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): authorize snapshot reads, bound the prune, give the marker a lifecycle
`dbt_job_id` is caller-supplied and selected straight from `dbt_graph_snapshot`,
which carries no RLS — so a caller who could see the script could read any run's
model set and relation paths, which a dynamic alias or schema can encode. Both
graph queries now require the job itself to be visible, in the authed
transaction, the same gate `raw_code` already applies to the script that
produced it.
The drift check compared against the deployed graph alone, which misses the way
back: a run at root B republishes the path-keyed `asset` usages at B, and
returning the profile to A then matches the deploy and skips the refresh,
leaving those usages at B while dbt builds A. It reads the most recent ingest
for the version instead — the one that last wrote them — ordered rather than an
arbitrary `LIMIT 1`.
The prune anti-joined every non-deployed node and edge with no age predicate, so
each run scanned the whole retained sidecar and concurrent runs duplicated it.
All three deletes share one age bound again, with the sentinel spelled as a
literal so the partial indexes apply — a bound parameter cannot be proven to
match the index predicate.
`dbt_graph_snapshot` was the one dbt table nothing in the script lifecycle
deleted: no `script` foreign key and absent from both `clear_dbt_manifest*`
sites. A marker outliving its rows is read as a snapshot with no nodes, and its
digest still answers the suppression check, so an identical run would write
nothing and then render an empty graph. It cascades like the rows now and both
clears take it.
Also: the preview cleared `exclude` rather than inheriting it, since previewing
a model the run excluded reached dbt as `--select m --exclude m`; and
`terminalize_running_relations` no longer claims to cover a killed worker, which
never reaches it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): quote profile names, record where usages were published, stop polling the graph
A profile name comes from the project's own `dbt_project.yml` and a target from
the descriptor, and both were interpolated into `profiles.yml` as bare YAML —
including as mapping keys. A name like `prod # hidden` truncates the mapping and
a newline opens a sibling key of the author's choosing. Both are rendered as
quoted scalars now, as are the BigQuery keyfile's keys, with a test that asserts
the document still parses to exactly the keys we wrote.
The drift check read the most recent ingest, which latches: a run that returns
to the deployed root re-ingests but stores no snapshot (its digest matches the
version's), so the moved run's rows stay newest and every later run pays an
extra parse and ingest. The publisher now records the root it published the
path-keyed usages at, which is the only thing that answers "where do the current
usages point" — the deploy's own root goes stale as soon as a run republishes.
The run page polled `/assets/graph` every two seconds alongside progress, so it
re-sent every node's SQL for the length of a run — hundreds of KB a tick on a
real project, for a graph that a dynamic descriptor re-ingests exactly once
before the build. It fetches once more shortly after mount and then polls
progress alone.
Node results carry `outcome` beside `status`. `status` stays dbt's own word, but
dbt owns that vocabulary — 1.x and 2.x differ on casing and `no-op` arrived in a
minor release — so publishing only it would force a break or a lie the first
time it moves. `outcome` is the stable half a downstream script branches on.
Also: the worker's dbt entry points are `pub(crate)`, since nothing outside the
crate calls them and they resolve secrets and launch processes; and the snapshot
gate records that it is RLS-only where `/jobs/run_progress` also honours a
share-link token, which is a gap in what a shared page shows rather than in what
it protects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(dbt): pin the graph storage invariants against a real database
Every defect review found in this area was DB-shaped — which row set a read
resolves to, which rows a clear takes, whether a snapshot exists at all — and
none of it is reachable from a unit test on a pure function. Four rounds
established these answers and nothing guarded them, which is why each round kept
finding another.
Six cases, on the harness the repo already uses for schema-shaped behaviour:
an identical run stores no snapshot and leaves no marker; a differing run keeps
its own while the version's is untouched; an empty run graph is still a snapshot
rather than an absent one; clearing one version leaves the others whole; the
path-wide clear takes the markers with it; and the sweep ages out run snapshots
while never touching a version's own graph.
`IngestedNode` gains `Default` so a test can state the two fields a case is
about rather than the eighteen it is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* perf(dbt): bound a script's stored graphs by deploy count
Run snapshots expire on a clock, but a VERSION's graph could not: its reader is
every finished run of that version, and a run page is as old as its job. So
nothing reclaimed them — a deploy graph went only when its `script` row was hard
deleted, which Windmill does not routinely do. A CI deploying on every commit
added a full model set with SQL bodies per commit, forever: roughly 200 KB a
deploy for a 200-model project, which is gigabytes a year across an instance.
Bounded by COUNT instead of age, since age is the thing that cannot be right
here. The newest 50 deploys per path keep their graph and older ones are
reclaimed, making growth `versions x models` rather than unbounded in time.
Generous on purpose: reaching the bound empties that version's run pages, so it
exists to stop unbounded growth rather than to be hit in normal use. Ordered by
the script's own `created_at`, so a late-finishing job re-ingesting an old
version cannot promote it.
Pinned by a test that deploys past the bound and asserts both halves: the count
holds, and the newest version is always among the survivors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): let the run page know when its snapshot has landed
The one-shot graph refetch was wrong: a dynamic descriptor's ingest happens
before the build but after cloning, dependency install and parse, so a fixed
delay either fires too early — and the run page then shows the deployed models
for the whole run, never that run's own — or keeps re-sending the whole graph
for the length of it. Neither is a timing problem to tune; the page had no way
to tell "the snapshot is not written yet" from "this run has none".
`/assets/graph` answers that directly: `dbt_snapshot_job` is the job the dbt half
resolved from, when one was asked for and found. The page polls the graph until
that is its own job, and stops. A static descriptor never snapshots, so an
attempt cap ends it there rather than polling for the run's duration.
`dbt_node.relation_root` is gone. The drift check moved to the marker's
`published_relation_root`, which left the column written on every node and read
by nothing.
`outcome` was published as the stable half of the result contract, but the
in-tree consumer still ranked and coloured from dbt's own word — so the field
existed and nothing used it. `statusRank` takes it, `DbtRunResult` passes it, and
`classifyStatus` is documented as the fallback for results that predate it and
for the live event stream, which carries dbt's word alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(dbt): record what a share-link viewer actually sees
The comment at the snapshot gate said the graph "falls back to the deployed
set", which is only the rarer half of it. A share link is an extra grant for a
logged-in user who lacks access to the job, so the usual case is no read on the
script either — and then the `live` CTE matches nothing and the whole dbt half
comes back empty. A blank Models panel over working progress rows, not a
fallback.
`docs/dbt-runtime.md` now carries the analysis a follow-up needs: that relaxing
this leaks nothing, because `v2_job_completed.result` already gives that viewer
every node's `unique_id` and `relation_name` — the graph's only incremental
exposure is `raw_code`, which is gated separately on seeing the script. And the
shape of the fix: `OptViewToken` and `validate_view_token` are self-contained
enough to move into `windmill-api-auth`, which `windmill-api-assets` already
depends on, after which the gate can honour a token for that job's snapshot
alone while `raw_code` stays where it is.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): keep model SQL behind the scripts:read scope, and unbreak CI
`/assets/graph` is authorized as `assets:read`, and RLS decides whether the
caller can see the script that produced a node — but RLS is not a scoped
token's grants. A token deliberately narrowed to `assets:read` could therefore
read model source and repository paths for scripts outside its `scripts:read`
paths. The same `build_scope_path_predicate` the macro endpoint already applies
now gates `raw_code` and `original_file_path`; the relation's shape is
unaffected, only its body is withheld.
`DbtAdapter::ALL` exists for the tests that must cover every adapter, so it is
dead in a release build and `-D warnings` failed all four backend checks on it.
It is `#[cfg(test)]` now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): stop the graph poll at the ingest, and type `limit` in the schema
The poll's stop condition was a snapshot appearing, with a 40-attempt cap
behind it — so a STATIC descriptor, which never snapshots, took the cap every
time and re-fetched the whole graph forty times. That is most of what removing
the poll was meant to save, and static is the common case.
The ingest runs BEFORE the build, so the first model to report progress proves
it has already happened: a snapshot absent by then is one this run never
writes. Progress arriving is now the second exit, and the cap is only a
backstop for a run that reports none at all.
`limit` is declared `Typ::Int` but `dbt_arg_schema` had no integer arm, so the
run form and the generated clients saw an untyped default and offered no
numeric control for a value the worker clamps. Covered by the schema test.
`relationOutcome` still re-derived from dbt's word while `statusRank` had moved
to `outcome`; both read it now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): make a retry prove its arguments still resolve the same
The saved arguments are the ones SUBMITTED, so a `$var:` in them is re-resolved
on retry. The identity did not cover the resolved values, so a variable that
changed between the failed run and the retry was accepted — and which graph the
retry then used depended on WHERE it landed: a worker holding the local
snapshot replays the saved manifest, while a database restore reparses with the
new value. Placement decided whether the resumed failures described the
relations being built.
The identity gains a digest of the resolved arguments, and is compared in two
halves because resolution happens between them. Project, warehouse, engine and
env are checkable up front; the arguments are not, because a retry request
carries only `dbt_command` and the ones to compare are the SAVED arguments after
this caller has re-resolved them. Comparing the whole string up front would have
refused every retry — which is what the obvious version of this fix does.
A row written before the digest existed has no last segment, and still restores
rather than being refused.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): keep pre-upgrade retries working, and stop losing a late snapshot
Splitting the identity on its last `|` read a pre-upgrade row's env digest as an
arguments digest and left only `<run_identity>` as the prefix, so every saved
failure on an upgraded instance became unretryable — a regression the previous
commit's own test missed by using an identity with no `|` in it at all, which is
not what an old one looks like. The digest is tagged (`|args=`) rather than
positional, and the test now uses a real pre-upgrade identity.
The graph poll gave up after a bounded number of tries, but provisioning and
`dbt deps` precede the ingest and can outlast that on a cold worker — and the
engines that emit no node events never produce the progress that ends it early.
A finished run now reloads the graph unconditionally, and the poll's own exit
issues one last load: progress proves the ingest happened, not that the previous
tick saw it, and dbt's compile window is wider than one tick.
A `dbt retry` restores the failed run's arguments inside the worker and they are
never written back to the retry job, whose own args are just
`{"dbt_command": "retry"}` — so previewing a row on a retry's page ran without
the vars the run used. The result now carries the invocation's arguments as
SUBMITTED, so a `$var:` stays a reference and no resolved value is published.
The deploy-count sweep ran instance-wide on every dbt run: `FROM script WHERE
language = 'dbt'` has no index to stand on, and both orphan deletes are the
complement of every partial index here. It is scoped to the running script's
`(workspace_id, path)` — which `index_script_on_path_created_at` serves — and
the orphan deletes only run when a marker actually went.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): hide dbt from module-less pickers, and stabilise the retry digests
`processLangs` feeds every language picker, including flow steps and app inline
scripts. Those are raw bodies with nowhere to carry a module bundle, and a dbt
script IS its bundle — so choosing dbt there produced a job that could only fail
once the worker looked for `dbt_project.yml`. Those two surfaces use
`processInlineLangs`, which drops the languages that need modules; a flow still
reaches dbt the way it reaches any script, by path to a deployed one.
`graph_digest` moved to SHA-256 because it is persisted and compared by a later
worker, and `DefaultHasher` is documented as unstable across Rust releases — but
the retry identity's own digests were left on it, and they are persisted in
`dbt_run_state.identity` for exactly the same comparison. A toolchain bump would
have refused every saved failure as a different project. All three go through
one `stable_digest`, length-prefixed so no split of the same bytes collides.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): enforce the tag scope on snapshot reads, reset state between runs
A tag scope is an orthogonal hard restriction: a token limited to some tags must
not read a job outside them however else it is authorized. The snapshot lookup
went through `v2_job` RLS alone, which knows nothing about tags, so such a token
could still retrieve a run's model set and its dynamic relation paths. The same
predicate `require_job_read_access` applies for the progress half of the page is
applied here — `get_scope_tags` is already public in `windmill-api-auth`, and it
is `None` for an unscoped caller, so a normal session pays nothing.
SvelteKit reuses the run graph between run ids, and `graphTries`, `polled` and
`raw` all describe the previous job: a spent retry count stopped the next run's
snapshot poll before it began, and stale progress coloured its models with
another run's statuses. All three reset when the graph key changes.
Also a wrapped string literal missing its backslashes, which put two ~22-space
runs in the middle of the retry-refusal message.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* docs(dbt): put each digest helper's rationale on its own function
Inserting `stable_digest` above `split_identity` split that function's doc, so
five lines describing where the identity divides ended up introducing the
hasher. Each is back on the function it describes, stated once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(dbt): read the run-pinned graph through the job, not the asset graph
Pinning the asset graph to one run is job-scoped data, but `dbt_job_id` sat on
`/assets/graph`, authorized as `assets:read`. The job-read contract —
`require_job_read_access` — is five parts that pull in opposite directions (tag
scope restrictive, `created_by` permissive, app-embed restrictive-overriding,
view token permissive, RLS underneath), so plain RLS is neither a stricter nor a
looser approximation of it. Restating the parts near the graph query kept leaving
one out: first the job check entirely, then the share-link asymmetry, then the
tag scope, and the app-embed restriction was still missing and failing open.
The helper cannot be called from `windmill-api-assets`, because `windmill-api`
depends on that crate. So move the read instead of the check: the run-pinned
graph is now `GET /w/{w_id}/jobs/dbt_graph/{id}` in `windmill-api`, on the same
gate as the `run_progress` it colours, and `/assets/graph` has no `dbt_job_id`
parameter at all.
- `asset_graph_for` takes the job as an argument from an already-authorized
caller; the route handler passes `None`.
- Extract the graph response into a `AssetGraph` component schema, now that two
paths return it.
- The run page fetches the job route when it has a job id.
* fix(dbt): charge assets:read on the run-graph route, trust the job gate in SQL
Round 13 findings on the route moved last commit.
The scope domain comes from the URL segment, so putting the read under `/jobs`
asked a scoped token for `jobs:read` alone while returning asset-graph data that
`/assets/graph` charges `assets:read` for. A token narrowed to polling run status
could read workspace topology, and the missing-job fallback made it cheaper still
— any random UUID skipped the job gate. Both scopes are now required: the job
gate reaches this run, `assets:read` reaches asset data at all.
The `chosen` CTE re-decided job visibility under plain RLS after the caller had
already passed `require_job_read_access`. It could only disagree, and did so
silently by falling back to the deployed graph — a share-link viewer entitled to
the run was shown a different run's model set. Dropped; the contract is that a
job reaching `asset_graph_for` is already authorized.
Also: the flow editor's `+` insert menu still offered dbt (the third
`processLangs` caller, missed when the other two moved to `processInlineLangs`),
the docs still described the deleted `dbt_job_id` parameter, and the new handler
had again been inserted between `get_run_progress`'s doc comment and its
function.
* fix(dbt): resolve a pinned run's version from the job row, not script RLS
Local codex review of the branch.
A share-link viewer is entitled to the run and usually has no grant on the
project — that is what the link works around. The graph's `live` CTE resolved the
version by selecting `script` inside the viewer's RLS transaction, so it answered
for their access to the project rather than for the run they were given: the
Models panel came back blank beneath working progress rows.
A pinned run now takes its path and hash from the job row the handler already
read after authorizing the job, so `live` does not consult `script` at all.
`raw_code` keeps its own `EXISTS` against `script`, so the model bodies stay
behind access to the project. Verified under RLS as an unprivileged role: the
shape query goes 0 rows -> 1, the `raw_code` gate stays 0.
Taking the version from the job also means a caller can no longer pin one
project's version while naming another's run, since `dbt_script_hash` is ignored
when a job is given.
The run page's graph fetch is a raw `fetch`, which bypasses the interceptor that
adds `X-View-Token` to generated-client calls, so a shared page was refused
before any of this mattered; it goes through `appendViewToken` now.
Also trims three comments to the AGENTS.md limit, dropping drafting-history
rationale that belongs in docs/dbt-runtime.md.
* fix(dbt): snapshot vars-overridden runs, carry the pinned version everywhere
Second local codex pass.
A `vars` run argument overrides the descriptor's, and vars drive `enabled`,
alias, schema, database and materialization — so such a run builds relations the
deployed graph does not describe. It now snapshots under its own job id, which
per-job keying makes safe: the version's graph stays for runs that did not
override. The old comment claimed gating on it would strand the override's graph
for the next default run, which was true only when the write went to the
deployed slot.
Two sites still read the caller's `dbt_script_hash` instead of the version
resolved from the job, so `/jobs/dbt_graph/{id}` without that redundant
parameter dropped models the run's version had and a later deploy removed.
The `dbt_snapshot_job` marker re-checked `v2_job` under RLS — the recheck the
graph query itself drops. A share-link viewer got the right graph and a null
marker, so the run page refetched it 40 times before giving up.
`wmill script preview` read the bundle with the generic `__mod` suffix and
script-module parsing, so previewing a `.dbt.yaml` omitted the project and failed
on the missing `dbt_project.yml`. It uses the same suffix and verbatim read as
deploy.
* fix(dbt): key retry state by principal, not by script path alone
`dbt_run_state` held one row per (workspace, script path), and a retry replaces
the caller's arguments with the saved ones. Anyone able to run the script could
therefore retry whoever ran it last, replaying that run's literal `select` and
`vars` against the warehouse and publishing them as their own job's
`invocation_args`. Running the script was already theirs to do; seeing another
principal's arguments was not.
`permissioned_as` joins the key, so a retry resumes only state written under the
same authority. Two runs sharing an authority can already act for each other, so
this is the boundary that matches the rest of the job model.
* test(dbt): pin what a caller without access to the project sees of its run
The share-link case had no regression guard, and every fix in this area touched
one of its two halves: the graph's SHAPE has to survive a caller who cannot read
the script, and the model SQL must not.
Two cases against a real database, calling `asset_graph_for` as a member with no
grant on the project's folder: pinned to a run, the models render and `raw_code`
is withheld; unpinned, the same caller sees nothing of it, so making the first
work did not relax the second.
Both assertions were checked by mutation — reverting the `live` bypass empties
the graph, and dropping the `raw_code` script gate leaks `select 1` — so neither
passes on the code it is meant to catch.
* fix(dbt): key the worker-local retry cache by principal too
Keying `dbt_run_state` by `permissioned_as` left its worker-local twin keyed by
workspace and script path alone, so the boundary held only where the database row
was consulted. An agent worker never reads that table — `Connection::Http` leaves
`latest_job` as `None` — so there the local cache was the whole boundary and it
had none: the next principal to retry the script on that worker restored the
previous one's `select` and `vars`.
Also records the sqlx `--all-targets` trap in the update-sqlx skill: it is needed
for queries inside tests, and in a CE checkout it aborts on `tests/otel.rs`
(EE-only `otel_ee`) after having already emptied the cache.
* fix(dbt): log a dropped retry-state save, correct the run-progress contract
Saving retry state is best-effort — losing it costs a retry, not the run that
just finished — but `.ok()` dropped the reason too. The only symptom was `dbt
retry` reporting nothing to resume, which reads as a bug in retry rather than a
failed write. Found by running a real failing build against a worker whose
binary predated the `permissioned_as` column: the insert violated NOT NULL and
said nothing.
The run-progress endpoint's OpenAPI description promised an empty list for a
caller who cannot see the job. It is refused instead; an empty list means the job
recorded nothing yet or is unknown here.
* fix(dbt): return the retry-state write failure the warning was added to report
`save_run_state` discarded the insert result, so the caller's warning could never
fire and a lost retry row stayed silent — the symptom being `dbt retry` finding
nothing on another worker.
The error is held rather than returned at once: the worker-local copy is what an
agent worker resumes from, so a failed insert must not cost that too. Every exit
after it surfaces it, including the ones that give up on the local save.
* fix(dbt): decide a retry's graph from its restored args, keep local state in step
Three from the seventh local review.
A retry submits only `dbt_command`, so the vars-override check ran against an
empty argument set and left `graph_is_per_run` false. The failed run's arguments
are restored afterwards, and those are what the retry builds with — an overridden
one wrote no snapshot for its own job and its page fell back to the deployed
graph, showing the wrong enabled models, aliases and schemas. The decision is
re-asked once the restore has happened.
A failed durable write no longer publishes the worker-local generation either.
`restore` accepts a local generation only when the database row names it, so
publishing one the database never recorded made this worker reject its own newest
state and resume the previous run's — its selection and vars, or "nothing to
retry" if that one had succeeded. An agent worker attempts no durable write, so
it keeps its local copy as before.
A rename that also converts away from dbt moved the old path's retry state onto
the new one, reinstating what the conversion had just cleared and leaving one
user's arguments and results under a path no dbt script occupies. It moves only
while the destination stays dbt, and clears the source otherwise.
* fix(dbt): drop retry state when a run produced none, let module pushes fail loudly
Three from the eighth local review.
A run that never wrote `run_results.json` — cancelled, timed out, or dead before
dbt got there — left the PREVIOUS run's state authoritative in both the database
and the local pointer, so a later `dbt retry` resumed that older invocation's
failed nodes. Producing nothing resumable now clears both copies, so neither can
answer for the other.
`wmill sync push` wrapped the descriptor lookup and its deployment in one
try/catch meant for a missing parent. Any API failure or invalid descriptor was
reported as "no parent found" and swallowed, so a module-only push exited zero
with the remote project unchanged. Only the lookup is tolerated now.
Also condenses a comment that narrated how earlier status comparisons behaved.
* fix(dbt): forget retry state on pre-build exits too, drop cascade claims
A dynamic run whose pre-build `dbt parse` or graph ingest fails returns before
the save that clears stale state, so the previous run stayed authoritative in
both the database and the local pointer and `dbt retry` resumed ITS failed nodes
— writing relations the run that just failed never touched. Both exits now
invalidate, through one helper shared with the no-artifact case.
Two frontend comments described dbt producer rows as driving cascade dispatch.
The executor returns before dispatch for every dbt job and deployment rejects
`table://` subscriptions, so they promised behaviour that cannot occur; they
describe the lineage and ownership that is actually retained.
* fix(dbt): let the database decide retry state where it is reachable
A SQL worker treated "no `dbt_run_state` row" as no opinion and accepted any
worker-local generation. But no row is the authoritative answer that the last
invocation left nothing resumable, so a local pointer that outlived it — an
unlink that failed, a process killed between the delete and the removal, a stale
cache — resurrected a replaced run and let `dbt retry` write relations it never
touched. An agent worker keeps accepting its local copy: it has no authority to
consult.
Invalidation failures are logged rather than dropped, since a silent one is
exactly what leaves the pointer behind.
* docs(dbt): record how to run an agent worker locally, keep archived graphs
Every step of standing one up fails as something else: a normal build cannot
start one at all, the server's routes need a separate feature, and all three
token mistakes surface as a bare 401 on the agent with the reason only in the
server log. Written down with the error each produces.
Also keeps a dbt script's graph when it is ARCHIVED rather than deleted. The
pinned read resolves versions through a CTE that already skips archived rows, so
clearing bought nothing and emptied the Models panel of every completed run of
the project. Deletion still clears it.
* feat(dbt): let an agent worker publish its graph, through one endpoint
An agent worker was refused any dbt script whose profile comes from a Windmill
resource — the common case — because it could neither read the stored relation
root to check for drift nor re-ingest a corrected one.
Those look like two needs but collapse into one: verification exists only to
decide whether the stored graph still describes reality, so a worker that can
PUBLISH never has to ask. It stores what it just parsed.
`POST /api/agent_workers/dbt_graph/{workspace_id}` is the whole addition. It
wraps the same `replace_dbt_manifest` the SQL path calls, so digest suppression,
the marker write and retention cannot drift between the two transports, and it
refuses a job the token's tags do not cover. `IngestedManifest`/`IngestedNode`
gain Deserialize to cross the wire.
Two guards go, both now false: the pre-build refusal, and the `Connection::Sql`
gate added earlier to stop a `vars` override grounding an agent run.
Live progress stays SQL-only — that is a per-model event stream, and routing it
through the API would mean a round trip per node.
* docs(dbt): warn that a differing cargo feature set swaps the shared binary
* docs(dbt): record the verified agent-worker behaviour and the tmpfs quota trap
An agent worker now runs a dbt job end to end, retries, and publishes its graph
— confirmed with a dynamic descriptor whose per-run snapshot came back through
the new endpoint. The doc said it was refused; that was true before the endpoint
existed.
Also `WINDMILL_DIR`: on a dev box the job dies with `Disk quota exceeded (os
error 122)` writing the project's files while `df` shows free space AND free
inodes, because /tmp is a tmpfs carrying a per-USER quota. Point the worker at a
real disk rather than trying to clean up beneath it.
* chore(dbt): pin the EE revision carrying the agent graph endpoint
* fix(dbt): bind the published graph to the job, break the completed-page poll loop
Five from the thirteenth local review.
The EE endpoint took `script_path` and `script_hash` from the payload and checked
only that the supplied job carried one of the agent's tags, so an agent holding
any matching-tag job could name another script and replace its graph. Both are
read from the verified queue row now and the request carries only the job id. A
raw preview has no version, so it no-ops rather than 422ing before dbt runs.
`IngestedManifest`/`IngestedNode` take `#[serde(default)]`: they were
serialize-only, and a field the serializer skips made the whole manifest
unparseable on the receiving side.
A completed run page fetched the graph forever — `load()` assigns `raw`, which
recomputes `settled`, which re-entered the same effect. The final fetch is keyed
to the job by a plain (non-reactive) variable, and `settled` is read untracked.
The pin now names a revision that compiles: the previous one still called
`authed.tags()`, a method that does not exist, because both that fix and the JSON
response landed after it was committed.
* fix(dbt): keep a run snapshot out of the script's deployed ownership
Everything `persist_ingest` writes after the manifest is keyed by PATH — one row
set per script, describing what is deployed there. A run snapshot was still
reaching it, so a one-off `vars` override republished that invocation's relations
as the script's ownership and the workspace graph stayed on the override's
schemas and aliases: an ordinary run of a static descriptor never ingests again
to correct it, so only a redeploy would. A snapshot now stops after recording its
own rows.
The row preview also selected a bare model name, which dbt resolves across every
installed package while `show` takes a single node — a project model sharing its
name with a package's was previewed wrongly or refused. It selects the
package-qualified FQN.
* fix(dbt): forget stale retry state when preparation itself fails
`prepare_project` runs before every path that could clear it, and it fails for
reasons unrelated to the saved run — a profile that stopped resolving, a
provision cancelled, packages that will not install. The invocation still left
nothing resumable, so the previous one must not stay authoritative: a repaired
project would otherwise let `dbt retry` rebuild an older run's selection and
write relations the latest invocation never reached. A retry is exempt, since it
is trying to use that state and failing to prepare says nothing about it.
Also corrects the runtime doc, which still described agent workers as unable to
run dynamic descriptors or Windmill-resolved profiles. They publish their graph
through the API now; what they do not get is live progress and a durable retry
row, and the doc says so.
* fix(dbt): spell the whole FQN for preview, bound retained retry generations
The FQN selector added last commit was `<package>.<name>`, but a dbt FQN is the
resource's path within its package and the matcher must consume the selector and
end on equal lengths — so it matched nothing for a model under `models/marts/`,
which is the layout most projects use and the one this repo's own complex fixture
has. The middle segments come from `original_file_path`, whose first element is
the resource root the FQN excludes. Without a path it falls back to the bare
name: ambiguous across packages, but a selector dbt resolves rather than rejects.
Tested on a nested model, which is the input that separates the three spellings.
Superseded retry generations were removed only when a later run published one,
and never inside the hour-long grace period — so a burst left a manifest and a
results copy per run with nothing afterwards to collect them. At most four now
sit in the grace window, oldest evicted first.
* fix(dbt): scope preview state to the run, seed the project on a language switch
Previews are keyed by `unique_id`, which is the same string for the same model in
every run, and the run-change effect reset only the graph and progress. Opening a
second run of one project therefore showed the previous run's rows immediately,
and `runPreview` treated them as cached and refused to fetch. A generation
counter also drops a preview that resolves after navigation, which the reset
alone cannot catch.
The dbt project was seeded only by the empty-script bootstrap, but dbt is in the
ordinary language picker: reaching it by switching a draft produced a script with
no `dbt_project.yml`, which the runtime refuses to deploy or run. Both entry
points seed now, and neither touches modules that already exist.
* chore(dbt): cache the agent graph endpoint's query for the EE offline build
* fix(dbt): publish the graph a moved profile built
A run snapshot stopped before everything `persist_ingest` keys by PATH, which is
right for a one-off `vars` override and wrong for the other two reasons a run
re-ingests. `graph_is_per_run` was one bool for all of them, and the profile
drift check both sets it and reads back what the publisher recorded: a profile
moved A->B was detected by every run forever, each paying a `dbt parse` for a
snapshot nobody reads while the asset rows went on naming schema A.
The reason is carried now (`GraphRefresh`), and it decides both writes. Drift is
the version's own move, so it rewrites the VERSION's graph and republishes the
ownership that ends the drift; a dynamic descriptor snapshots under its job id
and still publishes; anything the CALLER scoped — an overridden `vars`, a
narrowed `select` — snapshots and publishes nothing, so one invocation's subset
can neither stand as what the script owns nor drop the models it left out from
the version's graph. Where they meet the caller wins, and the next ordinary run
settles the drift.
A restore also rebuilt `run_results.json` by copying the generation directory a
second time, so a burst of saves pruning it mid-restore left `dbt retry` with
nothing to resume and a job that reported success. It is written from the bytes
the restore already read; a manifest that went the same way falls back to the
parse a database restore pays anyway, and a generation that vanished before
either read falls back to the database's row for that same run instead of
reporting there is nothing to retry.
* fix(dbt): select a row preview by package, not by file path
The preview built dbt's FQN by dropping one segment of `original_file_path`,
which assumes the model root is `models/`. A project setting
`model-paths: ["src/models"]` turned `src/models/marts/orders.sql` into
`pkg.models.marts.orders`, and dbt's matcher — equal lengths, compared from the
front — resolves that to nothing: the preview came back empty for every model in
the project.
It selects `<name>,package:<pkg>` instead. The comma is dbt's intersection
operator, so this names the node by its own name and the package it belongs to,
which is what the FQN was reaching for and needs no knowledge of the resource
root. Verified on dbt-core 1.12, dbt-core 2.0.0-alpha.5 and fusion
2.0.0-preview.202, including a package shipping a model whose name the root
project also uses.
* fix(dbt): refuse a lockfile version that is not one, keep a named selector
Two things a preview reaches that a deploy does not vouch for.
A raw preview submits its own `lock`, so `engine_version` arrives from the
caller and was interpolated straight into the engine cache path — `../..` in it
made the download, extraction and rename land anywhere the worker can write,
and provisioning runs on the host rather than inside the dbt jail. Both it and
`adapter_version` (a pip requirement) are now accepted only as a plain version
token.
`effective_selector` also read any submitted `select`/`exclude` as an override
of the descriptor's named selector. The generated run form posts a default back
for every field the caller left untouched, and a selector descriptor's `select`
default is `[]` — so pressing Test, saving a schedule or firing a webhook built
the WHOLE project instead of `--selector nightly`. An override is now one that
DIFFERS from the descriptor's own value; a run that wants the whole project
despite the selector asks with `["*"]`.
* fix(dbt): let a moved profile settle, from the runs that actually happen
Two ways the drift check could never come to rest, both verified against a real
run of a real project on a normal worker.
`add_caller_args` read any submitted `select`/`exclude` as a caller's narrowing.
The generated run form posts a default back for every field left untouched, so
every run from the UI, a schedule, a webhook or a flow step carried them and was
marked caller-scoped: with the profile moved A->B, each one stored its models
under its own job id and left the workspace graph — and the root the check reads
back — at A. Since no UI run omits the field, the "an ordinary run settles it"
escape hatch was unreachable. Both this and `effective_selector` now ask one
question, `selection_is_overridden`: DIFFERENT from the descriptor's, not merely
submitted.
The root was also recorded beside the path-keyed publication rather than beside
the graph it describes, so a version that cannot claim the path — an older one
run by hash, a deploy overtaken by a newer one — rewrote its graph at the moved
root and recorded nothing. Its next run then compared against a root that was
absent or two moves stale and skipped the refresh its own run page needed. It is
written wherever the deployed row's graph is.
Verified end to end: same UI-shaped arguments before and after, the moved
profile now republishes (asset rows and version graph both move to the new
schema), a second run detects nothing and re-parses nothing, and moving the
profile back settles it again.
`prune_dbt_run_graphs` also ran from runs alone, while a deploy writes a whole
node set of its own, `raw_code` per model included: a project redeployed on
every push by CI and run nightly kept one full graph per push until the next
run, and one deployed but never run kept them for good.
* fix(dbt): drop a self-dependent effect in the run graph
`previewGen` was `$state` written by the effect that also reads it, three lines
under a `finalLoadFor` that is a plain `let` for exactly that reason. Nothing
reactive reads it — the only reads are inside `runPreview`, a plain async
function — so it becomes a plain `let` too.
* fix(dbt): pin a retry to the engine versions it resolved
`run_identity` carried the engine KIND but not the version it resolved, nor the
dbt-core 1.x adapter's. Redeploy an unchanged project after a release and it
locks a newer dbt or adapter while the saved `run_results.json` still passes the
check, so `dbt retry` feeds one version's artifacts to another — the exact
reproducibility the lockfile exists to hold. Both resolved versions are in the
identity now; a real failure and retry still resumes.
Also drops three comments that outlived what they describe: two said `[]`
clears a descriptor's selector, which `selection_is_overridden` reversed, and
one pointed at an agent-worker guard that no longer exists — the agent path
reaches the ingest deliberately and publishes through the API.
* fix(dbt): discard a run graph the page has already navigated away from
The component is reused across runs, so a slow `/jobs/dbt_graph` or progress
response could land after the reset and put the previous run's models, statuses
and failure state on the current run's page, where nothing would fetch again to
correct it. Every response is now checked against the generation it was
requested under — the counter the preview path already used, renamed for what
it means.
The graph poll also backs off. Neither of its stops is reachable for a whole
class of runs — `dbt_snapshot_job` never matches a static descriptor, and
`polled` stays empty for the engines that emit no node events — so an ordinary
run walked to the cap, re-sending every model's SQL 40 times in two minutes.
* fix(dbt): forget the previous run when the durable save fails, keep quoting
`save_run_state` returns the database error when its upsert fails, which leaves
run N-1's row and local generation in place: same project, same arguments, so a
`dbt retry` matches them and resumes an older attempt's failed nodes against
this checkout — the outcome the no-results branch twelve lines above calls
`invalidate_run_state` to prevent, reached by another door. It now goes through
the same call. Best effort, since the delete goes to the database that just
refused a write, but the local pointer is what a retry landing back here reads.
The run page also rejoined a relation's parts after `splitRelation` stripped
their quotes, so the one name the button exists to paste —
`"wh"."analytics.v2"."Order Items"` — was copied as something no client
resolves. It copies `relation_name` verbatim.
And a source on a finished run was called another project's: the check that
guards against two projects claiming one relation asks whether this run executed
the node, and a run executes no sources — they appear in no `run_results.json`.
Nothing materializes a source, so that warning could never be true of one.
Docs: the `vars`-override paragraph still said such a run does not refresh the
graph, which the table above it contradicts — it refreshes under its job id and
publishes nothing.
* fix(dbt): keep the asset rows and the version's models describing one graph
The workspace graph takes an asset's relations from the path-keyed `asset` rows
and its models, SQL, tests and lineage from the version's `dbt_node`/`dbt_edge`.
A dynamic descriptor published the former while storing the latter under its own
job id, so a placeholder that moved an alias or a schema left the current graph
with assets no model stands behind — nothing dbt contributes to them survives.
Ownership is published exactly when the VERSION's graph was written now, which
is the only state in which the two agree. Two cases are settled elsewhere by
design: an override's relations are a one-off, and a dynamic descriptor at a
moved profile keeps the deploy's ownership until a redeploy — its runs each show
their own models and it re-parses regardless, so the undetected drift costs it
nothing it was not already paying.
An agent worker has no durable row, so its local `current` pointer is the whole
of what a retry reads — and every local publication failure returned success
with the PREVIOUS run's pointer still in place. Where a row exists that is
harmless (`restore` takes a local generation only when the row names it), so the
abandonment is scoped to the agent case.
A failed `dbt show` also cleared the retry state: the preparation-failure exempts
`retry` but not a read-only command, and the run page's row preview is exactly
that, run as the principal the state is keyed by — so a preview that could not
provision took the retry away from the run being looked at.
Frontend: the run-change reset left `loading` and `failed` behind, so the gap
before the next run's answer rendered "no models in the asset graph" — a claim
about the descriptor — over a project that is fine. And three derivations argued
from "the graph is the current deploy", which the pinned endpoint made untrue;
each is still needed, for the version-graph rewrite and retention reasons now
written down.
* fix(cli): let --skip-scripts cover a script's module files
The module shortcut in `elementsToMap` maps the file and `continue`s before
every skip filter, and a module is deployed as part of its parent script — so
`wmill sync push --skip-scripts` still pushed the script whenever one of its
modules changed, and pull still overwrote them locally. Harmless while a module
was a rare helper file; every file of a dbt project is one of these now.
* docs(dbt): a dynamic descriptor's ownership stays the deploy's
* fix(dbt): read the run out of a failure whose message has braces of its own
`parseDbtRun` anchored on the FIRST `{` in the error message and parsed
everything after it. The worker appends the structured result after the error
text, and dbt's errors carry braces — a Jinja template, the compiled SQL, an
adapter's own JSON — so the failures most worth reading were the ones whose
summary and per-node outcomes the run page dropped. Every brace is tried now,
bounded, and the first that parses as a run wins.
Pins the EE revision that gives the agent publish endpoint the deleted-version
guard the SQL path takes: deletion is soft, the foreign key still accepts graph
rows, and the pinned graph query serves non-live versions, so an agent finishing
during a delete put a deleted project's model SQL back on screen. The query is
byte-identical to `persist_ingest`'s, so the offline cache already covers it —
verified with a full-EE `SQLX_OFFLINE=true` check.
* fix(dbt): seed a project when a modular draft switches to dbt
`seedDbtProject` returned whenever the draft carried any module at all, so a
modular script holding a `helper.ts` reached dbt with none of what dbt needs:
the project view is read-only, and the worker refuses a bundle without
`dbt_project.yml`, so that draft could neither run nor deploy. Keyed on the
project file now, and the seed goes in under whatever is already there — the
previous language's helpers are inert to dbt and the user's to remove.
Also records this runtime's schema in `backend/summarized_schema.txt`: the
`table` asset kind, the `dbt` script language, the five dbt tables and the
`materialization_status` enum the progress table uses.
* docs(dbt): move the pipeline-membership rationale out of the deploy path
* fix(dbt): gate a pinned run's model SQL on the version it belongs to
The `EXISTS` against `script` is the only thing standing between a share-link
viewer and the project's source, and it matched the workspace and path alone.
`extra_perms` is a grant on a ROW: archive a version that granted someone
access, recreate the path with narrower permissions, and that stale grant
satisfied the probe while the query returned the NEW version's `raw_code`. Both
probes name the hash now. The regression test drives exactly that shape and
fails without it, returning `select 2` to a caller granted only on the archived
version.
* fix(dbt): keep a delimiter an identifier escaped by doubling
Every dialect these relations come from escapes its own delimiter by doubling
it, and both split functions closed the quoted section on the first half and
reopened on the second: `"schema"."a""b"` came out as `a.b`. The manifest keeps
the real spelling, so the run wrote its per-model status and row counts under an
asset path no graph node has — the node simply never moves, which is the failure
mode this splitter exists to prevent.
Fixed in the worker and in its frontend mirror, which have to agree, with a case
per delimiter on both sides.
* fix(dbt): key retry state by the caller, not only by the principal it runs as
An `on_behalf_of` script executes every caller's job as its owner, so
`permissioned_as` names one principal for all of them and the retry state — the
durable row and the worker-local generation both — collapsed onto a single
entry. After one caller's run failed, the next could submit `dbt_command: retry`
and resume it: their arguments replayed against the warehouse, and handed back
through `invocation_args`. Nothing else separated them, and on an agent worker
the local directory is the whole boundary.
`created_by` joins the key in both places. For an ordinary script it changes
nothing — `permissioned_as` is already that caller — and a run that was itself
superseded was never resumable anyway.
Includes the offline cache for the four changed queries and the three the
pinned-graph regression test added last commit, which had none: `prepare`
without `--all-targets` does not compile test targets, so CI's
`SQLX_OFFLINE=true ... --all-targets` would have failed on them.
* fix(dbt): compare the schema too when reporting a relation that moved
`relationDrift` compared the leaf name alone, and the move it exists to report —
a profile repointed at another schema, which a later run then writes into the
version's graph — leaves every model's name exactly where it was. So the one
case that reliably produces a graph naming relations this run did not write was
the one case the notice stayed silent for.
The schema segment joins the comparison, qualified against qualified: an
unqualified one means the target's own database, which the relation names
anyway, so comparing that would report a move on every node.
* fix(dbt): bound the retry state now that it is keyed per caller
Keying by `created_by` fixed one caller resuming another's run and created a
growth problem doing it: a shared `on_behalf_of` script kept one row and one
worker directory for everyone who had ever run it, and the generation prune only
bounds files INSIDE a directory.
Three bounds, none of them new machinery. A run with nothing failed or skipped
saves nothing — `dbt retry` builds from those nodes alone, so that state could
only ever be refused — while still clearing what the previous run left, since
its failures are no longer what last happened here. The rows expire on the same
30-day clock as a run snapshot, swept per path by the prune every dbt job
already spawns. And the worker-local directories are swept there too, by the age
of the pointer a save rewrites, because their digest names neither the script
nor the caller.
* docs(dbt): the retry state is worker-affine only on an agent worker
* fix(auth): only the server may set a token label that names a user
`create_token_internal` wrote `NewToken.label` verbatim, and the auth layer reads
some labels as an IDENTITY: `username_override_from_label` maps
`ephemeral-script-end-user-<name>` to exactly `<name>`, which then becomes
`created_by` on every job that token pushes. The label is free-form request
input, so any member could mint a token that speaks as somebody else — the shape
`require_job_read_access` already works around when it refuses to trust
`username_override` and falls back to an RLS probe, and the one that made dbt's
retry-state key (`created_by`) forgeable for an `on_behalf_of` script.
The labels are refused where request input enters: the member-facing
`tokens/create`, and `impersonate`, which names its subject in
`impersonate_email` and has no business renaming the caller too. The legitimate
producers are unaffected — a job's own token comes from `create_token_for_owner`
in the worker, and native triggers and app-embed tokens build their labels
themselves rather than accepting one.
`Ephemeral lsp token` stays allowed: its override is the fixed sentinel `lsp`,
not a name the caller chose, and the editor mints exactly that label through this
endpoint for its language server. The test pins the two lists together, so an arm
added to `username_override_from_label` that lets a label choose a name fails
until it is reserved too.
* Revert "fix(auth): only the server may set a token label that names a user"
This reverts commit
|
||
|
|
81b23a2ba0 |
feat: make the fork lineage the only deploy relationship (#10410)
* feat: make the fork lineage the only deploy relationship `workspace_settings.deploy_to` (2023) and `workspace.parent_workspace_id` (2025) both expressed "which workspace does this one deploy into". Fork creation and dev-workspace attach seeded both, but nothing kept them in agreement, so every reader picked one and they disagreed. Drop `deploy_to`. A migration folds surviving pairs into the lineage: a sole claimant on a target with no dev workspace becomes that target's dev workspace and keeps its own job tags, while many-to-one pairs become plain forks. Pairs that the lineage cannot express -- dangling target, self-reference, chain, mutual -- are reported and left unlinked. Job tags were never lineage-aware: `per_workspace_tag` mapped any parented workspace to its parent while `$workspace` interpolated the raw id, so a fork running a script tagged `<tag>-$workspace` produced a tag no worker serves and the job queued forever. Both paths now resolve to the nearest ancestor whose id an admin would provision workers for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve unconvertible deploy links and sweep tag caches on reparent Review findings on the deploy_to unification: - convert chains instead of discarding them, and keep whatever the lineage cannot express in workspace_deploy_to_unmigrated so the down migration can restore it - ignore soft-deleted workspaces when choosing between a dev workspace and a plain fork; an archived claimant was demoting live pairs - mirror attach_dev_workspace's git-sync strip, which the migration skipped - sweep the tag cache over whole subtrees on rename and delete: tag resolution now walks ancestors, so a nested fork kept a tag nothing serves - call a dev workspace a dev workspace in the settings copy - redirect a root away from ?tab=deploy_to instead of rendering an empty target Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: detect lineage cycles and record archived links in the deploy_to migration Second review round on the unification: - detect cycles over the lineage as it would exist after conversion, not over the deploy_to graph alone: a root whose target was one of its own forks closed a loop that no deploy_to edge revealed - record an archived source's link instead of filtering it out entirely, which dropped it with the column - treat a fork whose deploy_to merely repeats its parent as redundant rather than reporting every pre-existing fork as unmigrated - read the row count from the lineage update rather than the git-sync one - sweep the tag cache when archiving a dev workspace, the last site that mutates is_dev_workspace without one Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: resolve $workspace on preprocessed flow tags regardless of $args Third review round on the unification: - a flow tag containing only `$workspace` skipped interpolation entirely on the preprocessed path, because the branch that ran it keys on `$args`. The raw tag was written back and named a queue no worker serves. Resolve `$workspace` before the branch and leave `$args` to it. - record the new table's foreign key in the schema summary - describe what the archive tag sweep actually does: the dev flag is cleared for any archived workspace, which is why it is unconditional Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the deploy_to leftovers table only when it holds something * fix: sweep tag caches on archive only where the dev flag actually changes * feat: broadcast lineage changes and walk ws_specific ancestors only - propagate tag-cache invalidation across processes over notify_events: the cache is per-process, so replicas kept resolving stale lineage for the TTL. The listener clears the whole cache rather than tracking ids, since a single mutation invalidates an unbounded set of descendants and lineage changes are rare admin actions. - narrow list_ws_specific_versions to ancestors: walking down as well made a root fan out over its entire live fork subtree, and each member costs an identity lookup plus an RLS switch and probe. Ancestors are bounded by the fork depth limit. - probe the leftovers table unqualified so rollback restores on a PG_SCHEMA install, where search_path is not public - drop the nativets client method for the removed edit_deploy_to endpoint Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: let a prod see its dev workspace in ws_specific, and stop the walk oscillating Descending into plain forks made a root fan out over its whole live fork subtree, but a dev workspace is the paired editable environment rather than a throwaway copy, so a prod should still see it. There is at most one per parent and attach rejects nested dev chains, so that edge stays bounded. The edges run both ways, so the recursion never converged: it bounced parent<->dev until the depth cap on every call, 33 rows for a two-member set. A visited-path guard ends the walk when nothing new is reachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep dev pairings unnested, gate the delete broadcast, cover the ws_specific walk Fifth review round: - a root that already owns a dev workspace no longer converts: linking it under its deploy target would leave that dev nested beneath a fork, the shape attach_dev_workspace refuses to create. The link is preserved instead. - broadcast a lineage change on delete only when descendants are orphaned. Deleting a leaf, which ephemeral fork churn does constantly, changes nobody else's resolution and was making every replica drop its whole tag cache. - call list_ws_specific_versions in a test. plpgsql defers everything past a raw parse to the first call, so replaying the migration only proved it parses. - use unwrap_or_default for the descendant sweeps, which run after the transaction has committed; a transient failure must not fail the request - trim the traversal comment to the four-line limit Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: cache the renamed tally query and clear instance alerts on conversion The integration test's query was never cached: `cargo sqlx prepare` without --all-targets skips test targets entirely, and renaming its fixture workspace changed the query text. Regenerated with --all-targets --features all_sqlx_features,private, which is what lets the EE-gated otel test compile. Also from review: - clear error_handler_fallback_to_instance_alerts on converted workspaces. Dispatch ignores it once a parent exists, but the settings page keeps submitting the stored true, which the API rejects on a fork. - restore the schema summary row to the file's name: columns format and put it back in alphabetical order Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: never cache an unresolvable tag workspace, and unadvertise the removed endpoint - lookup_tag_workspace cached a "no row" result as self-resolution. A rename resolves the new id before its row lands, so a fork could be pinned to its own wm-fork-* id -- which nothing serves -- for the whole TTL, and its schedules kept re-pushing onto that dead tag. Fall back for the call without caching, matching how the error path already behaved. - change_workspace_id swept its children but never itself. Sweep the new and old ids and broadcast unconditionally, since a rename always changes lineage. - openapi-deref.{json,yaml} are served to clients via include_str!, so they were advertising edit_deploy_to after it started 404ing. The audit-action enum keeps the entry: historical rows still carry it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: align the served YAML spec with the JSON one and correct two comments - the YAML deref lost the removed path but kept deploy_to on get_settings, so the two served specs disagreed. Both are now identical. - the rename-sweep comment blamed cached-unresolvable lookups, which the same commit stopped caching. The real reason is that workspace ids are reclaimable, so a new id can carry a previous occupant's resolution. - the instance-alert comment claimed the settings page submits the stored true and gets a 400. It hides the option on a fork and sends false; the hazard is the value outliving the pairing and re-enabling alerts after a detach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 82da6cb2bafeda18acd6b70c599013a12117ecb0 This commit updates the EE repository reference after PR #694 was merged in windmill-ee-private. Previous ee-repo-ref: f9ddf6a75aa13d1c13a3d7216a361a96f75ca435 New ee-repo-ref: 82da6cb2bafeda18acd6b70c599013a12117ecb0 Automated by sync-ee-ref workflow. * fix: grant the deploy_to preservation table to the windmill roles * test: drop the one-shot migration tests, keep the ws_specific execution guard The two conversion tests replayed the migration against the fully-migrated schema, which is not how it runs -- in production it runs mid-sequence against the schema as of that point. A later migration touching workspace or workspace_settings would break them without breaking anything real, and sqlx checksums already freeze a released migration. They earned their keep finding the archived-claimant and nested-dev cases during development; there is nothing left for them to guard. list_ws_specific_versions is different: it is live, no caller exercises it, and plpgsql only parses a function body until first call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: invalidate a reclaimed fork id cluster-wide without flushing every entry Gating the delete broadcast on orphaned descendants stopped leaf churn flushing every replica, but fork ids are reclaimable: the deleting process invalidated locally while every other replica kept the old parent for the TTL, so a job pushed in a recreated fork routed to the previous parent's tag. The broadcast payload now carries meaning. A workspace id drops that one entry, used for leaf deletion where exactly one id changed what it denotes. The `*` sentinel drops everything, used for attach, detach, archive, rename and deletions that orphan descendants -- reshaping a subtree no single id names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: name the right broadcast for each invalidation case * docs: attach does invalidate the tag cache; the resolver walks the whole chain --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 5
|
4d3ff0299f |
feat: mark failed jobs as resolved so handled failures stop showing red (#10319)
* feat: mark failed jobs as resolved so handled failures stop showing red Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: constrain auto-resolve to the proven retry chain and honor resolved filter everywhere Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply resolved filter to queue-union, concurrency and delete paths, bound note Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: sweep resolutions on workspace delete, verify helper args, enforce UI limits Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: count resolution note in characters on both sides of the API Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: skip the queue lookup for cancel-all under the resolved-only filter Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: converge retry auto-resolution from either commit order, keep notes on re-resolve Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: correct the idempotency claim on the retry auto-resolve sweep Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: gate resolution notes and attribution behind enterprise, add note popover Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: hide resolution from operators, exclude flow steps, enforce EE licence at runtime Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: add job_resolution.automatic to the summarized schema Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: preserve stored attribution when re-resolving without a valid licence Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: condense the attribution-preservation comment to four lines Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: validate resolution notes by code point instead of a UTF-16 maxlength Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the resolution popover open when a note is rejected Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: offer to resolve the original failure after a successful re-run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: verify supersession server-side and stop re-runs overwriting notes Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: apply tag scope to the superseding run Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: exclude obscured cross-workspace runs from resolution actions Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
3bd9f05938 |
fix(alerts): identify server replica in low-disk alert + per-host dedup tag (#10143)
* fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
84141add1d |
feat(pipelines): workspace duckdb macro libraries (// macros / // use) (#9890)
* feat(pipelines): parse duckdb macro-library annotations (// macros, // use) * feat(pipelines): duckdb macro registry tables + deploy-path validation and writes * feat(pipelines): inject workspace duckdb macros into consumer jobs at run time * feat(pipelines): surface macro libraries and lib-consumer edges in asset graph api * feat(frontend): macro-library nodes, lib-consumer edges and scaffold in pipeline graph * docs: mark dbt gap #7 (packages/macros) shipped via workspace macro libraries * fix(pipelines): review fixes - char-safe parsing, local macros win, fork clone, trust-model docs * feat(frontend): duckdb macro autocomplete + workspace macro explorer drawer * fix(pipelines): address CI review - use-setup retention, splice past local defs, orphan filter, full consumer rescan, index-keyed strip * fix(pipelines): inject provider library setup for implicitly-called macros too * fix(pipelines): rls-gate macro listing + honor library-level // use transitively * fix(pipelines): weave injected macros around local definitions by bind order * fix(pipelines): injected library setup always runs before user blocks * perf(pipelines): cache macro registry per workspace with notify-event invalidation * perf(pipelines): disable macro registry cache on cloud |
||
|
hugocasaandClaude Opus 4.8
|
b4b0c6a93e |
feat: add dev workspaces paired with a lockable prod workspace (#9793)
* feat: add dev workspaces paired with a lockable prod workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate dev-workspace prod-lock on admin and prevent attach cycles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: redirect locked-prod edits into the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make dev-workspace settings tab available on CE (was EE-gated) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: lock prod against forking too and funnel edits to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: open dev item page on edit and tailor dev-workspace lock messages Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: prevent nested dev workspaces and hide dev option when one exists Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: drop the redundant already-has-dev hint on the fork form Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: badge dev workspaces and sort them ahead of forks in the tree/switcher Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label dev workspaces as 'Dev workspace of X' instead of 'Fork of X' Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label edit as 'Edit in <dev>', cover editor headers, auto-expand dev in tree Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: split prod lock into separate block-deploy and prevent-forking toggles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: make resources/variables workspace-specific from compare page Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: steer AI-chat sessions to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: refine session fork options and lock guidance for dev/prod Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: session picker reads prod's real rules, default to current ws Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: copy members into forks and clarify dev-workspace root labeling Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: place the workspace id field under the fork name Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address dev-workspace review findings and harden fork detection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate sqlx offline cache Restores entries dropped during the origin/main merge and adds the dev-workspace queries (is_dev_workspace, ws_specific, has_parent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address second-round dev-workspace review findings Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Pi and Codex review findings on dev-workspace endpoints Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate locked-dev git-branch fork on admin and validate ws_specific path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear prod dev-lock when deleting an attached dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: consolidate dev-workspace migration and scope all-group join to attach Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: restore dev-workspace CHECK into consolidated migration and scope all-group join Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: drop copy_members from the dev-workspace attach path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: dev-workspace lifecycle/auth fixes from Codex review round Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: explicit create-in-other for workspace-specific items Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other strictly create-only (never overwrite target) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: return 403 (not 401) for dev-workspace permission denials Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: allow attaching a same-family fork as a dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: emphasize the go-to-dev action in the no-direct-deploy alert Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: seed a resource's linked variables when creating it in the other workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: judge workspace deploy/fork locks against the user's identity in that workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: clarify create-in help text in workspace-specific panel Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: admin-gate dev-workspace creation and harden lock/seed edges Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: preserve a staged fork's source on picker create-mode re-entry Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear dev flag on archive and check dev existence server-side Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other atomically create-only via direct create Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: create-only resource insert, ws-specific list scopes, archive lock guard Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: reserve the dev_workspace_lock protection-rule name from the public API Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: reattach create_protection_rule doc comment to its function Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: make dev-archive pairing teardown atomic with the archive Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: follow deploy_to on root rename; show dev pairing to non-member prod admins Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: copy creator metadata on fork; invalidate fork routing cache on rename Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: accept g/ paths in set_ws_specific; gate copy_members to dev workspaces Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
7fa924e67e |
fix: correct flow conversation pagination (#8919)
* fix: remove conversation after_id filter * fix: implement message after_id cursor * fix: use persisted cursor for chat polling Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: simplify message cursor ordering Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: use cte for message cursor Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update SQLx metadata * fix: use monotonic flow message cursor * Update SQLx metadata * fix: tighten flow message cursor pagination Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update SQLx metadata --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
ce3e676f4a |
feat: list external JWT tokens in instance settings (#8783)
* [ee] feat: add external JWT tokens listing in instance settings Add the ability for superadmins to view all external JWT tokens that have been used for authentication, along with their claim metadata. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref.txt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: move external JWT tokens listing to users tab - Move list endpoint from /oidc/ext_jwt_tokens to /users/ext_jwt_tokens - Display as a sub-tab below the instance Users tab, only shown when tokens exist - Use DataTable's built-in load-more pattern for pagination - Add "Recently active only" toggle (tokens used in the last 30 days) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add dev_override cargo feature to windmill-common * feat: show placeholder for legacy external JWT entries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 62a462461271b900351c18b0ab1ca78651154b2a This commit updates the EE repository reference after PR #524 was merged in windmill-ee-private. Previous ee-repo-ref: 7b493a337abe00a47cf9d94847babe3cb3a6799f New ee-repo-ref: 62a462461271b900351c18b0ab1ca78651154b2a Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
|
c57c769dea |
feat: add CI test scripts with auto-trigger on deploy (#8736)
* feat: add CI test scripts with auto-trigger on deploy Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: fix annotation parser early return and handle renames correctly Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move CI test results to top of script/flow detail pages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: improve CI test results spacing, icon, and remove pass label Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: support one-line annotation and use script/path format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: move CI test trigger logic to EE Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: move CI badge next to New badge and add deduplicated CI summary Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add CI test e2e tests and fix nullable column annotations Add integration tests for CI test annotation parsing (creates/removes ci_test_reference rows) and the CI test results API (single + batch endpoints). Add backend test for auto-trigger on deploy (private+python). Fix sqlx LEFT JOIN LATERAL nullable column annotations in get_ci_test_results and get_ci_test_results_batch queries — sqlx cannot infer nullability from LATERAL subqueries, causing runtime decode errors when no matching job exists. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix build/sqlx * fix * feat: CI test improvements and templates - Fix windmill-dep-map/private feature propagation in worker, api-scripts, and api-flows Cargo.toml so CI test triggers actually fire in EE mode - Clone ci_test_reference rows during workspace fork - Add polling to CiTestResults component (refetch every 3s while running) - Add running state and auto-refresh to ForkWorkspaceBanner CI summary - Add yellow "CI test" badge on script list rows and detail page - Fix Library badge border color (remove indigo border override) - Add CI Test TypeScript and CI Test Python templates in ScriptBuilder - Update sqlx offline cache - Add debug tracing for CI test trigger in worker_lockfiles Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add missing children prop to WorkspaceDeployLayout Fixes svelte-fast-check type error when passing named snippets as children content inside the component tag. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback - Remove empty wrapper divs around CiTestResults, move mb-4 into component - Add batch endpoint size cap (max 200 items) - Add ON DELETE CASCADE to ci_test_reference workspace FK (new migration) - Downgrade CI test trigger logs from info to debug - Fix false-positive polling: only treat status='running' as running, not null status (CiTestResults, CompareWorkspaces, ForkWorkspaceBanner) - Fix test numbering in integration tests Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to latest EE commit Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to d9d68c2406df0b59f413ea0b2cb24780a9817d04 This commit updates the EE repository reference after PR #516 was merged in windmill-ee-private. Previous ee-repo-ref: d7ccd9b86da99ec056a0e8708e3637d64290387a New ee-repo-ref: d9d68c2406df0b59f413ea0b2cb24780a9817d04 Automated by sync-ee-ref workflow. * fix: treat queued jobs (job_id set, null status) as running Jobs that have been pushed but not yet picked up by a worker have a job_id but null status. Treat these as 'running' to avoid showing misleading 'pass' badges or '0 passing'. Tests that were never triggered (no job_id, null status) remain neutral/hidden. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: hugocasa <hugo@casademont.ch> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
c4c9ef5fd7 |
feat: add optional labels to scripts, flows, apps, schedules, triggers (#8609)
* feat: add optional labels to scripts, flows, apps, raw apps, schedules, and triggers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update sqlx cache, make labels optional in openapi, regenerate system prompts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add minimal labels input UI to script, flow, and schedule editors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce gap between summary and labels input
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add labels to script/flow detail pages and summary/path popover
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: move labels inside SummaryPathDisplay trigger for clickable area, reduce gap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: display labels inline to the right of summary, not below
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: increase gap between summary and labels
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add labels to resources/variables, make labels nullable, add home page label filter badges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add labels to workspace export/import, resources, variables + test coverage
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: make migration idempotent, regenerate sqlx cache after merge
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pass labels in script create and flow create/update API calls
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add labels input UI to resource and variable editors
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove negative margin from LabelsInput to prevent overlap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add top and left margin to LabelsInput for better spacing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: reduce left margin on LabelsInput
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: widen label input to w-32
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use inline-flex so LabelsInput doesn't stretch full width
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove flex-wrap so label input stays on same line as badges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add label filter presets to resources, variables, and schedules search
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use max-w-32 on label input to prevent stretching
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pull labels closer to summary with negative top margin
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: increase negative margin to pull labels even closer to summary
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: pass labels in schedule create/update API calls
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use COALESCE to preserve existing labels when not provided in schedule/flow update
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add labels to CreateResource, EditResource, CreateVariable, EditVariable in OpenAPI spec
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: display label badges on resource and variable list pages
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: display label badges on schedule and all trigger list pages
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add folder and label presets to schedules search filter
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: apply user_folders_only filter on all workspaces including admins
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add label presets to resources and variables search filters
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: derive folder presets from loaded items, not all workspace folders
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add label query parameter to resource and variable list endpoints in OpenAPI
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: display label filter badges inline with folder filters on home page
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Revert "feat: display label filter badges inline with folder filters on home page"
This reverts commit
|
||
|
a46aa641f9 |
feat: add R language support (#8263)
* feat: add R language support Add R as a new supported scripting language in Windmill, following the same pattern used for Ruby. Includes: - Backend: ScriptLang::Rlang enum variant, DB migration, tree-sitter-r parser crate with tests, WASM parser binding, R executor with NSJail sandboxing, job dispatch and signature parsing - Frontend: language picker, R icon, syntax highlighting, editor bar insertions (Sys.getenv, get_variable, get_resource), schema inference, init code template, BETA badge - CLI: .r extension mapping, sync support, bootstrap template R scripts use `main <- function(...)` syntax, jsonlite for JSON serialization, and system curl for the Windmill client helper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add R package resolution and installation Parse library()/require() calls from R scripts to extract dependencies. Resolve versions from CRAN, cache lockfiles in pip_resolution_cache, and install packages to a shared R library cache. The run step sets R_LIBS_USER so installed packages are available to the script. - Parser: parse_r_requirements() extracts package names from AST - Executor: resolve() generates lockfile, install() installs from CRAN - Worker lockfiles: wire up R resolve for dependency jobs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add nsjail sandboxing for R resolve and install phases Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: fix R get_variable/get_resource and add sandbox annotation + e2e tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: fix R arg inference with JS fallback parser and get_variable/get_resource Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix flake * nsjail * nits * fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits - Suppress renv verbose output during resolve and install (controlled by #verbose annotation) - Filter renv from install list (already loaded, causes noisy restart message) - Log compact "resolved N packages" instead of full renv.lock JSON - Add R (r-base, r-cran-renv) to DockerfileFull and DockerfileFullEe - Use disable_rl for nsjail install config (R compiles from source) - Reduce default concurrency from 20 to 5 - Add rlang to openflow.openapi.yaml - Fix MainArgSignature (no_main_func -> auto_kind) after main merge Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * final * fix: remove accidental R install from multiplayer Dockerfile Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: remove R from Windows build and DockerfileExtra Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: rename R migration to avoid timestamp collision with trigger_filter_logic Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * all * fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: add clear error when Rscript binary is missing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: fix type errors in R fallback parser, use format! in wrap(), add R system prompts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: pyranota <pyra@duck.com> |
||
|
|
55ad0ff5c4 |
fix: use resource-level scope overrides during OAuth2 token refresh (#8540)
* fix: use resource-level scope overrides during OAuth2 token refresh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref.txt Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 6db424512b0d02f86489e85f0026581b7637d6e6 This commit updates the EE repository reference after PR #484 was merged in windmill-ee-private. Previous ee-repo-ref: c9277992608537155a9505a089aca91403d91159 New ee-repo-ref: 6db424512b0d02f86489e85f0026581b7637d6e6 Automated by sync-ee-ref workflow. * fix: restore non-enterprise sqlx cache entries deleted by update_sqlx.sh Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update sqlx cache for latest EE changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rename migration to avoid timestamp collision with trashbin Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: collapse duplicate match arms and simplify effective_scopes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszel
|
69ce946241 | feat: add trashbin system for soft-deleting items (#8519) | ||
|
Ruben FiszelandClaude Opus 4.6
|
31d6660d56 |
feat: script module mode with CLI sync, preview, and WAC UI improvements (#8380)
* feat: add script module mode with folder model for Bun and Python Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add missing modules field to RawCode in bun_executor Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * sqlx * feat: enrich WAC templates with checkpoint and replay semantics Add prominent comments explaining that all computation must happen inside task/step/taskScript or it will be replayed on resume/retry. Clarify that waitForApproval does not hold a worker and that approve/reject URLs are available in the timeline step details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(cli): script module sync idempotency, per-module hash tracking, and preview support - Fix pull→push idempotency: use `??` instead of `||` for module lock field so empty strings are preserved (matches API's `lock: ""`) - Add per-module hash tracking in wmill-lock.yaml following the flow inline script pattern (SCRIPT_TOP_HASH + per-module subpath hashes) - Selective module lock regeneration: only regenerate locks for modules whose content actually changed, not all modules - Use unfiltered rawWorkspaceDependencies for module hashes to match what updateModuleLocks passes to fetchScriptLock - Show changed module names in stale script output for clarity - Add module support to `script preview` command: read modules from __mod/ folder and pass them in the preview API request - Add preview tests for taskScript pattern (flat and folder layout) - Update test assertion for module stale detection output Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): WAC UI improvements — reorder templates, module tab rename, import consolidation - Reorder WAC template buttons: TypeScript before Python in ScriptBuilder, CreateActionsScript, and CreateActionsFlow - Remove dropdown items from +Script button (simplify to direct link) - Move "Import Workflow-as-Code" to +Flow dropdown with dedicated drawer - Add module tab rename: pencil icon on hover opens popover with validation, fixed-width icon container prevents layout shift Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: remaining module-mode changes from working branch - Backend parser updates for WAC detection - CLI sync/types updates for raw app path and module support - Frontend UI polish (Dev.svelte, ScriptRow, script hash page) - Test fixture updates Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(cli): add test for module modification detection in generate-metadata Verifies that modifying a single module file re-triggers stale detection and only the changed module is listed, not all modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): critical fixes from PR review - Fix hardcoded dev path in bun_executor.rs WAC v2 wrapper — use "windmill-client" import instead of absolute filesystem path - Fix missed no_main_func → auto_kind rename in parser TS test - Add modules column to clone_script SQL (windmill-common and windmill-api-workspaces) so cloned scripts retain their modules - Add modules: None to RawCode structs in worker tests - Restore complete sqlx cache (merge main's cache + our new queries) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix clone warning treated as error in CI Change `.clone()` on double reference to `*k` dereference in scripts.rs hash implementation. Update sqlx cache with new query hashes from modified clone_script SQL. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use published parser wasm versions for CI build The local file:// paths for windmill-parser-wasm-py and windmill-parser-wasm-ts don't exist in the Cloudflare Pages build environment. Revert to published npm versions (1.655.0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): update parser wasm packages to 1.657.2 Use newly published windmill-parser-wasm-ts and windmill-parser-wasm-py v1.657.2 which include auto_kind/WAC detection changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): regenerate package-lock.json for npm ci compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(frontend): use main's lockfile as base, update only parser wasm packages Regenerating package-lock.json from scratch pulled different dependency versions causing svelte-check type errors. Instead, start from main's lockfile and only update the two changed packages. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): add modules column to fetch_script_for_update query The Script<SR> struct has a modules field (FromRow), but fetch_script_for_update didn't SELECT modules, causing a runtime error "no column found for name: modules" when the worker processed dependency jobs. This was the root cause of the relock_skip test timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix script module execution for Python and Bun - Fix modules not passed through job queue: inject _MODULES into PushArgs.extra when pushing Code jobs so worker can extract them - Fix Python module imports: use relative imports (from .helper) and add sys.path.insert for module directory in wrapper - Fix Python tests: use relative imports and empty lock to prevent pip from resolving module names as packages - Add local file check in Bun loader for module resolution - Ignore Bun module test (bundle mode loader integration tracked separately) - Add missing modules column to fetch_script_for_update query Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): remove unnecessary empty lock in Python module tests Relative imports (from .helper) are not parsed as pip packages, so the empty lock workaround is not needed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(backend): fix module execution for Python and Bun — all tests pass Python modules: - Use relative imports (from .helper import greet) since scripts run as packages - Add sys.path.insert for module directory in wrapper to ensure local modules take precedence over pip packages with same name Bun modules: - Use bundled output (./out/main.js) as wrapper import when modules are present — the bundled output has module content inlined by Bun.build, avoiding runtime loader resolution issues - Add local file check in loader.bun.js onResolve to short-circuit API URL resolution for module files on disk Job queue: - Inject _MODULES into PushArgs.extra when pushing Code jobs so the worker can extract them at execution time Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: address PR review — simplify, fix correctness, remove dead code Critical fixes: - Replace all CLI `no_main_func` references with `auto_kind` (string) to match the backend migration and API changes - Remove duplicated `compute_python_module_dir` in worker.rs, use the canonical version from python_executor.rs High priority: - Auto-create `__init__.py` in intermediate directories for nested Python modules so imports like `from .utils.math import add` work without users manually creating __init__.py files - Remove redundant `sys_path_insert` — relative imports use Python's package system, not sys.path Medium: - Fix lock file base name extraction: use regex to strip only the final extension (`.replace(/\.[^.]+$/, '')`) instead of `indexOf(".")` which breaks for files like `helper.test.ts` Simplification: - Remove dead `{#if false}` Popover block in ScriptEditor.svelte - Guard loader.bun.js local file check to only run for relative paths (matching the Windows loader pattern) - Add clarifying comment on Bun dual mechanism (build + run phases) - Add maintenance comment on manual Hash impl for NewScript Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: final review fixes — stale cleanup, baseName, auto_kind export - Fix sync.ts baseName extraction using indexOf(".") → regex (same fix as script.ts/metadata.ts, missed this instance) - Add stale module file cleanup in writeModulesToDisk: removes files from __mod/ that are no longer in the modules map before writing, fixing the pull→push cycle that couldn't delete modules - Log warning when _MODULES serialization fails in job push instead of silently dropping modules - Use strict equality (===) for auto_kind comparison - Exclude auto_kind from workspace export — it is auto-detected by the parser at deploy time from script content Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(cli): remove auto_kind from push, comparison, and metadata auto_kind is auto-detected by the parser at deploy time, so the CLI should not send it, compare it, or write it to script.yaml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove erroneously added backend/backend/.sqlx directory Duplicate .sqlx cache was committed at the wrong nested path. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review feedback + fix CI dead_code warning Frontend (ScriptEditor.svelte): - Fix switchToMain() missing lastSyncedCode update — prevents stale code sync on external changes while editing a module tab - Fix formatAction saving module code to main script's localStorage draft — now saves main code when on a module tab - Fix non-null assertion on inferModuleLang in renameModule — fall back to original language instead of force unwrap - Remove redundant activeModuleTab truthy check in runTest CLI (script.ts): - Clean up empty directories after removing stale module files in writeModulesToDisk Backend: - Add path traversal guard in write_module_files — reject module paths containing ".." - Fix dead_code warning on auto_kind field in workspace export struct Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(frontend): improve auto_kind UX + address review findings - Rename "Include without main function" toggle to "Include library scripts" in script list (ItemsList.svelte) - Update NoMainFuncBadge: "No main" → "Library" with clearer tooltip - Filter module file extensions by main script language — Python scripts only allow .py modules, TypeScript only .ts, etc. - Split flushModuleState into flushModuleContent (no UI side-effect) and flushModuleState (flush + reset tab), reducing duplication - Dynamic placeholder and hint text in add module popover based on main script language Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
  
|
f2be625348 |
feat: store hashed tokens instead of plaintext (#8217)
* feat: store hashed tokens in the token table instead of plaintext
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address review issues in token hash migration
- Update all base.sql fixtures to include token_hash/token_prefix columns
- Keep plaintext token for webhook tokens (needed for URL reconstruction)
- Restore get_token_by_prefix to query DB for webhook tokens
- Fix down migration to delete NULL-token rows before restoring NOT NULL
- Update parser fixture standalone schema
- Update EE dedicated_worker_ee.rs to use token_hash/token_prefix
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore sqlx offline cache (only add new query files)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: keep writing plaintext token column for backward compat
Write to token column alongside token_hash until MIN_VERSION_SUPPORTS_TOKEN_HASH
(1.649.0) is reached. This ensures older workers can still authenticate
during rolling upgrades. Remove the separate UPDATE in new_webhook_token
since create_token_internal now writes plaintext directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: branch on MIN_VERSION to write plaintext token or null
Check MIN_VERSION_SUPPORTS_TOKEN_HASH at runtime: write plaintext to
token column while old workers exist, switch to NULL once all workers
are >= 1.649.0.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: set MIN_VERSION_SUPPORTS_TOKEN_HASH to 1.650.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use token_hash for email lookup and expiry notifications
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: rotate webhook tokens instead of recovering plaintext from DB
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: use token_hash for native trigger token lookups and deletes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* refactor: drop webhook_token_prefix from native_trigger table
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: backward compat for token rotation and make webhook_token_hash NOT NULL
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: prevent panic on short superadmin secret token prefix
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: prevent panic on short superadmin secret token prefix
Replace all `token[0..TOKEN_PREFIX_LEN]` slicing with
`token.get(..TOKEN_PREFIX_LEN).unwrap_or(token)` to prevent
panics when a token shorter than 10 chars is provided (e.g.
malformed Authorization header, short superadmin secret).
Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>
* fix: prevent panic on short token prefix slicing
Replace all `token[0..TOKEN_PREFIX_LEN]` with safe
`token.get(..TOKEN_PREFIX_LEN).unwrap_or(token)` to prevent panics
on malformed tokens shorter than 10 characters.
Co-authored-by: hugocasa <hugocasa@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Revert "fix: prevent panic on short superadmin secret token prefix"
This reverts commit
|
||
|
|
ec20d76216 |
feat: add auto_commit option to Kafka triggers with advanced UI badges (#8317)
* feat: add auto_commit option to Kafka triggers with manual commit API
Add ability to disable auto-commit on Kafka triggers so users can
manually commit offsets after processing messages. This prevents
message loss when processing fails.
Changes:
- Add `auto_commit` column to kafka_trigger table (default true)
- Add POST /kafka_triggers/commit_offsets/{path} endpoint using
BaseConsumer with manual assign() to avoid rebalance
- Enrich trigger_info payload with partition and offset fields
- Conditionally commit based on auto_commit setting
- Add auto-commit toggle to frontend Kafka trigger config
- Add commitKafkaOffsets helpers to Python and TypeScript SDKs
- Add integration tests for auto_commit DB defaults
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: use DB-based pending commits for kafka manual offset commit
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: pass trigger_path to all v2 preprocessors, secure commit_offsets endpoint, fix commit semantics
- Add trigger_path to v2 preprocessor event for all trigger types (kafka, nats, sqs, mqtt, gcp, postgres, websocket, http, email)
- Secure commit_offsets endpoint: infer trigger from job token (OptJobAuthed) instead of requiring trigger path parameter
- Fix auto_commit: only commit offset after successful job push
- Fix pending commits: commit offset+1 (Kafka semantics) and use CommitMode::Sync
- Update TS/Python clients and frontend preprocessor templates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add advanced section badges and reorganize kafka trigger settings
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove dead wm_trigger assertions from kafka e2e test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* refactor: remove unused advancedCollapsed state from all trigger editors
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update ref
* chore: update ee-repo-ref to ed2c9d360e6fab866b9744cc79f50038d1fc7152
This commit updates the EE repository reference after PR #452 was merged in windmill-ee-private.
Previous ee-repo-ref: 5b31116a1d5a042c6a780732901cfd89584d1773
New ee-repo-ref: ed2c9d360e6fab866b9744cc79f50038d1fc7152
Automated by sync-ee-ref workflow.
* fix: use path-based auth for kafka commit_offsets endpoint
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update ee-repo-ref to fcd3ea52b0cc94fbe1159baf662a38da947456de
This commit updates the EE repository reference after PR #457 was merged in windmill-ee-private.
Previous ee-repo-ref: b3a5c33c92cb1b2caf7a65986d71da291ff72a35
New ee-repo-ref: fcd3ea52b0cc94fbe1159baf662a38da947456de
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
|
||
|
hugocasaandClaude Opus 4.6
|
e56ccd200b |
feat: token expiration notifications (#8190)
* feat: add token expiration notifications via email, critical alerts, and webhooks - Monitor loop checks for tokens expiring within 7 days and sends email notifications to token owners. Tracks notification state via new `expiry_notified` column on the token table to avoid duplicates. - When tokens expire and are deleted, owners are also notified. - Critical alerts (in-app UI) are gated behind a new instance setting `critical_alerts_on_token_expiry` (off by default); emails are always sent regardless of the setting. - Add TokenExpiringSoon and TokenExpired webhook message variants for workspace webhook integrations. - Frontend: show expiration badges and a warning banner on the tokens table for tokens expiring within 30 days. - Exclude session and ephemeral tokens from all notifications. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: use separate token_expiry_notification table for dedup - Replace `expiry_notified` column on token table with a dedicated `token_expiry_notification` table (token, expiration) - Insert notification row on token creation via shared `register_token_expiry_notification()` helper - Delete notification row atomically when sending the notification - Clean up orphaned rows in `delete_expired_items()` - No FK constraint to avoid cascade overhead on token deletions - Add index on expiration column for efficient range queries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: calendar-based expiration badge and move notification cleanup - Fix daysUntilExpiration to compare calendar dates instead of time diff - Move notification row cleanup from delete_expired_items to check_expiring_tokens to keep it off the hot path - Use simple expiration <= now() index scan instead of NOT EXISTS join Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
32c4b474f9 |
feat: add fileset resource type support
Add a new "fileset" resource type that represents a collection of files stored as a relpath→content map. This enables resource types to manage multiple files (e.g., config directories, template sets) instead of just a single file. Backend: - Add is_fileset column to resource_type table - Update CRUD operations and workspace duplication to handle is_fileset - Add integration tests for fileset resource types Frontend: - Add FilesetEditor component with file explorer + Monaco editor - Extract shared FileExplorer component from RawAppSidebar (dedup) - Add fileset toggle to EditableSchemaWrapper - Show fileset editor in ResourceEditor and ApiConnectForm - Show folder icon for fileset resource types in IconedResourceType CLI: - Support fileset resources in sync pull (expand to .fileset/ directory) - Support fileset resources in sync push (reconstruct from directory) - Handle !inline_fileset YAML tag in resource resolution Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
2dad2b43a4 |
internal: Improve instructions for claude (#7921)
* Improve CLAUDE.md instructions and compact DB schema summary - Add code validation instructions (cargo check, npm run check) to all CLAUDE.md files with guidance to use only relevant feature flags - Reference backend/CLAUDE.md and frontend/CLAUDE.md paths from root - Add database querying guidance (psql commands for exact table info) - Compact summarize_schema.py output: inline columns, shorten types, one-line enums, drop indexes (use psql \d for exact info) - Fix FK parsing for multi-line ALTER TABLE statements - Result: schema summary reduced from 1514 lines/40KB to 194 lines/23KB * cleaning * fix: use prefix-based type abbreviations and filter CONSTRAINT pseudo-columns - Change TYPE_ABBREVIATIONS matching from exact to prefix-based so parametrized types (character(64) -> char(64)) and array types (integer[] -> int[], real[] -> float[]) are properly abbreviated - Skip CONSTRAINT lines inside CREATE TABLE blocks that were being incorrectly matched as columns by the column regex - Update summarized_schema.txt to reflect both changes Co-authored-by: centdix <centdix@users.noreply.github.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: centdix <centdix@users.noreply.github.com> |
||
|
hugocasaandwindmill-internal-app[bot]
|
60858d1e20 |
feat: workspace dedicated workers (#7741)
* feat: workspace dedicated workers * ref * chore: update ee-repo-ref to a18ac31062ac092cb9a5fc87629e217d97f4911d This commit updates the EE repository reference after PR #398 was merged in windmill-ee-private. Previous ee-repo-ref: 98cfe3fef764d9d815d326d5056c734a03689d33 New ee-repo-ref: a18ac31062ac092cb9a5fc87629e217d97f4911d Automated by sync-ee-ref workflow. * fix(frontend): workspace script in flow steps --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
73c4ce3012 |
feat: add workspace success handler with 60s TTL caching (#7598)
* feat: add workspace success handler with 60s TTL caching Add a new workspace-level success handler that triggers when jobs complete successfully, similar to the existing error handler. Key features: - Database migration adds success_handler and success_handler_extra_args columns - API endpoint for configuring success handler in workspace settings - 60s TTL caching to avoid DB trips when checking handler existence - Handler only triggers for top-level Script/Flow/SingleStepFlow jobs (not previews) - Frontend UI with ScriptPicker and "Create from template" button for bunnative scripts - EE warning alerts for CE users on success handler and critical alerts sections - Backend tests for cache behavior and settings CRUD operations The success handler receives: path, email, result, job_id, is_flow, workspace_id, and started_at. It runs as the g/success_handler group. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: update RawCode struct in tests to use new field names Update test to use ConcurrencySettingsWithCustom and DebouncingSettings instead of the deprecated individual fields. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update backend/windmill-api/src/workspaces.rs Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> * test: improve success handler test to verify job creation Update test_success_handler_triggered_on_success to actually verify that: - A success handler job is created in v2_job table - The job has the correct runnable_path (configured script) - The job runs as success_handler@windmill.dev - The job has correct parent_job and root_job references - The trigger starts with 'success.handler.' This provides better test coverage for the success handler feature. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: fix success handler test and root_job parameter - Fix test to use JobPayload::ScriptHash (JobKind::Script) instead of JobPayload::Code (JobKind::Preview) since previews don't trigger handlers - Add required 'lock' column to script inserts for deployed scripts - Fix duplicate lines in workspaces.rs - Set root_job correctly in push_success_handler - Update test assertions to match actual behavior (root_job optimization) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * sqlx * refactor: reduce duplication in workspace handler cache logic - Extract database queries into fetch_error_handler_from_db and fetch_success_handler_from_db helper functions - Use shared WORKSPACE_HANDLER_CACHE_TTL_SECONDS constant for 60s TTL - Reduces code from ~116 lines to ~54 lines while maintaining identical behavior Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Add cache propagation warning to workspace success handler UI Note that changes may take up to 60 seconds to propagate due to caching. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> |
||
|
centdix
|
2d0e65b7ca |
internal: give database schemas information to Claude (#5961)
* add schema * give claude the schema * use summarized schema * add indexes * add usage comments * Remove backend/schema.sql from remote |