mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-07 00:02:18 +00:00
83f3d7f910b331c09f60cc9ff556728afa3dec07
6658
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
   
|
83f3d7f910 |
feat(licensing): enforce offline license seat cap (#9845)
* [ee] feat(licensing): enforce offline license seat cap Companion to windmill-ee-private. Aligns the offline-license seat count with the billing model and adds real-time enforcement when usage exceeds the cap. OSS side carries the ee_oss stubs, the reactivation cap-check call site, the regenerated SQLx cache, and the EE ref bump. - Exclude instance-disabled users (password.disabled) and service accounts from the seat count. Deactivating a user now frees a seat. - Service accounts no longer consume seats (no check at creation). - Hard-block reactivation when it would exceed the cap. - Invalidate the license (halting jobs) when seat usage exceeds the cap, mirroring CU-cap enforcement; recovers when usage drops back under or a higher-cap key is loaded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): bump EE ref for reactivation seat-check fixes Points to the EE companion commit that fixes reactivation double-counting and preserves the original seat alert tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] fix(licensing): reactivation seat delta includes pending invites Bumps the EE ref and drops the now-orphaned usr-only cache entry; the reactivation check reuses the existing usr ∪ workspace_invite query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * [ee] test(licensing): bump EE ref for offline seat-cap tests Adds #[sqlx::test] coverage for the offline seat counting and cap-check logic; EE-only (runtime queries, no cache change). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: update ee-repo-ref to f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 This commit updates the EE repository reference after PR #637 was merged in windmill-ee-private. Previous ee-repo-ref: b2622e3afc2fe1fe3e2ec978ca46cf9decf91b82 New ee-repo-ref: f814c3f75308c1ef1e4526d8d0eeb360ce16abe4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
|
6b79bddd42 |
fix(s3_proxy): preserve URL-encoding on Hive-partition proxy writes (#9848)
* [ee] fix(s3_proxy): preserve URL-encoding on forward re-sign for Hive-partition keys Bump ee-repo-ref to pull the EE fix for SigV4 SignatureDoesNotMatch on DuckLake Hive-partition writes through the S3 proxy. The forward re-sign leg rebuilt the upstream URI from the decoded object key (literal `=`) instead of the still-encoded request path (`%3D`), diverging from how S3/minio canonicalizes the key. Companion EE commit c6b110f. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 1a98119b0b8b8548601983c9e5ab091150f0b180 This commit updates the EE repository reference after PR #638 was merged in windmill-ee-private. Previous ee-repo-ref: c6b110fd3b3591a5c3f09952c388c42bd5766188 New ee-repo-ref: 1a98119b0b8b8548601983c9e5ab091150f0b180 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
b4b0c6a93e |
feat: add dev workspaces paired with a lockable prod workspace (#9793)
* feat: add dev workspaces paired with a lockable prod workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate dev-workspace prod-lock on admin and prevent attach cycles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: redirect locked-prod edits into the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make dev-workspace settings tab available on CE (was EE-gated) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: lock prod against forking too and funnel edits to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: open dev item page on edit and tailor dev-workspace lock messages Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: prevent nested dev workspaces and hide dev option when one exists Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: drop the redundant already-has-dev hint on the fork form Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: badge dev workspaces and sort them ahead of forks in the tree/switcher Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label dev workspaces as 'Dev workspace of X' instead of 'Fork of X' Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: label edit as 'Edit in <dev>', cover editor headers, auto-expand dev in tree Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: split prod lock into separate block-deploy and prevent-forking toggles Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: make resources/variables workspace-specific from compare page Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: steer AI-chat sessions to the dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: refine session fork options and lock guidance for dev/prod Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: session picker reads prod's real rules, default to current ws Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: copy members into forks and clarify dev-workspace root labeling Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: place the workspace id field under the fork name Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address dev-workspace review findings and harden fork detection Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore: regenerate sqlx offline cache Restores entries dropped during the origin/main merge and adds the dev-workspace queries (is_dev_workspace, ws_specific, has_parent). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address second-round dev-workspace review findings Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Pi and Codex review findings on dev-workspace endpoints Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate locked-dev git-branch fork on admin and validate ws_specific path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear prod dev-lock when deleting an attached dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: consolidate dev-workspace migration and scope all-group join to attach Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: restore dev-workspace CHECK into consolidated migration and scope all-group join Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: drop copy_members from the dev-workspace attach path Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: dev-workspace lifecycle/auth fixes from Codex review round Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: explicit create-in-other for workspace-specific items Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other strictly create-only (never overwrite target) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: return 403 (not 401) for dev-workspace permission denials Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: allow attaching a same-family fork as a dev workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: emphasize the go-to-dev action in the no-direct-deploy alert Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: seed a resource's linked variables when creating it in the other workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: judge workspace deploy/fork locks against the user's identity in that workspace Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style: clarify create-in help text in workspace-specific panel Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: admin-gate dev-workspace creation and harden lock/seed edges Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: preserve a staged fork's source on picker create-mode re-entry Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: clear dev flag on archive and check dev existence server-side Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: make create-in-other atomically create-only via direct create Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: create-only resource insert, ws-specific list scopes, archive lock guard Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: reserve the dev_workspace_lock protection-rule name from the public API Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: reattach create_protection_rule doc comment to its function Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor: make dev-archive pairing teardown atomic with the archive Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: follow deploy_to on root rename; show dev pairing to non-member prod admins Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: copy creator metadata on fork; invalidate fork routing cache on rename Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: accept g/ paths in set_ws_specific; gate copy_members to dev workspaces Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
83ed011e26 |
feat(object-store): make GCS service account key optional for Workload Identity (#9842)
build_gcs_client always called `.with_service_account_key(...)`, so an
absent key (the settings UI stores "no key" as the empty JSON object `{}`)
was handed to the builder and failed to parse instead of falling through
to the object_store crate's InstanceCredentialProvider. Skip the call when
the key is blank so GCS uses the instance's ambient credentials (GKE
Workload Identity / the GCP metadata server).
"Blank" (empty/whitespace/`{}`/`null`) is centralized in a shared
`gcs_service_account_key_is_blank` predicate so the build path and the
non-super-admin connectivity-test SSRF guard (`validate_object_storage_test`)
agree on what counts as "no key" — otherwise a blank key would bypass the
guard yet still trigger the ambient-credential fallback, letting an
untrusted caller probe arbitrary buckets with the server's instance role.
Also clarify the settings UI hint that the key may be left empty for
ambient credentials, and add regression tests for the blank-key build path
and the guard.
Fixes WIN-2110
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
 Ruben Fiszelandrubenfiszel
|
a9ffdb996b |
chore(main): release 1.743.0 (#9837)
* chore(main): release 1.743.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
|
9b65161c64 |
fix(gcp): require token verification for authenticated push delivery (#9834)
* fix(gcp): require token verification for authenticated push delivery Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2 This commit updates the EE repository reference after PR #636 was merged in windmill-ee-private. Previous ee-repo-ref: 8c63d487c486002baf09c77ab937fd77a91765eb New ee-repo-ref: 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
c91027824b |
feat(pipeline): AI-chat data-pipeline editor (route + in-session) + home surfacing (#9805)
* feat(pipeline): AI chat tools to build pipeline nodes with diff/approval Add a data-pipeline AI chat experience modeled on the flow editor and surfaced through the dev-gated global chat (no new chat panel). The /pipeline editor registers PipelineAIChatHelpers on the AIChatManager; while it is open the global mode layers pipeline tools, a pipeline prompt section, and the helpers on top of the full global tool set (behavior is unchanged when no pipeline editor is open). New tools (frontend/src/lib/components/copilot/chat/pipeline/core.ts): - get_pipeline_graph / read_pipeline_node — read the live graph and bodies - build_pipeline_node / edit_pipeline_node — stage changes as AI-pending drafts - remove_pipeline_node — drop a staged proposal - test_pipeline_node — preview-run a node (requires confirmation) Tools never deploy: they stage drafts flagged aiPending, rendered on the canvas with an accent ring and reviewed via Accept all / Reject all (the flow editor's GlobalReviewButtons). Accept commits the drafts; Reject reverts to a pre-AI snapshot, preserving earlier accepted drafts. Auto-accept is gated on the chat autonomy mode. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): teach the global/session chat to author data pipelines Without an open /pipeline editor the session chat had no pipeline concept, so "create a data pipeline" loaded flow instructions and built a flow. Add a first-class pipeline authoring path: - system_prompts/base/pipeline-base.md — what a data pipeline is (a DAG of annotated scripts wired by storage assets, NOT a flow) and how to author the // pipeline / // on / // materialize annotations; wired through generate.py as getPipelinePrompt() (regenerated prompts.ts/index.ts). - global/core.ts — new get_instructions subject "pipeline", and a global-prompt rule disambiguating data pipelines from flows so the model routes correctly. - ai_evals/cases/global.yaml — two global cases (single node, two-node chain) asserting pipeline-annotated script drafts and forbidding write_flow, guarding the pipeline-vs-flow conflation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline): show & build pipelines in the AI session preview Add a 'pipeline' session preview target so the session AI can show the data-pipeline graph for a folder and build nodes in-pane: - open_preview now accepts kind="pipeline" (path = folder); SessionTarget / EDITOR_TARGET_KINDS widen accordingly. The slot/codec load model stays flow|script|raw_app — pipeline bypasses it with its own fetch/draft state. - New PipelineEditorView.svelte mounts in the session pane: fetches the folder graph, overlays AI drafts, renders AssetGraphCanvas + the Accept/Reject review buttons, and registers PipelineAIChatHelpers on the *session-scoped* manager (via getAiChatManager) so build_pipeline_node / edit_pipeline_node + the diff/approval work inside the session too. - System prompt nudges the model to open the pipeline preview and use the staging tools while building. Verified end-to-end with a real model: the session AI called open_preview, the graph mounted in the side panel, then build_pipeline_node staged a node on the session canvas with its schedule trigger and ducklake output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): share the AI editor logic between route page and session Consolidate the duplicated data-pipeline AI logic onto a single shared layer so the route editor and the in-session preview behave identically and the session gains the full code editor. - New pipelineAiHelpers.ts: createPipelineAiHelpers(deps) owns the propose/edit/ remove/accept/reject/test staging + the per-turn snapshot bookkeeping that powers Reject. Callers inject accessors for their own draft Map and graph. - Route page (/pipeline/[folder]) drops its ~250-line inline AI-helper block and wires the shared factory via deps (folder/workspace/graph/drafts + focus, ensureEditable, run-started). Its shell — persistence, navigation guard, activity, cascade, trigger drawers — is untouched. - Session PipelineEditorView uses the same factory and now renders the real AssetGraphDetailsPane (code editor + live overlays + test), so a node built in a session opens with its source, matching the route editor. Verified: route page hydrates/renders drafts unchanged; in a session the AI opened the pipeline preview, built a node, and its code showed in the details pane. check:fast clean, 197 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): externalize editor state into PipelineEditorState (step 1) Introduce PipelineEditorState — the data-pipeline analogue of the flow editor's flowStore. It owns the draft Map, the live editor overlays, and the selection, with callback-safe methods (handleDraftPersist / handleAnnotationsChange / … ), so a single editor can be rendered by both the route page and the session. This commit lands the store and points the in-session PipelineEditorView at it (no behaviour change — the session already had these inline). Next steps move the route page onto the store and a shared <PipelineGraphEditor>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): point the route editor at PipelineEditorState (step 1) Move the route page's draft Map, live editor overlays, selection, and the draft-persist / live-change handlers onto the shared PipelineEditorState (`pe`), referencing them as `pe.*` in place. No behaviour change — persistence, graph resolution, run dispatch, AI staging, and deploy all stay on the page and now read/write the externalized state. This is the data-pipeline analogue of the flow editor's flowStore: the route page and the in-session preview now share one source of editor truth, setting up the shared <PipelineGraphEditor> in the next steps. Verified: the page hydrates its DB draft, renders the overlay graph, the toolbar counts (Save all (N)) track pe.drafts, and selecting a node opens it in the details pane. check:fast clean, 84 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): render the route editor via shared PipelineGraphEditor (step 2) Extract the canvas + details-pane editor body into PipelineGraphEditor.svelte, the data-pipeline analogue of FlowBuilder. The route page now delegates its Splitpanes block to it, passing the externalized PipelineEditorState plus its run/cascade/trigger/deploy callbacks; the component owns pane sizing, selection/details-open derivation, and the canvas+details rendering. Root-caused the earlier ts2769 "$props() No overload" to a prop named `state` colliding with the `$state` rune (`let x = $state(...)` parsed as a store auto-subscription on the prop) — the prop is now `editor`. Net: the route page sheds ~310 lines of template/state; behaviour preserved. Verified: the page hydrates its DB draft, renders the graph, opens the draft in the details pane (live code editor + Test), pane sizing works. check:fast clean, 24 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): move draft autosave into PipelineGraphEditor (step 3) Fold the per-user `data_pipeline` DraftService bundle autosave (hydrate + debounced persist + localStorage crash mirror) into PipelineGraphEditor, gated by a `persistDrafts` prop — FlowBuilder's parameterized-autosave shape. The route page passes `persistDrafts` + `folder` and reads `editor.loadedFromDbDraft` for its AutosaveIndicator; the in-session preview will leave persistence off. Also restores the `untrack(...)` wrapping on the pane-sizing $effect (dropped when the editor body was extracted in step 2). Without it the Pane `bind:size` feedback loops the effect and pegs the main thread when the details pane is closed — a latent hang in the step-2 commit. check:fast clean, 24 pipeline tests pass. Note: browser revalidation was not possible this session (the Playwright MCP browser was reset); the autosave is a verbatim port and the untrack fix is the original working form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(pipeline): render the session preview via shared PipelineGraphEditor (step 4) Point the in-session PipelineEditorView at the shared PipelineGraphEditor instead of its own inline canvas + details pane. The session now renders the exact same editor body as the route page — gaining the full details/code pane — while opting out of persistence (persistDrafts=false) and the run/cascade/trigger/bounded affordances (their callbacks are omitted, so those controls hide). Building nodes + the Accept/Reject diff still work via the AI helpers. Also fixes issues surfaced by a full `svelte-check` while wiring this up: - PipelineGraphEditor: edit mode opened the details pane unconditionally (a step-2 regression); restored the route's "open only on selection/draft" behaviour. - Route page passed an `isOperator` prop the component doesn't accept (step-2; caught only by full check, not check:fast). - SessionItemNotFound: narrow its `kind` to exclude `pipeline` (pipeline targets never slot-load, so they can't 404 through it) — closes the SessionTarget-widen fallout. - PipelineEditorView: cast the resolveGraph base to AssetGraphResponse. Full `svelte-check` now clean across all pipeline/session files; 137 unit tests pass. (Browser revalidation still pending — Playwright MCP was unavailable this session; see the smoke-test note on the PR.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): stop an infinite microtask loop when persisting a no-output draft handleDraftPersist short-circuits when the open draft's content + inferred writes are unchanged. The writes check compared `d.outputAssets?.length === writes.length`, but a no-output draft has `outputAssets: undefined` (so `?.length` is `undefined`) while the details pane infers an empty `writes: []` (length 0). `undefined === 0` is false, so it never short-circuited: every persist re-wrote the drafts Map with an equivalent object, which gave `activeDraft.script` a new identity → the pane re-emitted its overlays → the graph re-derived → persist fired again. A self- sustaining microtask loop that pegged the renderer and froze the tab on any pipeline carrying a no-output draft (e.g. hydrating one from the saved data_pipeline draft on load). It hangs rather than throwing effect_update_depth_ exceeded because it cycles across microtasks, not within one reactive flush. Fix: coalesce the undefined length to 0 so "no outputs" compares equal to an empty inferred-writes list. Adds pipelineEditorState.test.ts covering the idempotency (fails without the fix) plus the change/no-change cases. Root-caused by instrumenting the reactive churn: every iteration reassigned drafts/liveContent/liveBodyAssets/liveAnnotations/displayGraph with identical values — pure reference churn off the drafts re-write. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ai-chat): make the agent open the pipeline editor before building nodes In a session, the GLOBAL system prompt only *advised* opening the pipeline preview ("show its graph with open_preview ... prefer those tools once it is open"), so the agent routinely skipped it: on a plain "build a data pipeline" request it reached for write_script and staged plain script drafts, and the canvas editor never opened. build_pipeline_node / edit_pipeline_node are only registered once the preview is open, so skipping open_preview also loses the canvas-staged Accept/Reject diff-approval flow entirely. Make the guidance imperative: open_preview(kind="pipeline", path=<folder>) is the FIRST step before creating any node (an empty or not-yet-created folder is fine — create_folder first if needed), and pipeline nodes go through build_pipeline_node / edit_pipeline_node, never write_script. This also clears the agent's "the folder might not exist" hesitation that pushed it toward write_script. Verified live (same plain prompt, before/after): before it used write_script with no editor; after, the agent opens the editor first and stages a canvas-highlighted node with Accept all / Reject all. The guidance is gated on previewTools (session-only), so it doesn't affect the non-preview global eval cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pipeline): preserve in-session pipeline drafts across editor hide/show The session preview's PipelineEditorState lived in the PipelineEditorView component with persistDrafts=false. Hiding the editor sets editorVisible=false, which makes `hasEditor` false and the `{#if hasEditor}` block unmount the view — discarding its component-local store. Showing it again remounted a fresh, empty one, so the pipeline the AI had built in the session vanished. Move the PipelineEditorState onto the per-session SessionRuntime (like the flow / script / raw_app editors, which already host their state there and take {runtime}), so it survives the pane unmount on hide and across session switches. The runtime is keyed by session id and only dropped on session deletion. Because the instance is now reused, guard against a retarget to a different folder: PipelineEditorView resets the state when `path` changes to a new folder (a same-folder remount keeps the drafts). Adds `folder` + `reset()` to the store. Verified: build a node in a session → Close editor → Show editor → the staged node, its wiring, the details-pane code, and Accept/Reject all re-appear. Full svelte-check clean; 139 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(pipeline-ai): clearer diff + persistent review banner on the canvas The AI review affordance had two problems on the pipeline canvas: - The floating Accept-all / Reject-all bar sat bottom-center, where it collided with the minimap once the canvas narrowed on node selection — reading as "the buttons vanished when I select a node". - Every staged draft rendered with the same blue ring, so it wasn't clear what the review would actually change (a plain manual draft looked the same as an AI proposal). Replace the floating bar with a top-left review banner (z-30, clear of the controls and minimap) that stays put regardless of selection and spells out the pending counts. Color the diff per node: a proposal that adds a node that isn't deployed rings green with a "new" chip; one that edits an already-deployed node rings amber with an "edited" chip. Plain manual drafts keep the neutral gray dashed border, so only the green/amber nodes read as part of the Accept/Reject set. aiPendingKind is resolved in resolveGraph (deployed runnable present → modified, else added) and forwarded through the canvas to the node. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): persist in-session pipeline proposals across reload/switch Staged AI proposals lived only in the per-session runtime's in-memory PipelineEditorState (persistDrafts=false), so a page reload — and an LRU-evicted runtime on session switch — dropped them, leaving the canvas and the Accept/Reject review empty even though the chat still showed the nodes as staged. Enable the same per-folder DB-draft persistence the route page uses for the in-session editor. To keep hide/show cheap and race-free, hydration is now gated per editor instance (PipelineEditorState.hydratedFromDb) rather than per component mount: the runtime-hosted instance hydrates ONCE when fresh (reload / evicted runtime) and then keeps its in-memory drafts across the editor pane unmounting on hide — re-reading the DB on every remount would race a not-yet-flushed autosave and drop a just-staged draft. A folder retarget resets the flag so the new folder re-hydrates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): make Reject all work for rehydrated proposals rejectAll only reverted paths tracked in the in-memory aiSnapshots map, which is rebuilt empty on each editor mount. After a reload (or session switch into a fresh runtime) the proposals are restored from the persisted draft but have no snapshot, so Reject all was a no-op on exactly the nodes it should discard. Sweep any still-pending draft without a snapshot and discard it (revertPath with no snapshot deletes the path; for an edit of a deployed node that correctly falls back to the deployed body). Adds unit coverage for accept/reject including the no-snapshot case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): keep proposals visible while the graph reloads on switch The session editor pane is LRU-capped (MAX_WARM_EDITORS), so returning to a session whose pane was evicted remounts PipelineEditorView with a fresh graphRes resource (loading=true, current=undefined). The deployed-graph loading spinner gated the whole canvas, so the staged proposals and the Accept/Reject review banner vanished until the re-fetch resolved — read as "the proposal disappears when I switch sessions". Only show the loading/error placeholder when there are no drafts to display. When the runtime already holds staged drafts, render the editor immediately: resolveGraph overlays them on an empty base so the proposals + banner stay visible, and the deployed nodes fill in when the fetch completes. Verified with a 4s-delayed graph fetch — proposals render through the load with no spinner. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(pipeline-ai): apply AI node edits directly as drafts, no approve/reject The canvas-level Accept all / Reject all review (aiPending proposals, the green/amber diff ring + "new"/"edited" chips, and the review banner) didn't fit the pipeline editor. Match the flow/script editor instead: build/edit apply directly as ordinary unsaved drafts on the canvas, which the user then deploys — there is no separate approval step. Removed across the surface: - aiPending / aiPendingKind on the runnable node + resolveGraph seeding + canvas forwarding; AI-built nodes now render with the existing plain unsaved-draft dashed styling. - the review banner, count derivations, and hasAiPending/onAccept/onReject props from PipelineGraphEditor and both consumers (route page + session view). - acceptAll/rejectAll/hasPending and the per-turn snapshot bookkeeping from the shared helpers; removeProposedNode now just discards the unsaved draft at a path (undo a build). acceptAllProposals/rejectAllProposals/ hasPendingProposals dropped from the PipelineAIChatHelpers interface and the manager's auto-accept hook. - accept/reject language from the tool descriptions, return messages, and the system-prompt section. Tests updated; pipeline + AssetGraph suites pass (142). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(drafts-diff): support data_pipeline diffs + fix blank empty-summary row Two issues in the session "Drafts" diff drawer (DraftDiffDrawer): - Clicking a `data_pipeline` bundle row threw "Draft diff not supported for kind data_pipeline" (utils_draft_deploy.ts) — there was no handler for the kind, so it fell to the OVERLAY_GETTERS lookup and errored. The bundle has no deployed counterpart (each node deploys individually as a script), so diff it node-by-node: surface each node's draft body keyed by path, folding in the deployed body as the "before" when a node edits a deployed script. - A draft row whose summary is an empty string (e.g. the app draft) rendered with no title at all: WorkspaceItemRow's single-line branch used `summary ?? secondary`, and `??` doesn't treat '' as absent, so it showed the empty summary instead of the path. Use `||` so an empty summary falls back to the path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(drafts-diff): explode data_pipeline bundle into per-node subitems A data_pipeline draft is a bundle of node-script drafts, so a single row diffed the whole thing as one blob. Explode it in DraftDiffDrawer into one script row per node, nested under the bundle's `…/data_pipeline` folder so they read as the pipeline's subitems — each with its own path and a proper script Content/Metadata code diff. The node's draft body is the "after"; its deployed body (when the node is already deployed) is the "before", so edits show as line diffs and new nodes as added. A single bundle row (via the getDraftDiffValues data_pipeline fallback) is kept only for the case where the bundle can't be read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(pipeline-ai): simplify — drop vestigial approve/reject scaffolding & redundant field Review pass over the PR, removing complexity left from the approve/reject removal and the shared-component refactor (all behavior-preserving): - Inline the `acceptPendingEdits` pass-through into `acceptPendingFlowEdits` and revert the now-inert `autoAcceptEditsAvailable` GLOBAL+pipeline widening (pipeline edits are direct drafts — nothing to auto-accept). - Fix the global system prompt: pipeline tools "apply directly as unsaved drafts (no accept/reject)", not "proposals the user Accepts or Rejects". - Collapse the redundant `outputAsset` (singular) into `outputAssets`, removing a whole resolveGraph fallback tier; simplify propose/editNode. - Drop the single-field `PipelineAiHelpersHandle` wrapper (callers just destructured `{ helpers }`); inline the misleading `isoNow()` helper. - Remove the now-unreachable `data_pipeline` branch in getDraftDiffValues (the drafts drawer explodes bundles per-node; an unreadable bundle is skipped) and the "Step N consolidation" drafting narration. - Un-export internal-only types; reuse `storageKey`; refresh stale comments that still referenced proposals / the review banner / diff-approval. svelte-check clean; 141 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline): tooltip clarifying the Create/Save button deploys The accent button in the asset-graph details pane ("Create" for a new script, "Save" for an existing one) is really a deploy, but had no tooltip explaining that. Add a title — "Deploy this new script to the workspace" / "Deploy your changes to this script" — keeping the create-vs-update label distinction while making clear both deploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): document the `materialize` annotation in the pipeline prompt The model invented "materialize run" because the prompt only mentioned `// materialize <uri>` in passing. Spell out what it is in both the in-app pipeline prompt (getPipelinePromptSection) and the base prompt (pipeline-base.md, regenerated): a MANAGED output where the runtime writes the table around a single SELECT (no manual CREATE/INSERT); replace (default) vs `append` vs `key=<col>` strategies; `manual` to opt out (track-only); and its pairing with `// partitioned …` (runs once per partition, `{partition}` token substituted at run time). Explicitly: materialize is an output declaration, not a command — there is no "materialize run". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline-ai): trigger drawers in the AI session preview Bring the route page's native-trigger affordances to the in-session pipeline editor by reusing the shared <PipelineTriggerEditors> (no duplication of the drawer UI). Clicking a "Schedule · Missing — no trigger row" node (or edit/delete on an attached trigger, webhook, data-upload) now opens the same drawers the full editor uses, instead of doing nothing. Draft nodes get the same "save the script first" guard (a trigger row needs a deployed script). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline-ai): run buttons + live run state in the AI session preview Wire the per-node Run button and live run-state badges into the in-session pipeline editor, reusing the shared folder-scoped job poll (useActiveRunnableIds) the route page uses — node badges, the event log, and the zero-latency "running" hint all come from it. The session runs one node at a time (preview for an unsaved draft, the deployed version otherwise), skipping the route page's cascade/deploy-queue machinery the AI-session UX doesn't need. Verified: a node's Run button dispatches a job and the badge updates live from the poll. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(pipeline): label the node deploy button "Deploy" (was Create/Save) Users read "Create" and asked whether it deploys. It does — and the main script editor's DeployButton already says "Deploy", so this is the consistent term. Use "Deploy" for both the new-script and existing-script cases; the new-vs-changes nuance stays in the button's tooltip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(home): surface data pipelines as units, including bundle-phase drafts Treat a data pipeline as one home entry instead of scattering its member scripts: - The home "Pipeline · f/<folder>" entry now also covers bundle-phase pipelines — a folder that so far only exists as a `data_pipeline` draft — not just deployed ones, so a pipeline shows up the moment its first node is drafted (union listPipelineFolders + data_pipeline draft folders). - Pipeline-member scripts (`auto_kind='pipeline'`) are filtered out of the individual scripts list; they're represented by their pipeline's entry. - Tree view injects pipeline folders so they (and their "Pipeline" entry) still appear when their only scripts are hidden members or they have none deployed yet. Verified in both list and tree view: app_groups (deployed member folded) and a draft-only nyc_transit both show as pipelines; the member script no longer lists individually. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(scripts): compute auto_kind for draft-only pipeline nodes A never-deployed pipeline node (a script draft starting with `// pipeline`) had no script row, so list_scripts synthesized it with `auto_kind: None` — and the home page therefore couldn't tell it was a pipeline member, listing it individually instead of folding it into its pipeline. Parse the draft content the same way the create path does (`parse_pipeline_annotations(...).in_pipeline`) and set `auto_kind = "pipeline"` on the synthesized draft-only row, so draft nodes fold into their pipeline like deployed members. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(search): hide pipeline-member scripts from global search The Ctrl+k global search listed pipeline-member scripts (`auto_kind='pipeline'`) individually. Filter them out — they're reached through their pipeline, matching the home page. Deployed members carry auto_kind from the script row; draft-only members now do too (computed from draft content in list_scripts), so both are excluded here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address PR review findings Session run dispatch (the one real bug): - runNode now passes `_wmill_skip_asset_dispatch: true` for a single-node run of a deployed node unless the user chose "run + downstream" (cascade) — previously a single Run could fan out to downstream deployed scripts via the backend asset dispatcher and fire side-effecting production runs. - onRunProducer guards `kind === 'script'`; onTestStateChange only clears the run hint for the script the pane finished (not a different in-flight node); clear the hint on folder retarget; gate the background poll on isActiveSession so hidden warm panes don't poll; note the PipelineTriggerEditors workspace coupling. Home page pipeline surfacing: - Fold pipeline-member folders into `pipelineFolders` (captured in loadScripts) so a members-only / draft-only-`// pipeline` folder still shows its pipeline entry instead of vanishing; and don't render the empty-state when only pipelines remain (they aren't part of the text filter). - Insert injected tree folders in name order instead of prepending. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): make clear `// materialize` is DuckDB + DuckLake only The model put `// materialize` on a python3 node, which deploy rejects ("only supported for DuckDB scripts"). The prompt only implied SQL ("write the body as a single SELECT") without stating the hard constraint. Spell it out in both the in-app prompt and pipeline-base.md: `// materialize` is DuckDB-only and its target must be a DuckLake table; for python3/bun/postgresql nodes, write the output via the SDK instead and let it be inferred — reach for duckdb when a node should materialize a DuckLake table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): fix stale comment — session now wires run + trigger affordances Addresses review: the comment still claimed the session 'opts out of the run/cascade/trigger/bounded affordances', but run buttons + trigger drawers were wired in. Describe the current state (wires run + triggers; omits only cascade/bounded/add-script). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address Codex review — test_pipeline_node dispatch + tree search - [P1] testNode (the test_pipeline_node tool) ran a deployed node via runScriptByPath without `_wmill_skip_asset_dispatch`, so previewing one node could fan out to downstream deployed subscribers and run side-effecting scripts. Add the skip flag (test is always single-node) + a regression test. - [P2] Home tree view injected pipeline folders — and rendered their Pipeline row — even during a text search, surfacing unrelated pipelines. Gate both the TreeViewRoot injection and TreeView's hasPipeline on `!isSearching`, matching the list view which hides pipeline rows on a query. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): keep the pipeline prompt after update_user_instructions rebuildGlobalSystemMessage (called by the update_user_instructions tool) rebuilt only the base Global prompt, dropping the pipeline-editor section that configureGlobalMode appends. So after the chat remembered an instruction, the next GLOBAL turn lost the active /pipeline/<folder> context + direct-draft/ materialize guidance while pipeline tools stayed registered. Re-append the pipeline section here when a pipeline editor is registered. Addresses Codex review [P2]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(home): gate pipeline entries by kind/archived/owner filters Codex review [P2]: pipeline rows/folders rendered independently of the item filters, so a pipeline still showed under the Flows/Apps tabs, in the archived view, and outside a selected owner. Add `visiblePipelineFolders` applying the same gates the items get (kind ∈ {all, script}, not archived, owner-prefix match) and route the list rows, tree injection, and empty-state check through it. Pipelines are always `f/<folder>`, so the user-folder toggle and kind=script keep including them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): address review — route folder-switch state, AI node guards, diff identity claude[bot] [P1]: the route page's in-app folder switcher navigates same-route (no remount), but nothing reset PipelineEditorState — so folder A's drafts displayed under B and autosave persisted them into B's bundle, and B never hydrated. Reset pe on folder change (mirror the session retarget), and guard the shared hydrateDrafts against a stale folder result landing after a retarget. codex/claude [P2]: build_pipeline_node (proposeNode) only checked drafts.has — now rejects a path outside the open folder and one colliding with an existing deployed node (model should edit_pipeline_node). + 3 regression tests. codex/claude [P2]: exploded pipeline-node diff rows shared `script/<path>` with a standalone script draft at the same path, colliding in the {#each} key + value cache. Add an explicit unique `key` (the distinct bundle-nested path) on DiffRow; pipeline nodes set/look up by it while `path` stays the real edit target. claude [P2]: session AI test_pipeline_node now arms the live run badge (onRunStarted), matching the route page. nit: pipelineAiHelpers.test uses afterEach(restoreAllMocks) instead of an unreachable inline mockRestore. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline): harden AI node mutations + close home label-filter / rename gaps Codex [P1] (AI mutations trust model paths) — fully scoped now: - editNode validates the open folder too (proposeNode already did), via a shared assertInFolder; an edit_pipeline_node for f/other/* no longer persists an unrelated script into the current folder's data_pipeline bundle. - both build_pipeline_node and edit_pipeline_node now require the `// pipeline` annotation (assertPipelineAnnotation) so a staged draft is definitionally a pipeline member, not a silently-non-member script. + tests. (proposeNode's folder + deployed-collision guards landed in the prior commit.) Codex [P2] home label filter — visiblePipelineFolders ignored labelFilter, so a label selection still showed every pipeline (and the empty-state fell through to render pipeline rows). Pipelines carry no labels, so a label filter hides them. Codex [P2] session rename — PipelineEditorView now wires onScriptRenamed (repoint selection + refetch), matching the route page; a persisted-script rename no longer leaves the canvas on the old path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(pipeline-ai): language-specific comment prefix for annotations Codex [P2]: the tool schema and prompt told the model to write `// pipeline` / `// on` / `// materialize` regardless of language, and pipeline-base.md grouped SQL with `#`. A `//` (or `#`) annotation line is invalid in a DuckDB/Postgres node — it passes the frontend parser (which strips `//`/`--`/`#`) but is a SQL syntax error at deploy/run. Make the guidance language-specific everywhere: `--` for SQL (duckdb/postgresql), `#` for python3/bash, `//` for bun/TS — the `//` in examples is the TS form to translate. Regenerated the prompt outputs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): re-scope Global prompt on folder switch + language-aware base prompt Codex [P2] x2: - The route page resets editor state on an in-app folder switch, but the Global chat's system message kept the old `/pipeline/<folder>` scope (the helper methods read the reactive folder, but the prompt string is only rebuilt on Global-mode reconfigure). Rebuild it on folder change so the next turn targets the new folder. - The pre-editor base Global prompt (seen before open_preview/get_instructions) still showed TS-only `// pipeline` / `// on`. Make it language-aware (`--` SQL, `#` Python/Bash, `//` TS) so the model can't draft invalid DuckDB/Postgres nodes before the pipeline tools are registered. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): authoritative new-node probe + SQL-correct eval checklist Codex [P2] x2: - build_pipeline_node's collision check relied on the resolved graph, which can be empty while the session preview races open_preview (a build could shadow a deployed node before the graph loads) and only covered pipeline runnables, not a non-pipeline script at the same path. Add an authoritative backend probe (ScriptService.getScriptByPath): any deployed script at the path → reject with "use edit_pipeline_node". + regression test (empty graph, deployed script). - The DuckLake eval judgeChecklist required the exact `// pipeline` annotation, which would penalize the now-correct `-- pipeline` SQL output (or reward invalid DuckDB syntax). Make both cases syntax-aware. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): rebuild Global prompt on session preview folder retarget Codex [P2]: open_preview(kind="pipeline", path="B") can retarget an existing pipeline preview from folder A to B without remounting. The retarget effect resets editor state and the helper methods read the new path, but the registration effect only depends on isActiveSession, so the Global system message stayed scoped to /pipeline/A. Mirror the route-page fix: rebuild the global system message on retarget (gated on isActiveSession — only the active session's helpers are registered; a hidden session reconfigures when it next becomes active). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(pipeline-ai): edit_pipeline_node preserves deployed script metadata Codex [P1]: editNode kept only the deployed script's language and staged a fresh makePipelineScript draft with empty hash/summary/description/tag/schema/settings. Deploying that edit from the pane (auto_parent) would update the script while wiping its metadata, and the route "Save all" path (no parent_hash) could hit the backend path-conflict branch on the occupied path. Base the draft on the existing draft's / deployed script object and replace ONLY content (+ inferred output assets), preserving hash and metadata. + regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszelandrubenfiszel
|
96c0ff65bd |
chore(main): release 1.742.0 (#9830)
* chore(main): release 1.742.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
|
75ba81b2d2 |
fix(audit): don't read pg_authid from an elevated context in S3 export migration (#9832)
* fix(audit): don't read pg_authid from an elevated context in S3 export migration Migration 20260626132251 aborted instance startup on managed Postgres (e.g. Cloud SQL) with "Modifying pg_authid or pg_auth_members is not allowed in elevated context": the audit S3 export "oldest in-flight xact_start" floor probe calls pg_has_role(...), which reads pg_authid, and managed providers forbid that read from an elevated context. The migration ran the probe inline in its UPDATE, so the whole migration — and the instance boot — failed. Extract the probe into a shared SQL function audit_logs_s3_oldest_inflight_ts() that returns the oldest in-flight xact_start (when cluster-wide stats are visible) or NULL otherwise. The pg_has_role read is wrapped in a plpgsql BEGIN/EXCEPTION subtransaction, so a pg_authid failure returns NULL (callers fall back to a conservative 7-day window / reject) instead of aborting. is_superuser (a GUC, no catalog read) is checked first to short-circuit. The migration's trigger and UPDATE, the OSS backfill try_start, and the EE exporter/startup anchor (companion windmill-ee-private PR) all route through it. Because 20260626132251 already shipped, it is added to the potentially_stale list in windmill-api/src/db.rs: on startup the stale _sqlx_migrations row (checksum mismatch) is deleted and the fixed, idempotent migration re-applies, so already-migrated instances upgrade without a checksum-mismatch boot failure. Fixes WIN-2108 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 95352c13c4c82247d8cfd80936f9203aeb079802 This commit updates the EE repository reference after PR #635 was merged in windmill-ee-private. Previous ee-repo-ref: 136f49a52af922868acac33abf8198913a9e835c New ee-repo-ref: 95352c13c4c82247d8cfd80936f9203aeb079802 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
c0768de0ac |
fix: close unauthenticated DAP debugger program-mode launch bypass (#9829)
The /ws_debug debugger WebSocket gated JWT signature verification on inline `code` being present (`if (code && REQUIRE_SIGNED_REQUESTS)`), so a `program`-mode launch (naming an arbitrary server-side file path that is read and executed) skipped verification entirely — even with REQUIRE_SIGNED_DEBUG_REQUESTS=true. The WS handshake also performed no Origin check, allowing cross-origin (CSWSH) drive-by from a malicious page. - Enforce signing on every launch in both handlers (Python + Bun/TS): reject program-mode outright and require+verify a token for inline code. - Add opt-in DEBUG_ALLOWED_ORIGINS allowlist enforced at the WS handshake. - Default docker-compose REQUIRE_SIGNED_DEBUG_REQUESTS to true. - Update THREAT_MODEL T8/EP15 to reflect the root cause and mitigation. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
003a262a4e |
feat: column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) (#9814)
* feat: column-level lineage for ducklake pipelines via // column annotation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: auto-derive column lineage from DuckDB SQL AST (annotation as override) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: clarify column-lineage inference is server-side; drafts use annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): surface inferred column lineage in live pipeline drafts Threads the DuckDB SQL-AST column lineage (from the WASM asset parser) through ScriptEditor -> details pane -> page -> resolveGraph, merged with // column annotations (annotation wins) so the live preview matches the deployed graph. Takes effect once windmill-parser-wasm-asset is republished with the inference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(frontend): bump windmill-parser-wasm-asset to 1.740.0 for SQL column-lineage inference Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: column-lineage inference now runs live (WASM) too, merged with annotations Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): transitive column-lineage trace (impact analysis) Stitches every producer's column_lineage into a pipeline-wide column graph (columnLineageGraph.ts) and replaces the single-hop diagram with an interactive ColumnLineageTrace: select an asset to see its columns' full upstream/downstream lineage across scripts; click any column to highlight its complete transitive impact set (forward + backward) and dim the rest. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on column lineage (parse-fallback, node-id, perf, leak) - backend: DuckDB SQL parse failure now falls back to `// column` annotation lineage instead of dropping it (Codex P1) - columnLineageGraph: collision-proof JSON node ids; deterministic first-write output anchoring when a producer has multiple ducklake writes (cubic P2 ×2) - pipeline page: gate buildColumnGraph to a ducklake-asset selection so it doesn't rebuild on every editor keystroke (cubic P2) - ScriptEditor: clear inferredColumnLineage on parse error so it can't leak across a script switch (cubic P2) - AssetGraphEdge: widen badge stacking offset 12px->18px to fully clear (cubic P3) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: resolve JOIN inputs + anchor column lineage to // materialize target Addresses the second Codex review pass (two P1s): - SQL inference now walks JOINed tables: build_from_maps maps every FROM entry AND its joins into the alias map, and single-table attribution requires no joins. `SELECT o.x, c.y FROM a o JOIN b c` now resolves c.y (was dropped). - The column graph anchors a producer's lineage to its declared // materialize target (surfaced on the runnable node) instead of guessing a ducklake write-edge, which is unordered for deployed graphs and ambiguous for multi-output scripts. Falls back to a write-edge when no materialize target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: gate column-lineage badge to the // materialize target write-edge The canvas badge keyed on `e.asset_kind === 'ducklake'`, so a multi-output producer showed the same column mapping on every ducklake write-edge. Use the same materialize-target anchor as buildColumnGraph: the badge lands only on the declared output's edge, falling back to the ducklake write-edge when there's no materialize annotation. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: build column trace from displayGraph so View hides draft lineage The transitive column trace was built from graphWithDraft regardless of mode, so in View with drafts hidden it could surface draft `// column` lineage the deployed canvas doesn't show. Build it from `displayGraph` (the graph the canvas actually renders) so the trace matches: draft overlays in edit / show-drafts, deployed-only in plain View. (Codex P2) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: don't infer column lineage for local/temp staging CTAS A CTAS into a local/temp staging table isn't the materialized output, but its projection was inferred and (flat) column_lineage anchored to the script's // materialize target — so staging columns showed up as the final asset's. Gate inference to the actual output: a top-level managed-materialize SELECT, or a CTAS/CREATE VIEW whose target resolves to a real asset. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: scope inferred column lineage to one output asset Inference accumulated columns from every output-producing query into one flat list, all anchored (frontend) to the script's // materialize target — so an auxiliary CTAS into a different asset showed its columns on the materialized one. Tag each inferred entry with its output asset and, in parse_assets, scope the list to the // materialize target (keeping untagged top-level-SELECT entries); with no declared target, drop inference when entries span multiple output assets rather than attribute them to an arbitrary one. Parser-internal — no wire change. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: treat CREATE TEMP TABLE/VIEW as local even under an active USE A one-part temp name under `USE dl` resolved to an asset (ducklake://…/tmp) before being registered local, so a final SELECT reading it invented `final.total <- warehouse/tmp.amt` (a phantom DuckLake column) and recorded a phantom asset. track_table_definition now registers any temporary table/view as local up front, bypassing active-asset resolution; CreateTable/CreateView pass their `temporary` flag. (Codex P1) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben Fiszelandrubenfiszel
|
9172a0945b |
chore(main): release 1.741.0 (#9804)
* chore(main): release 1.741.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
|
577ceeee86 |
perf(audit): re-anchor S3 audit export on enable + opt-in backfill (#9818)
* [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill
The S3/GCS audit-log export's steady-state query filters by `age(xmin)`
(unindexable), so the only scan bound is the timestamp floor. On a fresh
enable the floor was epoch, and on a re-enable the cursor resumed from its
pre-disable position — either way the first run scanned the whole
`audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan
never completes: the cursor never advances, nothing is exported, and the
repeated full scans saturate the database.
Re-anchor on enable (EE companion, windmill-ee-private#634):
- New trigger migration records a recent timestamp floor instead of the epoch
sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on
re-enable, so the export always resumes from ~now and never rescans history.
Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade.
Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints):
- Exports a chosen `[from, to)` window on demand, scanning strictly by
`timestamp` (the partition key) in bounded keyset pages — each query is an
index scan capped at one page (verified via EXPLAIN: later partitions
`never executed`, ~11ms/page), so it stays well under any statement timeout
regardless of window size. Writes alongside the steady-state objects under
logs/audit/, without touching the xmin cursor.
- POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise),
GET /settings/audit_logs_s3_backfill_status.
Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a
backlog" (the cursor is capped and catching up), and updates the setting
description to point operators at the backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): heartbeat backfill lease per object; bump EE ref
Address review (cubic): persist progress (refreshing the lease heartbeat) after
every object PUT in the backfill page loop, not only once per page, so the gap
between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads
and another replica can't re-claim mid-page and run a concurrent backfill.
Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain
regression into the single audit e2e test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject unstable backfill windows; bump EE ref
Address review (P1): the backfill keyset-pages over rows visible at scan time
and declares completion when the scan runs dry, but a row's `timestamp` is its
inserting transaction's `xact_start`. A window whose upper bound is recent or in
the future could silently omit a transaction that started inside `[from, to)`
but commits after the scan passed that timestamp. `try_start` now rejects any
`to` newer than the oldest in-flight `xact_start` (everything strictly older
than the oldest running transaction is committed and stable), using the same
trustworthy stats gating as the exporter's floor (restricted role / 2PC → a
7-day-old cutoff).
Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor
Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts
to now()-7d while leaving the old last_xmin in place. On an instance that
enabled export on the old code >7 days ago and got stuck before the first
successful batch, the next run would filter post-enable rows older than 7 days
out via `timestamp >= ts_floor` while still advancing last_xmin over the
interval — silently dropping them (the same floor-vs-cursor loss class fixed
elsewhere in this PR), and contradicting the "nothing committed after enabling
is skipped" guarantee.
A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be
arbitrarily old, so any recent floor prunes rows the cursor then skips, and an
epoch floor reintroduces the full scan). Re-anchor it to the migration's current
snapshot xmin instead — exactly like a fresh enable — so the export resumes
cleanly from ~now and the never-exported pre-upgrade window is recovered via the
opt-in backfill rather than silently dropped. Reword the setting description so
it no longer implies the disabled/legacy window is covered by the cursor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(audit): end-to-end integration tests for the object-store backfill
The backfill previously had only SQL-level/EXPLAIN validation. Add real
integration tests (in-memory object store, sqlx::test) exercising the public
path:
- backfill_exports_window_in_pages: with the page size forced to 2 rows, a
settled 3-day window is exported across multiple keyset pages; asserts every
in-window row lands exactly once, rows outside [from,to) are excluded, a day
that straddles a page boundary yields more than one object, progress counts
match, and a re-run is idempotent (deterministic keys overwritten, no dupes).
- backfill_rejects_unstable_window: a future/live `to` is rejected as unstable,
a window safely in the past is accepted.
Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a
handful of rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(audit): note backfill scope is audit_partitioned only
Make explicit that, like the steady-state export, the backfill reads only
audit_partitioned; the pre-partitioning `audit` table is intentionally out of
scope (not a missed case).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill windows before the partitioned boundary
Address review (Codex P1): the backfill reads only audit_partitioned, but
pre-partitioning history lives in the legacy `audit` table (still read by audit
list/get via UNION ALL, and retained for the configured period — 365 days by
default on EE). Since the setting text points operators at this API for
"pre-existing history", a window overlapping legacy rows would report completion
while silently omitting them.
Per the decision to not export the legacy table, reject instead of silently
omit: try_start now rejects a `from` earlier than the oldest audit_partitioned
timestamp (every legacy row predates the partition cutover, so a `from` at/after
that boundary can never overlap them). Reworded the setting text to scope the
backfill to the partitioned era. Added a regression test, plus an RAII guard
(cubic P2) so the test-only globals are restored even if an assertion panics.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): backfill object keys per-window; require trustworthy settled cutoff
Address review (two P1s):
- Object-key overwrite loss: keys were `dt=<day>/audit_backfill_<min_id>.ndjson`.
A narrower, overlapping backfill can start a day's page at the same first row
(same min_id) but hold fewer rows, and `put` would overwrite a broader run's
object — silently dropping the rows only that object held. Include the
requested window in the key so different ranges write disjoint objects (same
window re-runs stay idempotent; consumers dedupe overlapping rows by id). New
regression test (verified red→green).
- Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role
lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old
now()-7d fallback could still let an old transaction commit rows inside an
accepted window after the scan, so a "complete" backfill silently missed them.
Since a backfill asserts completeness, reject in those cases instead of
falling back. (The continuous exporter keeps its 7-day fallback — it only
claims bounded lag.)
Also makes the tests robust under the parallel runner: run_backfill takes the
store as a param, so tests pass a local in-memory store (no global
OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment
Address review (1 P1 + 2 P2):
- Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when
audit_partitioned is empty, so an upgraded instance with legacy `audit` rows
but no partitioned rows yet would accept a window and complete with zero rows,
silently omitting the legacy rows. Check the legacy `audit` table directly:
reject any window that overlaps a legacy row (subsumes the boundary check and
covers the empty-partitioned case). Test updated accordingly.
- openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!)
so /openapi.{yaml,json} expose the new backfill endpoints.
- Migration comment (P2): trim the PR-history narration to the durable
constraints (why a recent floor and a monotonic cursor are required), per
AGENTS.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d
This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private.
Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89
New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
ade74b297f |
feat: capture managed-materialize output schema as asset metadata (#2a) (#9812)
* feat: capture managed-materialize output schema as asset metadata (#2a) After a managed `// materialize` run, capture the producer's output schema via a DESCRIBE folded into the existing one-row summary read (no extra round-trip) and persist it in a new versioned `materialized_asset_schema` sidecar table. This is the producer-side capture that pipeline parity gap #2b (save-time consumer-ref contract enforcement) will read back. - materialized_asset_schema sidecar (asset-level grain), versioned: a new version row is inserted only when the captured column set changes. - output_schema column added to the materialize summary codegen. - worker extracts + records the schema on a successful materialize. - /assets/asset_schemas read endpoint exposing the evolution history. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address CI review on schema capture (partition col, order, status gate) - exclude the synthetic `_wm_partition` column from the captured schema for partitioned assets, so the recorded contract is the producer's logical output, not Windmill's storage detail (claude/cubic P1). - make the captured column list explicitly ordered (`row_number()` over the DESCRIBE + `list(... ORDER BY)`), so the `list()` aggregate can't reorder columns and spuriously bump the schema version (cubic P2). - gate the API `record_materialization` schema upsert on a `Materialized` status, so a failed/running write (or a client attaching a schema to one) can't advance the schema history (cubic P2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address Codex review (manual-mode schema gate + auth contract docs) - gate output_schema extraction on the managed (`Some((Some(_), _))`) path so a `// materialize manual` run — whose result is the user's own query output — can't persist a caller-shaped `output_schema` into materialized_asset_schema (Codex P2). Verified e2e: a manual run returning a fabricated `output_schema:[{injected,EVIL}]` records the partition but writes no schema version, while the managed path still captures normally. - document the authorization contract on the new public `record_asset_schema` and `list_asset_schemas` helpers: they perform no access control (mirroring the materialized_partition siblings) and require callers to pass a workspace-authorized executor (Codex P1). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(frontend): schema-history tab on the ducklake asset node (#2a) Adds a "Schema" tab to DucklakeAssetPanel surfacing the captured output-schema versions persisted by the materialize run. Master-detail (mirrors the History tab): the version list (newest first, newest auto-selected) shows column count + snapshot + capture time; selecting a version renders its column/type table. Reads the GET /assets/asset_schemas endpoint via raw fetch, matching the sibling PartitionStatusGrid convention (these materialization endpoints are not in the generated client). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat: schema tab is strategy-aware (history vs fixed schema) Only a whole-table `replace` producer (CREATE OR REPLACE) can change columns run-to-run; `append`/`merge`/partitioned writes INSERT into a fixed-schema table, so their schema is pinned at first materialize and the "history" framing is degenerate (always one version). - backend: surface the managed `materialize_strategy` (`replace`/`append`/ `merge`) on the asset-graph runnable node, alongside the existing `partition_kind` (same parse-from-annotation path). - frontend: the pipeline page derives `schemaCanEvolve` for the selected asset from its write-producer (`replace` && not partitioned) and threads it to the Schema tab. Evolvable → master-detail version history; fixed → a single current-schema table with a short "schema is fixed" note. Unknown defaults to evolvable so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: schemaCanEvolve fails open on unknown producer strategy Previously a producer present but missing `materialize_strategy` (e.g. a draft-overlay runnable, synthesized without the field) fell through to canEvolve=false, hiding captured history behind the fixed-schema view — contradicting the "unknown defaults to evolvable" intent. Now the fixed view shows only when *every* producer is a known insert-style write (append/merge, or partitioned replace); any producer with unknown (missing) strategy is treated as evolvable, so real history is never hidden. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
40110bc715 |
fix: skipped suspend step no longer parks the flow forever (#9821)
* fix: skipped suspend step no longer parks the flow forever A flow step that declares a `suspend` (approval) but is skipped via `skip_if` was leaving the flow stuck waiting for a resume that would never arrive. Suspend gates the *next* step: before pushing step N, `needs_resume` checks whether step N-1 declared a non-zero `suspend` and finished as `Success`. A step skipped via `skip_if` is also recorded as `FlowStatusModule::Success` (with `skipped: true`), so `needs_resume` treated a skipped approval gate as a real one and parked the flow waiting for an event that nothing ever sends — until the suspend timeout (up to 24h). The skip is most visible when the skipped suspend step is followed by a branch/subflow: the flow appears stuck on the *following* predicate node with a generic resume button, while none of the branch/subflow steps ran. Fix: honor the `skipped` flag in `needs_resume` and do not gate the next step on a suspend that was skipped. Adds regression test `skipped_suspend_step_does_not_block_next_step` (times out without the fix, completes with it). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: reword regression test comment as a current invariant Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
1c4bb8af14 |
test: de-flake asset-dispatch by bypassing cross-DB script-hash caches (#9820)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
3cda447621 |
fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751) (#9813)
* fix(flows): reject corrupt step paths at deploy + atomic cache writes (#9751) A flow step could execute an unrelated (and in the reported case, destructive) script at runtime even though every stored definition looked correct. A forensic dump traced it to two issues: - Deploy accepted absolute/local step paths. `wmill sync push` from a feature- branch checkout under /tmp baked an absolute path (`/tmp/.../ops/scripts/clean_device/...`) into a step's `value.path`. Persisted verbatim, it mis-resolved to an unrelated script at runtime. - The on-disk cache write was neither truncating nor atomic. `FsBackedCache::put` used `write+create`, so a shorter overwrite left stale trailing bytes and concurrent writers could interleave into a torn file — a corrupt cached blob that a worker then scheduled from. Fixes: - Reject non-workspace flow step paths (must be u/, f/, g/ or hub/) in `validate_flow_value` (covers create_flow + update_flow, recursively through loops/branches/AI-agent tools) and early in the CLI `pushFlow`. - Make `FsBackedCache::put` write a unique temp file (truncate + fsync) then atomically rename it over the target, cleaning up on error. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(flows): validate failure/preprocessor module paths + sub-flow paths in CLI Address PR review (cubic + claude): - Backend `validate_flow_value` is the authoritative guard but only walked `modules`; extend it to also validate `failure_module` and `preprocessor_module` (which can themselves be sub-flows/loops/branches), so an absolute path there can't be persisted. - CLI preflight only collected `type: "script"` paths; now collects sub-flow (`type: "flow"`) step paths too (recursively, incl. failure/preprocessor), so the comment's claim matches the behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cli): include AI-agent tool step paths in flow path preflight Address Codex review: collectStepPaths skipped aiagent tools, so a bad path in a tool fell through to the API error instead of the local fail-fast. The backend already validates these (traverse_modules walks AIAgent tools); this aligns the CLI early-error with it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(flows): make failure/preprocessor path test key explicit The test used `slot:` as a json! key. json! does interpolate an ident key to its variable's value (json!({slot:1}) with slot="failure_module" => {"failure_module":1}), so the test was correct and exercised the validation — but the behavior is subtle, so build the key explicitly via serde_json::Map to remove ambiguity (review nit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(cache): use a UUID temp name for atomic put (shared-volume safe) Address Codex (P1): pid+counter temp names collide across container PID namespaces on a shared cache volume (same pid, PUT_SEQ resets to 0 per process), so two workers could truncate/clobber the same temp file before rename. Use a random UUID suffix (matching worker.rs's atomic-write helpers) — globally unique, so the cross-process temp-file hazard is closed. Also trims the comment to the AGENTS.md <=4-line limit (Pi nit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
|
43bb676dc5 |
fix: ping job during volume setup to prevent false zombie restarts (#9803)
* fix: ping job during volume setup to prevent false zombie restarts Volume mount setup (S3 lease acquisition wait + download) runs synchronously before the language executor spawns the child process and its ping loop, leaving the job ping frozen. A slow lease wait or cold S3 download could exceed ZOMBIE_JOB_TIMEOUT (default 60s) and get the job falsely restarted as a zombie. Heartbeat the job ping throughout volume setup. EE companion: windmill-labs/windmill-ee-private#633 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 7b92c8e0de4cfc6d986499d60a5f79cd1c6b9d0b This commit updates the EE repository reference after PR #633 was merged in windmill-ee-private. Previous ee-repo-ref: 32e6b9a25f4ec3ea87f429b3d6279f9287a24de7 New ee-repo-ref: 7b92c8e0de4cfc6d986499d60a5f79cd1c6b9d0b Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben Fiszelandrubenfiszel
|
b7a227f860 |
chore(main): release 1.740.0 (#9776)
* chore(main): release 1.740.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
0dbd9c1231 |
perf: eliminate dual-connection DB pool contention across worker, queue, and api (#9798)
* perf: eliminate dual-connection DB pool contention across worker, queue, and api Reuse the held transaction (or move pool reads before begin()) instead of checking out a second pool connection while a tx is open, extending the fix from #9789/#7861. Targets the per-worker pool (max 5) hot paths plus several server-pool API handlers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: pass owned pool to get_email_from_permissioned_as in http trigger handler The generified signature takes impl PgExecutor; the http trigger handler passed &db where db is already &DB, yielding &&Pool which does not impl PgExecutor (only surfaced under the full feature set in CI). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: keep RLS-exposed reads on the non-RLS pool and isolate flow-eval reads in a savepoint Addresses review of the dual-connection sweep: - worker_flow: wrap the stop_after_all_iters_if reads in a SAVEPOINT. The caller swallows the error and keeps using tx, so a DB read failure must not leave the outer transaction aborted (it would fail the later commit). Matches the previous pool-read semantics. - Revert reads that were moved onto an RLS (user_db) transaction back to the non-RLS pool, since RLS row-visibility/role context can change results: push_scheduled_job (email/tag/settings lookups; reachable with a user_db tx from api-schedule/api-flows), push_inner native-retry dedicated_worker routing (RLS isolation variants), resources.rs app-namespace folder auto-create (non-admins must not be blocked), and the script archive/delete UPDATEs. Non-RLS db.begin() reuse and move-before-begin are kept. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: failpoint proving the stop_after_all_iters_if savepoint isolates an aborted read Adds a worker-crate failpoints feature and a data-driven hook: when the stop_after_all_iters_if expr is the magic sentinel, the in-evaluation read runs SELECT 1/0 to abort its (savepoint) transaction. The test asserts the flow still completes (iteration marked failed) — which only holds if the savepoint keeps the outer status-update transaction committable. Without the savepoint the abort would poison the outer tx and the job would never complete. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
b28f974e50 |
fix: opt out of Deno minimum-dependency-age for private npm registries (#9802)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
ba768fee88 |
feat(api): add structured endpoint for flow logs (#9797)
Add `GET /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}` as a
JSON alternative to `get_flow_all_logs`. It returns the same flow log
tree as an array of per-job entries (job_id, label, kind, step path,
depth, parent module type, sibling index/count, and resolved logs)
instead of a single delimited text blob, so callers can render or
process logs per-step without parsing the `=== ... ===` markers.
The shared auth, recursive-CTE query, and label-building logic is
extracted into `collect_flow_log_entries`; the existing text endpoint
now formats those entries and produces byte-identical output.
Fixes WIN-2102
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
|
5549bdc67a |
fix(debounce): never supersede a running debounce survivor (#9780)
* fix(debounce): never supersede a running debounce survivor
Companion to the windmill-ee-private change in upsert_debounce_key.
With debounce_args_to_accumulate + a concurrent_limit, a message arriving
while its debounce survivor is already running was marked completed/skipped
("Debounced Running by ...") and the running survivor deleted from the
queue, silently dropping accumulated elements. A slow step + concurrent
limit keeps the survivor running for a long window, so any arrival during
it was lost. The fix leaves a running survivor untouched and starts a fresh
debounce window for the late arrival.
Adds regression coverage in windmill-queue/tests/debounce_test.rs (push,
flow post-preprocessing, no-accumulation, committed-running, and
max-count-window cases) and refreshes the SQLx cache for the changed
upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): add missing SQLx cache for test-only running-flag query
The cargo_test CI job compiles the test target with SQLX_OFFLINE=true; the
new regression tests use `UPDATE v2_job_queue SET running = true ...` which
was not in the offline cache (the library-only `cargo sqlx prepare` skipped
test targets). check_oss/check_ee passed because they don't build tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): harden running-survivor guard against concurrent arrivals
Companion to windmill-ee-private: switch the running-state check to a
correlated EXISTS on the post-conflict-lock holder so two late arrivals
racing after a survivor started running can't both spawn independent
windows (the row lock serializes them; the second debounces into the
first's fresh window).
Adds a concurrent regression test
(test_debounce_concurrent_arrivals_after_running_survivor) asserting
exactly one late arrival survives and the other is debounced, and refreshes
the SQLx cache for the updated upsert_debounce_key queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): serialize upsert per key (simpler, race-free)
Companion to windmill-ee-private: the running-survivor guard and batch
chaining are now protected by a per-key advisory lock instead of
snapshot-sensitive single-statement SQL. This closes a concurrent-arrival
data-loss race where a debounced late arrival's args could be dropped
because the batch lookup couldn't see the predecessor's just-committed
batch row.
Extends test_debounce_concurrent_arrivals_after_running_survivor to pull the
survivor and assert its accumulation includes BOTH racing late arrivals
(shared batch), and refreshes the SQLx cache for the rewritten queries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic upsert robust to concurrent pull-time key deletion
Companion to windmill-ee-private: keep upsert_debounce_key a single atomic
INSERT ... ON CONFLICT DO UPDATE so a chaining push cannot fail when the
worker pull path concurrently deletes the holder's debounce_key (the prior
read+UPDATE split could hit "no row updated"). Adds
test_debounce_push_races_key_deletion_by_pull (races a chaining push against
the key deletion 50x, asserts the push never errors) and refreshes the SQLx
cache.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(debounce): claim-based exactly-once batch consumption
Eliminates the rare duplicate/loss when two survivors land on one debounce
batch (a narrow push/pull race), without locking the worker pull hot path.
- migration: v2_job_debounce_batch gains consumed_at + consumed_by.
- pull side (maybe_apply_debouncing): instead of deleting the batch on consume,
a survivor atomically claims its own row + any unclaimed siblings (stamping
consumed_by = itself) and accumulates exactly the rows it claimed. A second
survivor of the same batch finds its row already consumed by another job and
runs empty (no duplicate); a re-pulled survivor recognizes its own prior claim
and keeps its accumulated args; a never-batched job (CE/legacy) keeps its own
args. Non-accumulate debounce paths still hard-delete their batch rows.
- complete_debounced_job (EE companion) never completes a running predecessor,
so its in-flight run is not killed (no loss); the claim then prevents the
duplicate the guard would otherwise allow.
- monitor: GC sweep deletes consumed batch rows past a 1h grace.
Together with the running-survivor guard this makes debounce accumulation
exactly-once. Adds tests: batch_consumed_exactly_once, repull_keeps_accumulated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): exhaustive edge cases + tighten consumed-batch GC grace
Tighten the consumed debounce-batch GC grace 1h -> 10min: per-op cost of the
claim is unchanged (an indexed mark is as cheap as the old delete), so the only
cost of retaining consumed rows is table growth, which a shorter grace bounds
under high-throughput debounce (a survivor that could still reference a row is
pulled long before 10min; GC is not correctness-critical since a re-pull whose
row was swept falls back to its persisted args).
Adds edge-case tests: never-batched keeps own args (CE fallback), concurrent
claim partitions a batch disjointly (exactly-once under real concurrency),
three survivors -> first takes all / rest run empty, non-accumulate debounce
hard-deletes its batch rows (no leak), and the GC sweep deletes only
past-grace consumed rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): port the #9781 regression case, flow-node guard, full-path bench
- Port the regression from #9781
(test_post_preprocessing_debounce_into_running_survivor_loses_message):
post-preprocessing survivor accumulates + runs, a later same-key message must
start a new batch (survive) not be folded into the running survivor. Exercises
the full EE path via jobs_ee::maybe_debounce_post_preprocessing.
- Add the third EE entry point's guard:
test_flow_node_debounce_running_survivor_not_superseded (maybe_debounce_flow_node).
- Add an #[ignore] full-source throughput bench (bench_debounce_full_path) driving
the real maybe_debounce + maybe_apply_debouncing end-to-end.
All debounce tests exercise the real jobs_ee implementation (run with
--features private,enterprise); none stub it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): scalar-arg accumulation + GC-then-repull no-loss
Close two accumulation edge gaps (both run on --features private,enterprise,
exercising the real jobs_ee path):
- accumulate bare-scalar values (the T | T[] union fallback): each scalar is
wrapped and accumulated into the survivor's list.
- GC reclaiming a survivor's consumed batch row before a re-pull must not lose
data: the re-pull finds no row and keeps its already-persisted accumulated
args (had_row=false fallback), rather than running empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(debounce): real-worker end-to-end accumulation test
Drives the full real path on --features enterprise,deno_core,private: push 3
same-key debounced flow jobs (real push() -> maybe_debounce collapses the
batch), a real worker pulls the survivor (real pull() -> maybe_apply_debouncing
claim+accumulate) and executes the deno flow, then asserts the executed result
is the full accumulated set [1,2,3] and the two superseded messages are skipped.
Complements the in-process unit tests with a genuine worker-execution run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): atomic claim+persist, GC only non-queued rows; reword comment
Address review findings:
- [P1] Claim and accumulated-args persist are now in one transaction. Before,
a crash between stamping batch rows consumed_by=self and the `UPDATE v2_job
SET args` could let a zombie re-pull see its own prior claim and keep only its
own args (dropping the siblings it had claimed). Wrapping claim + accumulate +
persist in a tx makes them commit together or roll back together (re-pull then
re-claims cleanly).
- [P1] GC of consumed batch rows now also requires the job to no longer be in
v2_job_queue. A consumed sibling can stay queued well past any time grace under
a concurrency limit / backlog; reclaiming its marker by age alone let its
eventual pull treat it as never-batched and re-run its item (a duplicate).
Keeping the row until the job leaves the queue preserves the "already consumed"
signal. Test extended with a still-queued consumed row that must survive GC.
- [P2] Drop "Customer" attribution from a test doc comment (AGENTS.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(debounce): emit accumulation log after committing the claim transaction
append_logs opened a second pool connection while the claim transaction (and its
batch row locks) were still held; under concurrent debounced pulls that risks
pool-exhaustion stalls/timeouts. Defer the log line until after tx.commit().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
This commit updates the EE repository reference after PR #631 was merged in windmill-ee-private.
Previous ee-repo-ref: 30d740e619fad219108ec4b4c6a9d67c1ab42d46
New ee-repo-ref: 6aabd7c5ce53b9153be05c3e7bc9a76eadb1a48a
Automated by sync-ee-ref workflow.
* fix(debounce): claim whole batch in one UPDATE (no deadlock); assert test setup
Both Codex (P1) and Claude (P2) flagged a deadlock: the claim used two writable
CTEs (claim_self then claim_rest), locking the self row before siblings, so two
survivors of the same batch pulled concurrently acquired row locks in opposite
order and PostgreSQL aborted one with deadlock_detected (a transient pull error
on exactly the two-survivors race this path handles).
Replace with a single `UPDATE ... WHERE debounce_batch = (...) AND consumed_at IS
NULL RETURNING id` that claims the whole batch: both transactions lock rows in
the same scan order, so one simply waits and re-evaluates under EvalPlanQual.
A `claimed_self` flag (EXISTS id = self in the claimed set) plus the `mine`
snapshot still distinguishes fresh-claim / consumed-by-other / own-re-pull.
Also assert add_survivor_to_batch_of actually inserts a row (rows_affected == 1)
so a mis-set-up test can't pass vacuously.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
6664ce6dc0 |
fix(frontend): apply script editor timeout to preview/Test runs (#9794)
The custom timeout configured in the script editor settings was only
honored for deployed script runs: it is persisted on the script row and
passed as custom_timeout when running by hash/path. Preview ("Test")
runs derive their timeout solely from the `timeout` query param of
/jobs/run/preview, which the editor never sent, so Test silently fell
back to the instance default.
Forward the editor's timeout setting through ScriptBuilder ->
ScriptEditor -> JobLoader.runPreview as the preview run's timeout query
param. The backend already clamps custom_timeout against the instance
max in resolve_job_timeout, so previews get the same ceiling as deployed
runs.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
9d61e4e59e |
feat: self-host docs search for chat, mcp, cli; drop inkeep (#9772)
* feat: self-host docs search for chat, mcp and cli; remove inkeep
Embed a vendored docs snapshot (llms.txt/llms-full.txt) in the backend and
serve ranking + page rendering from GET /api/docs/{search,page}. The AI chat,
the MCP searchDocs/readDocsPage tools, and 'wmill docs' all consume it, so docs
search works with no runtime egress and is no longer EE-gated. Removes the
inkeep proxy. EE companion deletes inkeep_ee.rs (ee-repo-ref bumped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: name read_docs_page param `url` instead of `path`
search_docs returns each hit's `Source` URL, so the read tool now takes a
`url` argument to match — the AI/MCP loop reads "search gives a Source URL,
read takes that url" rather than copying a `Source:` URL into a `path` slot.
A bare `/docs/...` path is still accepted and canonicalized before lookup.
Regenerated openapi-deref, the MCP endpoint tools, and the frontend client.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: add scheduled workflow to refresh the vendored docs snapshot
The backend embeds docs_snapshot/*.gz at build time, so the in-product docs
corpus is otherwise only as fresh as the last manual fetch.sh run. This adds a
weekly (and manually dispatchable) job that re-runs fetch.sh, sanity-checks the
result against truncation/garbage, and opens a PR via the internal app when the
snapshot changed — so a human reviews the docs diff before it rides into the
next release build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: make docs tool-result strings caller-neutral
The search/page endpoints back three differently-named consumers (the AI chat
`read_docs_page` tool, the MCP `readDocsPage` tool, and the `wmill docs` CLI),
so the shared rendered text shouldn't name one of them. Refer to "the docs
page-reading tool" and its `url` argument instead, and add tests pinning the
caller-neutral follow-up guidance.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: point ee-repo-ref at inkeep-removal companion rebased on EE main
The companion branch now carries only the inkeep_ee.rs deletion on top of EE
main (was based on the native-job-retry EE line, which polluted the EE PR diff).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(docs): expose docs:read in token catalog; precompute lowercased corpus
Addresses two review nits on the self-hosted docs PR:
- docs:read was enforced (ScopeDomain::Docs) but missing from the token scope
catalog (token.rs ALL_SCOPES), so it couldn't be selected when creating a
standard scoped token in the UI — leaving scope-restricted CLI/MCP docs use
effectively ungrantable. Add a read-only "Documentation" group (no write
surface) and a test asserting it is exposed.
- search ran page.body.to_lowercase() on the whole corpus per query. Lowercase
body/title/description once at parse time (into the OnceLock corpus) and scan
the precomputed copies instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* chore: update ee-repo-ref to 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
This commit updates the EE repository reference after PR #630 was merged in windmill-ee-private.
Previous ee-repo-ref: c7ec3a0c2fa38d4cb5e50bf0265eef4710de4860
New ee-repo-ref: 27a4f41b8e5603d6e444efcfc420bd1c44a07eed
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
11d83ab1ec |
fix(python): serialize concurrent installs into shared wheel cache dir (#9787)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
6c71c33470 |
fix(python): re-verify wheel RECORD on local cache reuse (once per worker) (#9775)
A corrupt local pip cache entry (e.g. wmill==1.739.0 missing s3_reader.py after out-of-band file loss on a persistent/shared cache volume) was trusted indefinitely: handle_python_reqs only checked the .valid.windmill marker on the reuse fast path. verify_wheel_record already guarded the install and S3-pull paths, but never ran again once the marker existed. Re-verify the wheel RECORD on the first reuse of each cache entry per worker process and repair (wipe + reinstall) on failure. A VERIFIED_VENVS in-memory set makes every subsequent reuse skip the scan, so the warm-cache hot path keeps paying only its original single stat. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
aa098c70c0 |
perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes (#9786)
* perf: drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: document delete_jobs auth contract and workspace-scope jobs_export purge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
962758c02d |
fix: pass SSL cert env vars to uv python install (#9790)
`install_python` cleared the subprocess environment via `env_clear()` and forwarded only a subset of variables, omitting `SSL_CERT_FILE` (from `PY_INDEX_CERT`/`PIP_INDEX_CERT`) and `UV_NATIVE_TLS` (from `PY_NATIVE_CERT`). This caused `invalid peer certificate: UnknownIssuer` errors when downloading managed Python runtimes in environments with corporate/private CAs. Forward both variables, mirroring the sibling `find_python` method and the pip install path in `python_executor.rs`. Fixes WIN-2100 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
754cae956a |
fix: use transaction for parallel_monitor_lock DELETE in last-iteration path (#9789)
In the last-iteration path of a parallel for-loop (nindex == len), the DELETE FROM parallel_monitor_lock ran on the pool (db) while the transaction tx (begun earlier) was still held. This dual-connection pattern requires 2 simultaneous connections from the per-worker pool (default max 5) and can trigger "pool timed out while waiting for an open connection" under concurrent load. Run the DELETE on the held transaction (&mut *tx) instead, matching the fix PR #7861 applied to other queries in this file. The transaction is committed shortly after, so including the DELETE in it is safe and consistent with the non-last-iteration path. Fixes WIN-2099 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
12f92e3ab7 |
[ee] feat(backend): native script retry without one-step-flow wrapping (#9688)
* feat(backend): native script retry without one-step-flow wrapping Schedules and data pipelines that retry a single script previously wrapped it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a v2_job_status row, and UI projection complexity. This adds native retry on a plain JobKind::Script job. - RetrySettings: flatten Retry into a deduped retry_settings table, carried via the existing runnable_settings_handle (lazy, off the hot path). - push() materializes a bare-script-with-retry SingleStepFlow into a native Script job (gated on min-version + no handlers/retry_if). - add_completed_job re-pushes the next attempt on failure with backoff, tracking the attempt counter in v2_job_queue.extras and the chain via parent_job; schedule completion handlers fire only on the terminal attempt. - frontend: ScriptRetryChain shows the attempt chain on the run page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): native retry_if eval + per-occurrence schedule handlers Extends native script retry to the two cases that previously stayed on the one-step-flow path: - retry_if: evaluated natively on the failure path via a feature-gated windmill-jseval dep (quickjs) over the failure result + flow_input; push materializes such policies natively only when quickjs is available. - on_failure_times / on_recovery: apply_schedule_handlers now resolves each past scheduled occurrence's terminal status across its native-retry chain (root OR any parent_job=root child succeeded) and excludes the current occurrence, so the counting is per-occurrence rather than per-attempt. All scheduled-script retries now go native (schedule.rs gate removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): always materialize retry_if natively; unsupported without quickjs retry_if is evaluated by the worker (which always has quickjs), not the pusher, so gating materialization on the pusher's feature was wrong. The flow path was never a real fallback either — the flow runtime needs quickjs to evaluate retry_if too. retry_if now always goes native; on a worker without quickjs it is unsupported and fails closed (no retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(backend): un-park asset-cascade (pipeline) retry Native retry resolves the blocker that parked pipeline retry: a retried subscriber is now a Script job (not a one-step flow / flow step), so it stays eligible for asset dispatch and can trigger its own downstream on recovery. - scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset edges (was dropped with a TODO warning). - asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job, so native-retry attempts dispatch on success while flow steps stay excluded. - tests: retry-bearing subscriber now dispatches as a native Script carrying the policy in runnable_settings_handle; native-retry attempt is eligible. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): cap native retry interval, lazy result serialization, idempotent retry push Hardening from a self-review of the native retry path: - Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path (evaluate_retry); the exponential formula could otherwise schedule up to ~18h vs the flow path's 6h. - Serialize the failure result lazily (only when a retry_if policy needs it), so the common failure no longer pays the serialization on the failure path. - Push each retry with a deterministic id per (root, attempt). If a worker dies between enqueueing the retry and finalizing the current attempt, the reaper re-handles the attempt and lands here again — push rejects the duplicate id, so the retry is enqueued exactly once (no double-retry). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): defer schedule handlers idempotently on retry-push replay (review P1) Address local-review findings: - P1: retry_pending was derived from the retry push *result*, so on a worker crash + reaper replay the duplicate-id push returned Err → retry_pending flipped to false → apply_schedule_handlers fired for the non-terminal attempt (and the terminal attempt later fired them again). Pre-check whether the deterministic retry id already exists and report it as pending without re-pushing, so the handler-deferral invariant is crash-idempotent too. - P2: refresh the stale 'wrap the script in a one-step flow' comment in the asset-cascade retry push — it now materializes a native Script. - Add RetrySettings <-> Retry round-trip unit tests (clamping edges). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): native retry chain + per-occurrence status sqlx tests Close the two integration-test gaps flagged in local review: - chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts) and asserts crash-replay idempotency (the P1 fix: a replayed completion reports pending without double-enqueueing). - per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but- recovered occurrence counts as success, retries (parent_job set) are excluded from occurrence counting, and the current occurrence is excluded. - canceled_job_does_not_retry: cancellation wins over a pending retry. Runtime sqlx API (no .sqlx cache entry needed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): exclude schedule handlers from the retry-attempt chain The retry chain listed all script children of the root by parent_job, but schedule completion handlers (on_failure/on_recovery/on_success) are also script children — when the occurrence has no retries, the handler's parent is the root itself, so a successful, never-retried job rendered a bogus 'Retries (1)' badge pointing at the handler. Filter children to re-runs of the same script (matching script_hash); real retries keep the root's hash, handlers run a different script. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): surface schedule handlers on the run page Extend the run-page chain component with schedule completion handlers: - A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/ on_success runs that fired for that occurrence (found as children of the terminal attempt, identified by their synthetic created_by). - A handler's own run page now shows a 'Failure/Recovery/Success handler' label with a link back to the run it handled and its schedule. on_recovery and on_success share created_by, disambiguated by the recovery-only error_started_at arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare An earlier `cargo sqlx prepare` on this branch ran before #8801's folder_default_permissioned_as test merged in, so it pruned the 3 query caches that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main. * fix(backend): only cascade assets from native retry attempts, not handlers (review P1) is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script child became asset-eligible — including schedule/error/recovery handlers (Script jobs with parent_job set and no flow_step_id). A handler that declares assets would then trigger a cascade the old parent_job IS NULL guard prevented. Gate parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as its chain parent (handlers run a different script). Runtime query, no sqlx cache. * fix(backend): cache the private-gated retry_setting asset-dispatch test query The same prepare-without-private that dropped the folder_default caches also pruned the cache for the retry_setting_dispatches_subscriber_as_native_script test query (asset_trigger_dispatch.rs:721). Regenerated with --features private. * fix(backend): exclude handler children from per-occurrence recovery (review) A scheduled occurrence's on_failure/on_success handler runs as a successful child (parent_job = occurrence), and the per-occurrence success EXISTS counted ANY successful child — so a failed occurrence whose error handler succeeded was marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in the merge) and on_failure_times counting. EE query now scopes the EXISTS to same-runnable children (only native retry attempts); regenerate sqlx cache + bump ee-repo-ref. native_retry_test gains a handler-child regression case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(backend): scheduled-script retry is a native Script, not SingleStepFlow test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a native Script. Update both to assert kind='script' and that the retry policy is carried via runnable_settings_handle. * fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic) Address cubic CI review: - P1: the SingleStepFlow->native Script materialization dropped dedicated_worker, so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve it from the script row in push so the materialized Script keeps the dedicated tag. - P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the u32->i16 // retry count narrowing (scripts.rs) instead of wrapping. * fix(backend): use a retry-specific signal, not runnable equality (codex review) Address Codex CI review: - P1: is_native_retry_attempt treated any same-runnable parented Script child as a retry. WAC v2 inline children have that exact shape, so an inline child of an asset producer would cascade. Use a retry-specific signal instead: the job carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no flow_innermost_root_job. Apply the same flow_innermost guard to the EE per-occurrence EXISTS (WAC inline children must not count as a recovery). - P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate now resolves as 'retry pending' (re-check on the duplicate-id error) instead of flipping retry_pending to false and firing handlers early. - Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(backend): explicit native_retry_attempt marker, drop heuristics Replace the per-site "is this a retry?" inference (parent_job + runnable match + flow_innermost / retry_settings) with one explicit marker: a sparse native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker also carries the attempt counter (previously in v2_job_queue.extras), so it's the single source of truth. - asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker. - EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost. - maybe_enqueue: reads/writes the marker (persistent) instead of queue extras. - Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap. - Eliminates handler / WAC-inline-child misclassification by construction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex) The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but the periodic monitor.rs path deleted v2_job rows without it, orphaning markers. Add the same WHERE job_id = ANY(...) sweep there. * fix(backend): widen native_retry_attempt.attempt to integer (cubic) The smallint column was cast to/from u32 and could wrap a retry chain longer than i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32 attempt count, so no narrowing occurs on the maybe_enqueue read/write path. * feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT - Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi), computed from the native_retry_attempt marker. The run-page chain now filters retry attempts by is_retry instead of the script_hash heuristic, so WAC v2 inline children (same script, parent_job) no longer render as retries (codex). - Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep the plain catch-and-continue matching the other side-table deletes; the table is created by a startup migration so it always exists when cleanup runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but does not select is_retry, which errored with "no column found". Only the list endpoint populates the marker; #[sqlx(default)] lets every other query omit the column and default to None. * feat(backend): select is_retry in single-job GET too for consistency The list endpoint already exposes the marker; populate it on the single-job GET (both completed and queued variants) as well so a run loaded directly reflects its retry status. #[sqlx(default)] stays as a safety net for any other query. * feat(backend): reap orphaned native_retry_attempt markers via periodic sweep The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so direct job deletions (workspace/job delete, schedule clearing) would leave marker rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which must then be remembered for every future path), reap orphans in the periodic delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the job). The table is sparse so the anti-join drives off it and probes v2_job by PK — cheap. Retention still sweeps markers inline (keeps the table small so this stays cheap); a transient orphan is harmless (nothing reads is_retry for a gone job). * fix(frontend): include flow handlers in retry chain handler row (codex) Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...), whose handler job is a flow, not a script. The chain fetched children with jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts are still selected by is_retry and handlers by created_by, so both kinds surface. * fix(backend): carry concurrency/debouncing settings into native retries maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings ::default(), dropping the script/pipeline concurrency settings the failed job carried in its runnable_settings_handle. A retry of a concurrency-limited script then inserted no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and pass them in the payload, which push forwards to the materialized retry. Adds a regression test asserting the retry's handle resolves to the concurrency settings. * fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex) P1a (schedule.rs): the scheduled-retry materialization fetched the script's concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings ::default() into the SingleStepFlow payload, so the root attempt's handle held only the retry policy and the whole chain ran unbounded. Pass the fetched settings. Regression test asserts the root handle resolves to retry + concurrency. P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract — it is pub only for the integration test; the sole production caller is the worker completion path passing a DB-derived, already-authorized MiniCompletedJob. * docs(backend): attach native-retry auth contract to the function itself (codex) The doc block was merged with eval_retry_if's doc and bound to that function, leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if keeps its own doc; the native-retry + authorization contract now sits directly above maybe_enqueue_native_script_retry. * docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex) - Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since 1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the served spec couldn't see the field. Now current at 1.739.0. - schedule.rs: a retry_if gate is evaluated at failure time and fails closed without quickjs (no retry); it does not fall back to a flow path. - windmill-types jobs.rs: is_retry is selected by both the list and single-job GET endpoints (not list-only). * docs(backend): fix remaining stale retry_if/quickjs comments (codex) The retry_if block and the push materialization comments claimed push keeps retry_if on a flow path / the worker always has quickjs. The code always materializes native retry and the no-quickjs eval_retry_if path fails closed — correct the comments to that constraint. * docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex) - Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated and the job does not retry (no one-step-flow fallback). - jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence failure/recovery status across the retry chain, so the old 'restricted to schedules whose handlers don't need per-occurrence counting' claim is dropped. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
cd42c6ca18 |
fix: decrypt secret variables via external backend in common resolvers (#9784)
`get_variable_or_self`, `get_variable_or_self_as`, `get_secret_value_as_admin` (and `transform_json_unchecked`'s `$var:` branch) in windmill-common always ran the raw `variable.value` through `decrypt()`. With an external secret backend (HashiCorp Vault / Azure Key Vault / AWS Secrets Manager) configured, that column holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 ciphertext, so base64 decoding failed with `Invalid byte 36, offset 0` (the `$`). This broke GitHub App git sync (git_sync_ee.rs) and any other consumer of these resolvers when an external backend is active. Move backend resolution (`get_secret_backend`, `get_secret_value`, `is_*_stored_value`, caching) into `windmill-common::secret_backend::resolver` so the low-level variable resolvers can route external markers through the configured backend's `get_secret()` instead of `decrypt()`. The windmill-store and windmill-api `secret_backend_ext` modules now re-export these from windmill-common (single source of truth / single backend cache) and keep only their write-side helpers. No `_ee.rs` files change. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 Diego ImbertandClaude Opus 4.8
|
170cd79aaf |
fix: allow hyphens in postgresql database name validation (#9782)
* fix: allow hyphens in postgresql database name validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover hyphen acceptance in validate_dbname Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
f6998ec54c |
feat: data tests for ducklake pipeline materialization (#9708)
* feat: data tests for ducklake pipeline materialization Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data_test count badge on pipeline graph nodes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: surface annotation badges (incl. data_test) on deployed pipeline nodes Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): keep materialize output edge when editing the producer in the pipeline graph The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: run all data tests in one pass with a structured per-test result Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): data-test pass/fail checklist in the job result DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): move data-test badge onto the producer→asset edge with run status The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(frontend): render custom data-test scripts as their own clickable graph nodes A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): anchor edge badge on routed path + a11y text on test icons Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests Address review (cubic) findings: - P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact. - P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script. - P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: fail loud if fewer data-test outcomes recovered than declared Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: relationships data test same-lake reuse + schema-qualified target quoting Address Codex/Pi review (two P1s in the relationships codegen): - A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references. - A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator. Adds tests for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target Address Codex review nits (both P2): - resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch. - AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
hugocasaandClaude Opus 4.8
|
88fca6a8c1 |
fix: enforce containment of python module dir for preview jobs (#9704)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
d131d754e1 |
feat: ducklake time-travel UX (snapshot history + AT VERSION reads) (#9709)
* feat: ducklake time-travel UX (snapshot history + AT VERSION reads) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: catalog-qualify ducklake time-travel FROM hints (lake. prefix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: render ducklake snapshot_time (microseconds since epoch) correctly Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: merge ducklake History + Query into one master-detail tab Snapshot list (left) selects the version previewed in the read-only grid (right); newest auto-selected. Copy-clause moved to the preview's SQL line. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scope ducklake snapshot history to the table's versions Catalog-wide snapshots predate a table's creation; previewing AT a version before the table existed errored ("Table ... does not exist at version N"). The DUCKLAKE_SNAPSHOTS marker now takes the table and lists only snapshots from its first creation onward. Also: narrower snapshot-list pane on large screens (target a fixed width, not a fixed fraction). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: load ducklake preview columns at the pinned version + reset on asset switch Addresses CI review (codex/pi P1, cubic P2): - Historical previews loaded current-schema columns, so an AT(VERSION) read enumerating a column added in a later snapshot failed. Now DESCRIBE-loads the column set at the pinned version; the read is gated on columns matching the current version to avoid a stale-colDefs race on version switch. - selectedVersion no longer sticks across assets: the panel is keyed on path (remounts per asset) and effectiveVersion falls back to newest when the pick isn't in the current list. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: match History tab UI (master-detail, full-FROM copy) after merge Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: handle catalog-only ducklake asset paths (no table segment) parseDbInputFromAssetSyntax threw on a catalog-only path like 'ducklake://main' (undefined.split('.')) — a real graph node (e.g. a consumer of the whole catalog). It now returns a table-less input instead of throwing, and DucklakeAssetPanel renders only the partition grid (no per-table history/time-travel) for table-less nodes. Adds parser unit tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: escape ducklake catalog name in client-built time-travel DESCRIBE fetchDucklakeColumnsAtVersion interpolated the catalog name into an ATTACH string literal without escaping; double single-quotes (mirrors backend escape_sql_literal) so a quote-containing catalog name can't break out. Also fixed the v1.x docs checklist line to match the shipped full-FROM copy affordance. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszelandrubenfiszel
|
920f5688ca |
chore(main): release 1.739.0 (#9746)
* chore(main): release 1.739.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
b5bd8245d8 |
fix: reject symlink traversal in job-dir path validation (#9713)
* fix: reject symlink traversal in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover dangling symlink in job-dir path validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: close symlink-traversal bypass via in-bounds `..` in path check Walk the normalized relative path instead of raw user components, so an in-bounds `..` (e.g. `foo/../evil/payload`) can no longer drift the walk past a planted symlink. Adds regression coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 GuilhemandClaude Opus 4.8
|
42c5e7a3fc |
feat: scope AI sessions per workspace root with lifecycle reconcile (#9734)
* feat: scope AI sessions per workspace family with lifecycle reconcile Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: centralize session reconcile trigger + extract pure lifecycle decision Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf: remove unused workspace family index * refactor: scope sessions by workspace root id, drop family_id column Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sessions): preserve user-archived sessions when archiving their workspace archiveSessionsForWorkspace tagged every session archivedByWorkspace, including ones the user had already archived by hand, so a later workspace unarchive auto-restored them. Skip already-archived sessions so only workspace-archived ones are tagged, matching decideSessionLifecycle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: archived-session banner with unarchive, suppress workspace-gone banner while archived Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: re-root sub-fork sessions on reconcile when an ancestor is deleted Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: group AI sessions by workspace family with show-all-workspaces filter Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: revert unrelated AIProviderPicker cosmetic changes Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: hide per-session unarchive when workspace is gone, show move/discard instead Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: GC attached files on lifecycle delete + reconcile on sidebar fork delete Addresses Codex review: deleteSessionsForWorkspace/reconcile delete now GC linked files (deleteItemsForSession), matching deleteSession; sidebar deleteFork now reconciles so surviving child forks re-root off the deleted ancestor. Also de-flaked post-rehydrate reads in the IndexedDB tests via vi.waitFor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't strand user if post-delete reconcile throws; refresh stale warmSessions comment Addresses auto-review P2s: wrap reconcileAfterWorkspaceChange in deleteFork so the parent switch + navigation always runs even on reconcile failure; correct the warmSessions comment which no longer holds under 'Show all workspaces'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: don't fail/strand fork archive+delete when client session cleanup throws Addresses cubic P1/P2 on forks/compare: the workspace archive/delete is authoritative; wrap the best-effort session cleanup + reconcile so a local IndexedDB failure neither falsely reports failure nor blocks navigation away from the gone fork. Mirrors the SidebarContent fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop drafting-history aside from reconcileAfterWorkspaceChange comment Addresses auto-review P2: keep the refresh-before-reconcile invariant, drop the 'which they did inconsistently' narration per AGENTS.md (comments record constraints, not drafting history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: clean up sessions on fork-id reuse + make all workspace-mutation cleanup best-effort Addresses Codex P1s: (1) CreateWorkspaceInner 'permanently delete existing fork' (id-reuse) now drops local sessions for that id so they don't resurface on the recreated fork; (2) workspace_settings archive/delete and SidebarContent child-delete loop + main delete now treat post-mutation session cleanup as best-effort, so a local IndexedDB failure can't strand the user or abort remaining deletes (matching the compare-page fix). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: make fork-reuse session cleanup fire-and-forget (non-blocking) Addresses cubic P2: don't await the best-effort cleanup so a slow IndexedDB op can't block the delete/reuse flow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: drop previous user's transient drafts on user change Addresses Pi P1: hydrateSessions preserved transient (unsent) drafts across user changes, so user A's draft + its pending fork/workspace state bled into user B's list and got reused by createSession. onUserChange now drops transients when the email changes; reconcile (intra-user) still preserves them. Regression test added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
288318ac26 |
fix(apps): realign legacy raw-app drafts to raw_app draft kind (#9761)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
f5828780fd |
fix(backend): resolve folder_labels search_path on non-public (PG_SCHEMA) schemas (#9758)
* fix(backend): strip search_path=public from folder_labels migrations for non-public schema The folder-labels migrations (20260610151334_folder_labels, 20260614075900_dedup_folder_labels) define `folder_labels(...)` with `SET search_path = public` in their `CREATE FUNCTION` bodies. When Windmill runs in a non-public schema (PG_SCHEMA), PostgreSQL validates the function body against the `public` schema, where the `folder` table lacks the new `labels` column, failing with `column "labels" does not exist`. Add both migrations to OVERRIDDEN_MIGRATIONS, stripping the `SET search_path = public` clause so the function inherits the current search_path (which resolves the correct schema). Same regression and fix pattern as PR #5400. Fixes WIN-2093 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): pin folder_labels search_path FROM CURRENT instead of stripping it Keep the SECURITY DEFINER injection hardening while resolving the correct schema on non-public (PG_SCHEMA) installs: FROM CURRENT snapshots the migration connection's search_path at function creation time (public on normal installs, the custom schema otherwise) instead of dropping the pin and inheriting the caller's search_path at call time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(backend): repair migration to re-pin folder_labels search_path on applied instances Instances that already applied the folder-labels migrations with the hardcoded SET search_path = public have a folder_labels function pinned to public. On a non-public (PG_SCHEMA) schema that reads the wrong folder table at runtime; the OVERRIDDEN_MIGRATIONS fix only helps instances that have not applied them yet. Add a CREATE OR REPLACE ... SET search_path FROM CURRENT migration that re-pins the function to the migration connection's schema. No-op on public installs (re-pins to public) and idempotent on already-correct ones. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
8912e21d15 |
perf(monitor): vacuum job_perms/job_result_stream right after each orphan sweep (#9753)
A customer's top-load query was the job_perms orphan sweep (cleanup_job_perms_orphaned: 6.9s mean, 41s max). The cost is discovery, not deletion (~2.1ms per row deleted): the NOT EXISTS anti-join seq-scans the whole job_perms heap to find a few orphans, and that scan tracks the heap's physical size. job_perms / job_result_stream_v2 get one row per job and are drained only by these per-cycle sweeps, so they churn hard — but the bulk vacuuming_tables() runs only ~hourly, so dead tuples bloat the heap between bulk vacuums. Reclaim right after each sweep instead: VACUUM (SKIP_LOCKED) the swept table when it deleted rows. Plain VACUUM (not FULL) takes only SHARE UPDATE EXCLUSIVE so concurrent job creates/reads proceed; the visibility map skips unchanged pages so repeated runs are cheap; SKIP_LOCKED means HA replicas don't pile up (one vacuums, the rest skip). Benchmarked ~7x: a bloated 268MB job_perms heap swept in 35ms vs 5ms vacuumed. Chosen over an autovacuum reloptions migration so the behavior is explicit and lives with the sweep it pairs with. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
55bed4abcf |
perf(audit): adaptive timestamp floor for S3 audit-log export (#9752)
* perf(audit): adaptive timestamp floor for S3 audit-log export (ee) EE change in windmill-ee-private (src/ee.rs); this OSS commit carries the regenerated sqlx cache for the new oldest-in-flight query and bumps ee-repo-ref.txt to the EE branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to ed89574be9117cda5e2d7d9de02cb5db066e93e3 This commit updates the EE repository reference after PR #628 was merged in windmill-ee-private. Previous ee-repo-ref: 8a7f645c0a194a284fe19dd20dbe79dd0733dfdb New ee-repo-ref: ed89574be9117cda5e2d7d9de02cb5db066e93e3 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
043c2c05b7 |
fix: forbid superadmin job tokens from global user and token management (#9715)
* fix: forbid superadmin job tokens from global user and token management Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: extend superadmin job token guard to offboard and export routes Apply forbid_superadmin_job_token to offboard_global_user and export_global_users, the remaining global user-management routes that were gated only by require_super_admin. Offboarding can delete a user along with their tokens, password, invites and instance-group membership, and export returns every user's password_hash, so both must be unreachable by a superadmin job token. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszelandrubenfiszel
|
9e4cf139b1 |
chore(main): release 1.738.0 (#9735)
* chore(main): release 1.738.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> |
||
|
hugocasaandClaude Opus 4.8
|
cbf54d4eb4 |
fix: preserve fork parent linkage on workspace id change (#9716)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
9793d01575 |
feat: add resource and infrastructure telemetry (#9737)
* feat(telemetry): disclose resource and infra usage stats When minimal telemetry is disabled, the stats payload now includes resource counts (workspaces, scripts per language, flows, workflows as code, low-code and raw apps) and, on EE only, infrastructure info (container runtime, database size, max connections, RDS detection). Update the telemetry disclosure in instance settings accordingly: resource counts are listed for both CE and EE; infra info is shown only on EE since it is collected only there. Bump the EE ref and add the sqlx cache for the new queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(telemetry): expand EE infra disclosure and add sysinfo dep Disclose the expanded EE infrastructure telemetry (deployment mode, host OS/arch/CPU/memory, filesystem space, Postgres version and connection counts, object storage backend, sandboxing and retention settings) in instance settings. Add sysinfo as a windmill-common dependency for host memory and filesystem stats, bump the EE ref, and add the sqlx cache for the new queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(telemetry): focus EE infra disclosure on wrapping platform Drop the single-server host details (OS, arch, CPU, memory, filesystem) and tuning config from the EE infra disclosure, and revert the sysinfo dependency they required. Reflect managed-database-provider detection in place of the RDS flag. Bump the EE ref and update the sqlx cache for the revised queries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(telemetry): drop deployment mode and worker count from disclosure Remove deployment mode and worker count from the EE infra disclosure to match the backend, and bump the EE ref. They reflect only the node sending telemetry, not the deployment topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to 6d3301507db50818f1683dac3941d3e0cf1152a7 This commit updates the EE repository reference after PR #627 was merged in windmill-ee-private. Previous ee-repo-ref: d30e7d18d14992598a97356d0ed13f7d5d585115 New ee-repo-ref: 6d3301507db50818f1683dac3941d3e0cf1152a7 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev> |
||
|
hugocasaandClaude Opus 4.8
|
24446e8009 |
fix: allow object storage test for non-super-admins, harden on cloud (#9739)
* fix: allow non-super-admin object storage test, harden SSRF surface on cloud Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: validate effective object storage host to close region/bucket SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: validate gcs_base_url/token_uri in GCS service account key to close SSRF bypass Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: match url scheme case-insensitively in object storage host validation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
e90b2be8fa |
perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744)
The expired-job retention loop re-scanned the same oldest rows on every batch. When the oldest completed jobs are undeletable (children of a still-active root flow), the ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20 batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch, ~180s/cleanup-cycle on a 1.5M-row prefix). Carry a completed_at watermark (max deleted) across batches and re-apply it as completed_at >= floor so each batch resumes past the already-processed prefix. Also skip the v2_job join entirely when no old root flow is active (the common case), since nothing is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms. The watermark only ever skips rows the current run already deleted, was protecting, or skip-locked — all deferred to the next run, identical to the unbounded scan's row set (verified: union of batched deletes == single delete, 0 diff). Mirrored in windmill-api-settings log_cleanup. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
984ea728d9 |
fix: pipeline annotation false-positives from body comments (#9736)
* fix: reject pipeline `# tag` annotation false-positives on regular comments `parse_pipeline_annotations` treats any comment line starting with `# tag <text>` as a worker-tag annotation. In Python scripts, ordinary English comments beginning with "# tag ..." were misinterpreted: values over 50 chars failed the `script.tag` INSERT (varchar(50)), and shorter ones silently overrode the script's worker tag. Worker tags are single-word identifiers (e.g. `heavy`, `gpu`), so reject any candidate that contains whitespace or exceeds 50 characters. Mirror the same validation in the TS parity parser and add regression tests on both sides. Fixes WIN-2090 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: restrict pipeline annotation scan to the leading comment header The root cause of the `# tag` false-positive is broader than the `tag` keyword: `parse_pipeline_annotations` scanned every comment line in the whole file, so any body comment matching an annotation grammar (`on`, `freshness`, `tag`, `retry`, ...) was misinterpreted. The `tag` case was the most visible because an over-length value crashed the `script.tag` INSERT (varchar(50)). Windmill's other comment-directive parsers (BashAnnotations::sandbox_image, ssh_target) already scan only the leading comment header and stop at the first line of real code. Align parse_pipeline_annotations (and its TS mirror) with that convention: skip blank lines, break on the first non-comment line. This eliminates body-comment false-positives for every annotation, not just `tag`. The `tag` whitespace/length guard from the previous commit is kept as defense for prose that sits in the header itself. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |