From c91027824be1f1f49cdd14148baf6aad092a1dd0 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Mon, 29 Jun 2026 22:33:50 +0200 Subject: [PATCH 01/21] feat(pipeline): AI-chat data-pipeline editor (route + in-session) + home surfacing (#9805) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(pipeline): AI chat tools to build pipeline nodes with diff/approval Add a data-pipeline AI chat experience modeled on the flow editor and surfaced through the dev-gated global chat (no new chat panel). The /pipeline editor registers PipelineAIChatHelpers on the AIChatManager; while it is open the global mode layers pipeline tools, a pipeline prompt section, and the helpers on top of the full global tool set (behavior is unchanged when no pipeline editor is open). New tools (frontend/src/lib/components/copilot/chat/pipeline/core.ts): - get_pipeline_graph / read_pipeline_node — read the live graph and bodies - build_pipeline_node / edit_pipeline_node — stage changes as AI-pending drafts - remove_pipeline_node — drop a staged proposal - test_pipeline_node — preview-run a node (requires confirmation) Tools never deploy: they stage drafts flagged aiPending, rendered on the canvas with an accent ring and reviewed via Accept all / Reject all (the flow editor's GlobalReviewButtons). Accept commits the drafts; Reject reverts to a pre-AI snapshot, preserving earlier accepted drafts. Auto-accept is gated on the chat autonomy mode. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(pipeline): teach the global/session chat to author data pipelines Without an open /pipeline editor the session chat had no pipeline concept, so "create a data pipeline" loaded flow instructions and built a flow. Add a first-class pipeline authoring path: - system_prompts/base/pipeline-base.md — what a data pipeline is (a DAG of annotated scripts wired by storage assets, NOT a flow) and how to author the // pipeline / // on / // materialize annotations; wired through generate.py as getPipelinePrompt() (regenerated prompts.ts/index.ts). - global/core.ts — new get_instructions subject "pipeline", and a global-prompt rule disambiguating data pipelines from flows so the model routes correctly. - ai_evals/cases/global.yaml — two global cases (single node, two-node chain) asserting pipeline-annotated script drafts and forbidding write_flow, guarding the pipeline-vs-flow conflation. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(pipeline): show & build pipelines in the AI session preview Add a 'pipeline' session preview target so the session AI can show the data-pipeline graph for a folder and build nodes in-pane: - open_preview now accepts kind="pipeline" (path = folder); SessionTarget / EDITOR_TARGET_KINDS widen accordingly. The slot/codec load model stays flow|script|raw_app — pipeline bypasses it with its own fetch/draft state. - New PipelineEditorView.svelte mounts in the session pane: fetches the folder graph, overlays AI drafts, renders AssetGraphCanvas + the Accept/Reject review buttons, and registers PipelineAIChatHelpers on the *session-scoped* manager (via getAiChatManager) so build_pipeline_node / edit_pipeline_node + the diff/approval work inside the session too. - System prompt nudges the model to open the pipeline preview and use the staging tools while building. Verified end-to-end with a real model: the session AI called open_preview, the graph mounted in the side panel, then build_pipeline_node staged a node on the session canvas with its schedule trigger and ducklake output. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): share the AI editor logic between route page and session Consolidate the duplicated data-pipeline AI logic onto a single shared layer so the route editor and the in-session preview behave identically and the session gains the full code editor. - New pipelineAiHelpers.ts: createPipelineAiHelpers(deps) owns the propose/edit/ remove/accept/reject/test staging + the per-turn snapshot bookkeeping that powers Reject. Callers inject accessors for their own draft Map and graph. - Route page (/pipeline/[folder]) drops its ~250-line inline AI-helper block and wires the shared factory via deps (folder/workspace/graph/drafts + focus, ensureEditable, run-started). Its shell — persistence, navigation guard, activity, cascade, trigger drawers — is untouched. - Session PipelineEditorView uses the same factory and now renders the real AssetGraphDetailsPane (code editor + live overlays + test), so a node built in a session opens with its source, matching the route editor. Verified: route page hydrates/renders drafts unchanged; in a session the AI opened the pipeline preview, built a node, and its code showed in the details pane. check:fast clean, 197 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): externalize editor state into PipelineEditorState (step 1) Introduce PipelineEditorState — the data-pipeline analogue of the flow editor's flowStore. It owns the draft Map, the live editor overlays, and the selection, with callback-safe methods (handleDraftPersist / handleAnnotationsChange / … ), so a single editor can be rendered by both the route page and the session. This commit lands the store and points the in-session PipelineEditorView at it (no behaviour change — the session already had these inline). Next steps move the route page onto the store and a shared . Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): point the route editor at PipelineEditorState (step 1) Move the route page's draft Map, live editor overlays, selection, and the draft-persist / live-change handlers onto the shared PipelineEditorState (`pe`), referencing them as `pe.*` in place. No behaviour change — persistence, graph resolution, run dispatch, AI staging, and deploy all stay on the page and now read/write the externalized state. This is the data-pipeline analogue of the flow editor's flowStore: the route page and the in-session preview now share one source of editor truth, setting up the shared in the next steps. Verified: the page hydrates its DB draft, renders the overlay graph, the toolbar counts (Save all (N)) track pe.drafts, and selecting a node opens it in the details pane. check:fast clean, 84 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): render the route editor via shared PipelineGraphEditor (step 2) Extract the canvas + details-pane editor body into PipelineGraphEditor.svelte, the data-pipeline analogue of FlowBuilder. The route page now delegates its Splitpanes block to it, passing the externalized PipelineEditorState plus its run/cascade/trigger/deploy callbacks; the component owns pane sizing, selection/details-open derivation, and the canvas+details rendering. Root-caused the earlier ts2769 "$props() No overload" to a prop named `state` colliding with the `$state` rune (`let x = $state(...)` parsed as a store auto-subscription on the prop) — the prop is now `editor`. Net: the route page sheds ~310 lines of template/state; behaviour preserved. Verified: the page hydrates its DB draft, renders the graph, opens the draft in the details pane (live code editor + Test), pane sizing works. check:fast clean, 24 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): move draft autosave into PipelineGraphEditor (step 3) Fold the per-user `data_pipeline` DraftService bundle autosave (hydrate + debounced persist + localStorage crash mirror) into PipelineGraphEditor, gated by a `persistDrafts` prop — FlowBuilder's parameterized-autosave shape. The route page passes `persistDrafts` + `folder` and reads `editor.loadedFromDbDraft` for its AutosaveIndicator; the in-session preview will leave persistence off. Also restores the `untrack(...)` wrapping on the pane-sizing $effect (dropped when the editor body was extracted in step 2). Without it the Pane `bind:size` feedback loops the effect and pegs the main thread when the details pane is closed — a latent hang in the step-2 commit. check:fast clean, 24 pipeline tests pass. Note: browser revalidation was not possible this session (the Playwright MCP browser was reset); the autosave is a verbatim port and the untrack fix is the original working form. Co-Authored-By: Claude Opus 4.8 (1M context) * refactor(pipeline): render the session preview via shared PipelineGraphEditor (step 4) Point the in-session PipelineEditorView at the shared PipelineGraphEditor instead of its own inline canvas + details pane. The session now renders the exact same editor body as the route page — gaining the full details/code pane — while opting out of persistence (persistDrafts=false) and the run/cascade/trigger/bounded affordances (their callbacks are omitted, so those controls hide). Building nodes + the Accept/Reject diff still work via the AI helpers. Also fixes issues surfaced by a full `svelte-check` while wiring this up: - PipelineGraphEditor: edit mode opened the details pane unconditionally (a step-2 regression); restored the route's "open only on selection/draft" behaviour. - Route page passed an `isOperator` prop the component doesn't accept (step-2; caught only by full check, not check:fast). - SessionItemNotFound: narrow its `kind` to exclude `pipeline` (pipeline targets never slot-load, so they can't 404 through it) — closes the SessionTarget-widen fallout. - PipelineEditorView: cast the resolveGraph base to AssetGraphResponse. Full `svelte-check` now clean across all pipeline/session files; 137 unit tests pass. (Browser revalidation still pending — Playwright MCP was unavailable this session; see the smoke-test note on the PR.) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(pipeline): stop an infinite microtask loop when persisting a no-output draft handleDraftPersist short-circuits when the open draft's content + inferred writes are unchanged. The writes check compared `d.outputAssets?.length === writes.length`, but a no-output draft has `outputAssets: undefined` (so `?.length` is `undefined`) while the details pane infers an empty `writes: []` (length 0). `undefined === 0` is false, so it never short-circuited: every persist re-wrote the drafts Map with an equivalent object, which gave `activeDraft.script` a new identity → the pane re-emitted its overlays → the graph re-derived → persist fired again. A self- sustaining microtask loop that pegged the renderer and froze the tab on any pipeline carrying a no-output draft (e.g. hydrating one from the saved data_pipeline draft on load). It hangs rather than throwing effect_update_depth_ exceeded because it cycles across microtasks, not within one reactive flush. Fix: coalesce the undefined length to 0 so "no outputs" compares equal to an empty inferred-writes list. Adds pipelineEditorState.test.ts covering the idempotency (fails without the fix) plus the change/no-change cases. Root-caused by instrumenting the reactive churn: every iteration reassigned drafts/liveContent/liveBodyAssets/liveAnnotations/displayGraph with identical values — pure reference churn off the drafts re-write. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ai-chat): make the agent open the pipeline editor before building nodes In a session, the GLOBAL system prompt only *advised* opening the pipeline preview ("show its graph with open_preview ... prefer those tools once it is open"), so the agent routinely skipped it: on a plain "build a data pipeline" request it reached for write_script and staged plain script drafts, and the canvas editor never opened. build_pipeline_node / edit_pipeline_node are only registered once the preview is open, so skipping open_preview also loses the canvas-staged Accept/Reject diff-approval flow entirely. Make the guidance imperative: open_preview(kind="pipeline", path=) is the FIRST step before creating any node (an empty or not-yet-created folder is fine — create_folder first if needed), and pipeline nodes go through build_pipeline_node / edit_pipeline_node, never write_script. This also clears the agent's "the folder might not exist" hesitation that pushed it toward write_script. Verified live (same plain prompt, before/after): before it used write_script with no editor; after, the agent opens the editor first and stages a canvas-highlighted node with Accept all / Reject all. The guidance is gated on previewTools (session-only), so it doesn't affect the non-preview global eval cases. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(pipeline): preserve in-session pipeline drafts across editor hide/show The session preview's PipelineEditorState lived in the PipelineEditorView component with persistDrafts=false. Hiding the editor sets editorVisible=false, which makes `hasEditor` false and the `{#if hasEditor}` block unmount the view — discarding its component-local store. Showing it again remounted a fresh, empty one, so the pipeline the AI had built in the session vanished. Move the PipelineEditorState onto the per-session SessionRuntime (like the flow / script / raw_app editors, which already host their state there and take {runtime}), so it survives the pane unmount on hide and across session switches. The runtime is keyed by session id and only dropped on session deletion. Because the instance is now reused, guard against a retarget to a different folder: PipelineEditorView resets the state when `path` changes to a new folder (a same-folder remount keeps the drafts). Adds `folder` + `reset()` to the store. Verified: build a node in a session → Close editor → Show editor → the staged node, its wiring, the details-pane code, and Accept/Reject all re-appear. Full svelte-check clean; 139 pipeline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(pipeline-ai): clearer diff + persistent review banner on the canvas The AI review affordance had two problems on the pipeline canvas: - The floating Accept-all / Reject-all bar sat bottom-center, where it collided with the minimap once the canvas narrowed on node selection — reading as "the buttons vanished when I select a node". - Every staged draft rendered with the same blue ring, so it wasn't clear what the review would actually change (a plain manual draft looked the same as an AI proposal). Replace the floating bar with a top-left review banner (z-30, clear of the controls and minimap) that stays put regardless of selection and spells out the pending counts. Color the diff per node: a proposal that adds a node that isn't deployed rings green with a "new" chip; one that edits an already-deployed node rings amber with an "edited" chip. Plain manual drafts keep the neutral gray dashed border, so only the green/amber nodes read as part of the Accept/Reject set. aiPendingKind is resolved in resolveGraph (deployed runnable present → modified, else added) and forwarded through the canvas to the node. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): persist in-session pipeline proposals across reload/switch Staged AI proposals lived only in the per-session runtime's in-memory PipelineEditorState (persistDrafts=false), so a page reload — and an LRU-evicted runtime on session switch — dropped them, leaving the canvas and the Accept/Reject review empty even though the chat still showed the nodes as staged. Enable the same per-folder DB-draft persistence the route page uses for the in-session editor. To keep hide/show cheap and race-free, hydration is now gated per editor instance (PipelineEditorState.hydratedFromDb) rather than per component mount: the runtime-hosted instance hydrates ONCE when fresh (reload / evicted runtime) and then keeps its in-memory drafts across the editor pane unmounting on hide — re-reading the DB on every remount would race a not-yet-flushed autosave and drop a just-staged draft. A folder retarget resets the flag so the new folder re-hydrates. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): make Reject all work for rehydrated proposals rejectAll only reverted paths tracked in the in-memory aiSnapshots map, which is rebuilt empty on each editor mount. After a reload (or session switch into a fresh runtime) the proposals are restored from the persisted draft but have no snapshot, so Reject all was a no-op on exactly the nodes it should discard. Sweep any still-pending draft without a snapshot and discard it (revertPath with no snapshot deletes the path; for an edit of a deployed node that correctly falls back to the deployed body). Adds unit coverage for accept/reject including the no-snapshot case. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): keep proposals visible while the graph reloads on switch The session editor pane is LRU-capped (MAX_WARM_EDITORS), so returning to a session whose pane was evicted remounts PipelineEditorView with a fresh graphRes resource (loading=true, current=undefined). The deployed-graph loading spinner gated the whole canvas, so the staged proposals and the Accept/Reject review banner vanished until the re-fetch resolved — read as "the proposal disappears when I switch sessions". Only show the loading/error placeholder when there are no drafts to display. When the runtime already holds staged drafts, render the editor immediately: resolveGraph overlays them on an empty base so the proposals + banner stay visible, and the deployed nodes fill in when the fetch completes. Verified with a 4s-delayed graph fetch — proposals render through the load with no spinner. Co-Authored-By: Claude Opus 4.8 * refactor(pipeline-ai): apply AI node edits directly as drafts, no approve/reject The canvas-level Accept all / Reject all review (aiPending proposals, the green/amber diff ring + "new"/"edited" chips, and the review banner) didn't fit the pipeline editor. Match the flow/script editor instead: build/edit apply directly as ordinary unsaved drafts on the canvas, which the user then deploys — there is no separate approval step. Removed across the surface: - aiPending / aiPendingKind on the runnable node + resolveGraph seeding + canvas forwarding; AI-built nodes now render with the existing plain unsaved-draft dashed styling. - the review banner, count derivations, and hasAiPending/onAccept/onReject props from PipelineGraphEditor and both consumers (route page + session view). - acceptAll/rejectAll/hasPending and the per-turn snapshot bookkeeping from the shared helpers; removeProposedNode now just discards the unsaved draft at a path (undo a build). acceptAllProposals/rejectAllProposals/ hasPendingProposals dropped from the PipelineAIChatHelpers interface and the manager's auto-accept hook. - accept/reject language from the tool descriptions, return messages, and the system-prompt section. Tests updated; pipeline + AssetGraph suites pass (142). Co-Authored-By: Claude Opus 4.8 * fix(drafts-diff): support data_pipeline diffs + fix blank empty-summary row Two issues in the session "Drafts" diff drawer (DraftDiffDrawer): - Clicking a `data_pipeline` bundle row threw "Draft diff not supported for kind data_pipeline" (utils_draft_deploy.ts) — there was no handler for the kind, so it fell to the OVERLAY_GETTERS lookup and errored. The bundle has no deployed counterpart (each node deploys individually as a script), so diff it node-by-node: surface each node's draft body keyed by path, folding in the deployed body as the "before" when a node edits a deployed script. - A draft row whose summary is an empty string (e.g. the app draft) rendered with no title at all: WorkspaceItemRow's single-line branch used `summary ?? secondary`, and `??` doesn't treat '' as absent, so it showed the empty summary instead of the path. Use `||` so an empty summary falls back to the path. Co-Authored-By: Claude Opus 4.8 * feat(drafts-diff): explode data_pipeline bundle into per-node subitems A data_pipeline draft is a bundle of node-script drafts, so a single row diffed the whole thing as one blob. Explode it in DraftDiffDrawer into one script row per node, nested under the bundle's `…/data_pipeline` folder so they read as the pipeline's subitems — each with its own path and a proper script Content/Metadata code diff. The node's draft body is the "after"; its deployed body (when the node is already deployed) is the "before", so edits show as line diffs and new nodes as added. A single bundle row (via the getDraftDiffValues data_pipeline fallback) is kept only for the case where the bundle can't be read. Co-Authored-By: Claude Opus 4.8 * refactor(pipeline-ai): simplify — drop vestigial approve/reject scaffolding & redundant field Review pass over the PR, removing complexity left from the approve/reject removal and the shared-component refactor (all behavior-preserving): - Inline the `acceptPendingEdits` pass-through into `acceptPendingFlowEdits` and revert the now-inert `autoAcceptEditsAvailable` GLOBAL+pipeline widening (pipeline edits are direct drafts — nothing to auto-accept). - Fix the global system prompt: pipeline tools "apply directly as unsaved drafts (no accept/reject)", not "proposals the user Accepts or Rejects". - Collapse the redundant `outputAsset` (singular) into `outputAssets`, removing a whole resolveGraph fallback tier; simplify propose/editNode. - Drop the single-field `PipelineAiHelpersHandle` wrapper (callers just destructured `{ helpers }`); inline the misleading `isoNow()` helper. - Remove the now-unreachable `data_pipeline` branch in getDraftDiffValues (the drafts drawer explodes bundles per-node; an unreadable bundle is skipped) and the "Step N consolidation" drafting narration. - Un-export internal-only types; reuse `storageKey`; refresh stale comments that still referenced proposals / the review banner / diff-approval. svelte-check clean; 141 unit tests pass. Co-Authored-By: Claude Opus 4.8 * feat(pipeline): tooltip clarifying the Create/Save button deploys The accent button in the asset-graph details pane ("Create" for a new script, "Save" for an existing one) is really a deploy, but had no tooltip explaining that. Add a title — "Deploy this new script to the workspace" / "Deploy your changes to this script" — keeping the create-vs-update label distinction while making clear both deploy. Co-Authored-By: Claude Opus 4.8 * docs(pipeline-ai): document the `materialize` annotation in the pipeline prompt The model invented "materialize run" because the prompt only mentioned `// materialize ` in passing. Spell out what it is in both the in-app pipeline prompt (getPipelinePromptSection) and the base prompt (pipeline-base.md, regenerated): a MANAGED output where the runtime writes the table around a single SELECT (no manual CREATE/INSERT); replace (default) vs `append` vs `key=` strategies; `manual` to opt out (track-only); and its pairing with `// partitioned …` (runs once per partition, `{partition}` token substituted at run time). Explicitly: materialize is an output declaration, not a command — there is no "materialize run". Co-Authored-By: Claude Opus 4.8 * feat(pipeline-ai): trigger drawers in the AI session preview Bring the route page's native-trigger affordances to the in-session pipeline editor by reusing the shared (no duplication of the drawer UI). Clicking a "Schedule · Missing — no trigger row" node (or edit/delete on an attached trigger, webhook, data-upload) now opens the same drawers the full editor uses, instead of doing nothing. Draft nodes get the same "save the script first" guard (a trigger row needs a deployed script). Co-Authored-By: Claude Opus 4.8 * feat(pipeline-ai): run buttons + live run state in the AI session preview Wire the per-node Run button and live run-state badges into the in-session pipeline editor, reusing the shared folder-scoped job poll (useActiveRunnableIds) the route page uses — node badges, the event log, and the zero-latency "running" hint all come from it. The session runs one node at a time (preview for an unsaved draft, the deployed version otherwise), skipping the route page's cascade/deploy-queue machinery the AI-session UX doesn't need. Verified: a node's Run button dispatches a job and the badge updates live from the poll. Co-Authored-By: Claude Opus 4.8 * feat(pipeline): label the node deploy button "Deploy" (was Create/Save) Users read "Create" and asked whether it deploys. It does — and the main script editor's DeployButton already says "Deploy", so this is the consistent term. Use "Deploy" for both the new-script and existing-script cases; the new-vs-changes nuance stays in the button's tooltip. Co-Authored-By: Claude Opus 4.8 * feat(home): surface data pipelines as units, including bundle-phase drafts Treat a data pipeline as one home entry instead of scattering its member scripts: - The home "Pipeline · f/" entry now also covers bundle-phase pipelines — a folder that so far only exists as a `data_pipeline` draft — not just deployed ones, so a pipeline shows up the moment its first node is drafted (union listPipelineFolders + data_pipeline draft folders). - Pipeline-member scripts (`auto_kind='pipeline'`) are filtered out of the individual scripts list; they're represented by their pipeline's entry. - Tree view injects pipeline folders so they (and their "Pipeline" entry) still appear when their only scripts are hidden members or they have none deployed yet. Verified in both list and tree view: app_groups (deployed member folded) and a draft-only nyc_transit both show as pipelines; the member script no longer lists individually. Co-Authored-By: Claude Opus 4.8 * feat(scripts): compute auto_kind for draft-only pipeline nodes A never-deployed pipeline node (a script draft starting with `// pipeline`) had no script row, so list_scripts synthesized it with `auto_kind: None` — and the home page therefore couldn't tell it was a pipeline member, listing it individually instead of folding it into its pipeline. Parse the draft content the same way the create path does (`parse_pipeline_annotations(...).in_pipeline`) and set `auto_kind = "pipeline"` on the synthesized draft-only row, so draft nodes fold into their pipeline like deployed members. Co-Authored-By: Claude Opus 4.8 * feat(search): hide pipeline-member scripts from global search The Ctrl+k global search listed pipeline-member scripts (`auto_kind='pipeline'`) individually. Filter them out — they're reached through their pipeline, matching the home page. Deployed members carry auto_kind from the script row; draft-only members now do too (computed from draft content in list_scripts), so both are excluded here. Co-Authored-By: Claude Opus 4.8 * fix(pipeline): address PR review findings Session run dispatch (the one real bug): - runNode now passes `_wmill_skip_asset_dispatch: true` for a single-node run of a deployed node unless the user chose "run + downstream" (cascade) — previously a single Run could fan out to downstream deployed scripts via the backend asset dispatcher and fire side-effecting production runs. - onRunProducer guards `kind === 'script'`; onTestStateChange only clears the run hint for the script the pane finished (not a different in-flight node); clear the hint on folder retarget; gate the background poll on isActiveSession so hidden warm panes don't poll; note the PipelineTriggerEditors workspace coupling. Home page pipeline surfacing: - Fold pipeline-member folders into `pipelineFolders` (captured in loadScripts) so a members-only / draft-only-`// pipeline` folder still shows its pipeline entry instead of vanishing; and don't render the empty-state when only pipelines remain (they aren't part of the text filter). - Insert injected tree folders in name order instead of prepending. Co-Authored-By: Claude Opus 4.8 * docs(pipeline-ai): make clear `// materialize` is DuckDB + DuckLake only The model put `// materialize` on a python3 node, which deploy rejects ("only supported for DuckDB scripts"). The prompt only implied SQL ("write the body as a single SELECT") without stating the hard constraint. Spell it out in both the in-app prompt and pipeline-base.md: `// materialize` is DuckDB-only and its target must be a DuckLake table; for python3/bun/postgresql nodes, write the output via the SDK instead and let it be inferred — reach for duckdb when a node should materialize a DuckLake table. Co-Authored-By: Claude Opus 4.8 * docs(pipeline-ai): fix stale comment — session now wires run + trigger affordances Addresses review: the comment still claimed the session 'opts out of the run/cascade/trigger/bounded affordances', but run buttons + trigger drawers were wired in. Describe the current state (wires run + triggers; omits only cascade/bounded/add-script). Co-Authored-By: Claude Opus 4.8 * fix(pipeline): address Codex review — test_pipeline_node dispatch + tree search - [P1] testNode (the test_pipeline_node tool) ran a deployed node via runScriptByPath without `_wmill_skip_asset_dispatch`, so previewing one node could fan out to downstream deployed subscribers and run side-effecting scripts. Add the skip flag (test is always single-node) + a regression test. - [P2] Home tree view injected pipeline folders — and rendered their Pipeline row — even during a text search, surfacing unrelated pipelines. Gate both the TreeViewRoot injection and TreeView's hasPipeline on `!isSearching`, matching the list view which hides pipeline rows on a query. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): keep the pipeline prompt after update_user_instructions rebuildGlobalSystemMessage (called by the update_user_instructions tool) rebuilt only the base Global prompt, dropping the pipeline-editor section that configureGlobalMode appends. So after the chat remembered an instruction, the next GLOBAL turn lost the active /pipeline/ context + direct-draft/ materialize guidance while pipeline tools stayed registered. Re-append the pipeline section here when a pipeline editor is registered. Addresses Codex review [P2]. Co-Authored-By: Claude Opus 4.8 * fix(home): gate pipeline entries by kind/archived/owner filters Codex review [P2]: pipeline rows/folders rendered independently of the item filters, so a pipeline still showed under the Flows/Apps tabs, in the archived view, and outside a selected owner. Add `visiblePipelineFolders` applying the same gates the items get (kind ∈ {all, script}, not archived, owner-prefix match) and route the list rows, tree injection, and empty-state check through it. Pipelines are always `f/`, so the user-folder toggle and kind=script keep including them. Co-Authored-By: Claude Opus 4.8 * fix(pipeline): address review — route folder-switch state, AI node guards, diff identity claude[bot] [P1]: the route page's in-app folder switcher navigates same-route (no remount), but nothing reset PipelineEditorState — so folder A's drafts displayed under B and autosave persisted them into B's bundle, and B never hydrated. Reset pe on folder change (mirror the session retarget), and guard the shared hydrateDrafts against a stale folder result landing after a retarget. codex/claude [P2]: build_pipeline_node (proposeNode) only checked drafts.has — now rejects a path outside the open folder and one colliding with an existing deployed node (model should edit_pipeline_node). + 3 regression tests. codex/claude [P2]: exploded pipeline-node diff rows shared `script/` with a standalone script draft at the same path, colliding in the {#each} key + value cache. Add an explicit unique `key` (the distinct bundle-nested path) on DiffRow; pipeline nodes set/look up by it while `path` stays the real edit target. claude [P2]: session AI test_pipeline_node now arms the live run badge (onRunStarted), matching the route page. nit: pipelineAiHelpers.test uses afterEach(restoreAllMocks) instead of an unreachable inline mockRestore. Co-Authored-By: Claude Opus 4.8 * fix(pipeline): harden AI node mutations + close home label-filter / rename gaps Codex [P1] (AI mutations trust model paths) — fully scoped now: - editNode validates the open folder too (proposeNode already did), via a shared assertInFolder; an edit_pipeline_node for f/other/* no longer persists an unrelated script into the current folder's data_pipeline bundle. - both build_pipeline_node and edit_pipeline_node now require the `// pipeline` annotation (assertPipelineAnnotation) so a staged draft is definitionally a pipeline member, not a silently-non-member script. + tests. (proposeNode's folder + deployed-collision guards landed in the prior commit.) Codex [P2] home label filter — visiblePipelineFolders ignored labelFilter, so a label selection still showed every pipeline (and the empty-state fell through to render pipeline rows). Pipelines carry no labels, so a label filter hides them. Codex [P2] session rename — PipelineEditorView now wires onScriptRenamed (repoint selection + refetch), matching the route page; a persisted-script rename no longer leaves the canvas on the old path. Co-Authored-By: Claude Opus 4.8 * docs(pipeline-ai): language-specific comment prefix for annotations Codex [P2]: the tool schema and prompt told the model to write `// pipeline` / `// on` / `// materialize` regardless of language, and pipeline-base.md grouped SQL with `#`. A `//` (or `#`) annotation line is invalid in a DuckDB/Postgres node — it passes the frontend parser (which strips `//`/`--`/`#`) but is a SQL syntax error at deploy/run. Make the guidance language-specific everywhere: `--` for SQL (duckdb/postgresql), `#` for python3/bash, `//` for bun/TS — the `//` in examples is the TS form to translate. Regenerated the prompt outputs. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): re-scope Global prompt on folder switch + language-aware base prompt Codex [P2] x2: - The route page resets editor state on an in-app folder switch, but the Global chat's system message kept the old `/pipeline/` scope (the helper methods read the reactive folder, but the prompt string is only rebuilt on Global-mode reconfigure). Rebuild it on folder change so the next turn targets the new folder. - The pre-editor base Global prompt (seen before open_preview/get_instructions) still showed TS-only `// pipeline` / `// on`. Make it language-aware (`--` SQL, `#` Python/Bash, `//` TS) so the model can't draft invalid DuckDB/Postgres nodes before the pipeline tools are registered. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): authoritative new-node probe + SQL-correct eval checklist Codex [P2] x2: - build_pipeline_node's collision check relied on the resolved graph, which can be empty while the session preview races open_preview (a build could shadow a deployed node before the graph loads) and only covered pipeline runnables, not a non-pipeline script at the same path. Add an authoritative backend probe (ScriptService.getScriptByPath): any deployed script at the path → reject with "use edit_pipeline_node". + regression test (empty graph, deployed script). - The DuckLake eval judgeChecklist required the exact `// pipeline` annotation, which would penalize the now-correct `-- pipeline` SQL output (or reward invalid DuckDB syntax). Make both cases syntax-aware. Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): rebuild Global prompt on session preview folder retarget Codex [P2]: open_preview(kind="pipeline", path="B") can retarget an existing pipeline preview from folder A to B without remounting. The retarget effect resets editor state and the helper methods read the new path, but the registration effect only depends on isActiveSession, so the Global system message stayed scoped to /pipeline/A. Mirror the route-page fix: rebuild the global system message on retarget (gated on isActiveSession — only the active session's helpers are registered; a hidden session reconfigures when it next becomes active). Co-Authored-By: Claude Opus 4.8 * fix(pipeline-ai): edit_pipeline_node preserves deployed script metadata Codex [P1]: editNode kept only the deployed script's language and staged a fresh makePipelineScript draft with empty hash/summary/description/tag/schema/settings. Deploying that edit from the pane (auto_parent) would update the script while wiping its metadata, and the route "Save all" path (no parent_hash) could hit the backend path-conflict branch on the occupied path. Base the draft on the existing draft's / deployed script object and replace ONLY content (+ inferred output assets), preserving hash and metadata. + regression test. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 (1M context) --- ai_evals/cases/global.yaml | 60 + backend/windmill-api-scripts/src/scripts.rs | 10 +- .../lib/components/WorkspaceItemRow.svelte | 6 +- .../AssetGraph/AssetGraphDetailsPane.svelte | 6 +- .../AssetGraph/PipelineGraphEditor.svelte | 477 +++++++ .../AssetGraph/pipelineAiHelpers.test.ts | 143 +++ .../assets/AssetGraph/pipelineAiHelpers.ts | 319 +++++ .../AssetGraph/pipelineEditorState.svelte.ts | 184 +++ .../AssetGraph/pipelineEditorState.test.ts | 70 + .../assets/AssetGraph/resolveGraph.test.ts | 23 +- .../assets/AssetGraph/resolveGraph.ts | 17 +- .../lib/components/assets/AssetGraph/types.ts | 1 + .../copilot/chat/AIChatManager.svelte.ts | 116 +- .../components/copilot/chat/global/core.ts | 33 +- .../copilot/chat/pipeline/core.test.ts | 171 +++ .../components/copilot/chat/pipeline/core.ts | 328 +++++ .../src/lib/components/home/ItemsList.svelte | 92 +- .../src/lib/components/home/TreeView.svelte | 8 +- .../lib/components/home/TreeViewRoot.svelte | 37 +- .../search/GlobalSearchModal.svelte | 16 +- .../sessions/DraftDiffDrawer.svelte | 96 +- .../sessions/PipelineEditorView.svelte | 381 ++++++ .../sessions/SessionItemNotFound.svelte | 8 +- .../components/sessions/SessionWrapper.svelte | 11 +- .../sessions/WorkspaceDiffDrawer.svelte | 6 + .../sessions/sessionRuntime.svelte.ts | 11 + .../sessions/sessionState.svelte.ts | 10 +- .../(logged)/pipeline/[folder]/+page.svelte | 1139 +++++------------ system_prompts/auto-generated/index.d.ts | 1 + system_prompts/auto-generated/index.ts | 5 + system_prompts/auto-generated/prompts.d.ts | 1 + system_prompts/auto-generated/prompts.ts | 62 + system_prompts/base/pipeline-base.md | 60 + system_prompts/generate.py | 8 + 34 files changed, 3008 insertions(+), 908 deletions(-) create mode 100644 frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte create mode 100644 frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts create mode 100644 frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts create mode 100644 frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts create mode 100644 frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/pipeline/core.test.ts create mode 100644 frontend/src/lib/components/copilot/chat/pipeline/core.ts create mode 100644 frontend/src/lib/components/sessions/PipelineEditorView.svelte create mode 100644 system_prompts/base/pipeline-base.md diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 31b097867b..66e5fd3cb7 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -1223,6 +1223,66 @@ - places it in team_a (writable by this non-admin user) and not team_b (read-only) - leaves the result as a draft only +- id: global-test-pipeline-create-node + prompt: |- + Set up the first step of a data pipeline at `f/evals/global/orders_ingest`. + On a schedule, it should pull raw orders and land them in a managed DuckLake + table so later steps can build on it. Keep it as an AI draft only — don't + deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + path: f/evals/global/orders_ingest + valueIncludes: + - pipeline + forbiddenDrafts: + - type: flow + pathStartsWith: f/evals/global/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - write_flow + - deploy_workspace_item + - delete_workspace_item + judgeChecklist: + - builds a data pipeline node as a script (not a flow) + - marks the script as a pipeline member with the pipeline annotation in the script's comment syntax (`-- pipeline` for a DuckDB/SQL node, not `// pipeline`) + - declares a schedule trigger and writes its output to a managed DuckLake table + - leaves the result as an AI draft and does not deploy or save it + +- id: global-test-pipeline-two-node-chain + prompt: |- + Build a small data pipeline in the `f/evals/global` folder: one step that + ingests orders into a DuckLake table, and a second step that reads that table + and writes a daily order-count rollup table. Wire the second step to run off + the first step's output. Keep everything as drafts — don't deploy. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json + runtime: + maxTurns: 14 + validate: + draftCountAtLeast: 2 + forbiddenDrafts: + - type: flow + pathStartsWith: f/evals/global/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - write_flow + - deploy_workspace_item + - delete_workspace_item + judgeChecklist: + - creates two data pipeline nodes as scripts (not a flow) in f/evals/global + - both scripts carry the pipeline annotation in their comment syntax (`-- pipeline` for DuckDB/SQL nodes, not `// pipeline`) + - the first ingests orders into a DuckLake table + - the second reads that same table and writes a daily rollup, wired to the first step's output asset + - leaves both as AI drafts without deploying + - id: global-path5-create-folder-then-draft prompt: |- Create a new shared folder called "analytics" for our data work, then draft a diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index 9aecd66947..5fd428bca3 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -408,6 +408,14 @@ async fn list_scripts( .and_then(|s| s.as_str()) .filter(|s| !s.is_empty() && *s != row.path.as_str()) .map(|s| s.to_string()); + // A draft-only pipeline node (`// pipeline`) has no deployed row to carry + // auto_kind, so compute it from the draft content — mirroring the create + // path — so the home page folds it into its pipeline like a deployed member. + let auto_kind = v + .get("content") + .and_then(|s| s.as_str()) + .filter(|c| parse_pipeline_annotations(c).in_pipeline) + .map(|_| "pipeline".to_string()); rows.push(ListableScript { hash: ScriptHash(0), path: row.path, @@ -429,7 +437,7 @@ async fn list_scripts( draft_only: Some(true), has_deploy_errors: false, ws_error_handler_muted: None, - auto_kind: None, + auto_kind, use_codebase: false, deployment_msg: None, kind, diff --git a/frontend/src/lib/components/WorkspaceItemRow.svelte b/frontend/src/lib/components/WorkspaceItemRow.svelte index 6e5900594f..deb8ddc27d 100644 --- a/frontend/src/lib/components/WorkspaceItemRow.svelte +++ b/frontend/src/lib/components/WorkspaceItemRow.svelte @@ -110,11 +110,13 @@ doesn't steal focus from a sibling search input (matches the picker).
{#if singleLine} +
- {summary ?? secondary} + {summary || secondary}
{:else if summary}
{summary}
diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte index 46ebaaf624..cd648c5faf 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte @@ -946,14 +946,18 @@ /> {/if} {#if !readOnly && isScriptView && script && !atLatestSavePoint} + {@const isCreate = isDraft && !script?.hash} {/if} {#if onHide} diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte new file mode 100644 index 0000000000..6c4611a54e --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineGraphEditor.svelte @@ -0,0 +1,477 @@ + + + + +
+ + {#if boundBar}{@render boundBar()}{/if} + {#if mode === 'edit'} + + {/if} + {#if prefetchingAssets} +
+ + Parsing assets… +
+ {/if} + {#if onTogglePanelHidden && (mode !== 'edit' || editor.selection != undefined || editor.activeDraftPath != undefined)} +
+
+ {/if} +
+ {#if detailsPaneOpen && workspace} + + {#if idleView && idlePane} + {@render idlePane()} + {:else} + onStartBoundedRunForOpen?.(editor.openScriptPath!) + : undefined} + {onRunCompleted} + {onTestStateChange} + {requestRemoveSignal} + {requestRunSignal} + {requestRunCascadeSignal} + {focusUploadSignal} + draftScript={activeDraft?.script} + {pathPrefix} + {onDraftPathChange} + {workspace} + onAnnotationsChange={editor.handleAnnotationsChange} + onAssetsChange={editor.handleAssetsChange} + onContentChange={editor.handleContentChange} + onDraftPersist={editor.handleDraftPersist} + onclose={onClose} + onHide={onTogglePanelHidden} + {onDiscard} + {onDraftSaved} + {onPersistedSaved} + {onScriptRenamed} + {onScriptRemoved} + /> + {/if} + + {/if} + diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts new file mode 100644 index 0000000000..a1a82870c8 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.test.ts @@ -0,0 +1,143 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { JobService, ScriptService } from '$lib/gen' +import { createPipelineAiHelpers, type PipelineDraft } from './pipelineAiHelpers' +import type { AssetGraphResponse } from './types' + +// Build a helper handle over an in-memory drafts Map, mirroring how the editor +// wires it. `getFolder` returns the bare folder name (as the route/session do). +function makeHandle( + initial: Array<[string, PipelineDraft]> = [], + runnables: Array<{ path: string }> = [] +) { + let drafts = new Map(initial) + let forgotten: string[] = [] + const handle = createPipelineAiHelpers({ + getFolder: () => 'x', + getWorkspace: () => 'w', + getResolvedGraph: () => + ({ assets: [], runnables, edges: [], triggers: [] }) as unknown as AssetGraphResponse, + getDrafts: () => drafts, + setDrafts: (next) => (drafts = next), + newDraftLocalId: () => 'id', + onForgetPath: (p) => forgotten.push(p) + }) + return { handle, drafts: () => drafts, forgotten: () => forgotten } +} + +afterEach(() => vi.restoreAllMocks()) + +const draft = (over: Partial = {}): PipelineDraft => + ({ localId: 'l', script: { content: '' } as any, ...over }) as PipelineDraft + +describe('pipeline AI direct-draft helpers', () => { + it('removeProposedNode discards the unsaved draft at a path', async () => { + const { handle, drafts, forgotten } = makeHandle([ + ['f/x/a', draft()], + ['f/x/b', draft()] + ]) + await handle.removeProposedNode('f/x/a') + expect(drafts().has('f/x/a')).toBe(false) + expect(drafts().has('f/x/b')).toBe(true) + expect(forgotten()).toContain('f/x/a') + }) + + it('removeProposedNode throws when there is no draft to discard', async () => { + const { handle } = makeHandle() + await expect(handle.removeProposedNode('f/x/missing')).rejects.toThrow() + }) + + it('getPipelineContext does not expose any pending/approval state', () => { + const { handle } = makeHandle([['f/x/a', draft()]]) + const ctx = handle.getPipelineContext() + expect(ctx).not.toHaveProperty('pendingProposals') + expect(handle).not.toHaveProperty('acceptAll') + expect(handle).not.toHaveProperty('rejectAll') + }) + + it('testNode on a deployed node never dispatches downstream subscribers', async () => { + // No draft at the path → runs the deployed version, which must carry + // `_wmill_skip_asset_dispatch` so a single-node test can't fire downstream. + const spy = vi.spyOn(JobService, 'runScriptByPath').mockResolvedValue('job-1' as any) + const { handle } = makeHandle() + await handle.testNode('f/x/deployed', { foo: 1 }) + expect(spy).toHaveBeenCalledWith( + expect.objectContaining({ + requestBody: expect.objectContaining({ _wmill_skip_asset_dispatch: true, foo: 1 }) + }) + ) + }) + + it('proposeNode rejects a path outside the open folder', async () => { + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ path: 'f/other/n', language: 'duckdb' as any, content: '' }) + ).rejects.toThrow(/open folder/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects content missing the pipeline annotation', async () => { + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ path: 'f/x/new', language: 'duckdb' as any, content: 'SELECT 1' }) + ).rejects.toThrow(/pipeline annotation/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects a path colliding with an existing draft', async () => { + const { handle } = makeHandle([['f/x/a', draft()]]) + await expect( + handle.proposeNode({ path: 'f/x/a', language: 'duckdb' as any, content: '-- pipeline' }) + ).rejects.toThrow(/already exists/) + }) + + it('proposeNode rejects a path colliding with an existing deployed node', async () => { + const { handle, drafts } = makeHandle([], [{ path: 'f/x/dep' }]) + await expect( + handle.proposeNode({ path: 'f/x/dep', language: 'duckdb' as any, content: '-- pipeline' }) + ).rejects.toThrow(/already exists/) + expect(drafts().size).toBe(0) + }) + + it('proposeNode rejects a path that is an already-deployed script when the graph has not hydrated', async () => { + // Empty graph (session preview can race open_preview), but a deployed script + // exists at the path — the backend probe must still catch it. + const spy = vi.spyOn(ScriptService, 'getScriptByPath').mockResolvedValue({} as any) + const { handle, drafts } = makeHandle() + await expect( + handle.proposeNode({ + path: 'f/x/deployed', + language: 'duckdb' as any, + content: '-- pipeline' + }) + ).rejects.toThrow(/already exists/) + expect(spy).toHaveBeenCalled() + expect(drafts().size).toBe(0) + }) + + it('editNode rejects a path outside the open folder', async () => { + const { handle, drafts } = makeHandle() + await expect(handle.editNode('f/other/foo', '-- pipeline')).rejects.toThrow(/open folder/) + expect(drafts().size).toBe(0) + }) + + it('editNode preserves the deployed script hash/metadata and replaces only content', async () => { + const deployed = { + hash: 'abc123', + path: 'f/x/node', + summary: 'My node', + description: 'desc', + tag: 'custom', + language: 'duckdb', + content: '-- pipeline\nSELECT 1' + } + vi.spyOn(ScriptService, 'getScriptByPath').mockResolvedValue(deployed as any) + const { handle, drafts } = makeHandle() + await handle.editNode('f/x/node', '-- pipeline\nSELECT 2') + const d = drafts().get('f/x/node') + expect(d?.script.hash).toBe('abc123') + expect(d?.script.summary).toBe('My node') + expect(d?.script.description).toBe('desc') + expect(d?.script.tag).toBe('custom') + expect(d?.script.content).toBe('-- pipeline\nSELECT 2') + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts new file mode 100644 index 0000000000..959992543f --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineAiHelpers.ts @@ -0,0 +1,319 @@ +import { JobService, ScriptService, type AssetKind, type Script, type ScriptLang } from '$lib/gen' +import { emptySchema, sendUserToast } from '$lib/utils' +import { inferAssets } from '$lib/infer' +import { extractWrites, type AssetWithAltAccessType } from '$lib/components/assets/lib' +import { assetUri, autoOutputAsset, type PipelineOutputKind } from './pipelineTemplates' +import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import type { AssetGraphResponse } from './types' +import type { + PipelineAIChatHelpers, + PipelineContext, + PipelineNodeSummary +} from '$lib/components/copilot/chat/pipeline/core' + +// ============================================================================ +// Shared data-pipeline AI helper layer. +// +// Both the full-page editor (/pipeline/[folder]) and the in-session preview +// (PipelineEditorView) drive the AI chat's pipeline tools through this factory, +// so the build/edit logic lives in exactly one place. Each caller injects +// accessors for its own draft Map and graph; this module owns the AI behaviour +// (build/edit/discard/test). AI edits apply directly as unsaved drafts — there +// is no separate approve/reject step. +// ============================================================================ + +/** + * An unsaved pipeline node draft. `localId` is a stable per-draft id preserved + * across renames (the page uses it to dedupe concurrent deploys). AI-built nodes + * and manually-created drafts are the same thing — an unsaved node on the canvas. + */ +export type PipelineDraft = { + localId: string + script: Script + outputAssets?: Array<{ kind: AssetKind; path: string }> +} + +export type PipelineAiHelperDeps = { + getFolder: () => string + getWorkspace: () => string | undefined + /** The draft-overlaid graph (resolveGraph output) the context summary reads. */ + getResolvedGraph: () => AssetGraphResponse + getDrafts: () => Map + setDrafts: (next: Map) => void + /** Stable id for a freshly-created draft (route page tracks deploys by it). */ + newDraftLocalId: () => string + /** Focus/select the node after it is staged (pan + open in the pane). */ + onProposeNode?: (path: string) => void + /** Throw (or switch to edit mode) when the surface can't accept AI edits. */ + ensureEditable?: () => void + /** Surface the draft overlay if it is hidden (the page's "show drafts" view). */ + onShowDrafts?: () => void + /** Forget per-path state when a draft is discarded. */ + onForgetPath?: (path: string) => void + /** Notify the caller a test run started so it can light up its run UI. */ + onRunStarted?: (jobId: string, path: string) => void +} + +export function makePipelineScript( + language: ScriptLang, + scriptPath: string, + content: string, + createdAt: string +): Script { + // Cast through unknown: a local draft only needs path/language/content/schema; + // the many readonly deployment fields on Script don't matter until createScript. + return { + hash: '', + path: scriptPath, + summary: '', + description: '', + content, + schema: emptySchema(), + is_template: false, + extra_perms: {}, + language, + kind: 'script', + created_by: '', + created_at: createdAt, + archived: false, + deleted: false, + starred: false + } as unknown as Script +} + +async function inferOutputAssets( + language: ScriptLang, + content: string +): Promise> { + try { + const inferred = await inferAssets(language, content) + if (inferred?.status === 'error') return [] + return extractWrites((inferred?.assets ?? []) as AssetWithAltAccessType[]) + } catch { + return [] + } +} + +export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIChatHelpers { + // A staged draft is always persisted into the OPEN folder's data_pipeline + // bundle, so a path outside the folder would silently land an unrelated script + // there. Both build and edit must stay scoped to the folder. + function assertInFolder(path: string) { + const folder = deps.getFolder() + if (folder && !path.startsWith(`f/${folder}/`)) { + throw new Error( + `Pipeline nodes must be in the open folder — use a path under 'f/${folder}/' (got '${path}').` + ) + } + } + + // A pipeline node IS its `// pipeline` annotation (it's what makes the deployed + // script a pipeline member). Reject content that lacks it so a staged draft + // isn't a non-member script the model can't see is broken until deploy. + function assertPipelineAnnotation(content: string) { + if (!parsePipelineAnnotations(content).inPipeline) { + throw new Error( + `Pipeline node content must declare the pipeline annotation on its own comment line ` + + `(\`// pipeline\`, or \`-- pipeline\` for SQL / \`# pipeline\` for Python).` + ) + } + } + + function buildContext(): PipelineContext { + const graph = deps.getResolvedGraph() + const drafts = deps.getDrafts() + const nodes: PipelineNodeSummary[] = graph.runnables + .filter((r) => r.usage_kind === 'script') + .map((r) => { + const draft = drafts.get(r.path) + const writes = graph.edges + .filter( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === r.path && + (e.access_type === 'w' || e.access_type === 'rw') + ) + .map((e) => assetUri({ kind: e.asset_kind, path: e.asset_path })) + const reads = graph.edges + .filter( + (e) => + e.runnable_kind === 'script' && + e.runnable_path === r.path && + (e.access_type === 'r' || e.access_type === 'rw') + ) + .map((e) => assetUri({ kind: e.asset_kind, path: e.asset_path })) + const triggers = graph.triggers + .filter((t) => t.runnable_kind === 'script' && t.runnable_path === r.path) + .map((t) => + t.trigger_kind === 'asset' + ? assetUri({ kind: t.asset_kind, path: t.asset_path }) + : t.trigger_kind + ) + return { + path: r.path, + language: draft?.script.language, + unsaved: r.unsaved ?? false, + summary: draft?.script.summary || undefined, + writes: [...new Set(writes)], + reads: [...new Set(reads)], + triggers: [...new Set(triggers)] + } + }) + return { + folder: deps.getFolder(), + mode: 'edit', + nodes, + assets: graph.assets.map((a) => assetUri({ kind: a.kind, path: a.path })) + } + } + + const helpers: PipelineAIChatHelpers = { + getPipelineContext: buildContext, + getNodeBody: async (path) => { + const draft = deps.getDrafts().get(path) + if (draft) return { language: draft.script.language, content: draft.script.content } + const workspace = deps.getWorkspace() + if (!workspace) return undefined + try { + const deployed = await ScriptService.getScriptByPath({ workspace, path }) + return { language: deployed.language, content: deployed.content } + } catch { + return undefined + } + }, + proposeNode: async ({ path, language, content, outputKind }) => { + deps.ensureEditable?.() + // build_pipeline_node creates a NEW node in the OPEN folder. Reject a path + // outside the folder (it would silently stage into this folder's bundle) + // and a path that collides with an existing node (the model should use + // edit_pipeline_node instead of shadowing a deployed node as a draft). + assertInFolder(path) + assertPipelineAnnotation(content) + const drafts = deps.getDrafts() + if (drafts.has(path)) { + throw new Error( + `A draft already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + if (deps.getResolvedGraph().runnables.some((r) => r.path === path)) { + throw new Error( + `A pipeline node already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + // Authoritative new-node check: the resolved graph may not have hydrated yet + // (the session preview can race open_preview), and it only lists pipeline + // runnables — so probe the backend. ANY deployed script at this path means + // "build new" would shadow it on deploy; the model should edit instead. + const workspace = deps.getWorkspace() + if (workspace) { + let deployedExists = false + try { + await ScriptService.getScriptByPath({ workspace, path }) + deployedExists = true + } catch { + // 404 → no deployed script at this path, safe to create a new node. + } + if (deployedExists) { + throw new Error( + `A script already exists at '${path}'. Use edit_pipeline_node to change it instead.` + ) + } + } + const inferred = await inferOutputAssets(language, content) + // Fall back to a seeded output (from the declared output_kind) when the + // body doesn't yet write anything inferable. + const seeded = + inferred[0] ?? + (outputKind + ? autoOutputAsset(outputKind as PipelineOutputKind, deps.getFolder(), language) + : undefined) + const next = new Map(drafts) + next.set(path, { + localId: deps.newDraftLocalId(), + script: makePipelineScript(language, path, content, new Date().toISOString()), + outputAssets: inferred.length > 0 ? inferred : seeded ? [seeded] : undefined + }) + deps.setDrafts(next) + deps.onShowDrafts?.() + deps.onProposeNode?.(path) + return { path } + }, + editNode: async (path, content) => { + deps.ensureEditable?.() + assertInFolder(path) + assertPipelineAnnotation(content) + const drafts = deps.getDrafts() + const existing = drafts.get(path) + // Base the edit on the existing draft's / deployed script object and replace + // ONLY the content — preserving hash, summary, description, tag, schema, and + // settings. Rebuilding a fresh script would wipe that metadata: deploying + // from the pane (auto_parent) would update the script while clearing it, and + // the route "Save all" path (no parent_hash) could hit the path-conflict + // branch on the occupied path. + let baseScript: Script + if (existing) { + baseScript = existing.script + } else { + const workspace = deps.getWorkspace() + if (!workspace) throw new Error('No workspace is selected.') + baseScript = await ScriptService.getScriptByPath({ workspace, path }) + } + const inferred = await inferOutputAssets(baseScript.language, content) + const next = new Map(drafts) + next.set(path, { + localId: existing?.localId ?? deps.newDraftLocalId(), + script: { ...baseScript, content }, + outputAssets: inferred.length > 0 ? inferred : existing?.outputAssets + }) + deps.setDrafts(next) + deps.onShowDrafts?.() + deps.onProposeNode?.(path) + }, + removeProposedNode: async (path) => { + if (!deps.getDrafts().has(path)) { + throw new Error(`No unsaved draft at '${path}' to discard.`) + } + const next = new Map(deps.getDrafts()) + next.delete(path) + deps.setDrafts(next) + deps.onForgetPath?.(path) + }, + testNode: async (path, args) => { + const workspace = deps.getWorkspace() + if (!workspace) return undefined + const draft = deps.getDrafts().get(path) + try { + let jobId: string + if (draft) { + // Un-deployed/edited body: preview-run the draft content so it can be + // tested before deploying. + jobId = await JobService.runScriptPreview({ + workspace, + requestBody: { + path, + content: draft.script.content, + language: draft.script.language, + args: args ?? {} + } + }) + } else { + // test_pipeline_node previews ONE node — never fan out to downstream + // deployed subscribers via the backend asset dispatcher (which would + // run side-effecting deployed scripts the user didn't ask for). + jobId = await JobService.runScriptByPath({ + workspace, + path, + requestBody: { ...(args ?? {}), _wmill_skip_asset_dispatch: true } + }) + } + deps.onRunStarted?.(jobId, path) + return jobId + } catch (e: any) { + sendUserToast(`Run failed: ${e?.body ?? e?.message ?? e}`, true) + return undefined + } + } + } + + return helpers +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts new file mode 100644 index 0000000000..bdecadff91 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.svelte.ts @@ -0,0 +1,184 @@ +import type { AssetKind, Script } from '$lib/gen' +import type { AssetWithAltAccessType } from '$lib/components/assets/lib' +import type { AssetGraphSelection } from './types' +import { + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations +} from './parsePipelineAnnotations' +import type { PipelineDraft } from './pipelineAiHelpers' + +// ============================================================================ +// Externalized pipeline-editor state — the data-pipeline analogue of the flow +// editor's `flowStore` / `flowStateStore`. It owns the in-flight draft Map, the +// live editor overlays, and the current selection: the substrate the route page +// editor and the in-session preview both render through (via the shared +// ). Persistence, graph resolution, run dispatch, and deploy +// stay with the consumer; this is a plain reactive bag so a consumer can +// read/mutate it without prop plumbing. +// ============================================================================ + +const EMPTY_ANNOTATIONS: PipelineAnnotations = parsePipelineAnnotations('') + +type LiveAnnotations = { scriptPath: string | undefined; annotations: PipelineAnnotations } +type LiveBodyAssets = { + scriptPath: string | undefined + assets: AssetWithAltAccessType[] + columnLineage?: ColumnLineage[] +} +type LiveContent = { scriptPath: string | undefined; content: string } + +export class PipelineEditorState { + /** In-flight drafts keyed by script path (manual + AI-staged). */ + drafts = $state>(new Map()) + /** Draft open in the details pane (mutually exclusive with `selection`). */ + activeDraftPath = $state(undefined) + /** The persisted node/asset selected on the canvas. */ + selection = $state(undefined) + + /** Live-parsed annotations of the open script (refreshed per keystroke). */ + liveAnnotations = $state({ + scriptPath: undefined, + annotations: EMPTY_ANNOTATIONS + }) + /** Live-inferred body read/write assets of the open script. */ + liveBodyAssets = $state({ scriptPath: undefined, assets: [] }) + /** The open draft's live editor buffer. */ + liveContent = $state({ scriptPath: undefined, content: '' }) + + /** Set true once a draft bundle was restored from the DB on load — drives the + * route toolbar's one-shot "Loaded from draft" hint. Written by the editor's + * autosave hydrate when persistence is enabled. */ + loadedFromDbDraft = $state(false) + + /** Folder this state is scoped to. Used by the in-session preview (where one + * instance is reused across editor hide/show) to detect a retarget to a + * different folder and reset, so stale drafts don't bleed across folders. */ + folder = $state(undefined) + + /** True once the DB draft bundle for the current folder has been hydrated + * into this instance. Gated per-instance (not per component mount) so the + * in-session preview hydrates ONCE when its runtime is fresh and then keeps + * the in-memory drafts across editor hide/show — re-reading the DB on every + * remount would race a not-yet-flushed autosave and drop a just-staged draft. + * Reset to false on a folder retarget so the new folder re-hydrates. */ + hydratedFromDb = $state(false) + + /** Clear all in-flight state. Used when the session preview retargets a + * different pipeline folder (a same-folder remount keeps the drafts). */ + reset = () => { + this.drafts = new Map() + this.activeDraftPath = undefined + this.selection = undefined + this.clearLiveOverlays() + this.loadedFromDbDraft = false + // Force a re-hydrate from the DB draft of the newly-targeted folder. + this.hydratedFromDb = false + } + + #nextDraftLocalId = 0 + // Arrow fields so `pe.method` can be passed straight as a callback (the + // details pane takes onDraftPersist / onAnnotationsChange / … by reference). + newDraftLocalId = (): string => { + this.#nextDraftLocalId += 1 + return `pe-${this.#nextDraftLocalId}` + } + + handleAnnotationsChange = (scriptPath: string | undefined, annotations: PipelineAnnotations) => { + this.liveAnnotations = { scriptPath, annotations } + } + handleAssetsChange = ( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) => { + this.liveBodyAssets = { scriptPath, assets, columnLineage } + } + handleContentChange = (scriptPath: string | undefined, content: string) => { + this.liveContent = { scriptPath, content } + } + + clearLiveOverlays = () => { + this.liveAnnotations = { scriptPath: undefined, annotations: EMPTY_ANNOTATIONS } + this.liveBodyAssets = { scriptPath: undefined, assets: [] } + this.liveContent = { scriptPath: undefined, content: '' } + } + + /** Drop per-path editor state when a path goes away. Does NOT touch + * consumer-owned per-path state (e.g. the route page's save errors — the + * route layers that on in its own wrapper). */ + forgetPath = (path: string) => { + if (this.activeDraftPath === path) this.activeDraftPath = undefined + if (this.selection?.kind === 'runnable' && this.selection.path === path) + this.selection = undefined + if (this.liveAnnotations.scriptPath === path) + this.liveAnnotations = { scriptPath: undefined, annotations: EMPTY_ANNOTATIONS } + if (this.liveBodyAssets.scriptPath === path) + this.liveBodyAssets = { scriptPath: undefined, assets: [] } + if (this.liveContent.scriptPath === path) + this.liveContent = { scriptPath: undefined, content: '' } + } + + discardDraft = (path: string) => { + if (!this.drafts.has(path)) return + const next = new Map(this.drafts) + next.delete(path) + this.drafts = next + this.forgetPath(path) + } + + /** Commit body edits + inferred outputs back into the drafts Map on pane + * teardown (deferred a microtask so a same-batch discard doesn't resurrect the + * entry). Verbatim port of the route page's `handleDraftPersist`. */ + handleDraftPersist = ( + p: string, + snapshot: { content: string; writes: { kind: AssetKind; path: string }[]; script?: Script } + ) => { + queueMicrotask(() => { + const d = this.drafts.get(p) + if (!d) { + if (!snapshot.script) return + const next = new Map(this.drafts) + next.set(p, { + localId: this.newDraftLocalId(), + script: snapshot.script, + outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined + }) + this.drafts = next + return + } + // `?? 0` is load-bearing: an undefined `outputAssets` (a no-output draft) + // vs an empty inferred `writes` both mean "no writes". Without the + // coalesce, `undefined === 0` is false, so this never short-circuits — + // every persist re-writes the drafts Map with an equivalent object, + // re-triggering the pane's emit → graph re-derive → persist, an infinite + // microtask loop (hangs the tab without an effect-depth throw). + const writesEqual = + (d.outputAssets?.length ?? 0) === snapshot.writes.length && + (d.outputAssets ?? []).every( + (a, i) => a.kind === snapshot.writes[i]?.kind && a.path === snapshot.writes[i]?.path + ) + if (d.script.content === snapshot.content && writesEqual) return + const next = new Map(this.drafts) + next.set(p, { + ...d, + script: { ...d.script, content: snapshot.content }, + outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined + }) + this.drafts = next + }) + } + + /** The draft open in the pane, if any. */ + get activeDraft(): PipelineDraft | undefined { + return this.activeDraftPath ? this.drafts.get(this.activeDraftPath) : undefined + } + + /** Whichever script is open — the active draft, or a selected persisted script. */ + get openScriptPath(): string | undefined { + if (this.activeDraftPath) return this.activeDraftPath + if (this.selection?.kind === 'runnable' && this.selection.runnable_kind === 'script') + return this.selection.path + return undefined + } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts new file mode 100644 index 0000000000..0fae3f836d --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineEditorState.test.ts @@ -0,0 +1,70 @@ +import { describe, it, expect } from 'vitest' +import { PipelineEditorState } from './pipelineEditorState.svelte' +import type { PipelineDraft } from './pipelineAiHelpers' +import type { AssetKind } from '$lib/gen' + +// handleDraftPersist defers its commit a microtask (so a same-batch discard can +// win); flush that microtask before asserting. +const flushMicrotasks = () => new Promise((resolve) => queueMicrotask(() => resolve())) + +function draft(content: string, outputAssets?: { kind: AssetKind; path: string }[]): PipelineDraft { + return { + localId: 'pe-1', + script: { path: 'f/x/n', language: 'duckdb', content } as PipelineDraft['script'], + outputAssets + } +} + +describe('PipelineEditorState.handleDraftPersist', () => { + // Regression: a no-output draft has `outputAssets: undefined`; the details pane + // infers an empty `writes: []`. Both mean "no writes", so persisting unchanged + // content+writes must be a no-op. The earlier `undefined === 0` length check made + // it false, so every persist re-wrote the drafts Map with an equivalent object — + // re-triggering the pane's emit → graph re-derive → persist, an infinite + // microtask loop that froze the tab. The Map reference must stay identical. + it('is idempotent for a no-output draft (undefined outputAssets vs empty inferred writes)', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 1', writes: [] }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + }) + + it('re-writes the drafts Map when the content actually changes', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { content: 'SELECT 2', writes: [] }) + await flushMicrotasks() + expect(pe.drafts).not.toBe(before) + expect(pe.drafts.get('f/x/n')?.script.content).toBe('SELECT 2') + }) + + it('re-writes the drafts Map when the inferred writes actually change', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([['f/x/n', draft('SELECT 1', undefined)]]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { + content: 'SELECT 1', + writes: [{ kind: 'resource' as AssetKind, path: 'f/x/out' }] + }) + await flushMicrotasks() + expect(pe.drafts).not.toBe(before) + expect(pe.drafts.get('f/x/n')?.outputAssets).toEqual([{ kind: 'resource', path: 'f/x/out' }]) + }) + + it('stays idempotent when outputAssets and inferred writes match (non-empty)', async () => { + const pe = new PipelineEditorState() + pe.drafts = new Map([ + ['f/x/n', draft('SELECT 1', [{ kind: 'resource' as AssetKind, path: 'f/x/out' }])] + ]) + const before = pe.drafts + pe.handleDraftPersist('f/x/n', { + content: 'SELECT 1', + writes: [{ kind: 'resource' as AssetKind, path: 'f/x/out' }] + }) + await flushMicrotasks() + expect(pe.drafts).toBe(before) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts index dd4c64b225..378c4e1469 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts @@ -66,11 +66,14 @@ describe('resolveGraph', () => { expect(resolveGraph(input({ base }))).toEqual(base) }) - it('draft: adds an unsaved runnable + write edge from the static outputAsset', () => { + it('draft: adds an unsaved runnable + write edge from outputAssets', () => { const drafts = new Map([ [ 'f/x/d', - { script: { content: '' }, outputAsset: { kind: 's3object' as const, path: '/out.json' } } + { + script: { content: '' }, + outputAssets: [{ kind: 's3object' as const, path: '/out.json' }] + } ] ]) const r = resolveGraph(input({ drafts })) @@ -93,22 +96,6 @@ describe('resolveGraph', () => { }) }) - it('draft: outputAssets snapshot wins over the static outputAsset', () => { - const drafts = new Map([ - [ - 'f/x/d', - { - script: { content: '' }, - outputAsset: { kind: 's3object' as const, path: '/old.json' }, - outputAssets: [{ kind: 's3object' as const, path: '/new.json' }] - } - ] - ]) - const r = resolveGraph(input({ drafts })) - expect(r.edges.map((e) => e.asset_path)).toContain('/new.json') - expect(r.edges.map((e) => e.asset_path)).not.toContain('/old.json') - }) - it('active draft: live body writes are authoritative over the snapshot', () => { const drafts = new Map([ [ diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts index b7c82822fa..e223379adb 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts @@ -15,7 +15,6 @@ import { /** Minimal structural shape of a pipeline draft `resolveGraph` needs. */ export type GraphDraft = { script: { content: string } - outputAsset?: { kind: AssetKind; path: string } outputAssets?: Array<{ kind: AssetKind; path: string }> } @@ -280,20 +279,15 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { unsaved: true } } - // Output asset(s): three-tier resolution. + // Output asset(s): two-tier resolution. // 1. Active draft (the body the user is editing right now): // live body inference is authoritative — renaming a // CREATE TABLE target or writeS3File path retires the // old output node and surfaces the new one as the user // types. - // 2. Inactive draft with a captured `outputAssets` snapshot - // (taken on the last pane transition): use those, so a - // draft the user already edited keeps its renamed outputs - // after they've clicked elsewhere. - // 3. Fallback to the static `outputAsset` seeded at draft - // creation — covers fresh drafts and parser misses (e.g. - // WIN-1943: wmill.writeS3File({s3, storage}) object form - // not yet detected by the TS parser). + // 2. Inactive draft: its captured `outputAssets` (inferred at + // creation/last edit, or the seeded output for a fresh draft + // whose body doesn't yet write anything inferable). const liveForThisDraft = liveBodyAssets.scriptPath === path const writeOuts: Array<{ kind: AssetKind; path: string }> = [] if (liveForThisDraft) { @@ -301,9 +295,6 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { } else if (d.outputAssets) { writeOuts.push(...d.outputAssets) } - if (writeOuts.length === 0 && d.outputAsset) { - writeOuts.push(d.outputAsset) - } // `// materialize ` declares a write output via annotation, not // the SQL body, so the body-inference tiers above miss it. Add it from // the live-parsed annotations so an edited materialize script keeps its diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index 9276f500af..274d218e82 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -49,6 +49,7 @@ export interface AssetGraphRunnableNode { // Synthesized by the page from a local draft; the script doesn't exist // in the DB yet. Drives a dashed/lower-opacity rendering to mirror how // unsaved triggers are styled — visually distinct from persisted nodes. + // AI-built nodes are plain drafts too (no separate pending/approval state). unsaved?: boolean } diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index fdba95ca7f..343965b709 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -88,6 +88,11 @@ import { type GlobalToolHelpers } from './global/core' import { isGlobalAiEnabled } from './global/gate' +import { + pipelineTools, + getPipelinePromptSection, + type PipelineAIChatHelpers +} from './pipeline/core' import { scopedKey, onUserChange, migrateLegacyLocalStorage } from '$lib/userScopedStorage' import { getLocalSetting, storeLocalSetting } from '$lib/utils' import { AttachedFilesStore } from './files/attachedFiles.svelte' @@ -254,6 +259,7 @@ export class AIChatManager { skipResponsesApi = false mode = $state(AIMode.NAVIGATOR) + pipelineAiChatHelpers = $state(undefined) readonly isOpen = $derived(chatState.size > 0) savedSize = $state(0) instructions = $state('') @@ -649,18 +655,11 @@ export class AIChatManager { ) switch (result) { case 'ok': - await this.historyManager.saveChat( - this.displayMessages, - this.messages, - this.contextUsage - ) + await this.historyManager.saveChat(this.displayMessages, this.messages, this.contextUsage) sendUserToast('Conversation compacted.') break case 'empty': - sendUserToast( - 'Compaction produced an empty summary — conversation left unchanged.', - true - ) + sendUserToast('Compaction produced an empty summary — conversation left unchanged.', true) break case 'error': sendUserToast('Failed to compact the conversation.', true) @@ -958,28 +957,7 @@ export class AIChatManager { this.tools = [searchDocsTool, readDocsPageTool, ...this.apiTools] this.helpers = {} } else if (mode === AIMode.GLOBAL) { - this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(mode), { - previewTools: this.isSessionChat, - skills: this.globalSkills - }) - this.tools = globalToolsFor({ sessionPreview: this.isSessionChat }) - this.helpers = { - ...(this.isSessionChat ? { sessionId: this.sessionId } : {}), - testActiveFlow: async (args?: Record) => - this.flowAiChatHelpers?.testFlow(args), - attachedFiles: this.attachedFiles, - getUserInstructions: () => getUserCustomPrompts()[AIMode.GLOBAL] ?? '', - setUserInstructions: (instructions: string) => { - const prompts = getUserCustomPrompts() - if (instructions.trim()) { - prompts[AIMode.GLOBAL] = instructions - } else { - delete prompts[AIMode.GLOBAL] - } - setUserCustomPrompts(prompts) - this.rebuildGlobalSystemMessage() - } - } satisfies GlobalToolHelpers + this.configureGlobalMode() void this.refreshGlobalSkills() } else if (mode === AIMode.APP) { const customPrompt = getCombinedCustomPrompt(mode) @@ -992,6 +970,43 @@ export class AIChatManager { // Fetch the workspace's AI skills and, if GLOBAL mode is still active, rebuild // the system message so the next chat-loop iteration advertises them. Ignore // stale resolves so workspace changes cannot overwrite newer skills. + // Build the global-mode system message, tools, and helpers, layering on the + // pipeline surface when a /pipeline editor has registered helpers. Centralized + // so changeMode, refreshGlobalSkills, and setPipelineHelpers stay consistent — + // each rebuild would otherwise drop the pipeline augmentation the others added. + private configureGlobalMode = () => { + const systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + previewTools: this.isSessionChat, + skills: this.globalSkills + }) + const baseHelpers: GlobalToolHelpers = { + ...(this.isSessionChat ? { sessionId: this.sessionId } : {}), + testActiveFlow: async (args?: Record) => this.flowAiChatHelpers?.testFlow(args), + attachedFiles: this.attachedFiles, + getUserInstructions: () => getUserCustomPrompts()[AIMode.GLOBAL] ?? '', + setUserInstructions: (instructions: string) => { + const prompts = getUserCustomPrompts() + if (instructions.trim()) { + prompts[AIMode.GLOBAL] = instructions + } else { + delete prompts[AIMode.GLOBAL] + } + setUserCustomPrompts(prompts) + this.rebuildGlobalSystemMessage() + } + } + const pipeline = this.pipelineAiChatHelpers + if (pipeline) { + systemMessage.content += getPipelinePromptSection(pipeline.getPipelineContext()) + this.tools = [...globalToolsFor({ sessionPreview: this.isSessionChat }), ...pipelineTools] + this.helpers = { ...baseHelpers, pipeline } + } else { + this.tools = globalToolsFor({ sessionPreview: this.isSessionChat }) + this.helpers = baseHelpers + } + this.systemMessage = systemMessage + } + refreshGlobalSkills = async (workspace = get(workspaceStore) ?? '') => { const refreshId = ++this.globalSkillsRefreshId const skills = await loadWorkspaceSkills(workspace) @@ -1000,10 +1015,7 @@ export class AIChatManager { } this.globalSkills = skills if (this.mode === AIMode.GLOBAL) { - this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { - previewTools: this.isSessionChat, - skills - }) + this.configureGlobalMode() } } @@ -1014,10 +1026,18 @@ export class AIChatManager { if (this.mode !== AIMode.GLOBAL) { return } - this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + const systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { previewTools: this.isSessionChat, skills: this.globalSkills }) + // Preserve the active pipeline-editor augmentation that configureGlobalMode + // adds — otherwise update_user_instructions (which calls this) would drop the + // /pipeline/ context + direct-draft/materialize guidance mid-session. + const pipeline = this.pipelineAiChatHelpers + if (pipeline) { + systemMessage.content += getPipelinePromptSection(pipeline.getPipelineContext()) + } + this.systemMessage = systemMessage } private expandGlobalSkillCommand = (instructions: string): string => { @@ -2132,7 +2152,7 @@ export class AIChatManager { moduleState && !moduleState.previewSuccess ? getStringError(moduleState.previewResult) : undefined, - getCode: () => module.value.type === 'rawscript' ? module.value.content : '', + getCode: () => (module.value.type === 'rawscript' ? module.value.content : ''), lang: module.value.language, path: module.id, ...editorRelated @@ -2176,6 +2196,28 @@ export class AIChatManager { } } + // Registered by the /pipeline editor while it is mounted. Rebuilds the global + // tool set so the pipeline tools appear (and disappear on unregister). Pipeline + // AI edits apply directly as drafts, so there is nothing to auto-accept. + // Returns a cleanup that tears the registration back down. + setPipelineHelpers = (pipelineHelpers: PipelineAIChatHelpers) => { + this.pipelineAiChatHelpers = pipelineHelpers + untrack(() => { + if (this.mode === AIMode.GLOBAL) { + this.configureGlobalMode() + } + }) + + return () => { + this.pipelineAiChatHelpers = undefined + untrack(() => { + if (this.mode === AIMode.GLOBAL) { + this.configureGlobalMode() + } + }) + } + } + /** * Refresh cached datatables from the app helpers (async) * Creates one context element per table (not per datatable) diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index b85e271632..90dcb655f4 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -56,6 +56,7 @@ import { createInlineScriptSession } from '../flow/inlineScriptsUtils' import { getDatatableSdkReference, getFlowPrompt, + getPipelinePrompt, getRawAppPrompt, getResourcePrompt, getScriptPrompt @@ -138,7 +139,9 @@ const INSTRUCTION_SUBJECTS = [ ] as const satisfies readonly WorkspaceItemType[] // `datatable` is not a workspace item type, but the model can request the // datatable SDK reference (the wmill.datatable() runnable API) the same way. -const INSTRUCTION_SUBJECTS_EXTRA = ['datatable'] as const +// `pipeline` likewise isn't an item type — a data pipeline is a set of +// annotated scripts in a folder, so it gets authoring guidance, not a CRUD type. +const INSTRUCTION_SUBJECTS_EXTRA = ['datatable', 'pipeline'] as const const ALL_INSTRUCTION_SUBJECTS = [...INSTRUCTION_SUBJECTS, ...INSTRUCTION_SUBJECTS_EXTRA] as const const MAX_LIST_LIMIT = 100 type ActiveGlobalEditorType = Extract @@ -171,7 +174,7 @@ const scriptLangSchema = z.enum($ScriptLang.enum) const getInstructionsSchema = z.object({ subject: instructionSubjectSchema.describe( - 'What to get authoring instructions for: a workspace item type (script, flow, resource, app) or "datatable" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don\'t need instructions — their tool schemas describe everything.' + 'What to get authoring instructions for: a workspace item type (script, flow, resource, app), "pipeline" for building a data pipeline (a DAG of annotated scripts wired by storage assets — NOT a flow), or "datatable" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don\'t need instructions — their tool schemas describe everything.' ), language: scriptLangSchema .optional() @@ -206,7 +209,9 @@ const updateUserInstructionsSchema = z.object({ .string() .min(1) .optional() - .describe("Required when operation is 'append': the instruction to add. Ignored for 'replace'."), + .describe( + "Required when operation is 'append': the instruction to add. Ignored for 'replace'." + ), old_string: z .string() .min(1) @@ -678,11 +683,13 @@ const deleteAppRunnableSchema = z.object({ const openPreviewSchema = z.object({ kind: z - .enum(['script', 'flow', 'raw_app']) + .enum(['script', 'flow', 'raw_app', 'pipeline']) .describe( - 'Item kind to preview. Use "raw_app" for code-based apps (created via init_app). The legacy drag-and-drop app builder ("app") is not previewable in the session panel — don\'t pass it.' + 'Item kind to preview. Use "raw_app" for code-based apps (created via init_app). Use "pipeline" to show the data-pipeline graph for a folder — here `path` is the folder name, not an item path. The legacy drag-and-drop app builder ("app") is not previewable in the session panel — don\'t pass it.' ), - path: z.string().describe('Workspace path of the item to preview.') + path: z + .string() + .describe('Workspace path of the item to preview, or the folder name when kind is "pipeline".') }) const getPreviewStatusSchema = z.object({}) @@ -825,6 +832,7 @@ Rules: - Variable values are never readable. For secrets, create a secret variable and reference it from resources as "$var:path/to/variable". - Use search_resource_types before write_resource. - Use get_instructions before writing scripts, flows, resources, or apps. For scripts, pass the target language. +- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on \` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow. - After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer drafts, so testing does not require deployment. - Use list_runs to find recent runs (optionally filtered by path, creator, label, or status), then get_job_logs with a returned id to inspect a specific run's logs — without starting a new test run. - When a required decision is ambiguous, use askUserQuestion with two to ten clear proposed answer strings instead of guessing. The user can also type a custom answer when none of the proposed answers fit. @@ -833,6 +841,7 @@ Rules: previewTools ? ` - After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited. +- Building a data pipeline: call open_preview(kind="pipeline", path="") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty or not-yet-created folder is fine (create_folder first if needed, then open it). Opening it registers build_pipeline_node / edit_pipeline_node — use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor. - When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app"). - get_app_runtime_logs only shows the app's browser console. For the server-side logs of a backend runnable the app invoked (a backend. call), call list_app_runs to get that run's job_id from the live preview, then get_job_logs with it. Use this when a backend call errors or returns something unexpected.` : '' @@ -1608,6 +1617,10 @@ Datatables are workspace-scoped managed PostgreSQL databases. In chat, explore a ${getDatatableSdkReference(lang)}` } +function getPipelineInstructions(): string { + return getPipelinePrompt() +} + function getInstructions(subject: InstructionSubject, language?: ScriptLang): string { switch (subject) { case 'script': @@ -1620,6 +1633,8 @@ function getInstructions(subject: InstructionSubject, language?: ScriptLang): st return getAppInstructions() case 'datatable': return getDatatableInstructions(language) + case 'pipeline': + return getPipelineInstructions() } } @@ -1672,7 +1687,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( getInstructionsSchema, 'get_instructions', - 'Get authoring guidance for scripts, flows, resources, apps, or the datatable SQL SDK (wmill.datatable()) used inside runnables.' + 'Get authoring guidance for scripts, flows, data pipelines, resources, apps, or the datatable SQL SDK (wmill.datatable()) used inside runnables.' ), fn: async ({ args, toolId, toolCallbacks }) => { const parsed = getInstructionsSchema.parse(args) @@ -2490,7 +2505,7 @@ function activeFlowTestFromCtx( export type OpenPreviewHandler = (req: { sessionId: string | undefined - kind: 'script' | 'flow' | 'raw_app' + kind: 'script' | 'flow' | 'raw_app' | 'pipeline' path: string }) => string @@ -2501,7 +2516,7 @@ export function setOpenPreviewHandler(handler: OpenPreviewHandler | undefined): } function openSessionPreview( - args: { kind: 'script' | 'flow' | 'raw_app'; path: string }, + args: { kind: 'script' | 'flow' | 'raw_app' | 'pipeline'; path: string }, sessionId: string | undefined ) { if (!openPreviewHandler) { diff --git a/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts b/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts new file mode 100644 index 0000000000..4f3e31986d --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/pipeline/core.test.ts @@ -0,0 +1,171 @@ +import { describe, it, expect, vi } from 'vitest' + +// `../shared` transitively pulls in the monaco editor (and its CSS), which the +// node test environment can't load — mirror the sibling chat tests' stub. +vi.mock('monaco-editor', () => ({ editor: {} })) + +import { + pipelineTools, + getPipelinePromptSection, + type PipelineAIChatHelpers, + type PipelineContext +} from './core' +import type { ToolCallbacks } from '../shared' + +function toolByName(name: string) { + const tool = pipelineTools.find((t) => t.def.function.name === name) + if (!tool) throw new Error(`tool ${name} not found`) + return tool +} + +function noopCallbacks(): ToolCallbacks { + return { setToolStatus: () => {}, removeToolStatus: () => {} } +} + +const sampleContext: PipelineContext = { + folder: 'analytics', + mode: 'edit', + nodes: [ + { + path: 'f/analytics/orders', + language: 'bun', + unsaved: true, + writes: ['ducklake://main/orders'], + reads: [], + triggers: ['schedule'] + } + ], + assets: ['ducklake://main/orders'] +} + +function makeHelpers(overrides: Partial = {}): { + helpers: { pipeline: PipelineAIChatHelpers } + calls: Record +} { + const calls: Record = {} + const record = + (name: string, ret?: any) => + (...args: any[]) => { + ;(calls[name] ??= []).push(args) + return ret + } + const pipeline: PipelineAIChatHelpers = { + getPipelineContext: () => sampleContext, + getNodeBody: async (path: string) => { + calls.getNodeBody = [...(calls.getNodeBody ?? []), [path]] + return { language: 'bun', content: 'export async function main() { return 1 }' } + }, + proposeNode: async (input) => { + calls.proposeNode = [...(calls.proposeNode ?? []), [input]] + return { path: input.path } + }, + editNode: async (path, content) => { + calls.editNode = [...(calls.editNode ?? []), [path, content]] + }, + removeProposedNode: record('removeProposedNode'), + testNode: async () => 'job-123', + ...overrides + } + return { helpers: { pipeline }, calls } +} + +describe('pipeline tools', () => { + it('exposes the expected tool surface', () => { + expect(pipelineTools.map((t) => t.def.function.name).sort()).toEqual([ + 'build_pipeline_node', + 'edit_pipeline_node', + 'get_pipeline_graph', + 'read_pipeline_node', + 'remove_pipeline_node', + 'test_pipeline_node' + ]) + }) + + it('get_pipeline_graph returns the live context as JSON', async () => { + const { helpers } = makeHelpers() + const out = await toolByName('get_pipeline_graph').fn({ + args: {}, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(JSON.parse(out)).toMatchObject({ folder: 'analytics' }) + }) + + it('build_pipeline_node forwards to proposeNode and does not deploy', async () => { + const { helpers, calls } = makeHelpers() + const out = await toolByName('build_pipeline_node').fn({ + args: { + path: 'f/analytics/clean', + language: 'bun', + content: '// pipeline\nexport async function main() {}', + output_kind: 'ducklake' + }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(calls.proposeNode?.[0]?.[0]).toMatchObject({ + path: 'f/analytics/clean', + language: 'bun', + outputKind: 'ducklake' + }) + expect(out).toContain('not deployed') + }) + + it('edit_pipeline_node reads then applies an exact find/replace', async () => { + const { helpers, calls } = makeHelpers({ + getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\nconst y = 2\n' }) + }) + await toolByName('edit_pipeline_node').fn({ + args: { path: 'f/analytics/orders', old_string: 'const x = 1', new_string: 'const x = 42' }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + expect(calls.editNode?.[0]?.[1]).toContain('const x = 42') + }) + + it('edit_pipeline_node surfaces a clear error when old_string is absent', async () => { + const { helpers } = makeHelpers({ + getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\n' }) + }) + await expect( + toolByName('edit_pipeline_node').fn({ + args: { path: 'f/analytics/orders', old_string: 'NOT THERE', new_string: 'x' }, + workspace: 'w', + helpers, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + ).rejects.toThrow(/was not found/) + }) + + it('mutation tools fail clearly when no pipeline editor is registered', async () => { + await expect( + toolByName('build_pipeline_node').fn({ + args: { path: 'f/a/b', language: 'bun', content: 'x' }, + workspace: 'w', + helpers: {}, + toolCallbacks: noopCallbacks(), + toolId: 't' + }) + ).rejects.toThrow(/No pipeline editor is open/) + }) + + it('test_pipeline_node requires confirmation', () => { + expect(toolByName('test_pipeline_node').requiresConfirmation).toBe(true) + }) +}) + +describe('getPipelinePromptSection', () => { + it('names the active folder and the direct-draft workflow', () => { + const section = getPipelinePromptSection(sampleContext) + expect(section).toContain('/pipeline/analytics') + expect(section).toContain('build_pipeline_node') + expect(section).toContain('directly as unsaved drafts') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/pipeline/core.ts b/frontend/src/lib/components/copilot/chat/pipeline/core.ts new file mode 100644 index 0000000000..f99bd0f210 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/pipeline/core.ts @@ -0,0 +1,328 @@ +import { z } from 'zod' +import { $ScriptLang } from '$lib/gen/schemas.gen' +import type { ScriptLang } from '$lib/gen' +import { createToolDef, executeTestRun, findAndReplace, type Tool } from '../shared' +import type { PipelineOutputKind } from '$lib/components/assets/AssetGraph/pipelineTemplates' + +// ============================================================================ +// Pipeline AI chat tools. +// +// These tools extend the GLOBAL chat mode when the user is on a /pipeline/ +// editor (the page registers `PipelineAIChatHelpers` on the AIChatManager). They +// let the model read the live pipeline graph and BUILD/EDIT pipeline nodes +// (scripts annotated with `// pipeline`). Mutations don't deploy: they apply +// directly as an unsaved DRAFT on the canvas — the same way the flow/script +// editor applies AI edits — which the user then deploys. There is no separate +// approve/reject step: the draft IS the change. +// +// The pipeline tools are added on top of the full global tool set, so docs +// search, datatable SQL, and workspace-item tools are already available +// alongside them — this file only carries the pipeline-graph-specific surface. +// ============================================================================ + +/** Compact, model-facing summary of one node in the pipeline graph. */ +export type PipelineNodeSummary = { + path: string + language?: ScriptLang + /** Has an unsaved local edit (draft) not yet deployed. */ + unsaved: boolean + summary?: string + /** Asset URIs this node writes (its outputs). */ + writes: string[] + /** Asset URIs this node reads (its inputs). */ + reads: string[] + /** Declared `// on ` execution-DAG bindings (asset URIs or native kinds). */ + triggers: string[] +} + +/** Compact, model-facing snapshot of the whole pipeline graph. */ +export type PipelineContext = { + folder: string + mode: 'view' | 'edit' + nodes: PipelineNodeSummary[] + /** All storage assets referenced by the graph, as URIs. */ + assets: string[] +} + +/** + * Bridge the pipeline page registers on the AIChatManager. Reads expose the live + * graph; writes apply directly as unsaved drafts (never deploy). Kept intentionally + * small — the page owns the draft Map and canvas rendering. + */ +export interface PipelineAIChatHelpers { + getPipelineContext: () => PipelineContext + /** Read a node's source (the in-flight draft body if one exists, else deployed). */ + getNodeBody: (path: string) => Promise<{ language: ScriptLang; content: string } | undefined> + /** Create a brand-new pipeline node as an unsaved draft on the canvas. */ + proposeNode: (input: { + path: string + language: ScriptLang + content: string + outputKind?: PipelineOutputKind + }) => Promise<{ path: string }> + /** Replace an existing node's body, applied as an unsaved draft. */ + editNode: (path: string, content: string) => Promise + /** Discard the unsaved draft at a path (undo a build_pipeline_node). */ + removeProposedNode: (path: string) => Promise + /** Preview-run a node (draft body preferred). Returns the started job id. */ + testNode: (path: string, args?: Record) => Promise +} + +/** Helper bag the pipeline tools receive from the manager in global mode. */ +export type PipelineToolHelpers = { pipeline?: PipelineAIChatHelpers } + +function requirePipeline(helpers: PipelineToolHelpers): PipelineAIChatHelpers { + if (!helpers?.pipeline) { + throw new Error( + 'No pipeline editor is open. Pipeline tools only work on a /pipeline/ page in edit mode.' + ) + } + return helpers.pipeline +} + +const scriptLangSchema = z.enum($ScriptLang.enum) + +const outputKindSchema = z + .enum(['none', 'datatable', 'ducklake', 'materialize', 's3_parquet', 's3_object']) + .describe( + 'Kind of output asset this node materializes, used to seed the output edge on the canvas before the body is parsed: "materialize"/"ducklake" → a DuckLake table, "datatable" → a Postgres data table, "s3_parquet"/"s3_object" → an S3 file, "none" → side-effect only. Defaults to none.' + ) + +// ---------------------------------------------------------------------------- +// Read tools +// ---------------------------------------------------------------------------- + +const getPipelineGraphSchema = z.object({}) + +const getPipelineGraphToolDef = createToolDef( + getPipelineGraphSchema, + 'get_pipeline_graph', + "Read the live pipeline graph for the open /pipeline/ editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG." +) + +const readPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the pipeline node (script) to read.') +}) + +const readPipelineNodeToolDef = createToolDef( + readPipelineNodeSchema, + 'read_pipeline_node', + 'Read the full source of one pipeline node (its in-flight draft body if it has unsaved edits, otherwise the deployed body). Use before edit_pipeline_node so edits target the exact current text.' +) + +// ---------------------------------------------------------------------------- +// Mutation tools (apply directly as unsaved drafts; never deploy) +// ---------------------------------------------------------------------------- + +const buildPipelineNodeSchema = z.object({ + path: z + .string() + .describe( + "Workspace path for the new node, e.g. f//. Use the open pipeline's folder. Must not collide with an existing node." + ), + language: scriptLangSchema.describe( + 'Script language. SQL-shaped data work uses duckdb (DuckLake/S3) or postgresql (data tables); bun/python3 for general transforms.' + ), + content: z + .string() + .describe( + "Full script source. Start it with the `pipeline` annotation as a top-of-file comment in the LANGUAGE'S comment syntax — `-- pipeline` for SQL (duckdb/postgresql), `# pipeline` for python3/bash, `// pipeline` for bun/TS — to mark it a pipeline member; declare inputs the same way (e.g. `-- on `), and write outputs via the wmill SDK / SQL so the lineage edges are inferred. A `// pipeline` line in a SQL node is a syntax error. Read existing node bodies first to match conventions." + ), + output_kind: outputKindSchema.optional() +}) + +const buildPipelineNodeToolDef = createToolDef( + buildPipelineNodeSchema, + 'build_pipeline_node', + 'Build a NEW pipeline node. It is applied directly as an unsaved draft on the canvas (a dashed node wired by its parsed asset reads/writes) — it does NOT deploy; the user deploys it. Prefer this over editing for new scripts.', + { strict: false } +) + +const editPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node to edit.'), + old_string: z.string().min(1).describe("Exact text to find in the node's current source."), + new_string: z.string().describe('Replacement text.'), + replace_all: z + .boolean() + .optional() + .default(false) + .describe( + 'When true, replace every exact match. When false, old_string must match exactly once.' + ) +}) + +const editPipelineNodeToolDef = createToolDef( + editPipelineNodeSchema, + 'edit_pipeline_node', + 'Edit an existing pipeline node by exact find/replace. The result is applied directly as an unsaved draft (does NOT deploy). Call read_pipeline_node first to get the exact current text.', + { strict: false } +) + +const removePipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node whose unsaved draft should be discarded.') +}) + +const removePipelineNodeToolDef = createToolDef( + removePipelineNodeSchema, + 'remove_pipeline_node', + 'Discard the unsaved draft at a path (undo a build_pipeline_node). Only affects the in-flight draft — to delete a deployed node, ask the user to do it on the canvas.' +) + +const testPipelineNodeSchema = z.object({ + path: z.string().describe('Workspace path of the node to preview-run.'), + args: z + .record(z.string(), z.any()) + .nullable() + .optional() + .describe('Arguments to pass to the script. Omit or pass null when none are needed.') +}) + +const testPipelineNodeToolDef = createToolDef( + testPipelineNodeSchema, + 'test_pipeline_node', + 'Preview-run one pipeline node (using its draft body when unsaved) and return the result/logs, without deploying. Requires user confirmation before it runs.', + { strict: false } +) + +// ---------------------------------------------------------------------------- +// Tool set +// ---------------------------------------------------------------------------- + +export const pipelineTools: Tool[] = [ + { + def: getPipelineGraphToolDef, + fn: async ({ helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + toolCallbacks.setToolStatus(toolId, { content: 'Reading pipeline graph...' }) + const ctx = pipeline.getPipelineContext() + toolCallbacks.setToolStatus(toolId, { + content: `Read pipeline graph (${ctx.nodes.length} node${ctx.nodes.length === 1 ? '' : 's'})`, + result: 'Success' + }) + return JSON.stringify(ctx, null, 2) + } + }, + { + def: readPipelineNodeToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path } = readPipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Reading node '${path}'...` }) + const node = await pipeline.getNodeBody(path) + if (!node) { + return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.` + } + toolCallbacks.setToolStatus(toolId, { content: `Read node '${path}'`, result: 'Success' }) + return JSON.stringify({ path, language: node.language, content: node.content }) + } + }, + { + def: buildPipelineNodeToolDef, + streamArguments: true, + showDetails: true, + showFade: true, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, language, content, output_kind } = buildPipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Building node '${path}'...` }) + await pipeline.proposeNode({ + path, + language: language as ScriptLang, + content, + outputKind: output_kind as PipelineOutputKind | undefined + }) + toolCallbacks.setToolStatus(toolId, { + content: `Added draft node '${path}'`, + result: 'Success' + }) + return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.` + } + }, + { + def: editPipelineNodeToolDef, + streamArguments: true, + showDetails: true, + showFade: true, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, old_string, new_string, replace_all } = editPipelineNodeSchema.parse(args) + const node = await pipeline.getNodeBody(path) + if (!node) { + return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.` + } + toolCallbacks.setToolStatus(toolId, { content: `Editing node '${path}'...` }) + const updated = findAndReplace( + node.content, + old_string, + new_string, + replace_all ?? false, + 'node source' + ) + await pipeline.editNode(path, updated) + toolCallbacks.setToolStatus(toolId, { + content: `Edited draft '${path}'`, + result: 'Success' + }) + return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).` + } + }, + { + def: removePipelineNodeToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path } = removePipelineNodeSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Discarding draft '${path}'...` }) + await pipeline.removeProposedNode(path) + toolCallbacks.setToolStatus(toolId, { + content: `Discarded draft '${path}'`, + result: 'Success' + }) + return `Discarded the unsaved draft at '${path}'.` + } + }, + { + def: testPipelineNodeToolDef, + requiresConfirmation: true, + confirmationMessage: 'Run pipeline node', + showDetails: true, + autoCollapseDetails: false, + fn: async ({ args, workspace, helpers, toolId, toolCallbacks }) => { + const pipeline = requirePipeline(helpers) + const { path, args: runArgs } = testPipelineNodeSchema.parse(args) + return executeTestRun({ + jobStarter: async () => { + const jobId = await pipeline.testNode(path, runArgs ?? undefined) + if (!jobId) { + throw new Error(`Could not start a run for node '${path}'.`) + } + return jobId + }, + workspace, + toolCallbacks, + toolId, + startMessage: `Starting run of '${path}'...`, + contextName: 'script' + }) + } + } +] + +/** + * Pipeline-specific guidance appended to the global system prompt when a + * /pipeline editor is open. Describes the annotation model and the direct-draft + * workflow so the model uses the pipeline tools rather than the generic + * write_script draft tools. + */ +export function getPipelinePromptSection(ctx: PipelineContext): string { + return ` + +Data Pipeline editor (ACTIVE): +- The user has the /pipeline/${ctx.folder} editor open. A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers. +- Annotations are top-of-file comments in the NODE'S OWN comment syntax: \`--\` for SQL (duckdb/postgresql), \`#\` for python3/bash, \`//\` for bun/TS. The \`//\` shown below is the TS form — translate it (a \`// pipeline\` line in a SQL node is a syntax error that won't deploy). +- A script becomes a pipeline node when its source starts with the \`// pipeline\` annotation. Declare execution-DAG inputs with \`// on \` (e.g. \`// on ducklake://main/orders\`). Outputs are inferred from what the body writes (wmill SDK calls / SQL CREATE TABLE / writeS3File); declare a managed output with \`// materialize \`. Optional badges: \`// partitioned \`, \`// freshness \`, \`// tag \`, \`// retry [delay]\`, \`// data_test ...\`. +- \`materialize\` (the managed output): \`// materialize \` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. IMPORTANT: \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake:///\`) — deploy rejects it on any other language or target. For a \`python3\`/\`bun\`/\`postgresql\` node, do NOT use \`// materialize\`; write the output via the SDK instead (e.g. \`wmill.writeS3File(...)\`, a \`CREATE TABLE\` in postgresql, or \`wmill.databaseUrlFromResource\`/ducklake helpers) and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize append\` INSERT-appends rows (incremental); \`// materialize key=\` merges/upserts on \`\`. \`// materialize manual \` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned \` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token inside any asset URI is substituted with the current partition value at run time. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run". +- Use get_pipeline_graph to see the current nodes/assets/triggers, and read_pipeline_node before editing one. +- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open. +- Reuse existing asset paths from the graph when wiring a downstream node to an upstream one (read the upstream's write asset, then \`// on\` that same URI). +- Only deploy when the user explicitly asks; the user deploys drafts from the canvas.` +} diff --git a/frontend/src/lib/components/home/ItemsList.svelte b/frontend/src/lib/components/home/ItemsList.svelte index b6ae577d4b..086760ac6e 100644 --- a/frontend/src/lib/components/home/ItemsList.svelte +++ b/frontend/src/lib/components/home/ItemsList.svelte @@ -13,6 +13,7 @@ type ListableRawApp } from '$lib/gen' import { resource } from 'runed' + import { getDraftItems } from '$lib/workspaceDrafts.svelte' import { userStore, workspaceStore } from '$lib/stores' import type uFuzzy from '@leeoniya/ufuzzy' import { @@ -73,23 +74,44 @@ type TableApp = TableItem type TableRawApp = TableItem - // Folders with ≥1 pipeline script (auto_kind='pipeline'). Used by - // TreeView to surface a "Pipeline" entry inside those folders. Cheap - // thanks to the partial index on script.auto_kind. + // Folders that are data pipelines, surfaced as their own "Pipeline" entry + // (the member scripts are folded into it, not listed individually). Two + // sources: deployed pipelines (folders with ≥1 `auto_kind='pipeline'` script, + // cheap via the partial index) AND bundle-phase pipelines that only exist as a + // `data_pipeline` draft so far — so a pipeline shows up the moment its first + // node is drafted, before anything is deployed. let pipelineFoldersRes = resource( () => $workspaceStore, async (ws) => { if (!ws) return new Set() + const folders = new Set() try { - const rows = await AssetService.listPipelineFolders({ workspace: ws }) - return new Set(rows.map((r) => r.folder)) + for (const r of await AssetService.listPipelineFolders({ workspace: ws })) + folders.add(r.folder) } catch { - // Decorative tree entry — degrade to "no pipelines" on failure. - return new Set() + // Decorative entry — degrade gracefully on failure. } + try { + for (const d of await getDraftItems(ws)) { + if (d.kind !== 'data_pipeline') continue + const m = d.path.match(/^f\/([^/]+)\/data_pipeline$/) + if (m) folders.add(m[1]) + } + } catch { + // Drafts unavailable — show deployed pipelines only. + } + return folders } ) - let pipelineFolders = $derived(pipelineFoldersRes.current ?? new Set()) + // Folders of pipeline-member scripts present in the current listing (captured + // in loadScripts before they're filtered out). Unioned in so a folder whose + // only pipeline node is a never-deployed `// pipeline` script draft — not in + // listPipelineFolders (deployed-only) nor a `data_pipeline` bundle — still gets + // a pipeline entry instead of vanishing. + let pipelineMemberFolders = $state(new Set()) + let pipelineFolders = $derived( + new Set([...(pipelineFoldersRes.current ?? []), ...pipelineMemberFolders]) + ) let scripts: TableScript[] | undefined = $state() let flows: TableFlow[] | undefined = $state() @@ -115,12 +137,26 @@ withoutDescription: true }) - scripts = loadedScripts.map((script: Script) => { - return { - canWrite: canWrite(script.path, script.extra_perms, $userStore) && !$userStore?.operator, - ...script - } - }) + // Pipeline-member scripts (`auto_kind='pipeline'`) are represented by their + // pipeline's entry, not listed individually — but capture their folders so + // the pipeline entry still surfaces (incl. a members-only / draft-only folder). + const memberFolders = new Set() + scripts = loadedScripts + .filter((script: Script) => { + if (script.auto_kind === 'pipeline') { + const m = script.path.match(/^f\/([^/]+)\//) + if (m) memberFolders.add(m[1]) + return false + } + return true + }) + .map((script: Script) => { + return { + canWrite: canWrite(script.path, script.extra_perms, $userStore) && !$userStore?.operator, + ...script + } + }) + pipelineMemberFolders = memberFolders loading = false } @@ -246,6 +282,25 @@ : undefined ) let filterUserFolders = $state(getLocalSetting(FILTER_USER_FOLDER_SETTING_NAME) == 'true') + + // Pipeline entries are rendered independently of the item list, so apply the + // same gates the items get — otherwise a pipeline would still show under the + // Flows/Apps tabs, in the archived view, under a label filter, or outside a + // selected owner. Pipelines are script-based units always at `f/`, so + // kind=script and the user-folder toggle always include them; kind=flow/app, + // archived, a label filter (pipelines carry no labels), and a non-matching + // owner exclude them. + let visiblePipelineFolders = $derived.by(() => { + if (archived) return new Set() + if (itemKind !== 'all' && itemKind !== 'script') return new Set() + if (labelFilter != undefined) return new Set() + if (ownerFilter == undefined) return pipelineFolders + return new Set( + [...pipelineFolders].filter( + (f) => `f/${f}` === ownerFilter || `f/${f}`.startsWith(ownerFilter + '/') + ) + ) + }) let includeWithoutMain = $state( getLocalSetting(INCLUDE_WITHOUT_MAIN_SETTING_NAME) ? getLocalSetting(INCLUDE_WITHOUT_MAIN_SETTING_NAME) == 'true' @@ -826,14 +881,17 @@ {#each new Array(6) as _} {/each} - {:else if filteredItems.length === 0} + {:else if filteredItems.length === 0 && (filter !== '' || visiblePipelineFolders.size === 0)} + {:else if treeView} loadScripts(includeWithoutMain)} on:flowChanged={loadFlows} @@ -850,7 +908,7 @@ {:else}
{#if filter === ''} - {#each [...pipelineFolders].sort() as folder (folder)} + {#each [...visiblePipelineFolders].sort() as folder (folder)} i && 'folderName' in i + // Hidden while searching: pipelines aren't part of the text filter (the list + // view hides their rows on a query too), so a folder matching the search + // shouldn't surface an unrelated Pipeline row. let hasPipeline = $derived( - depth === 0 && isFolderItem(item) && (pipelineFolders?.has(item.folderName) ?? false) + depth === 0 && + !isSearching && + isFolderItem(item) && + (pipelineFolders?.has(item.folderName) ?? false) ) const isFolder = isFolderItem diff --git a/frontend/src/lib/components/home/TreeViewRoot.svelte b/frontend/src/lib/components/home/TreeViewRoot.svelte index 59adf1aaa9..ef5cf25d58 100644 --- a/frontend/src/lib/components/home/TreeViewRoot.svelte +++ b/frontend/src/lib/components/home/TreeViewRoot.svelte @@ -24,7 +24,42 @@ let groupedItems: ReturnType | 'loading' = $state('loading') $effect(() => { items - untrack(() => (groupedItems = groupItems(items))) + pipelineFolders + isSearching + untrack(() => { + const grouped = groupItems(items) + // Ensure every pipeline folder is present at the top level so its + // "Pipeline" entry shows even when it has no listed items — a bundle-phase + // pipeline (only a draft so far) or a folder whose only scripts are + // pipeline members (folded into the pipeline, hidden from the list). + // Skip while searching: pipelines aren't part of the text filter (list view + // hides them on `filter !== ''`), so injecting them would surface unrelated + // folders in the results. + if (!isSearching) { + const present = new Set( + grouped + .filter((g) => 'folderName' in g) + .map((g) => (g as { folderName: string }).folderName) + ) + // Insert each missing pipeline folder among the existing folders in name + // order — `groupItems` already sorts user groups first then folders + // alphabetically, so inserting before the first greater-named folder + // keeps that ordering (rather than prepending out of order). + for (const folderName of [...(pipelineFolders ?? [])] + .filter((f) => !present.has(f)) + .sort()) { + const item = { folderName, items: [] } + const idx = grouped.findIndex( + (g) => + 'folderName' in g && + (g as { folderName: string }).folderName.localeCompare(folderName) > 0 + ) + if (idx < 0) grouped.push(item) + else grouped.splice(idx, 0, item) + } + } + groupedItems = grouped + }) }) diff --git a/frontend/src/lib/components/search/GlobalSearchModal.svelte b/frontend/src/lib/components/search/GlobalSearchModal.svelte index 19e71530e6..3e631a55e9 100644 --- a/frontend/src/lib/components/search/GlobalSearchModal.svelte +++ b/frontend/src/lib/components/search/GlobalSearchModal.svelte @@ -505,12 +505,16 @@ time: new Date(x.edited_at).getTime(), search_id: x.path })), - ...scripts.map((x) => ({ - ...x, - type: 'script' as 'script', - time: new Date(x.created_at).getTime(), - search_id: x.path - })), + // Pipeline-member scripts (`auto_kind='pipeline'`) are reached through + // their pipeline, not searched individually. + ...scripts + .filter((x) => x.auto_kind !== 'pipeline') + .map((x) => ({ + ...x, + type: 'script' as 'script', + time: new Date(x.created_at).getTime(), + search_id: x.path + })), ...apps.map((x) => ({ ...x, type: 'app' as 'app', diff --git a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte index 0716398131..0b529b2056 100644 --- a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte @@ -1,7 +1,7 @@ + +
+
+ + f/{path} + · data pipeline +
+
+ + {#if graphRes.loading && !graphRes.current && pe.drafts.size === 0} +
+ + Loading pipeline… +
+ {:else if graphRes.error && pe.drafts.size === 0} +
+ Failed to load pipeline: {graphRes.error.message} +
+ {:else} + + runNode(path, args)} + canRunByPath + onTestStateChange={(running) => { + const openPath = pe.openScriptPath + if (running && openPath) { + activeRunnable = { kind: 'script', path: openPath } + activeRunnables.arm(`script:${openPath}`) + activeRunnableJobId = undefined + } else if (!running && activeRunnable?.path === openPath) { + // Only clear the hint for the script the pane just finished — a + // canvas per-node run of a different script keeps its own hint. + activeRunnable = undefined + activeRunnableJobId = undefined + } + }} + onRunCompleted={() => { + activeRunnable = undefined + activeRunnableJobId = undefined + }} + onSelect={handleCanvasSelect} + onDraftSaved={afterSaved} + onPersistedSaved={afterSaved} + onScriptRemoved={async (removedPath) => { + pe.forgetPath(removedPath) + await graphRes.refetch() + }} + onScriptRenamed={async (oldPath, newPath) => { + // Repoint the selection so the canvas follows the renamed node instead + // of staying on the now-gone old path until an unrelated refetch. + if (pe.selection?.kind === 'runnable' && pe.selection.path === oldPath) { + pe.selection = { ...pe.selection, path: newPath } + } + await graphRes.refetch() + }} + onDiscard={() => { + if (pe.activeDraftPath) pe.discardDraft(pe.activeDraftPath) + }} + onClose={() => { + pe.selection = undefined + pe.activeDraftPath = undefined + pe.clearLiveOverlays() + }} + /> + {/if} +
+
+ + + graphRes.refetch()} +/> diff --git a/frontend/src/lib/components/sessions/SessionItemNotFound.svelte b/frontend/src/lib/components/sessions/SessionItemNotFound.svelte index dc1a088c53..77c5173813 100644 --- a/frontend/src/lib/components/sessions/SessionItemNotFound.svelte +++ b/frontend/src/lib/components/sessions/SessionItemNotFound.svelte @@ -3,7 +3,11 @@ import type { WorkspaceItem, WorkspaceItemKind } from '$lib/components/workspacePicker' import type { SessionTarget } from './sessionState.svelte' - const KIND_NOT_FOUND_LABEL: Record = { + // `pipeline` targets never hit this component (they aren't slot-loaded, so they + // can't 404 through SessionEditorTarget) — exclude it from the kinds here. + type NotFoundKind = Exclude + + const KIND_NOT_FOUND_LABEL: Record = { flow: 'Flow', script: 'Script', raw_app: 'Raw app' @@ -14,7 +18,7 @@ path, onNavigate }: { - kind: SessionTarget['kind'] + kind: NotFoundKind path: string onNavigate?: (item: WorkspaceItem) => void } = $props() diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index 027089d4ae..f62aa07926 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -27,6 +27,7 @@ import FlowEditorView from './FlowEditorView.svelte' import ScriptEditorView from './ScriptEditorView.svelte' import RawAppEditorView from './RawAppEditorView.svelte' + import PipelineEditorView from './PipelineEditorView.svelte' import SessionWorkspaceBar from './SessionWorkspaceBar.svelte' import SessionForkBar from './SessionForkBar.svelte' import SessionDraftBar from './SessionDraftBar.svelte' @@ -261,7 +262,8 @@ {@const hasTarget = session.target?.kind === 'flow' || session.target?.kind === 'script' || - session.target?.kind === 'raw_app'} + session.target?.kind === 'raw_app' || + session.target?.kind === 'pipeline'} {@const hasEditor = mountEditor && hasTarget && editorVisible} {#snippet inputPreface()} @@ -469,6 +471,13 @@ onNavigate={pickEditorTarget} isActiveSession={sessionState.currentSessionId === sessionId} /> + {:else if session.target.kind === 'pipeline'} + {/if}
diff --git a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte index c4099edd91..e6b4e5fd2f 100644 --- a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte @@ -15,6 +15,11 @@ /** Summary supplied by the data source. Preferred over the one derived * from the loaded diff value, and shown before that value loads. */ summary?: string + /** Explicit unique row identity, overriding the default `kind/path`. For a + * row whose `kind/path` isn't unique on its own (a pipeline-bundle node + * shares `script/` with a standalone script draft at the same path) + * while `path` must stay the real edit/display target. */ + key?: string } @@ -110,6 +115,7 @@ // e.g. a runnable rendered as `script` at `/runnables/foo` vs a real // script literally at that path. Prefix synthetic items so the {#each} key, // load cache, row id and nav identity never collide with a real DiffRow. + if ('key' in d && d.key) return d.key return ('appPath' in d ? 'rawapp:' : '') + `${d.kind}/${d.path}` } diff --git a/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts b/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts index d7d9f77630..f8a9e1045d 100644 --- a/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionRuntime.svelte.ts @@ -1,6 +1,7 @@ import { SvelteMap, SvelteSet } from 'svelte/reactivity' import { get } from 'svelte/store' import { AIChatManager, AIMode } from '$lib/components/copilot/chat/AIChatManager.svelte' +import { PipelineEditorState } from '$lib/components/assets/AssetGraph/pipelineEditorState.svelte' import { initFlow } from '$lib/components/flows/flowStore.svelte' import { AppService, @@ -79,6 +80,9 @@ export type SessionTargetKind = 'flow' | 'script' | 'raw_app' export interface SessionRuntime { readonly sessionId: string readonly manager: AIChatManager + // Pipeline target state — persists across editor hide/show (the pane unmounts + // on hide, so this can't be component-local) and across session switches. + readonly pipelineEditorState: PipelineEditorState // Kind-agnostic accessor over the per-kind load slots, for consumers (the // editor-target gate) that only need load state and not the typed store. slot(kind: SessionTargetKind): LoadSlot @@ -307,6 +311,12 @@ function createRuntime(session: Session): SessionRuntime { const rawApp: { val: SessionRuntime['rawApp']['val'] } = $state({ val: undefined }) const savedRawApp: { val: SessionRuntime['savedRawApp']['val'] } = $state({ val: undefined }) const rawAppSlot: LoadSlot = $state({ loadedPath: undefined, loading: false, notFound: false }) + + // Pipeline target state lives on the runtime (not the PipelineEditorView + // component) so the in-session drafts survive hide/show of the editor pane — + // the pane unmounts on hide, and a component-local store would be discarded. + const pipelineEditorState = new PipelineEditorState() + let runtimeLogRequester: RawAppRuntimeLogRequester | undefined = undefined let appRunsProvider: RawAppRunsProvider | undefined = undefined @@ -344,6 +354,7 @@ function createRuntime(session: Session): SessionRuntime { slot(kind: SessionTargetKind): LoadSlot { return kind === 'flow' ? flowSlot : kind === 'script' ? scriptSlot : rawAppSlot }, + pipelineEditorState, flowStore, flowStateStore, savedFlow, diff --git a/frontend/src/lib/components/sessions/sessionState.svelte.ts b/frontend/src/lib/components/sessions/sessionState.svelte.ts index e3d9e620f6..4e12d8622f 100644 --- a/frontend/src/lib/components/sessions/sessionState.svelte.ts +++ b/frontend/src/lib/components/sessions/sessionState.svelte.ts @@ -31,14 +31,18 @@ import type HistoryManager from '$lib/components/copilot/chat/HistoryManager.sve import { onUserChange } from '$lib/userScopedStorage' // Kinds the in-session editor pane can host. Legacy drag-and-drop apps are -// intentionally not previewable — only code-based 'raw_app' apps are. -export type SessionTarget = { kind: 'flow' | 'script' | 'raw_app'; path: string } +// intentionally not previewable — only code-based 'raw_app' apps are. A +// 'pipeline' target's `path` is the folder name (not a workspace item path): +// it hosts the data-pipeline graph editor for that folder, which uses its own +// fetch/draft model rather than the single-item load slots the other kinds share. +export type SessionTarget = { kind: 'flow' | 'script' | 'raw_app' | 'pipeline'; path: string } // Useful for filtering dropdowns / pickers to "items the side panel can open". export const EDITOR_TARGET_KINDS: ReadonlySet = new Set([ 'flow', 'script', - 'raw_app' + 'raw_app', + 'pipeline' ]) // Lifecycle status for a fork session. Git-parallel: diff --git a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte index 3056245e51..7933ef6d5e 100644 --- a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte @@ -4,7 +4,7 @@ import { page } from '$app/state' import Button from '$lib/components/common/button/Button.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' - import AssetGraphCanvas from '$lib/components/assets/AssetGraph/AssetGraphCanvas.svelte' + import PipelineGraphEditor from '$lib/components/assets/AssetGraph/PipelineGraphEditor.svelte' import { useActiveRunnableIds, isActiveEvent @@ -15,9 +15,7 @@ RunStatus } from '$lib/components/assets/AssetGraph/activeRunnables.svelte' import { usePipelineHistory } from '$lib/components/assets/AssetGraph/pipelineHistory.svelte' - import PipelineEventLog from '$lib/components/assets/AssetGraph/PipelineEventLog.svelte' import PipelineActivityPanel from '$lib/components/assets/AssetGraph/PipelineActivityPanel.svelte' - import AssetGraphDetailsPane from '$lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte' import PipelinePickerModal from '$lib/components/assets/AssetGraph/PipelinePickerModal.svelte' import { extractWrites, @@ -31,11 +29,7 @@ PipelineMode } from '$lib/components/assets/AssetGraph/types' import PipelineModeToggle from '$lib/components/assets/AssetGraph/PipelineModeToggle.svelte' - import { - parsePipelineAnnotations, - type ColumnLineage, - type PipelineAnnotations - } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations' + import { parsePipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations' import { buildColumnGraph, type ColumnLineageGraph @@ -68,9 +62,11 @@ type PipelineOutputKind, type DraftTriggerSource } from '$lib/components/assets/AssetGraph/pipelineTemplates' - import { decodeState, encodeState } from '$lib/utils' - import { DraftService } from '$lib/gen' - import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' + import { + createPipelineAiHelpers, + type PipelineDraft + } from '$lib/components/assets/AssetGraph/pipelineAiHelpers' + import { PipelineEditorState } from '$lib/components/assets/AssetGraph/pipelineEditorState.svelte' import AutosaveIndicator from '$lib/components/AutosaveIndicator.svelte' import { onMount, tick, untrack } from 'svelte' import { aiChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' @@ -98,14 +94,12 @@ type ScriptLang } from '$lib/gen' import { resource } from 'runed' - import { Pane, Splitpanes } from 'svelte-splitpanes' import { emptySchema, sendUserToast } from '$lib/utils' import type { Schema } from '$lib/common' import { beforeNavigate, goto } from '$app/navigation' import { fade } from 'svelte/transition' import { twMerge } from 'tailwind-merge' import Popover from '$lib/components/meltComponents/Popover.svelte' - import HideButton from '$lib/components/apps/editor/settingsPanel/HideButton.svelte' import { inferArgs, inferAssets } from '$lib/infer' import PipelineTriggerEditors from '$lib/components/assets/AssetGraph/PipelineTriggerEditors.svelte' @@ -115,7 +109,33 @@ const DATA_KINDS = ['s3object', 'ducklake', 'datatable', 'volume'] let folder = $derived(page.params.folder as string) - let selection = $state(undefined) + + // Externalized editor state (drafts, live overlays, selection), shared with + // the in-session pipeline preview via PipelineEditorState. Referenced as + // `pe.*` throughout; the persistence / graph / run logic below stays here. + const pe = new PipelineEditorState() + + // The in-app folder switcher navigates same-route (`/pipeline/`), which + // reuses this page component — nothing remounts. Mirror the session preview's + // retarget guard: reset the editor state on a folder change so folder A's drafts + // don't display under B (and aren't autosaved into B's bundle), and so + // hydratedFromDb flips back to false and B's draft bundle re-hydrates. + $effect(() => { + const f = folder + untrack(() => { + if (pe.folder !== f) { + if (pe.folder !== undefined) { + pe.reset() + // Re-scope the Global chat's pipeline prompt to the new folder. The + // helper methods already read the reactive folder, but the system + // message string was built for the old one and is only rebuilt when + // Global mode is reconfigured — so rebuild it here. + aiChatManager.rebuildGlobalSystemMessage() + } + pe.folder = f + } + }) + }) // Page mode, URL-addressable via `?mode=`. No param = view (the // default): deployed-only graph focused on past/live executions. @@ -129,10 +149,10 @@ // selection / open draft / hidden-pane state so view opens on Activity // rather than a stale details pane (mirrors toggleActivity's show path). if (m === 'view' && mode === 'edit') { - selection = undefined - activeDraftPath = undefined + pe.selection = undefined + pe.activeDraftPath = undefined panelHidden = false - liveAnnotations = EMPTY_LIVE_ANNOTATIONS + pe.liveAnnotations = EMPTY_LIVE_ANNOTATIONS } const url = new URL(page.url) if (m === 'view') url.searchParams.delete('mode') @@ -166,346 +186,44 @@ // output asset, and they all render on the graph simultaneously. // Saving removes a draft from the map; closing the pane keeps it so the // user can come back to it. - // Counter-based id source — sufficient for "stable across renames in - // this session"; doesn't need to survive a reload. (We have crypto. - // randomUUID() too but a short numeric id keeps localStorage tidy.) - let nextDraftLocalIdCounter = 0 - function newDraftLocalId(): string { - nextDraftLocalIdCounter += 1 - return `d${nextDraftLocalIdCounter}-${Date.now()}` - } + // The draft shape + draft Map + activeDraftPath now live in the shared + // PipelineEditorState (`pe`). `Draft` aliases the store's type so existing + // annotations keep working. + type Draft = PipelineDraft - type Draft = { - // Stable per-draft identifier, generated on first create and - // preserved across renames. Used to track concurrent deploys (a - // fast double-rename otherwise fires two saves that each leave a - // persisted script behind — the latest deploy archives the prior - // one keyed on this id). - localId: string - script: Script - // Undefined when the user picked `outputKind === 'none'` — the draft - // has no auto-generated output asset, so the graph overlay skips - // synthesizing a write edge for it. - outputAsset?: { kind: AssetKind; path: string } - // Inferred body writes from the last time this draft was open in - // the details pane. Captured on transition (selection change / - // pane close) so the canvas keeps showing the user's renamed - // outputs after they've clicked away. Falls back to outputAsset - // when undefined (initial state or parser miss). - outputAssets?: Array<{ kind: AssetKind; path: string }> - } - let drafts = $state>(new Map()) - - // Which draft (if any) is currently open in the details pane. When - // undefined and `selection` is set, the pane shows the persisted - // selection's script. Never both at once. - let activeDraftPath = $state(undefined) - - // Splitpanes sizes: bound so user-resized widths persist when the - // details pane is hidden + re-shown, or when switching between draft - // and persisted selections (previously the panes were sized inline, - // which made the right pane jump width when activeDraft was set - // since the left was hardcoded to 100%). When the right pane - // unmounts, svelte-splitpanes does NOT auto-stretch the remaining - // pane — the bound `leftPaneSize` stays at its last value and the - // other 40% renders as blank space. We explicitly set leftPaneSize - // to 100 in that state via a $derived effect below, and restore the - // stored split when the pane comes back. - let leftPaneSize = $state(60) - let rightPaneSize = $state(40) - let storedRightPaneSize = $state(40) + // Splitpane sizing + details-pane-open derivation live inside PipelineGraphEditor. // Explicit hide flag — keeps `selection` / `activeDraftPath` intact // so re-opening the pane re-uses them. Mirrors AppEditor's // hideRightPanel/showRightPanel pattern. let panelHidden = $state(false) - // Edit mode opens the pane only for a selection/draft; view mode - // keeps it open permanently — it shows the activity panel when nothing - // is selected and swaps to the details pane on node select. - let detailsPaneOpen = $derived( - mode === 'edit' - ? (selection != undefined || activeDraftPath != undefined) && !panelHidden - : !panelHidden - ) // View mode's idle pane is the activity feed; once a node is selected // (or the pane hidden) the only ways back were the pane's X and the // floating hide toggle — neither is named. The top-bar Activity toggle // is the explicit affordance: shows the feed from any state, hides the // pane when the feed is already showing. let activityShowing = $derived( - mode === 'view' && !panelHidden && selection == undefined && activeDraftPath == undefined + mode === 'view' && !panelHidden && pe.selection == undefined && pe.activeDraftPath == undefined ) function toggleActivity() { if (activityShowing) { panelHidden = true } else { - selection = undefined - activeDraftPath = undefined + pe.selection = undefined + pe.activeDraftPath = undefined panelHidden = false // Same reset as the pane's close button — clears the live // annotation overlay of whichever script was open. - liveAnnotations = EMPTY_LIVE_ANNOTATIONS + pe.liveAnnotations = EMPTY_LIVE_ANNOTATIONS } } - $effect(() => { - if (detailsPaneOpen) { - // Pane visible: reapply the stored split. We don't depend on - // rightPaneSize here (only stored) so user resizes via the - // splitter handle aren't immediately overridden. - const restore = storedRightPaneSize - untrack(() => { - rightPaneSize = restore - leftPaneSize = 100 - restore - }) - } else { - // About to hide. Stash the current right size so the next show - // restores it, then expand the left pane to fill — splitpanes - // won't do this automatically when a Pane unmounts. - untrack(() => { - if (rightPaneSize > 0) storedRightPaneSize = rightPaneSize - leftPaneSize = 100 - }) - } - }) - - // All of this folder's in-flight drafts live in ONE per-user DB draft - // (typ `data_pipeline`) keyed at the folder, so they sync across devices - // and surface in the global drafts list — replacing the prior - // browser-only blob. The `f//...` path drives the backend's - // folder-write access check. localStorage is kept as a synchronous crash - // mirror (no size cap, survives a hard close inside the debounce window) - // but is only ever READ for the one-time import below; the DB is the - // source of truth on load. - const PIPELINE_DRAFT_KIND = 'data_pipeline' as const + // Draft autosave (the data_pipeline DraftService bundle) lives inside + // PipelineGraphEditor now; the route just supplies its path to the indicator. let pipelineDraftPath = $derived(`f/${folder}/data_pipeline`) - let storageKey = $derived(`pipeline-${folder}`) - type PipelineDraftBundle = { drafts: Array<[string, Draft]>; activeDraftPath?: string } - // Gate the persist effect until the initial DB load resolves so empty - // pre-hydration state can't clobber the server copy. `lastPersistedBundle` - // holds the last value we pushed so an unchanged re-render (and the - // just-loaded value itself) isn't re-POSTed. - let draftsHydrated = $state(false) - let lastPersistedBundle: string | undefined = undefined - // True once a bundle was restored from the DB on load — drives the - // AutosaveIndicator's one-shot "Loaded from draft" hint. - let loadedFromDbDraft = $state(false) - - function restoreBundle(bundle: PipelineDraftBundle) { - if (Array.isArray(bundle.drafts)) { - const loaded = new Map() - for (const entry of bundle.drafts) { - if (entry && typeof entry[0] === 'string' && entry[1]?.script) { - const d = entry[1] as Draft - // Backfill localId for state persisted by older builds. - if (typeof d.localId !== 'string' || d.localId === '') { - d.localId = newDraftLocalId() - } - loaded.set(entry[0], d) - } - } - if (loaded.size > 0) drafts = loaded - } - if (typeof bundle.activeDraftPath === 'string') { - activeDraftPath = bundle.activeDraftPath - } - } - - // The pre-DB localStorage blob, for the one-time migration when the user - // has no DB draft yet. - function readLocalBundle(): PipelineDraftBundle | undefined { - if (typeof localStorage === 'undefined') return undefined - const raw = localStorage.getItem(`pipeline-${folder}`) - if (!raw) return undefined - try { - const state = decodeState(raw) - if (state && (Array.isArray(state.drafts) || typeof state.activeDraftPath === 'string')) { - return { - drafts: Array.isArray(state.drafts) ? state.drafts : [], - activeDraftPath: - typeof state.activeDraftPath === 'string' ? state.activeDraftPath : undefined - } - } - } catch (e) { - console.warn('failed to read local pipeline state', e) - } - return undefined - } - - onMount(() => { - void hydrateDrafts() - }) - - async function hydrateDrafts() { - const ws = $workspaceStore - const path = pipelineDraftPath - try { - let bundle: PipelineDraftBundle | undefined - let serverSavedAt: string | undefined - if (ws) { - const row = await DraftService.getOwnDraft({ - workspace: ws, - kind: PIPELINE_DRAFT_KIND, - path - }) - if (row?.value) { - bundle = row.value as PipelineDraftBundle - serverSavedAt = row.created_at - loadedFromDbDraft = true - } - } - // One-time migration: no DB draft yet, but an older build left a - // localStorage blob — adopt it and let the persist effect push it up. - let migratedFromLocal = false - if (!bundle) { - const local = readLocalBundle() - if (local) { - bundle = local - migratedFromLocal = true - } - } - if (bundle) restoreBundle(bundle) - // Seed the conflict baseline: server timestamp when loaded from the - // DB, none otherwise (first save omits last_sync → backend first-push - // branch). A local migration counts as "nothing server-side yet". - UserDraftDbSyncer.recordRemoteSync( - { workspace: ws ?? '', itemKind: PIPELINE_DRAFT_KIND, path }, - migratedFromLocal ? undefined : serverSavedAt - ) - // Record what we loaded so the first persist run is a no-op — UNLESS - // we migrated from localStorage, which must push to the DB once. - if (!migratedFromLocal) { - lastPersistedBundle = bundle ? JSON.stringify(bundle) : undefined - } - } catch (e) { - console.warn('failed to load pipeline drafts', e) - } finally { - draftsHydrated = true - } - } - - // Persist on change: debounced DB sync (UserDraftDbSyncer handles the - // debounce + optimistic-concurrency) plus a synchronous localStorage - // mirror for crash recovery before the first DB confirm. - $effect(() => { - // Track deps explicitly so Svelte 5 re-runs on mutation. - // For the active draft, also snapshot the latest live body writes - // at serialize time. Without this, edits made since the last - // pane-transition (the cleanup that calls onDraftPersist) are - // lost on reload — the draft restores with stale `outputAssets` - // and the graph drops the corresponding edges until the user - // re-opens the draft and types something new. - const liveWritesSnapshot = - liveBodyAssets.scriptPath != undefined && drafts.has(liveBodyAssets.scriptPath) - ? extractWrites(liveBodyAssets.assets) - : undefined - const liveWritesPath = liveBodyAssets.scriptPath - // Live editor buffer for the open draft. `onDraftPersist` only commits - // content into the Map on pane teardown, so without this overlay the - // autosave (and a crash reload) would lag a whole editing session behind. - const liveContentPath = - liveContent.scriptPath != undefined && drafts.has(liveContent.scriptPath) - ? liveContent.scriptPath - : undefined - const liveContentValue = liveContent.content - const serialized = Array.from(drafts.entries()).map(([p, d]) => { - const outputAssets = - liveWritesSnapshot != undefined && liveWritesPath === p - ? liveWritesSnapshot.length > 0 - ? liveWritesSnapshot - : undefined - : d.outputAssets - const script = - liveContentPath === p && d.script.content !== liveContentValue - ? { ...d.script, content: liveContentValue } - : d.script - if (script === d.script && outputAssets === d.outputAssets) { - return [p, d] as [string, Draft] - } - return [p, { ...d, script, outputAssets }] as [string, Draft] - }) - const activePath = activeDraftPath - const key = storageKey - const ws = $workspaceStore - const path = pipelineDraftPath - const hydrated = draftsHydrated - untrack(() => { - // Don't touch storage until the initial load settled. - if (!hydrated) return - const isEmpty = serialized.length === 0 && !activePath - const bundle: PipelineDraftBundle | undefined = isEmpty - ? undefined - : { drafts: serialized, activeDraftPath: activePath } - // localStorage crash mirror — synchronous, no debounce, no size cap. - try { - if (typeof localStorage !== 'undefined') { - if (isEmpty) localStorage.removeItem(key) - else - localStorage.setItem( - key, - encodeState({ drafts: serialized, activeDraftPath: activePath }) - ) - } - } catch (e) { - console.warn('failed to mirror pipeline state', e) - } - const serializedBundle = bundle ? JSON.stringify(bundle) : undefined - if (serializedBundle === lastPersistedBundle) return - lastPersistedBundle = serializedBundle - if (!ws) return - void UserDraftDbSyncer.save({ - workspace: ws, - itemKind: PIPELINE_DRAFT_KIND, - path, - // `null` deletes the bundle once the last draft is gone. - value: bundle ?? null, - auto: true - }) - }) - }) - - // Live-parsed annotations from whatever script is currently open in the - // details pane (draft or existing). Refreshed on every keystroke via - // `onAnnotationsChange`. Used to overlay unsaved schedule / trigger-asset - // edges onto the graph so the editor buffer and the graph stay in sync. - let liveAnnotations = $state<{ - scriptPath: string | undefined - annotations: PipelineAnnotations - }>({ - scriptPath: undefined, - annotations: { - inPipeline: false, - triggerAssets: [], - nativeTriggers: [], - dataTests: [], - columnLineage: [] - } - }) - - // Live-inferred body assets (read/write usages parsed by inferAssets - // — e.g. CREATE TABLE in SQL, loadS3File / writeS3File in TS/Python). - // Refreshed via onAssetsChange. We use the write subset as the - // authoritative output node set for drafts whose body has been edited - // past the seeded template; without this, renaming a CREATE TABLE - // target leaves the stale auto-output node on the graph. - let liveBodyAssets = $state<{ - scriptPath: string | undefined - assets: AssetWithAltAccessType[] - columnLineage?: ColumnLineage[] - }>({ scriptPath: undefined, assets: [] }) - - // The open draft's live editor buffer, emitted by the pane on every - // keystroke (`onContentChange`). The persist effect overlays it onto the - // Map's copy so autosave reflects in-progress edits — `onDraftPersist` - // only commits content into the Map on pane teardown, which would leave - // autosave a full editing session behind. - let liveContent = $state<{ scriptPath: string | undefined; content: string }>({ - scriptPath: undefined, - content: '' - }) - - // Canonical "empty" overlay literals, reused both as reset values for the - // live-* state above and as the no-overlay inputs to the deployed graph. + // The live editor overlays (annotations / body assets / content for the open + // script) now live in `pe`. The canonical "empty" literals stay here — they + // also seed the no-overlay inputs to the deployed graph below. const EMPTY_LIVE_ASSETS = { scriptPath: undefined, assets: [] } const EMPTY_LIVE_ANNOTATIONS = { scriptPath: undefined, @@ -518,16 +236,6 @@ } } - // Reset every live editor overlay (annotations / body assets / content) - // back to empty, unconditionally. Used by the leave-edit path so a stale - // buffer for the previously-open script can't leak into the view graphs. - // (forgetPath resets these per-path instead — see there.) - function clearLiveOverlays() { - liveAnnotations = EMPTY_LIVE_ANNOTATIONS - liveBodyAssets = EMPTY_LIVE_ASSETS - liveContent = { scriptPath: undefined, content: '' } - } - // Only-add cache of (script_path → body content) populated lazily by // `bodyFetchEffect` for every script in the current folder. We never // remove entries: stale keys (renamed-away, deleted) are simply ignored @@ -559,7 +267,9 @@ const g = graphRes.current if (!g) return { writes, reads } const liveAssetsForPath = (path: string) => - liveBodyAssets.scriptPath === path ? liveBodyAssets.assets : inferredAssetsByPath.get(path) + pe.liveBodyAssets.scriptPath === path + ? pe.liveBodyAssets.assets + : inferredAssetsByPath.get(path) for (const r of g.runnables) { if (r.usage_kind !== 'script') continue const assets = liveAssetsForPath(r.path) @@ -580,12 +290,12 @@ const out = new Map>() const g = graphRes.current if (!g) return out - const livePath = liveAnnotations.scriptPath + const livePath = pe.liveAnnotations.scriptPath for (const r of g.runnables) { if (r.usage_kind !== 'script') continue let kinds: Set if (r.path === livePath) { - kinds = new Set(liveAnnotations.annotations.nativeTriggers.map((n) => n.kind)) + kinds = new Set(pe.liveAnnotations.annotations.nativeTriggers.map((n) => n.kind)) } else { const body = bodiesByPath.get(r.path) if (!body) continue @@ -663,13 +373,17 @@ const script = buildDraft(language, scriptPath, triggers, outputKind, out, input) // Write the new draft into the map (structural update so Svelte // re-derives graphWithDraft) and focus it in the details pane. When - // the user picked `none`, `outputAsset` is undefined and the graph - // overlay skips synthesizing a write edge. - const next = new Map(drafts) - next.set(scriptPath, { localId: newDraftLocalId(), script, outputAsset: out }) - drafts = next - activeDraftPath = scriptPath - selection = undefined + // the user picked `none`, `out` is undefined and the graph overlay + // skips synthesizing a write edge. + const next = new Map(pe.drafts) + next.set(scriptPath, { + localId: pe.newDraftLocalId(), + script, + outputAssets: out ? [out] : undefined + }) + pe.drafts = next + pe.activeDraftPath = scriptPath + pe.selection = undefined // Follow the new node with a smooth pan. The id matches the runnable // node the canvas builds for a draft script (`script:`). @@ -713,6 +427,43 @@ aiChatManager.sendRequest({ instructions }) } + // ===================== AI chat pipeline integration ===================== + // The global AI chat (dev-gated) gains pipeline-building tools while this + // editor is mounted, via the helpers registered below. AI mutations don't + // deploy — they apply directly as unsaved drafts on the canvas (the same way + // the flow/script editor applies AI edits), which the user then deploys. The + // build/edit logic is shared verbatim with the in-session preview + // (PipelineEditorView) via createPipelineAiHelpers. + + const pipelineAiHelpers = createPipelineAiHelpers({ + getFolder: () => folder, + getWorkspace: () => $workspaceStore, + getResolvedGraph: () => graphWithDraft, + getDrafts: () => pe.drafts, + setDrafts: (next) => (pe.drafts = next), + newDraftLocalId: pe.newDraftLocalId, + onForgetPath: (path) => forgetPath(path), + onShowDrafts: () => (includeDrafts = true), + onProposeNode: (path) => focusPipelineNode(`script:${path}`), + ensureEditable: () => { + // Auto-enter edit so AI changes are visible/actionable, unless the user + // is an operator (no edit permission) — then refuse with a clear error. + if (isOperator) { + throw new Error('This pipeline is read-only for your role; AI edits are disabled.') + } + if (mode !== 'edit') setMode('edit') + }, + onRunStarted: (jobId, path) => { + activeRunnables.arm(`script:${path}`) + runsPendingJobId = jobId + runsRefreshKey++ + activeRunnable = { kind: 'script', path } + activeRunnableJobId = jobId + } + }) + + onMount(() => aiChatManager.setPipelineHelpers(pipelineAiHelpers)) + // Navigation guard state. `pendingNavigationUrl` holds the URL the user // tried to leave to so we can complete the navigation after they pick // "Save all" or "Discard all"; `bypassNavigationGuard` is the standard @@ -728,7 +479,7 @@ bypassNavigationGuard = false return } - if (drafts.size === 0) return + if (pe.drafts.size === 0) return // `leave` covers tab close / hard reload / cross-origin nav. SvelteKit // turns a cancelled leave into a browser-native "Leave site?" prompt, // which we explicitly don't want — match the rest of the editors and @@ -762,8 +513,8 @@ // Wipe every draft and the active selection so saved drafts and // stale active path don't bleed into the next page. localStorage // is overwritten by the persist effect on the next tick. - drafts = new Map() - activeDraftPath = undefined + pe.drafts = new Map() + pe.activeDraftPath = undefined saveErrors = new Map() const target = pendingNavigationUrl leaveModalOpen = false @@ -782,7 +533,7 @@ // page so they can deal with the failures via the bar's error // popover. Otherwise resume the navigation that triggered the // guard. - if (drafts.size === 0) { + if (pe.drafts.size === 0) { const target = pendingNavigationUrl leaveModalOpen = false pendingNavigationUrl = undefined @@ -871,10 +622,10 @@ } async function saveAllDrafts() { - if (!$workspaceStore || drafts.size === 0 || savingAll) return + if (!$workspaceStore || pe.drafts.size === 0 || savingAll) return savingAll = true const ws = $workspaceStore - const entries = [...drafts.entries()] + const entries = [...pe.drafts.entries()] // Snapshot what the preview promises for every draft before anything // deploys — used to verify the persisted graph below. const predicted = predictCascadeFacts(entries.map(([p]) => p)) @@ -905,22 +656,22 @@ // entries to keep insertion order stable. if (savedPaths.length > 0) { const next = new Map() - for (const [k, v] of drafts) { + for (const [k, v] of pe.drafts) { if (!savedPaths.includes(k)) next.set(k, v) } - drafts = next + pe.drafts = next // If the open draft just got deployed, transfer the focus to // its now-persisted runnable so the pane stays on the same // script the user was editing — otherwise the pane closes, // the canvas re-fits, and the user has to re-find their // script after every save. - if (activeDraftPath && savedPaths.includes(activeDraftPath)) { - selection = { + if (pe.activeDraftPath && savedPaths.includes(pe.activeDraftPath)) { + pe.selection = { kind: 'runnable', runnable_kind: 'script', - path: activeDraftPath + path: pe.activeDraftPath } - activeDraftPath = undefined + pe.activeDraftPath = undefined } await graphRes.refetch() // Verify only what actually deployed — failed drafts would @@ -939,10 +690,10 @@ } function discardDraft(path: string) { - if (!drafts.has(path)) return - const next = new Map(drafts) + if (!pe.drafts.has(path)) return + const next = new Map(pe.drafts) next.delete(path) - drafts = next + pe.drafts = next forgetPath(path) } @@ -961,18 +712,18 @@ // selection + per-path save errors. `bodiesByPath` keeps its entry // (only-add cache, harmless if stale). function forgetPath(path: string) { - if (activeDraftPath === path) activeDraftPath = undefined - if (selection?.kind === 'runnable' && selection.path === path) { - selection = undefined + if (pe.activeDraftPath === path) pe.activeDraftPath = undefined + if (pe.selection?.kind === 'runnable' && pe.selection.path === path) { + pe.selection = undefined } - if (liveAnnotations.scriptPath === path) { - liveAnnotations = EMPTY_LIVE_ANNOTATIONS + if (pe.liveAnnotations.scriptPath === path) { + pe.liveAnnotations = EMPTY_LIVE_ANNOTATIONS } - if (liveBodyAssets.scriptPath === path) { - liveBodyAssets = EMPTY_LIVE_ASSETS + if (pe.liveBodyAssets.scriptPath === path) { + pe.liveBodyAssets = EMPTY_LIVE_ASSETS } - if (liveContent.scriptPath === path) { - liveContent = { scriptPath: undefined, content: '' } + if (pe.liveContent.scriptPath === path) { + pe.liveContent = { scriptPath: undefined, content: '' } } clearSaveError(path) } @@ -983,13 +734,13 @@ // open and surface the conflict inline. function renameDraft(oldPath: string, newPath: string): boolean | string { if (oldPath === newPath) return true - const draft = drafts.get(oldPath) + const draft = pe.drafts.get(oldPath) if (!draft) return 'Draft not found' - if (drafts.has(newPath)) return 'Another draft already uses this path' + if (pe.drafts.has(newPath)) return 'Another draft already uses this path' const next = new Map() // Preserve insertion order: replace the entry at its original // position so the canvas / lists don't reshuffle on rename. - for (const [k, v] of drafts) { + for (const [k, v] of pe.drafts) { if (k === oldPath) { const updatedScript = { ...v.script, path: newPath } next.set(newPath, { ...v, script: updatedScript }) @@ -997,8 +748,8 @@ next.set(k, v) } } - drafts = next - if (activeDraftPath === oldPath) activeDraftPath = newPath + pe.drafts = next + if (pe.activeDraftPath === oldPath) pe.activeDraftPath = newPath // Path-keyed live overlays: re-key for the renamed draft so the // graph stays consistent between the moment we mutate `drafts` // here and the next editor event that re-emits annotations / @@ -1007,14 +758,14 @@ // re-applying live overlays against the same OLD path — leaving // phantom edges that displace the + node off the top of the // graph and shuffle the layout. - if (liveAnnotations.scriptPath === oldPath) { - liveAnnotations = { ...liveAnnotations, scriptPath: newPath } + if (pe.liveAnnotations.scriptPath === oldPath) { + pe.liveAnnotations = { ...pe.liveAnnotations, scriptPath: newPath } } - if (liveBodyAssets.scriptPath === oldPath) { - liveBodyAssets = { ...liveBodyAssets, scriptPath: newPath } + if (pe.liveBodyAssets.scriptPath === oldPath) { + pe.liveBodyAssets = { ...pe.liveBodyAssets, scriptPath: newPath } } - if (liveContent.scriptPath === oldPath) { - liveContent = { ...liveContent, scriptPath: newPath } + if (pe.liveContent.scriptPath === oldPath) { + pe.liveContent = { ...pe.liveContent, scriptPath: newPath } } // `inferredWritesByPath` / `inferredReadsByPath` / // `annotatedNativeKindsByPath` are derived from `g.runnables` × @@ -1055,7 +806,7 @@ >() function deployRenamedDraft(path: string) { - const draft = drafts.get(path) + const draft = pe.drafts.get(path) if (!draft) return const localId = draft.localId let state = deployQueue.get(localId) @@ -1078,7 +829,7 @@ try { while (true) { if (!$workspaceStore) break - const draft = drafts.get(path) + const draft = pe.drafts.get(path) if (!draft) break try { await saveDraft(path, draft, $workspaceStore) @@ -1111,12 +862,12 @@ // queued path is waiting, the next loop iteration will // pick it up and we keep the draft live. if (!state.queuedPath) { - const nextDrafts = new Map(drafts) + const nextDrafts = new Map(pe.drafts) nextDrafts.delete(path) - drafts = nextDrafts - if (activeDraftPath === path) { - selection = { kind: 'runnable', runnable_kind: 'script', path } - activeDraftPath = undefined + pe.drafts = nextDrafts + if (pe.activeDraftPath === path) { + pe.selection = { kind: 'runnable', runnable_kind: 'script', path } + pe.activeDraftPath = undefined } if (saveErrors.has(path)) { const nextErrors = new Map(saveErrors) @@ -1142,22 +893,22 @@ state.inflight = false // On the rare path where the draft is also gone (deployed + // no queued path), drop the slot to keep the map bounded. - if (!drafts.has(path) && !state.queuedPath) { + if (!pe.drafts.has(path) && !state.queuedPath) { deployQueue.delete(localId) } } } // Currently-open draft shape (if any) — fed into the details pane. - let activeDraft = $derived(activeDraftPath ? drafts.get(activeDraftPath) : undefined) + let activeDraft = $derived(pe.activeDraftPath ? pe.drafts.get(pe.activeDraftPath) : undefined) // Path of the script currently open in the details pane (draft or // persisted selection), used wherever run-routing / overlay logic needs // "the one script the user is editing right now". let openScriptPath = $derived( - activeDraftPath ?? - (selection?.kind === 'runnable' && selection.runnable_kind === 'script' - ? selection.path + pe.activeDraftPath ?? + (pe.selection?.kind === 'runnable' && pe.selection.runnable_kind === 'script' + ? pe.selection.path : undefined) ) @@ -1168,12 +919,12 @@ $effect(() => { if (mode !== 'edit') return if ( - selection?.kind === 'runnable' && - selection.runnable_kind === 'script' && - drafts.has(selection.path) + pe.selection?.kind === 'runnable' && + pe.selection.runnable_kind === 'script' && + pe.drafts.has(pe.selection.path) ) { - activeDraftPath = selection.path - selection = undefined + pe.activeDraftPath = pe.selection.path + pe.selection = undefined } }) // Symmetric demotion: view mode shows the deployed truth, so an open @@ -1185,95 +936,13 @@ if (mode !== 'view') return const d = activeDraft if (d && d.script.hash) { - selection = { kind: 'runnable', runnable_kind: 'script', path: d.script.path } - activeDraftPath = undefined + pe.selection = { kind: 'runnable', runnable_kind: 'script', path: d.script.path } + pe.activeDraftPath = undefined } }) - // Named handlers for the details pane's live callbacks. Inline arrows - // would be rebuilt on every parent re-render, and the pane's $effects - // track those refs as deps — combined with the drafts mutation in - // `handleDraftContentChange`, that creates a parent ↔ child feedback - // loop ("effect_update_depth_exceeded"). Named functions keep the - // prop reference stable so the $effects only re-fire on real - // content changes (e.g. handleContentChange below mutating drafts). - function handleAnnotationsChange( - scriptPath: string | undefined, - annotations: PipelineAnnotations - ) { - liveAnnotations = { scriptPath, annotations } - } - function handleAssetsChange( - scriptPath: string | undefined, - assets: AssetWithAltAccessType[], - columnLineage?: ColumnLineage[] - ) { - // Single update site for the live overlay. `inferredWritesByPath` - // / `inferredReadsByPath` are now derived from `liveBodyAssets` - // (for the open script) + `inferredAssetsByPath` (prefetched - // snapshot for every other script), so we don't have to write - // into those caches here — the derive picks up our update on the - // next reactive tick. - liveBodyAssets = { scriptPath, assets, columnLineage } - } - function handleContentChange(scriptPath: string | undefined, content: string) { - liveContent = { scriptPath, content } - } - function handleDraftPersist( - p: string, - snapshot: { - content: string - writes: { kind: AssetKind; path: string }[] - script?: Script - } - ) { - // Persist body edits + inferred outputs back into the drafts Map so - // they survive switching to another node and back (the details pane - // clones draftScript locally on every prop change, and `outputAsset` - // would otherwise stay frozen at the value seeded when the draft - // was opened — leaving a stale write edge on the canvas after the - // user has renamed a CREATE TABLE / writeS3File target). - // - // Deferred a microtask: this is called from the pane's $effect - // teardown, which observes the *previous* batch values — after a - // discard, `drafts` still appears to contain the discarded entry, - // and writing a map cloned from that stale read would resurrect it - // (the "discard needs two clicks" bug). One microtask later the - // batch has committed and the reads are fresh. - queueMicrotask(() => { - const d = drafts.get(p) - if (!d) { - // Unsaved edits to a *deployed* script (the pane only emits - // these when the buffer differs from the deployed content): - // promote to a draft so the work survives mode switches / - // selection changes, shows in the drafts chip, and deploys - // via Save all. The script snapshot carries the deployed - // hash, so saving chains a new version off it. - if (!snapshot.script) return - const next = new Map(drafts) - next.set(p, { - localId: newDraftLocalId(), - script: snapshot.script, - outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined - }) - drafts = next - return - } - const writesEqual = - d.outputAssets?.length === snapshot.writes.length && - (d.outputAssets ?? []).every( - (a, i) => a.kind === snapshot.writes[i]?.kind && a.path === snapshot.writes[i]?.path - ) - if (d.script.content === snapshot.content && writesEqual) return - const next = new Map(drafts) - next.set(p, { - ...d, - script: { ...d.script, content: snapshot.content }, - outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined - }) - drafts = next - }) - } + // The details pane's live callbacks (annotations / assets / content / persist) + // now live on `pe` as stable arrow fields — pass `pe.handleX` straight through. // Canvas callbacks, named so the prop refs stay stable across re-renders // (same rationale as the live-callback handlers above) and so the @@ -1297,13 +966,13 @@ s && s.kind === 'runnable' && s.runnable_kind === 'script' && - drafts.has(s.path) + pe.drafts.has(s.path) ) { - activeDraftPath = s.path - selection = undefined + pe.activeDraftPath = s.path + pe.selection = undefined } else { - activeDraftPath = undefined - selection = s + pe.activeDraftPath = undefined + pe.selection = s } } function handleAddScriptForAsset( @@ -1349,8 +1018,8 @@ return } if (info.runnable_kind !== 'script') return - activeDraftPath = undefined - selection = { kind: 'runnable', runnable_kind: 'script', path: info.path } + pe.activeDraftPath = undefined + pe.selection = { kind: 'runnable', runnable_kind: 'script', path: info.path } requestRemoveSignal++ } async function handleRunProducer(producer: { @@ -1386,7 +1055,7 @@ if (cascade) { const hasDraftInChain = producer.unsaved === true || - computeDownstreamClosure(graphWithDraft, producer.path).nodes.some((p) => drafts.has(p)) + computeDownstreamClosure(graphWithDraft, producer.path).nodes.some((p) => pe.drafts.has(p)) if (hasDraftInChain) { return await runDraftAwareCascade(producer.path) } @@ -1403,7 +1072,7 @@ const skipArg = cascade ? {} : { _wmill_skip_asset_dispatch: true } let jobId: string | undefined if (producer.unsaved) { - const draft = drafts.get(producer.path) + const draft = pe.drafts.get(producer.path) if (!draft?.script.content || !draft.script.language) return undefined jobId = await JobService.runScriptPreview({ workspace: $workspaceStore, @@ -1483,9 +1152,9 @@ let graphWithDraft = $derived.by(() => resolveGraph({ base: graphRes.current ?? EMPTY_GRAPH, - drafts, - liveBodyAssets, - liveAnnotations, + drafts: pe.drafts, + liveBodyAssets: pe.liveBodyAssets, + liveAnnotations: pe.liveAnnotations, inferredWritesByPath, inferredReadsByPath, annotatedNativeKindsByPath @@ -1523,11 +1192,11 @@ // Guard before reassigning so an already-empty overlay doesn't // needlessly invalidate the graph derives every mode toggle. if ( - liveAnnotations.scriptPath != undefined || - liveBodyAssets.scriptPath != undefined || - liveContent.scriptPath != undefined + pe.liveAnnotations.scriptPath != undefined || + pe.liveBodyAssets.scriptPath != undefined || + pe.liveContent.scriptPath != undefined ) { - clearLiveOverlays() + pe.clearLiveOverlays() } }) }) @@ -1536,20 +1205,11 @@ // preserved, so toggling back to edit keeps them). Tracks activeDraftPath // too, covering the onMount localStorage restore landing after a view load. $effect(() => { - if (mode === 'view' && !includeDrafts && activeDraftPath != undefined) { - activeDraftPath = undefined + if (mode === 'view' && !includeDrafts && pe.activeDraftPath != undefined) { + pe.activeDraftPath = undefined } }) - // Selection highlights the active draft (if any) or the user's picked - // node. Non-active drafts render without selection highlight but are - // still clickable to re-enter their edit pane. - let effectiveSelection = $derived( - activeDraftPath - ? { kind: 'runnable', runnable_kind: 'script', path: activeDraftPath } - : selection - ) - // Bumped after every successful run dispatch so AssetRunsPanel re-fetches // the listing immediately — the new (preview or script) job appears in // the history popover without waiting on its 3 s poll tick. @@ -1696,7 +1356,7 @@ // drafts (same condition as `displayGraph`). Otherwise — View mode with // drafts hidden — a bounded run must execute the *deployed* scripts the // user is looking at, not preview jobs from hidden local drafts. - const draft = mode === 'edit' || includeDrafts ? drafts.get(path) : undefined + const draft = mode === 'edit' || includeDrafts ? pe.drafts.get(path) : undefined if (draft) { if (!draft.script.content || !draft.script.language) { throw new Error(`draft ${path} has no content/language`) @@ -1961,7 +1621,7 @@ // running a draft via runScriptPreview creates a `preview`-kind job at // the same path, which the panel's listing query picks up. let selectionProducers = $derived.by(() => { - const sel = selection + const sel = pe.selection if (!sel || sel.kind !== 'asset') return [] return graphWithDraft.edges .filter((e) => { @@ -1991,7 +1651,7 @@ // ducklake-asset selection so it isn't rebuilt on every editor keystroke when // the trace UI isn't even shown. let columnGraph = $derived( - selection?.kind === 'asset' && selection.asset_kind === 'ducklake' + pe.selection?.kind === 'asset' && pe.selection.asset_kind === 'ducklake' ? buildColumnGraph(displayGraph) : EMPTY_COLUMN_GRAPH ) @@ -2008,7 +1668,7 @@ // without it) is treated as unknown → evolvable, so captured history is never // hidden behind a stale "fixed" verdict. let schemaCanEvolve = $derived.by(() => { - const sel = selection + const sel = pe.selection if (!sel || sel.kind !== 'asset' || sel.asset_kind !== 'ducklake') return true const producerPaths = new Set(selectionProducers.map((p) => p.path)) const producers = graphWithDraft.runnables.filter((r) => producerPaths.has(r.path)) @@ -2068,7 +1728,7 @@ // flow. Drafts have no deployed endpoint yet, so nudge the user to save // first (mirrors openMissingTriggerDrawer). function openWebhookDrawer(scriptPath: string) { - if (drafts.has(scriptPath)) { + if (pe.drafts.has(scriptPath)) { sendUserToast( `Save the script "${scriptPath}" first — webhooks only trigger the deployed version.`, true @@ -2090,12 +1750,12 @@ // Same mode gate as handleCanvasSelect: view mode targets the // deployed script (its run form runs the deployed version), even // when unsaved edits were promoted to a draft. - if (mode === 'edit' && drafts.has(scriptPath)) { - activeDraftPath = scriptPath - selection = undefined + if (mode === 'edit' && pe.drafts.has(scriptPath)) { + pe.activeDraftPath = scriptPath + pe.selection = undefined } else { - activeDraftPath = undefined - selection = { kind: 'runnable', runnable_kind: 'script', path: scriptPath } + pe.activeDraftPath = undefined + pe.selection = { kind: 'runnable', runnable_kind: 'script', path: scriptPath } } // Bump after a tick so the pane has reacted to the new selection/draft // and begun mounting the run form before it hunts for the S3 input. @@ -2110,7 +1770,7 @@ // time or silently bind to nothing. Surface that as a toast and // keep the drawer closed; the user needs to save the script first // (which also creates it under the new path if they renamed it). - if (drafts.has(scriptPath)) { + if (pe.drafts.has(scriptPath)) { sendUserToast( `Save the script "${scriptPath}" first — triggers can only be attached to deployed scripts.`, true @@ -2239,7 +1899,7 @@ g.runnables .filter((r) => r.usage_kind === 'script') .map((r) => r.path) - .filter((p) => !drafts.has(p) && !bodiesByPath.has(p)) + .filter((p) => !pe.drafts.has(p) && !bodiesByPath.has(p)) ) if (targets.length === 0) return let i = 0 @@ -2351,8 +2011,8 @@ anchored between the two flex-1 side groups so it stays centered, with breathing room on both sides. -->
- setMode(m)} /> - {#if mode === 'view' && drafts.size > 0} + setMode(m)} /> + {#if mode === 'view' && pe.drafts.size > 0} {/if}
@@ -2410,7 +2070,7 @@ {/snippet} {/if} - {#if mode === 'edit' && drafts.size > 0} + {#if mode === 'edit' && pe.drafts.size > 0} @@ -2420,7 +2080,7 @@ itemKind="data_pipeline" path={pipelineDraftPath} draftOnly - loadedFromDraft={loadedFromDbDraft} + loadedFromDraft={pe.loadedFromDbDraft} /> {/if} {/if} {#if mode === 'view'} @@ -2470,257 +2130,152 @@ Failed to load pipeline: {graphRes.error.message} {:else} - - -
- - {#if boundPick} - -
- -
- - {boundPickEnds.size === 0 - ? 'Click end node(s) to bound the run' - : `${boundScripts.length} script${boundScripts.length === 1 ? '' : 's'} up to ${boundPickEnds.size} end${boundPickEnds.size === 1 ? '' : 's'}`} - - - from {boundPickStart ? shortPath(boundPickStart) : ''} - -
- - + (panelHidden = !panelHidden)} + {prefetchingAssets} + hoveredPaths={activityHoverPaths} + selectedRunPaths={activitySelectPaths} + {activeRunnable} + activeRunnableIds={activeRunnables.ids} + runStates={mergedRunStates} + eventLogEvents={activeRunnables.events} + {runsRefreshKey} + {runsPendingJobId} + {boundPick} + validStartPaths={isOperator ? undefined : validStartPaths} + onStartBoundedRun={isOperator ? undefined : startBoundedRun} + onPickEnd={pickBoundEnd} + {panToNodeId} + onCreateMissingTrigger={mode === 'edit' ? openMissingTriggerDrawer : undefined} + onEditTrigger={mode === 'edit' ? openEditTriggerDrawer : undefined} + onDeleteTrigger={mode === 'edit' ? deleteAttachedTrigger : undefined} + onOpenWebhook={openWebhookDrawer} + onOpenDataUpload={openDataUploadRun} + onSelect={handleCanvasSelect} + onAddScriptForAsset={mode === 'edit' ? handleAddScriptForAsset : undefined} + onAddPipelineScript={mode === 'edit' ? handleAddPipelineScript : undefined} + onRunnableMenuRemove={mode === 'edit' ? handleRunnableMenuRemove : undefined} + onRunProducer={mode === 'edit' ? handleRunProducer : undefined} + onRequestEdit={isOperator ? undefined : () => setMode('edit')} + canRunByPath={openScriptHasDataUpload} + onRunByPath={runByPathLegit} + {selectionProducers} + selectionColumnGraph={pe.activeDraft ? EMPTY_COLUMN_GRAPH : columnGraph} + {schemaCanEvolve} + downstreamSubscribers={editedScriptDownstreamCount} + onStartBoundedRunForOpen={startBoundedRun} + canBoundedRunOpenScript={!!openScriptPath && + validStartPaths.has(openScriptPath) && + lineageDownstreamPaths.has(openScriptPath)} + onRunCompleted={() => { + activeRunnable = undefined + activeRunnableJobId = undefined + }} + onTestStateChange={(running) => { + const openPath = openScriptPath + if (running && openPath) { + activeRunnable = { kind: 'script', path: openPath } + activeRunnables.arm(`script:${openPath}`) + activeRunnableJobId = undefined + } else if (!running && activeRunnable?.path === openPath) { + activeRunnable = undefined + activeRunnableJobId = undefined + } + }} + {requestRemoveSignal} + {requestRunSignal} + {requestRunCascadeSignal} + focusUploadSignal={focusDataUploadSignal} + onDraftPathChange={renameDraft} + onClose={() => { + pe.selection = undefined + pe.activeDraftPath = undefined + pe.liveAnnotations = EMPTY_LIVE_ANNOTATIONS + }} + onDiscard={() => { + if (pe.activeDraftPath) discardDraft(pe.activeDraftPath) + }} + onDraftSaved={async (savedPath) => { + const predicted = predictCascadeFacts([savedPath]) + const nextDrafts = new Map(pe.drafts) + nextDrafts.delete(savedPath) + pe.drafts = nextDrafts + if (pe.activeDraftPath === savedPath) { + pe.selection = { kind: 'runnable', runnable_kind: 'script', path: savedPath } + pe.activeDraftPath = undefined + } + clearSaveError(savedPath) + await graphRes.refetch() + reportDeployDrift(predicted) + }} + onPersistedSaved={async (savedPath) => { + const predicted = predictCascadeFacts([savedPath]) + await graphRes.refetch() + reportDeployDrift(predicted) + }} + onScriptRenamed={async (oldPath, newPath) => { + if (pe.selection?.kind === 'runnable' && pe.selection.path === oldPath) { + pe.selection = { ...pe.selection, path: newPath } + } + await graphRes.refetch() + }} + onScriptRemoved={async (removedPath) => { + forgetPath(removedPath) + await graphRes.refetch() + }} + > + {#snippet boundBar()} + {#if boundPick} +
+ +
+ + {boundPickEnds.size === 0 + ? 'Click end node(s) to bound the run' + : `${boundScripts.length} script${boundScripts.length === 1 ? '' : 's'} up to ${boundPickEnds.size} end${boundPickEnds.size === 1 ? '' : 's'}`} + + + from {boundPickStart ? shortPath(boundPickStart) : ''} +
- {/if} - {#if mode === 'edit'} - - - {/if} - {#if prefetchingAssets} -
Cancel +
- {/if} - {#if mode !== 'edit' || selection != undefined || activeDraftPath != undefined} - -
-
- {/if} -
- {#if detailsPaneOpen && $workspaceStore} - - {#if mode !== 'edit' && selection == undefined && activeDraftPath == undefined} - - (activityHoverPaths = p ?? [])} - onSelectRun={(p) => (activitySelectPaths = p ?? [])} - /> - {:else} - setMode('edit')} - canRunByPath={openScriptHasDataUpload} - onRunByPath={runByPathLegit} - selection={activeDraft ? undefined : selection} - selectionProducers={activeDraft ? [] : selectionProducers} - selectionColumnGraph={activeDraft ? EMPTY_COLUMN_GRAPH : columnGraph} - {schemaCanEvolve} - {runsRefreshKey} - {runsPendingJobId} - {activeRunnable} - downstreamSubscribers={editedScriptDownstreamCount} - onStartBoundedRun={openScriptPath && - validStartPaths.has(openScriptPath) && - lineageDownstreamPaths.has(openScriptPath) - ? () => startBoundedRun(openScriptPath!) - : undefined} - onRunCompleted={() => { - activeRunnable = undefined - activeRunnableJobId = undefined - }} - onTestStateChange={(running) => { - // Bridge: ScriptEditor's Test button triggers the - // same canvas-level "is running" hint as the - // per-node Run button. The currently-edited script - // is whichever path is open in the pane (active - // draft, or the persisted-script selection). - const openPath = openScriptPath - if (running && openPath) { - activeRunnable = { kind: 'script', path: openPath } - // Mark the tested runnable as launched-from-here so the - // folder poll's catch-up pulse won't re-flash its edge a - // poll-interval after a fast job already finished (the - // edge is animated zero-latency by `activeRunnable`, and - // the test loader clears that the instant it completes). - // Also upgrades to the fast poll so the badge lands sooner. - activeRunnables.arm(`script:${openPath}`) - // Editor Test path clears via its own callbacks, not - // the job-id effect — drop any stale tracked id so a - // prior canvas run's completion can't clear this hint. - activeRunnableJobId = undefined - } else if (!running && activeRunnable?.path === openPath) { - activeRunnable = undefined - activeRunnableJobId = undefined - } - }} - {requestRemoveSignal} - {requestRunSignal} - {requestRunCascadeSignal} - focusUploadSignal={focusDataUploadSignal} - draftScript={activeDraft?.script} - {pathPrefix} - onDraftPathChange={renameDraft} - workspace={$workspaceStore} - onAnnotationsChange={handleAnnotationsChange} - onAssetsChange={handleAssetsChange} - onContentChange={handleContentChange} - onDraftPersist={handleDraftPersist} - onclose={() => { - // Close dismisses the pane but preserves drafts so - // the user can come back to them. Discarding is - // via the explicit "Discard" button in the pane. - selection = undefined - activeDraftPath = undefined - liveAnnotations = EMPTY_LIVE_ANNOTATIONS - }} - onHide={() => (panelHidden = true)} - onDiscard={() => { - if (activeDraftPath) discardDraft(activeDraftPath) - }} - onDraftSaved={async (savedPath) => { - // Snapshot the preview's promise while the draft - // overlay still exists (dropped from `drafts` below). - const predicted = predictCascadeFacts([savedPath]) - // Drop the now-deployed draft and hand focus to its - // persisted runnable so the pane stays open on the - // same script. `discardDraft` would clear - // activeDraftPath without setting selection — the - // canvas would deselect and the view reset on the - // next refetch. - const nextDrafts = new Map(drafts) - nextDrafts.delete(savedPath) - drafts = nextDrafts - if (activeDraftPath === savedPath) { - selection = { - kind: 'runnable', - runnable_kind: 'script', - path: savedPath - } - activeDraftPath = undefined - } - clearSaveError(savedPath) - await graphRes.refetch() - reportDeployDrift(predicted) - }} - onPersistedSaved={async (savedPath) => { - // Snapshot before the refetch replaces the base graph - // — the live editor overlay is the prediction here. - const predicted = predictCascadeFacts([savedPath]) - // Refresh the asset graph so the rows the deploy - // just inserted (from the body-asset write list we - // pass at save time) make it into base.edges. The - // in-memory `inferredWritesByPath` overlay - // dedupes against base, so the edge stays put - // instead of flickering when the ScriptEditor - // remounts on the new hash. - await graphRes.refetch() - reportDeployDrift(predicted) - }} - onScriptRenamed={async (oldPath, newPath) => { - // Repoint the selection at the new path before the - // graph refetches so the pane stays focused on the - // same script. Order matters: update selection - // first, then refetch — otherwise the resource - // driving the pane would briefly resolve to nothing. - if (selection?.kind === 'runnable' && selection.path === oldPath) { - selection = { ...selection, path: newPath } - } - await graphRes.refetch() - }} - onScriptRemoved={async (removedPath) => { - // Drop every path-keyed overlay / cache entry - // pointing at the now-archived runnable so - // resolveGraph doesn't keep emitting lineage - // edges or missing-trigger placeholders against - // a script that no longer exists. Without this - // the inferred writes / annotation maps would - // keep dragging phantom nodes onto the canvas - // until the next folder change. - forgetPath(removedPath) - await graphRes.refetch() - }} - /> - {/if} - - {/if} - + Run selection + +
+ {/if} + {/snippet} + {#snippet idlePane()} + (activityHoverPaths = p ?? [])} + onSelectRun={(p) => (activitySelectPaths = p ?? [])} + /> + {/snippet} + {/if}
@@ -2761,18 +2316,18 @@

- {drafts.size === 1 ? 'Unsaved draft' : `${drafts.size} unsaved drafts`} + {pe.drafts.size === 1 ? 'Unsaved draft' : `${pe.drafts.size} unsaved drafts`}

- You have {drafts.size === 1 + You have {pe.drafts.size === 1 ? 'a draft pipeline script' - : `${drafts.size} draft pipeline scripts`} that {drafts.size === 1 + : `${pe.drafts.size} draft pipeline scripts`} that {pe.drafts.size === 1 ? 'has' : 'have'} not been deployed yet. What would you like to do?

    - {#each [...drafts.keys()] as p} + {#each [...pe.drafts.keys()] as p}
  • {p}
    • {/each}
@@ -2787,7 +2342,9 @@ unifiedSize="sm" startIcon={{ icon: leaveSaving ? Loader2 : Save }} > - {leaveSaving ? 'Saving…' : `Save all (${drafts.size})`} + {leaveSaving ? 'Saving…' : `Save all (${pe.drafts.size})`}
\`). Deploy **rejects** \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`) or a non-DuckLake target. For a non-DuckDB node, do **not** use \`// materialize\` — write the output via the SDK (\`wmill.writeS3File(...)\`, a postgresql \`CREATE TABLE\`, ducklake helpers, …) and let it be inferred. Use \`duckdb\` when a node should materialize a DuckLake table. + +\`// materialize \` tells the runtime to write the node's output table **for you**: write the body as a single \`SELECT\` and the runtime wraps it in the create/replace — do **not** also write your own \`CREATE TABLE\` / \`INSERT\`. Write strategy: + +- no option → **replace** the whole table each run (full refresh; the only mode whose output columns may change); +- \`// materialize append\` → INSERT-append rows (incremental); +- \`// materialize key=\` → merge/upsert on \`\`. + +\`// materialize manual \` opts **out** of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. + +\`materialize\` pairs with partitioning for incremental pipelines: a \`// partitioned \` node runs **once per partition** (append/merge into a fixed-schema table), and the \`{partition}\` token inside any asset URI is substituted with the current partition value at run time. + +\`materialize\` is an output **declaration** on a node — not a command. There is no "materialize run". + +## How to build one in chat + +1. Put every node in the **same folder**: \`f//\`. The folder is the pipeline. +2. Author each node as a **script draft** with \`write_script\` (or \`edit_script\`), language chosen for the work: \`duckdb\` or \`postgresql\` for SQL-shaped data work, \`bun\`/\`python3\` for general transforms. SQL-heavy lakehouse steps usually use \`duckdb\`. +3. Start each body with \`// pipeline\`, then the \`// on\` input declarations, then the transform that writes the output. +4. **Chain nodes by asset URI**: read an upstream node's output asset, then \`// on \` in the downstream node so the edge forms. Reuse exact asset paths from existing nodes rather than inventing parallel ones. +5. Leave nodes as drafts unless the user asks to deploy. A pipeline only "runs" once its scripts are deployed and their triggers exist. + +When the user already has the \`/pipeline/\` editor open, prefer the dedicated \`build_pipeline_node\` / \`edit_pipeline_node\` tools (they stage reviewable, canvas-highlighted proposals). Outside the editor, use the standard script-draft tools with the annotations above. + +## Example (DuckDB → DuckLake, scheduled ingest + downstream transform) + +Node \`f/sales/orders_ingest\` (runs on a schedule, materializes a DuckLake table): + +\`\`\`sql +-- pipeline +-- on schedule +-- materialize ducklake://main/orders +SELECT * FROM read_csv('s3://raw/orders/*.csv') +\`\`\` + +Node \`f/sales/orders_daily\` (runs when \`orders\` is produced, writes a rollup): + +\`\`\`sql +-- pipeline +-- on ducklake://main/orders +-- materialize ducklake://main/orders_daily +SELECT date_trunc('day', ts) AS day, count(*) AS n +FROM ducklake.main.orders GROUP BY 1 +\`\`\` +`; + export const WORKFLOW_AS_CODE_BASE = `# Windmill Workflow-as-Code Writing Guide ## Scope diff --git a/system_prompts/base/pipeline-base.md b/system_prompts/base/pipeline-base.md new file mode 100644 index 0000000000..a0f2e2492a --- /dev/null +++ b/system_prompts/base/pipeline-base.md @@ -0,0 +1,60 @@ +# Data pipeline authoring + +A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates steps internally. A data pipeline is a set of **independent scripts**, each deployed on its own, that form a DAG by reading and writing shared **storage assets** (DuckLake tables, data tables, S3 objects, volumes, resources) and by declaring execution **triggers**. The pipeline is visualized and edited at `/pipeline/`; every node is a normal workspace script that happens to carry pipeline annotations. When the user asks for a "data pipeline" (or to "ingest / transform / materialize" data across steps), build pipeline-annotated scripts — do NOT build a flow. + +## What makes a script a pipeline node + +A script joins the pipeline when its source begins with the `pipeline` annotation as a top-of-file comment, **written in the script's own comment syntax** — `//` for TS/JS (bun), `--` for SQL (DuckDB/Postgres), `#` for Python/Bash. So it's `-- pipeline` in a DuckDB node, `# pipeline` in a Python node, `// pipeline` in a bun node. Every annotation below uses that same prefix (the `//` shown is the TS form). All other wiring is expressed as annotation comments near the top of the file: + +- `// on ` — declares an execution-DAG **input** (what triggers/feeds this node). `` is either: + - an **asset URI** (the node runs when that asset is produced upstream): `ducklake://main/orders`, `datatable://main/users`, `s3://`, `$res:f/folder/my_resource`, `volume://name/path`. + - a **native trigger kind**: `schedule`, `webhook`, `email`, `kafka`, `mqtt`, `nats`, `postgres`, `sqs`, `gcp`, or `data_upload` (a user-uploaded S3 file). For these the actual trigger row (cron, topic, …) is created separately; the annotation only declares the binding. +- **Outputs** are inferred from what the body writes — a `CREATE TABLE`, a `wmill.writeS3File(...)`, a DuckLake/datatable write. To declare a managed output explicitly, use `// materialize `. +- Optional badges: `// partitioned `, `// freshness ` (e.g. `1h`), `// tag `, `// retry [delay]`, `// data_test ...`. + +## Materialize (the managed output) + +> **`// materialize` is DuckDB-only**, and its target must be a DuckLake table (`ducklake:///
`). Deploy **rejects** `// materialize` on any other language (`python3`, `bun`, `postgresql`) or a non-DuckLake target. For a non-DuckDB node, do **not** use `// materialize` — write the output via the SDK (`wmill.writeS3File(...)`, a postgresql `CREATE TABLE`, ducklake helpers, …) and let it be inferred. Use `duckdb` when a node should materialize a DuckLake table. + +`// materialize ` tells the runtime to write the node's output table **for you**: write the body as a single `SELECT` and the runtime wraps it in the create/replace — do **not** also write your own `CREATE TABLE` / `INSERT`. Write strategy: + +- no option → **replace** the whole table each run (full refresh; the only mode whose output columns may change); +- `// materialize append` → INSERT-append rows (incremental); +- `// materialize key=` → merge/upsert on ``. + +`// materialize manual ` opts **out** of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. + +`materialize` pairs with partitioning for incremental pipelines: a `// partitioned ` node runs **once per partition** (append/merge into a fixed-schema table), and the `{partition}` token inside any asset URI is substituted with the current partition value at run time. + +`materialize` is an output **declaration** on a node — not a command. There is no "materialize run". + +## How to build one in chat + +1. Put every node in the **same folder**: `f//`. The folder is the pipeline. +2. Author each node as a **script draft** with `write_script` (or `edit_script`), language chosen for the work: `duckdb` or `postgresql` for SQL-shaped data work, `bun`/`python3` for general transforms. SQL-heavy lakehouse steps usually use `duckdb`. +3. Start each body with `// pipeline`, then the `// on` input declarations, then the transform that writes the output. +4. **Chain nodes by asset URI**: read an upstream node's output asset, then `// on ` in the downstream node so the edge forms. Reuse exact asset paths from existing nodes rather than inventing parallel ones. +5. Leave nodes as drafts unless the user asks to deploy. A pipeline only "runs" once its scripts are deployed and their triggers exist. + +When the user already has the `/pipeline/` editor open, prefer the dedicated `build_pipeline_node` / `edit_pipeline_node` tools (they stage reviewable, canvas-highlighted proposals). Outside the editor, use the standard script-draft tools with the annotations above. + +## Example (DuckDB → DuckLake, scheduled ingest + downstream transform) + +Node `f/sales/orders_ingest` (runs on a schedule, materializes a DuckLake table): + +```sql +-- pipeline +-- on schedule +-- materialize ducklake://main/orders +SELECT * FROM read_csv('s3://raw/orders/*.csv') +``` + +Node `f/sales/orders_daily` (runs when `orders` is produced, writes a rollup): + +```sql +-- pipeline +-- on ducklake://main/orders +-- materialize ducklake://main/orders_daily +SELECT date_trunc('day', ts) AS day, count(*) AS n +FROM ducklake.main.orders GROUP BY 1 +``` diff --git a/system_prompts/generate.py b/system_prompts/generate.py index 6ee1971f43..98c03c4367 100644 --- a/system_prompts/generate.py +++ b/system_prompts/generate.py @@ -2369,6 +2369,7 @@ def main(): flow_base = read_markdown_file(base_dir / "flow-base.md") resources_base = read_markdown_file(base_dir / "resources.md") raw_app_base = read_markdown_file(base_dir / "raw-app.md") + pipeline_base = read_markdown_file(base_dir / "pipeline-base.md") workflow_as_code_base = read_markdown_file(base_dir / "workflow-as-code.md") flow_cli = read_markdown_file(base_dir / "flow-cli.md") flow_chat_special_modules = read_markdown_file(base_dir / "flow-chat-special-modules.md") @@ -2435,6 +2436,7 @@ def main(): 'FLOW_BASE': flow_base, 'RESOURCES_BASE': resources_base, 'RAW_APP_BASE': raw_app_base, + 'PIPELINE_BASE': pipeline_base, 'WORKFLOW_AS_CODE_BASE': workflow_as_code_base, 'FLOW_CHAT_SPECIAL_MODULES': flow_chat_special_modules, @@ -2534,6 +2536,11 @@ export function getRawAppPrompt(): string { return prompts.RAW_APP_BASE; } +// Helper for data pipeline authoring (chat consumers) +export function getPipelinePrompt(): string { + return prompts.PIPELINE_BASE; +} + // Helper to get the datatable SQL SDK reference (wmill.datatable()). // Pass a language to get only that SDK; omit it to get both. export function getDatatableSdkReference(language?: string): string { @@ -2586,6 +2593,7 @@ export declare function getScriptPrompt(language: string): string; export declare function getFlowPrompt(): string; export declare function getResourcePrompt(): string; export declare function getRawAppPrompt(): string; +export declare function getPipelinePrompt(): string; export declare function getDatatableSdkReference(language?: string): string; export declare function getWorkflowAsCodePrompt(language?: string): string; """ From 9b65161c643bf3f120d2ebd82f786c17233a971b Mon Sep 17 00:00:00 2001 From: hugocasa Date: Mon, 29 Jun 2026 22:37:39 +0200 Subject: [PATCH 02/21] fix(gcp): require token verification for authenticated push delivery (#9834) * fix(gcp): require token verification for authenticated push delivery Co-Authored-By: Claude Opus 4.8 (1M context) * chore: update ee-repo-ref to 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2 This commit updates the EE repository reference after PR #636 was merged in windmill-ee-private. Previous ee-repo-ref: 8c63d487c486002baf09c77ab937fd77a91765eb New ee-repo-ref: 38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: windmill-internal-app[bot] --- backend/ee-repo-ref.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index ce3db8ebcb..4dd9981afc 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -95352c13c4c82247d8cfd80936f9203aeb079802 +38e87caeca6a1dce9e4f3fa029ac36dffb30f1b2 From a9ffdb996b418ad51a84973ff01dcd3f2c820f16 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Mon, 29 Jun 2026 22:41:38 +0200 Subject: [PATCH 03/21] chore(main): release 1.743.0 (#9837) * chore(main): release 1.743.0 * Apply automatic changes --------- Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com> --- CHANGELOG.md | 13 ++ backend/Cargo.lock | 172 +++++++++--------- backend/Cargo.toml | 4 +- .../parsers/windmill-parser-wasm/Cargo.lock | 48 ++--- .../parsers/windmill-parser-wasm/Cargo.toml | 2 +- backend/windmill-api/openapi.yaml | 2 +- benchmarks/lib.ts | 2 +- cli/src/core/constants.ts | 2 +- frontend/package-lock.json | 4 +- frontend/package.json | 2 +- lsp/Pipfile | 2 +- openflow.openapi.yaml | 2 +- .../WindmillClient/WindmillClient.psd1 | 2 +- python-client/wmill/pyproject.toml | 2 +- typescript-client/jsr.json | 2 +- typescript-client/package.json | 2 +- version.txt | 2 +- 17 files changed, 139 insertions(+), 126 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 361b6fbd91..77316bdff8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## [1.743.0](https://github.com/windmill-labs/windmill/compare/v1.742.0...v1.743.0) (2026-06-29) + + +### Features + +* **home:** redesign create-new popover and home header ([#9827](https://github.com/windmill-labs/windmill/issues/9827)) ([2493eaf](https://github.com/windmill-labs/windmill/commit/2493eaf031f30072637a297398674e761f039005)) +* **pipeline:** AI-chat data-pipeline editor (route + in-session) + home surfacing ([#9805](https://github.com/windmill-labs/windmill/issues/9805)) ([c910278](https://github.com/windmill-labs/windmill/commit/c91027824be1f1f49cdd14148baf6aad092a1dd0)) + + +### Bug Fixes + +* **gcp:** require token verification for authenticated push delivery ([#9834](https://github.com/windmill-labs/windmill/issues/9834)) ([9b65161](https://github.com/windmill-labs/windmill/commit/9b65161c643bf3f120d2ebd82f786c17233a971b)) + ## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28) diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 7072389c21..c154084c51 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -195,9 +195,9 @@ dependencies = [ [[package]] name = "arc-swap" -version = "1.9.1" +version = "1.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" +checksum = "c049c0be4daef0b145cb3555416b3b8ef5b7888a38aea1a3a155801fe7b0810b" dependencies = [ "rustversion", ] @@ -2601,9 +2601,9 @@ dependencies = [ [[package]] name = "curl-sys" -version = "0.4.89+curl-8.20.0" +version = "0.4.90+curl-8.21.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6d680779285438f2d0927485973ab45b212ea990bddb80de8a55a1e3c1d9ba22" +checksum = "97799a0d220bfb3361e0fe4936966ff8c4b24d65c3f06dfc70d7b680b44e7897" dependencies = [ "cc", "libc", @@ -6085,9 +6085,9 @@ dependencies = [ [[package]] name = "io-uring" -version = "0.7.12" +version = "0.7.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4d09b98f7eace8982db770e4408e7470b028ce513ac28fecdc6bf4c30fe92b62" +checksum = "9080b15e63775b9a2ac7dca720f7050a8b955e092ea0f6020a4a80f69998cdc0" dependencies = [ "bitflags 2.13.0", "cfg-if", @@ -12922,9 +12922,9 @@ dependencies = [ [[package]] name = "triomphe" -version = "0.1.15" +version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dd69c5aa8f924c7519d6372789a74eac5b94fb0f8fcf0d4a97eb0bfc3e785f39" +checksum = "b40688ea6389c8171614b25491f71d4a27946e0c7ce2da1c6de27e25abf1a0ae" dependencies = [ "serde", "stable_deref_trait", @@ -13734,7 +13734,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-nats", @@ -13816,7 +13816,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.742.0" +version = "1.743.0" dependencies = [ "async-stream", "async-trait", @@ -13849,7 +13849,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13862,7 +13862,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "argon2", @@ -14000,7 +14000,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14023,7 +14023,7 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14038,7 +14038,7 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14064,7 +14064,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.742.0" +version = "1.743.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14074,7 +14074,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14091,7 +14091,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14113,7 +14113,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14136,7 +14136,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14152,7 +14152,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14173,7 +14173,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14194,7 +14194,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14208,7 +14208,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-nats", @@ -14243,7 +14243,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14268,7 +14268,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14286,7 +14286,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14308,7 +14308,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14328,7 +14328,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14365,7 +14365,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14393,7 +14393,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.742.0" +version = "1.743.0" dependencies = [ "lazy_static", "serde", @@ -14405,7 +14405,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.742.0" +version = "1.743.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14430,7 +14430,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14444,7 +14444,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.742.0" +version = "1.743.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14477,7 +14477,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.742.0" +version = "1.743.0" dependencies = [ "chrono", "lazy_static", @@ -14491,7 +14491,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14510,7 +14510,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.742.0" +version = "1.743.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14612,7 +14612,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.742.0" +version = "1.743.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14631,7 +14631,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.742.0" +version = "1.743.0" dependencies = [ "regex", "serde", @@ -14646,7 +14646,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14670,7 +14670,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "futures", @@ -14687,7 +14687,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.742.0" +version = "1.743.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14703,7 +14703,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -14724,7 +14724,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -14755,7 +14755,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "arc-swap", @@ -14780,7 +14780,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-stream", @@ -14814,7 +14814,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "futures", @@ -14832,7 +14832,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.742.0" +version = "1.743.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14841,7 +14841,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -14853,7 +14853,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -14865,7 +14865,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "gosyn", @@ -14877,7 +14877,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -14889,7 +14889,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -14901,7 +14901,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "nu-parser", @@ -14912,7 +14912,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14923,7 +14923,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14935,7 +14935,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14946,7 +14946,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-recursion", @@ -14968,7 +14968,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -14980,7 +14980,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -14994,7 +14994,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15011,7 +15011,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -15024,7 +15024,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde", @@ -15036,7 +15036,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -15054,7 +15054,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15070,7 +15070,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15086,7 +15086,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde", @@ -15097,7 +15097,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-recursion", @@ -15136,7 +15136,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "const_format", @@ -15175,7 +15175,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.742.0" +version = "1.743.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15186,7 +15186,7 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-recursion", @@ -15220,7 +15220,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15244,7 +15244,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15277,7 +15277,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15310,7 +15310,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15330,7 +15330,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15364,7 +15364,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15400,7 +15400,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15423,7 +15423,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15447,7 +15447,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-nats", @@ -15471,7 +15471,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15506,7 +15506,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15534,7 +15534,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-trait", @@ -15559,7 +15559,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15578,7 +15578,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-once-cell", @@ -15688,7 +15688,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.742.0" +version = "1.743.0" dependencies = [ "bytes", "futures", diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 45682215cf..725f460df1 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.742.0" +version = "1.743.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.742.0" +version = "1.743.0" authors = ["Ruben Fiszel "] edition = "2021" diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 096d93163b..31e3ee26b6 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.742.0" +version = "1.743.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.742.0" +version = "1.743.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.742.0" +version = "1.743.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.742.0" +version = "1.743.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index e2afd2483b..62faf03ec3 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.742.0" +version = "1.743.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index f9a15ef72d..18b1914826 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.742.0 + version: 1.743.0 title: Windmill API contact: diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index f8cc748bdb..38995ac0e9 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.742.0"; +export const VERSION = "v1.743.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/core/constants.ts b/cli/src/core/constants.ts index 42d8109c49..7be2d4df35 100644 --- a/cli/src/core/constants.ts +++ b/cli/src/core/constants.ts @@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork"; // (e.g. utils.ts) can read it without importing main.ts and creating a circular // dependency (main → workspace → utils → main) that triggers a TDZ. // Re-exported from main.ts for backwards compatibility. -export const VERSION = "1.742.0"; +export const VERSION = "1.743.0"; diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 4a3fcf0696..0d66dfed67 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -1,12 +1,12 @@ { "name": "@windmill-labs/components", - "version": "1.742.0", + "version": "1.743.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@windmill-labs/components", - "version": "1.742.0", + "version": "1.743.0", "hasInstallScript": true, "license": "AGPL-3.0", "dependencies": { diff --git a/frontend/package.json b/frontend/package.json index e9ec640b02..0c122ffac7 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -1,6 +1,6 @@ { "name": "@windmill-labs/components", - "version": "1.742.0", + "version": "1.743.0", "scripts": { "dev": "vite dev", "dev:ui-builder": "mv static/ui_builder static/ui_builder.dev-disabled 2>/dev/null || true ; trap 'mv static/ui_builder.dev-disabled static/ui_builder 2>/dev/null || true' EXIT ; vite dev", diff --git a/lsp/Pipfile b/lsp/Pipfile index 7f9d00903a..1dc2caa039 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.742.0" +wmill = ">=1.743.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index f4342930e9..b2b394c836 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.742.0 + version: 1.743.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index e5b553e6f3..500f453c30 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.742.0' + ModuleVersion = '1.743.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index 5310b72896..508f2376c1 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.742.0" +version = "1.743.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 7ae2023a7c..650ccf5be0 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.742.0", + "version": "1.743.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index 8edc248c45..0a22c432d7 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.742.0", + "version": "1.743.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index bee795f5f0..9926885433 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.742.0 +1.743.0 From a27e814a03c615259381eaf684aa90d56569b0af Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 30 Jun 2026 10:10:31 +0200 Subject: [PATCH 04/21] feat: add copy-to-clipboard button to rendered Mermaid diagrams in AI chat (#9838) MermaidDisplay only showed the rendered SVG, hiding the raw source once rendering succeeded. Add a copy button in the showSvg branch mirroring the pattern in HighlightCode.svelte so the diagram source can be extracted. Fixes WIN-2109 Co-authored-by: Claude Opus 4.8 (1M context) --- .../copilot/chat/script/MermaidDisplay.svelte | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte index 06bcb63d98..4adf86e76d 100644 --- a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte @@ -1,6 +1,9 @@ {#if showSvg} -
- - {@html svg} +
+
{:else} From a37a144e81cf6b3de935688a617e9d0e1756004a Mon Sep 17 00:00:00 2001 From: hugocasa Date: Tue, 30 Jun 2026 17:49:57 +0200 Subject: [PATCH 05/21] fix(ai-chat): replay anthropic turns verbatim to keep thinking valid (#9843) Co-authored-by: Claude Opus 4.8 (1M context) --- .../components/copilot/chat/anthropic.test.ts | 164 ++++++++++++++++++ .../lib/components/copilot/chat/anthropic.ts | 94 +++++++--- 2 files changed, 230 insertions(+), 28 deletions(-) create mode 100644 frontend/src/lib/components/copilot/chat/anthropic.test.ts diff --git a/frontend/src/lib/components/copilot/chat/anthropic.test.ts b/frontend/src/lib/components/copilot/chat/anthropic.test.ts new file mode 100644 index 0000000000..9c9a4d0bb0 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/anthropic.test.ts @@ -0,0 +1,164 @@ +import { describe, expect, it, vi } from 'vitest' +import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' +import { convertOpenAIToAnthropicMessages } from './anthropic' + +// anthropic.ts pulls in the chat client/registry layer at import time; the +// converter under test is pure, so stub those side-effecting modules away. +vi.mock('../lib', () => ({ + getProviderAndCompletionConfig: vi.fn(), + workspaceAIClients: {} +})) + +vi.mock('../reasoningRegistry', () => ({ + applyReasoningToConfig: vi.fn() +})) + +vi.mock('./shared', () => ({ + processToolCall: vi.fn() +})) + +describe('convertOpenAIToAnthropicMessages', () => { + it('replays a captured assistant turn verbatim, skips the standalone text copy, and leaves the turn untouched', () => { + const anthropicContent = [ + { type: 'thinking', thinking: 'first', signature: 'sig-1' }, + { + type: 'server_tool_use', + id: 'srv_1', + name: 'web_search', + input: { query: 'nist password length' } + }, + { + type: 'web_search_tool_result', + tool_use_id: 'srv_1', + content: [{ type: 'web_search_result', title: 'NIST', url: 'https://nist.gov' }] + }, + { type: 'thinking', thinking: 'second', signature: 'sig-2' }, + { type: 'tool_use', id: 'tool_1', name: 'list_resources', input: {} } + ] + // Snapshot to assert the stored content is never mutated by the converter. + const anthropicContentSnapshot = JSON.parse(JSON.stringify(anthropicContent)) + + const messages: ChatCompletionMessageParam[] = [ + { role: 'user', content: 'find the nist length then list resources' }, + // Standalone text the streamer emits before the tool-call message. + { role: 'assistant', content: 'Let me search the web.' }, + { + role: 'assistant', + tool_calls: [ + { + id: 'tool_1', + type: 'function', + function: { name: 'list_resources', arguments: '{}' } + } + ], + _anthropicContent: anthropicContent + } as any, + { role: 'tool', tool_call_id: 'tool_1', content: 'resource A, resource B' } + ] + + const { messages: out } = convertOpenAIToAnthropicMessages(messages) + + // user + the verbatim assistant turn + the tool result; the standalone text is dropped. + expect(out).toHaveLength(3) + expect(out[0]).toEqual({ role: 'user', content: 'find the nist length then list resources' }) + + // Assistant turn replayed in original order, no reordering or dropped blocks. + expect(out[1].role).toBe('assistant') + expect((out[1].content as any[]).map((b) => b.type)).toEqual([ + 'thinking', + 'server_tool_use', + 'web_search_tool_result', + 'thinking', + 'tool_use' + ]) + // The verbatim turn must stay byte-identical — no cache_control injected into it, + // or a thinking-block signature would no longer validate. + expect(out[1].content).toEqual(anthropicContentSnapshot) + expect((out[1].content as any[]).some((b) => 'cache_control' in b)).toBe(false) + expect(anthropicContent).toEqual(anthropicContentSnapshot) + + // The cache breakpoint lands on the trailing tool result, not the assistant turn. + expect(out[2]).toEqual({ + role: 'user', + content: [ + { + type: 'tool_result', + tool_use_id: 'tool_1', + content: 'resource A, resource B', + cache_control: { type: 'ephemeral' } + } + ] + }) + }) + + it('converts a plain text assistant turn (no tools) and caches the trailing text block', () => { + const messages: ChatCompletionMessageParam[] = [ + { role: 'user', content: 'hello' }, + { role: 'assistant', content: 'hi there' } + ] + + const { messages: out } = convertOpenAIToAnthropicMessages(messages) + + expect(out).toHaveLength(2) + expect(out[0]).toEqual({ role: 'user', content: 'hello' }) + expect(out[1].role).toBe('assistant') + expect(out[1].content).toEqual([ + { type: 'text', text: 'hi there', cache_control: { type: 'ephemeral' } } + ]) + }) + + it('falls back to _anthropicThinkingBlocks for turns persisted before _anthropicContent', () => { + const thinkingBlocks = [{ type: 'thinking', thinking: 'reasoning', signature: 'sig-old' }] + + const messages: ChatCompletionMessageParam[] = [ + { role: 'user', content: 'do a thing' }, + // The standalone text persisted alongside an old-style turn must NOT be skipped + // (the fallback reconstruction relies on it for the assistant text). + { role: 'assistant', content: 'working on it' }, + { + role: 'assistant', + tool_calls: [ + { + id: 'tool_old', + type: 'function', + function: { name: 'list_resources', arguments: '{}' } + } + ], + _anthropicThinkingBlocks: thinkingBlocks + } as any + ] + + const { messages: out } = convertOpenAIToAnthropicMessages(messages) + + expect(out).toHaveLength(3) + expect(out[1]).toEqual({ role: 'assistant', content: 'working on it' }) + const content = out[2].content as any[] + // Thinking block re-injected first, then the tool_use. + expect(content.map((b) => b.type)).toEqual(['thinking', 'tool_use']) + expect(content[0]).toEqual(thinkingBlocks[0]) + expect(content[1]).toMatchObject({ type: 'tool_use', id: 'tool_old', name: 'list_resources' }) + }) + + it('caches a trailing tool result even when the prior turn used no captured content', () => { + const messages: ChatCompletionMessageParam[] = [ + { role: 'user', content: 'q' }, + { + role: 'assistant', + tool_calls: [ + { id: 't1', type: 'function', function: { name: 'list_resources', arguments: '{}' } } + ] + } as any, + { role: 'tool', tool_call_id: 't1', content: 'done' } + ] + + const { messages: out } = convertOpenAIToAnthropicMessages(messages) + + const last = out[out.length - 1] + expect(last.role).toBe('user') + expect((last.content as any[])[0]).toMatchObject({ + type: 'tool_result', + tool_use_id: 't1', + cache_control: { type: 'ephemeral' } + }) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/anthropic.ts b/frontend/src/lib/components/copilot/chat/anthropic.ts index 48b068039f..b6e7e57a17 100644 --- a/frontend/src/lib/components/copilot/chat/anthropic.ts +++ b/frontend/src/lib/components/copilot/chat/anthropic.ts @@ -286,15 +286,15 @@ export async function parseAnthropicCompletion( role: 'assistant', tool_calls: toolCallsToProcess } - // Preserve thinking blocks (with signatures) so the next request keeps the - // reasoning chain — Anthropic requires this when thinking is combined with tool - // use. They are re-injected by convertOpenAIToAnthropicMessages. - const thinkingBlocks = finalMessage.content.filter( - (b) => b.type === 'thinking' || b.type === 'redacted_thinking' - ) - if (thinkingBlocks.length > 0) { - ;(assistantWithTools as any)._anthropicThinkingBlocks = thinkingBlocks - } + // Preserve the assistant turn verbatim (thinking/redacted_thinking with their + // signatures, server_tool_use + web_search_tool_result, text and tool_use) in + // original order. Anthropic binds each thinking block's signature to the blocks + // that precede it in the latest assistant message, so when this turn is replayed + // to continue past its tool call it must be byte-identical: reordering thinking to + // the front or dropping the web-search blocks invalidates a later block's + // signature and the request 400s with "thinking blocks ... cannot be modified". + // convertOpenAIToAnthropicMessages replays this content as-is. + ;(assistantWithTools as any)._anthropicContent = finalMessage.content messages.push(assistantWithTools) addedMessages.push(assistantWithTools) @@ -323,7 +323,33 @@ export function convertOpenAIToAnthropicMessages(messages: ChatCompletionMessage let system: TextBlockParam[] | undefined const anthropicMessages: MessageParam[] = [] - for (const message of messages) { + // A streamed assistant turn that ends in tool calls is persisted as one or more + // standalone text messages followed by the tool-call message carrying + // _anthropicContent. That text is already inside _anthropicContent (replayed verbatim + // below), so drop the standalone copies — otherwise the text is duplicated and + // emitted ahead of the turn's thinking blocks. The scan stops at the preceding + // user/tool message, so only the current turn's own text is skipped. + const skipStandaloneText = new Set() + for (let i = 0; i < messages.length; i++) { + if (!(messages[i] as any)._anthropicContent) continue + for (let j = i - 1; j >= 0; j--) { + const m = messages[j] + if ( + m.role === 'assistant' && + typeof m.content === 'string' && + !m.tool_calls && + !(m as any)._anthropicContent + ) { + skipStandaloneText.add(j) + } else { + break + } + } + } + + for (let i = 0; i < messages.length; i++) { + const message = messages[i] + if (skipStandaloneText.has(i)) continue if (message.role === 'system') { const systemText = typeof message.content === 'string' ? message.content : JSON.stringify(message.content) @@ -345,10 +371,19 @@ export function convertOpenAIToAnthropicMessages(messages: ChatCompletionMessage typeof message.content === 'string' ? message.content : JSON.stringify(message.content) }) } else if (message.role === 'assistant') { + // Replay a captured assistant turn verbatim so its thinking-block signatures + // stay valid (see the _anthropicContent note where the streamed turn is stored). + const anthropicContent = (message as any)._anthropicContent + if (Array.isArray(anthropicContent) && anthropicContent.length > 0) { + anthropicMessages.push({ role: 'assistant', content: anthropicContent as any }) + continue + } + const content: any[] = [] - // Re-inject preserved thinking blocks first (Anthropic requires thinking to - // precede tool_use in the same assistant turn when thinking is enabled). + // Fallback for sessions persisted before _anthropicContent existed: re-inject + // the preserved thinking blocks first (Anthropic requires thinking to precede + // tool_use in the same assistant turn when thinking is enabled). const thinkingBlocks = (message as any)._anthropicThinkingBlocks if (Array.isArray(thinkingBlocks) && thinkingBlocks.length > 0) { content.push(...thinkingBlocks) @@ -404,26 +439,29 @@ export function convertOpenAIToAnthropicMessages(messages: ChatCompletionMessage } } - // Add cache_control to the last message content blocks + // Cache the conversation prefix: put an ephemeral breakpoint on the last content + // block of the last message. Each continuation only appends a tool result plus the + // next turn, so everything up to here is read from cache — which is what keeps + // replaying assistant turns verbatim (web-search results included) affordable. + // cache_control is valid on text/tool_use/tool_result blocks, but a thinking or + // redacted_thinking block must never be modified, so skip the breakpoint there. if (anthropicMessages.length > 0) { const lastMessage = anthropicMessages[anthropicMessages.length - 1] - if (Array.isArray(lastMessage.content)) { - // Add cache_control to the last content block - if (lastMessage.content.length > 0) { - const lastBlock = lastMessage.content[lastMessage.content.length - 1] - if (lastBlock.type === 'text') { - lastBlock.cache_control = { type: 'ephemeral' } - } - } - } else if (typeof lastMessage.content === 'string') { - // Convert string content to array format with cache_control + if (typeof lastMessage.content === 'string') { lastMessage.content = [ - { - type: 'text', - text: lastMessage.content, - cache_control: { type: 'ephemeral' } - } + { type: 'text', text: lastMessage.content, cache_control: { type: 'ephemeral' } } ] + } else if (Array.isArray(lastMessage.content) && lastMessage.content.length > 0) { + const lastIndex = lastMessage.content.length - 1 + const lastBlock = lastMessage.content[lastIndex] as any + if (lastBlock.type !== 'thinking' && lastBlock.type !== 'redacted_thinking') { + // Clone the block instead of mutating in place: the array may be a verbatim + // _anthropicContent turn that must stay unaltered for later requests. + lastMessage.content = [ + ...lastMessage.content.slice(0, lastIndex), + { ...lastBlock, cache_control: { type: 'ephemeral' } } + ] + } } } From 83ed011e264f20ffa66a7bf933f2fe3615cf6b67 Mon Sep 17 00:00:00 2001 From: Ruben Fiszel Date: Tue, 30 Jun 2026 17:57:26 +0200 Subject: [PATCH 06/21] feat(object-store): make GCS service account key optional for Workload Identity (#9842) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit build_gcs_client always called `.with_service_account_key(...)`, so an absent key (the settings UI stores "no key" as the empty JSON object `{}`) was handed to the builder and failed to parse instead of falling through to the object_store crate's InstanceCredentialProvider. Skip the call when the key is blank so GCS uses the instance's ambient credentials (GKE Workload Identity / the GCP metadata server). "Blank" (empty/whitespace/`{}`/`null`) is centralized in a shared `gcs_service_account_key_is_blank` predicate so the build path and the non-super-admin connectivity-test SSRF guard (`validate_object_storage_test`) agree on what counts as "no key" — otherwise a blank key would bypass the guard yet still trigger the ambient-credential fallback, letting an untrusted caller probe arbitrary buckets with the server's instance role. Also clarify the settings UI hint that the key may be left empty for ambient credentials, and add regression tests for the blank-key build path and the guard. Fixes WIN-2110 Co-authored-by: Claude Opus 4.8 (1M context) --- backend/windmill-api-settings/src/lib.rs | 26 ++++++++- backend/windmill-object-store/src/lib.rs | 58 ++++++++++++++++++- .../ObjectStoreConfigSettings.svelte | 5 +- 3 files changed, 86 insertions(+), 3 deletions(-) diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index a7e1322ad9..8339df7b76 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -406,7 +406,11 @@ async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Resul ) } ObjectSettings::Gcs(gcs) => { - if gcs.service_account_key.is_empty() { + // Mirror `build_gcs_client`'s blank-key check (shared predicate): a blank/`{}` key falls + // back to the instance's ambient credentials there, so it must be rejected here too — + // otherwise an untrusted caller could probe with the server's identity (the very + // SSRF/credential-exfil this function guards against). + if windmill_object_store::gcs_service_account_key_is_blank(&gcs.service_account_key) { return Err(error::Error::NotAuthorized( "Testing GCS storage without a service account key requires a super admin" .to_string(), @@ -2187,6 +2191,26 @@ mod object_storage_test_hardening { ); } + #[tokio::test] + async fn rejects_gcs_blank_service_account_key() { + // A blank key makes build_gcs_client fall back to the instance's ambient credentials, so an + // untrusted caller must not be allowed to test with it. The `serviceAccountKey` field is + // serialized via serde's `as_string` (`to_string` of the JSON value), so the settings UI's + // "no key" empty object arrives as `"{}"` and a null as `"null"` — both must be rejected. + for key in [serde_json::json!({}), serde_json::json!(null)] { + let settings: ObjectSettings = serde_json::from_value(serde_json::json!({ + "type": "Gcs", + "bucket": "b", + "serviceAccountKey": key + })) + .unwrap(); + assert!( + validate_object_storage_test(&settings).await.is_err(), + "blank key {key:?} should be rejected" + ); + } + } + fn ip(s: &str) -> IpAddr { s.parse().unwrap() } diff --git a/backend/windmill-object-store/src/lib.rs b/backend/windmill-object-store/src/lib.rs index cf30efe116..553eb69917 100644 --- a/backend/windmill-object-store/src/lib.rs +++ b/backend/windmill-object-store/src/lib.rs @@ -494,6 +494,23 @@ fn build_azure_blob_client( return Ok(Arc::new(store)); } +/// Whether a GCS `service_account_key` carries no static credentials, in which case the client +/// should fall back to the instance's ambient credentials (GKE Workload Identity / metadata server) +/// instead of being handed an unparseable key. Besides an empty/whitespace string, the settings UI +/// stores "no key" as an empty JSON object `{}` (and `serde_json` may yield `null`), so treat those +/// as absent too. Shared with the connectivity-test SSRF guard so both agree on what "no key" means. +pub fn gcs_service_account_key_is_blank(service_account_key: &str) -> bool { + let trimmed = service_account_key.trim(); + if trimmed.is_empty() { + return true; + } + match serde_json::from_str::(trimmed) { + Ok(serde_json::Value::Null) => true, + Ok(serde_json::Value::Object(map)) => map.is_empty(), + _ => false, + } +} + #[cfg(feature = "parquet")] async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result> { let gcs_resource = gcs_resource_ref.clone(); @@ -509,7 +526,12 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result
@@ -1129,6 +1436,27 @@ + + { + const it = createConfirm + createConfirm = undefined + if (it) createOnRemote(it) + }} + onCanceled={() => (createConfirm = undefined)} + > +

+ This copies the current value of {createConfirm?.path} + (including any secret value) from + {createConfirm?.onCurrent ? currentWorkspaceId : parentWorkspaceId} + into {createConfirm?.onCurrent ? parentWorkspaceId : currentWorkspaceId}. It stays + workspace-specific afterward, so later promotes won't overwrite it. If it already exists + there, it's left untouched and just marked workspace-specific. +

+ {:else}
No comparison data available
diff --git a/frontend/src/lib/components/DevWorkspaceSetting.svelte b/frontend/src/lib/components/DevWorkspaceSetting.svelte new file mode 100644 index 0000000000..bdba23c57f --- /dev/null +++ b/frontend/src/lib/components/DevWorkspaceSetting.svelte @@ -0,0 +1,191 @@ + + +{#if isDev && parentId} +
+

+ This is a dev workspace paired with root workspace {parentId}. Promote changes + from the home page banner or the Compare & Deploy page. +

+
+ +
+
+{:else if pairedDev} +
+

+ This workspace's dev workspace is {pairedDev.name} ({pairedDev.id}). Edits to this + workspace are redirected there. +

+
+ {#if pairedDev.isMember} + + {/if} + +
+
+{:else if parentId} +

+ Dev workspace pairing is only available for root workspaces. This workspace is a fork of + {parentId}. +

+{:else} +
+

+ Pair this workspace with a dev workspace: the same code with a different environment (resource + and variable values). Edits are made in the dev workspace and promoted here. +

+
+ Attach an existing workspace as dev +