Ruben Fiszel c91027824b feat(pipeline): AI-chat data-pipeline editor (route + in-session) + home surfacing (#9805) ...

* feat(pipeline): AI chat tools to build pipeline nodes with diff/approval

Add a data-pipeline AI chat experience modeled on the flow editor and
surfaced through the dev-gated global chat (no new chat panel).

The /pipeline editor registers PipelineAIChatHelpers on the AIChatManager;
while it is open the global mode layers pipeline tools, a pipeline prompt
section, and the helpers on top of the full global tool set (behavior is
unchanged when no pipeline editor is open).

New tools (frontend/src/lib/components/copilot/chat/pipeline/core.ts):
- get_pipeline_graph / read_pipeline_node — read the live graph and bodies
- build_pipeline_node / edit_pipeline_node — stage changes as AI-pending drafts
- remove_pipeline_node — drop a staged proposal
- test_pipeline_node — preview-run a node (requires confirmation)

Tools never deploy: they stage drafts flagged aiPending, rendered on the
canvas with an accent ring and reviewed via Accept all / Reject all (the
flow editor's GlobalReviewButtons). Accept commits the drafts; Reject reverts
to a pre-AI snapshot, preserving earlier accepted drafts. Auto-accept is gated
on the chat autonomy mode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipeline): teach the global/session chat to author data pipelines

Without an open /pipeline editor the session chat had no pipeline concept, so
"create a data pipeline" loaded flow instructions and built a flow. Add a
first-class pipeline authoring path:

- system_prompts/base/pipeline-base.md — what a data pipeline is (a DAG of
  annotated scripts wired by storage assets, NOT a flow) and how to author the
  // pipeline / // on / // materialize annotations; wired through generate.py as
  getPipelinePrompt() (regenerated prompts.ts/index.ts).
- global/core.ts — new get_instructions subject "pipeline", and a global-prompt
  rule disambiguating data pipelines from flows so the model routes correctly.
- ai_evals/cases/global.yaml — two global cases (single node, two-node chain)
  asserting pipeline-annotated script drafts and forbidding write_flow, guarding
  the pipeline-vs-flow conflation.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipeline): show & build pipelines in the AI session preview

Add a 'pipeline' session preview target so the session AI can show the
data-pipeline graph for a folder and build nodes in-pane:

- open_preview now accepts kind="pipeline" (path = folder); SessionTarget /
  EDITOR_TARGET_KINDS widen accordingly. The slot/codec load model stays
  flow|script|raw_app — pipeline bypasses it with its own fetch/draft state.
- New PipelineEditorView.svelte mounts in the session pane: fetches the
  folder graph, overlays AI drafts, renders AssetGraphCanvas + the
  Accept/Reject review buttons, and registers PipelineAIChatHelpers on the
  *session-scoped* manager (via getAiChatManager) so build_pipeline_node /
  edit_pipeline_node + the diff/approval work inside the session too.
- System prompt nudges the model to open the pipeline preview and use the
  staging tools while building.

Verified end-to-end with a real model: the session AI called open_preview,
the graph mounted in the side panel, then build_pipeline_node staged a node
on the session canvas with its schedule trigger and ducklake output.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): share the AI editor logic between route page and session

Consolidate the duplicated data-pipeline AI logic onto a single shared layer so
the route editor and the in-session preview behave identically and the session
gains the full code editor.

- New pipelineAiHelpers.ts: createPipelineAiHelpers(deps) owns the propose/edit/
  remove/accept/reject/test staging + the per-turn snapshot bookkeeping that
  powers Reject. Callers inject accessors for their own draft Map and graph.
- Route page (/pipeline/[folder]) drops its ~250-line inline AI-helper block and
  wires the shared factory via deps (folder/workspace/graph/drafts + focus,
  ensureEditable, run-started). Its shell — persistence, navigation guard,
  activity, cascade, trigger drawers — is untouched.
- Session PipelineEditorView uses the same factory and now renders the real
  AssetGraphDetailsPane (code editor + live overlays + test), so a node built in
  a session opens with its source, matching the route editor.

Verified: route page hydrates/renders drafts unchanged; in a session the AI
opened the pipeline preview, built a node, and its code showed in the details
pane. check:fast clean, 197 unit tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): externalize editor state into PipelineEditorState (step 1)

Introduce PipelineEditorState — the data-pipeline analogue of the flow editor's
flowStore. It owns the draft Map, the live editor overlays, and the selection,
with callback-safe methods (handleDraftPersist / handleAnnotationsChange / … ),
so a single editor can be rendered by both the route page and the session.

This commit lands the store and points the in-session PipelineEditorView at it
(no behaviour change — the session already had these inline). Next steps move the
route page onto the store and a shared <PipelineGraphEditor>.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): point the route editor at PipelineEditorState (step 1)

Move the route page's draft Map, live editor overlays, selection, and the
draft-persist / live-change handlers onto the shared PipelineEditorState (`pe`),
referencing them as `pe.*` in place. No behaviour change — persistence, graph
resolution, run dispatch, AI staging, and deploy all stay on the page and now
read/write the externalized state.

This is the data-pipeline analogue of the flow editor's flowStore: the route
page and the in-session preview now share one source of editor truth, setting up
the shared <PipelineGraphEditor> in the next steps.

Verified: the page hydrates its DB draft, renders the overlay graph, the toolbar
counts (Save all (N)) track pe.drafts, and selecting a node opens it in the
details pane. check:fast clean, 84 unit tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): render the route editor via shared PipelineGraphEditor (step 2)

Extract the canvas + details-pane editor body into PipelineGraphEditor.svelte,
the data-pipeline analogue of FlowBuilder. The route page now delegates its
Splitpanes block to it, passing the externalized PipelineEditorState plus its
run/cascade/trigger/deploy callbacks; the component owns pane sizing,
selection/details-open derivation, and the canvas+details rendering.

Root-caused the earlier ts2769 "$props() No overload" to a prop named `state`
colliding with the `$state` rune (`let x = $state(...)` parsed as a store
auto-subscription on the prop) — the prop is now `editor`.

Net: the route page sheds ~310 lines of template/state; behaviour preserved.
Verified: the page hydrates its DB draft, renders the graph, opens the draft in
the details pane (live code editor + Test), pane sizing works. check:fast clean,
24 pipeline tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): move draft autosave into PipelineGraphEditor (step 3)

Fold the per-user `data_pipeline` DraftService bundle autosave (hydrate +
debounced persist + localStorage crash mirror) into PipelineGraphEditor, gated
by a `persistDrafts` prop — FlowBuilder's parameterized-autosave shape. The route
page passes `persistDrafts` + `folder` and reads `editor.loadedFromDbDraft` for
its AutosaveIndicator; the in-session preview will leave persistence off.

Also restores the `untrack(...)` wrapping on the pane-sizing $effect (dropped
when the editor body was extracted in step 2). Without it the Pane `bind:size`
feedback loops the effect and pegs the main thread when the details pane is
closed — a latent hang in the step-2 commit.

check:fast clean, 24 pipeline tests pass. Note: browser revalidation was not
possible this session (the Playwright MCP browser was reset); the autosave is a
verbatim port and the untrack fix is the original working form.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(pipeline): render the session preview via shared PipelineGraphEditor (step 4)

Point the in-session PipelineEditorView at the shared PipelineGraphEditor instead
of its own inline canvas + details pane. The session now renders the exact same
editor body as the route page — gaining the full details/code pane — while opting
out of persistence (persistDrafts=false) and the run/cascade/trigger/bounded
affordances (their callbacks are omitted, so those controls hide). Building nodes
+ the Accept/Reject diff still work via the AI helpers.

Also fixes issues surfaced by a full `svelte-check` while wiring this up:
- PipelineGraphEditor: edit mode opened the details pane unconditionally (a step-2
  regression); restored the route's "open only on selection/draft" behaviour.
- Route page passed an `isOperator` prop the component doesn't accept (step-2;
  caught only by full check, not check:fast).
- SessionItemNotFound: narrow its `kind` to exclude `pipeline` (pipeline targets
  never slot-load, so they can't 404 through it) — closes the SessionTarget-widen
  fallout.
- PipelineEditorView: cast the resolveGraph base to AssetGraphResponse.

Full `svelte-check` now clean across all pipeline/session files; 137 unit tests
pass. (Browser revalidation still pending — Playwright MCP was unavailable this
session; see the smoke-test note on the PR.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipeline): stop an infinite microtask loop when persisting a no-output draft

handleDraftPersist short-circuits when the open draft's content + inferred writes
are unchanged. The writes check compared `d.outputAssets?.length === writes.length`,
but a no-output draft has `outputAssets: undefined` (so `?.length` is `undefined`)
while the details pane infers an empty `writes: []` (length 0). `undefined === 0`
is false, so it never short-circuited: every persist re-wrote the drafts Map with
an equivalent object, which gave `activeDraft.script` a new identity → the pane
re-emitted its overlays → the graph re-derived → persist fired again. A self-
sustaining microtask loop that pegged the renderer and froze the tab on any
pipeline carrying a no-output draft (e.g. hydrating one from the saved
data_pipeline draft on load). It hangs rather than throwing effect_update_depth_
exceeded because it cycles across microtasks, not within one reactive flush.

Fix: coalesce the undefined length to 0 so "no outputs" compares equal to an empty
inferred-writes list. Adds pipelineEditorState.test.ts covering the idempotency
(fails without the fix) plus the change/no-change cases.

Root-caused by instrumenting the reactive churn: every iteration reassigned
drafts/liveContent/liveBodyAssets/liveAnnotations/displayGraph with identical
values — pure reference churn off the drafts re-write.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ai-chat): make the agent open the pipeline editor before building nodes

In a session, the GLOBAL system prompt only *advised* opening the pipeline preview
("show its graph with open_preview ... prefer those tools once it is open"), so
the agent routinely skipped it: on a plain "build a data pipeline" request it
reached for write_script and staged plain script drafts, and the canvas editor
never opened. build_pipeline_node / edit_pipeline_node are only registered once
the preview is open, so skipping open_preview also loses the canvas-staged
Accept/Reject diff-approval flow entirely.

Make the guidance imperative: open_preview(kind="pipeline", path=<folder>) is the
FIRST step before creating any node (an empty or not-yet-created folder is fine —
create_folder first if needed), and pipeline nodes go through build_pipeline_node
/ edit_pipeline_node, never write_script. This also clears the agent's "the folder
might not exist" hesitation that pushed it toward write_script.

Verified live (same plain prompt, before/after): before it used write_script with
no editor; after, the agent opens the editor first and stages a canvas-highlighted
node with Accept all / Reject all. The guidance is gated on previewTools
(session-only), so it doesn't affect the non-preview global eval cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(pipeline): preserve in-session pipeline drafts across editor hide/show

The session preview's PipelineEditorState lived in the PipelineEditorView
component with persistDrafts=false. Hiding the editor sets editorVisible=false,
which makes `hasEditor` false and the `{#if hasEditor}` block unmount the view —
discarding its component-local store. Showing it again remounted a fresh, empty
one, so the pipeline the AI had built in the session vanished.

Move the PipelineEditorState onto the per-session SessionRuntime (like the flow /
script / raw_app editors, which already host their state there and take {runtime}),
so it survives the pane unmount on hide and across session switches. The runtime
is keyed by session id and only dropped on session deletion.

Because the instance is now reused, guard against a retarget to a different
folder: PipelineEditorView resets the state when `path` changes to a new folder
(a same-folder remount keeps the drafts). Adds `folder` + `reset()` to the store.

Verified: build a node in a session → Close editor → Show editor → the staged
node, its wiring, the details-pane code, and Accept/Reject all re-appear. Full
svelte-check clean; 139 pipeline tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pipeline-ai): clearer diff + persistent review banner on the canvas

The AI review affordance had two problems on the pipeline canvas:

- The floating Accept-all / Reject-all bar sat bottom-center, where it
  collided with the minimap once the canvas narrowed on node selection —
  reading as "the buttons vanished when I select a node".
- Every staged draft rendered with the same blue ring, so it wasn't clear
  what the review would actually change (a plain manual draft looked the
  same as an AI proposal).

Replace the floating bar with a top-left review banner (z-30, clear of the
controls and minimap) that stays put regardless of selection and spells out
the pending counts. Color the diff per node: a proposal that adds a node
that isn't deployed rings green with a "new" chip; one that edits an
already-deployed node rings amber with an "edited" chip. Plain manual
drafts keep the neutral gray dashed border, so only the green/amber nodes
read as part of the Accept/Reject set.

aiPendingKind is resolved in resolveGraph (deployed runnable present →
modified, else added) and forwarded through the canvas to the node.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): persist in-session pipeline proposals across reload/switch

Staged AI proposals lived only in the per-session runtime's in-memory
PipelineEditorState (persistDrafts=false), so a page reload — and an
LRU-evicted runtime on session switch — dropped them, leaving the canvas
and the Accept/Reject review empty even though the chat still showed the
nodes as staged.

Enable the same per-folder DB-draft persistence the route page uses for the
in-session editor. To keep hide/show cheap and race-free, hydration is now
gated per editor instance (PipelineEditorState.hydratedFromDb) rather than
per component mount: the runtime-hosted instance hydrates ONCE when fresh
(reload / evicted runtime) and then keeps its in-memory drafts across the
editor pane unmounting on hide — re-reading the DB on every remount would
race a not-yet-flushed autosave and drop a just-staged draft. A folder
retarget resets the flag so the new folder re-hydrates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): make Reject all work for rehydrated proposals

rejectAll only reverted paths tracked in the in-memory aiSnapshots map,
which is rebuilt empty on each editor mount. After a reload (or session
switch into a fresh runtime) the proposals are restored from the persisted
draft but have no snapshot, so Reject all was a no-op on exactly the nodes
it should discard. Sweep any still-pending draft without a snapshot and
discard it (revertPath with no snapshot deletes the path; for an edit of a
deployed node that correctly falls back to the deployed body). Adds unit
coverage for accept/reject including the no-snapshot case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): keep proposals visible while the graph reloads on switch

The session editor pane is LRU-capped (MAX_WARM_EDITORS), so returning to a
session whose pane was evicted remounts PipelineEditorView with a fresh
graphRes resource (loading=true, current=undefined). The deployed-graph
loading spinner gated the whole canvas, so the staged proposals and the
Accept/Reject review banner vanished until the re-fetch resolved — read as
"the proposal disappears when I switch sessions".

Only show the loading/error placeholder when there are no drafts to display.
When the runtime already holds staged drafts, render the editor immediately:
resolveGraph overlays them on an empty base so the proposals + banner stay
visible, and the deployed nodes fill in when the fetch completes. Verified
with a 4s-delayed graph fetch — proposals render through the load with no
spinner.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(pipeline-ai): apply AI node edits directly as drafts, no approve/reject

The canvas-level Accept all / Reject all review (aiPending proposals, the
green/amber diff ring + "new"/"edited" chips, and the review banner) didn't
fit the pipeline editor. Match the flow/script editor instead: build/edit
apply directly as ordinary unsaved drafts on the canvas, which the user then
deploys — there is no separate approval step.

Removed across the surface:
- aiPending / aiPendingKind on the runnable node + resolveGraph seeding +
  canvas forwarding; AI-built nodes now render with the existing plain
  unsaved-draft dashed styling.
- the review banner, count derivations, and hasAiPending/onAccept/onReject
  props from PipelineGraphEditor and both consumers (route page + session
  view).
- acceptAll/rejectAll/hasPending and the per-turn snapshot bookkeeping from
  the shared helpers; removeProposedNode now just discards the unsaved draft
  at a path (undo a build). acceptAllProposals/rejectAllProposals/
  hasPendingProposals dropped from the PipelineAIChatHelpers interface and
  the manager's auto-accept hook.
- accept/reject language from the tool descriptions, return messages, and the
  system-prompt section.

Tests updated; pipeline + AssetGraph suites pass (142).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(drafts-diff): support data_pipeline diffs + fix blank empty-summary row

Two issues in the session "Drafts" diff drawer (DraftDiffDrawer):

- Clicking a `data_pipeline` bundle row threw "Draft diff not supported for
  kind data_pipeline" (utils_draft_deploy.ts) — there was no handler for the
  kind, so it fell to the OVERLAY_GETTERS lookup and errored. The bundle has
  no deployed counterpart (each node deploys individually as a script), so
  diff it node-by-node: surface each node's draft body keyed by path, folding
  in the deployed body as the "before" when a node edits a deployed script.

- A draft row whose summary is an empty string (e.g. the app draft) rendered
  with no title at all: WorkspaceItemRow's single-line branch used
  `summary ?? secondary`, and `??` doesn't treat '' as absent, so it showed
  the empty summary instead of the path. Use `||` so an empty summary falls
  back to the path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(drafts-diff): explode data_pipeline bundle into per-node subitems

A data_pipeline draft is a bundle of node-script drafts, so a single row
diffed the whole thing as one blob. Explode it in DraftDiffDrawer into one
script row per node, nested under the bundle's `…/data_pipeline` folder so
they read as the pipeline's subitems — each with its own path and a proper
script Content/Metadata code diff. The node's draft body is the "after"; its
deployed body (when the node is already deployed) is the "before", so edits
show as line diffs and new nodes as added. A single bundle row (via the
getDraftDiffValues data_pipeline fallback) is kept only for the case where
the bundle can't be read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(pipeline-ai): simplify — drop vestigial approve/reject scaffolding & redundant field

Review pass over the PR, removing complexity left from the approve/reject
removal and the shared-component refactor (all behavior-preserving):

- Inline the `acceptPendingEdits` pass-through into `acceptPendingFlowEdits`
  and revert the now-inert `autoAcceptEditsAvailable` GLOBAL+pipeline widening
  (pipeline edits are direct drafts — nothing to auto-accept).
- Fix the global system prompt: pipeline tools "apply directly as unsaved
  drafts (no accept/reject)", not "proposals the user Accepts or Rejects".
- Collapse the redundant `outputAsset` (singular) into `outputAssets`,
  removing a whole resolveGraph fallback tier; simplify propose/editNode.
- Drop the single-field `PipelineAiHelpersHandle` wrapper (callers just
  destructured `{ helpers }`); inline the misleading `isoNow()` helper.
- Remove the now-unreachable `data_pipeline` branch in getDraftDiffValues
  (the drafts drawer explodes bundles per-node; an unreadable bundle is
  skipped) and the "Step N consolidation" drafting narration.
- Un-export internal-only types; reuse `storageKey`; refresh stale comments
  that still referenced proposals / the review banner / diff-approval.

svelte-check clean; 141 unit tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(pipeline): tooltip clarifying the Create/Save button deploys

The accent button in the asset-graph details pane ("Create" for a new script,
"Save" for an existing one) is really a deploy, but had no tooltip explaining
that. Add a title — "Deploy this new script to the workspace" / "Deploy your
changes to this script" — keeping the create-vs-update label distinction while
making clear both deploy.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(pipeline-ai): document the `materialize` annotation in the pipeline prompt

The model invented "materialize run" because the prompt only mentioned
`// materialize <uri>` in passing. Spell out what it is in both the in-app
pipeline prompt (getPipelinePromptSection) and the base prompt
(pipeline-base.md, regenerated): a MANAGED output where the runtime writes the
table around a single SELECT (no manual CREATE/INSERT); replace (default) vs
`append` vs `key=<col>` strategies; `manual` to opt out (track-only); and its
pairing with `// partitioned …` (runs once per partition, `{partition}` token
substituted at run time). Explicitly: materialize is an output declaration,
not a command — there is no "materialize run".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(pipeline-ai): trigger drawers in the AI session preview

Bring the route page's native-trigger affordances to the in-session pipeline
editor by reusing the shared <PipelineTriggerEditors> (no duplication of the
drawer UI). Clicking a "Schedule · Missing — no trigger row" node (or
edit/delete on an attached trigger, webhook, data-upload) now opens the same
drawers the full editor uses, instead of doing nothing. Draft nodes get the
same "save the script first" guard (a trigger row needs a deployed script).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(pipeline-ai): run buttons + live run state in the AI session preview

Wire the per-node Run button and live run-state badges into the in-session
pipeline editor, reusing the shared folder-scoped job poll
(useActiveRunnableIds) the route page uses — node badges, the event log, and
the zero-latency "running" hint all come from it. The session runs one node at
a time (preview for an unsaved draft, the deployed version otherwise),
skipping the route page's cascade/deploy-queue machinery the AI-session UX
doesn't need. Verified: a node's Run button dispatches a job and the badge
updates live from the poll.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(pipeline): label the node deploy button "Deploy" (was Create/Save)

Users read "Create" and asked whether it deploys. It does — and the main
script editor's DeployButton already says "Deploy", so this is the consistent
term. Use "Deploy" for both the new-script and existing-script cases; the
new-vs-changes nuance stays in the button's tooltip.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(home): surface data pipelines as units, including bundle-phase drafts

Treat a data pipeline as one home entry instead of scattering its member
scripts:

- The home "Pipeline · f/<folder>" entry now also covers bundle-phase
  pipelines — a folder that so far only exists as a `data_pipeline` draft —
  not just deployed ones, so a pipeline shows up the moment its first node is
  drafted (union listPipelineFolders + data_pipeline draft folders).
- Pipeline-member scripts (`auto_kind='pipeline'`) are filtered out of the
  individual scripts list; they're represented by their pipeline's entry.
- Tree view injects pipeline folders so they (and their "Pipeline" entry)
  still appear when their only scripts are hidden members or they have none
  deployed yet.

Verified in both list and tree view: app_groups (deployed member folded) and
a draft-only nyc_transit both show as pipelines; the member script no longer
lists individually.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(scripts): compute auto_kind for draft-only pipeline nodes

A never-deployed pipeline node (a script draft starting with `// pipeline`)
had no script row, so list_scripts synthesized it with `auto_kind: None` — and
the home page therefore couldn't tell it was a pipeline member, listing it
individually instead of folding it into its pipeline. Parse the draft content
the same way the create path does (`parse_pipeline_annotations(...).in_pipeline`)
and set `auto_kind = "pipeline"` on the synthesized draft-only row, so draft
nodes fold into their pipeline like deployed members.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(search): hide pipeline-member scripts from global search

The Ctrl+k global search listed pipeline-member scripts (`auto_kind='pipeline'`)
individually. Filter them out — they're reached through their pipeline, matching
the home page. Deployed members carry auto_kind from the script row; draft-only
members now do too (computed from draft content in list_scripts), so both are
excluded here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline): address PR review findings

Session run dispatch (the one real bug):
- runNode now passes `_wmill_skip_asset_dispatch: true` for a single-node run
  of a deployed node unless the user chose "run + downstream" (cascade) —
  previously a single Run could fan out to downstream deployed scripts via the
  backend asset dispatcher and fire side-effecting production runs.
- onRunProducer guards `kind === 'script'`; onTestStateChange only clears the
  run hint for the script the pane finished (not a different in-flight node);
  clear the hint on folder retarget; gate the background poll on isActiveSession
  so hidden warm panes don't poll; note the PipelineTriggerEditors workspace
  coupling.

Home page pipeline surfacing:
- Fold pipeline-member folders into `pipelineFolders` (captured in loadScripts)
  so a members-only / draft-only-`// pipeline` folder still shows its pipeline
  entry instead of vanishing; and don't render the empty-state when only
  pipelines remain (they aren't part of the text filter).
- Insert injected tree folders in name order instead of prepending.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(pipeline-ai): make clear `// materialize` is DuckDB + DuckLake only

The model put `// materialize` on a python3 node, which deploy rejects ("only
supported for DuckDB scripts"). The prompt only implied SQL ("write the body
as a single SELECT") without stating the hard constraint. Spell it out in both
the in-app prompt and pipeline-base.md: `// materialize` is DuckDB-only and its
target must be a DuckLake table; for python3/bun/postgresql nodes, write the
output via the SDK instead and let it be inferred — reach for duckdb when a
node should materialize a DuckLake table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(pipeline-ai): fix stale comment — session now wires run + trigger affordances

Addresses review: the comment still claimed the session 'opts out of the
run/cascade/trigger/bounded affordances', but run buttons + trigger drawers
were wired in. Describe the current state (wires run + triggers; omits only
cascade/bounded/add-script).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline): address Codex review — test_pipeline_node dispatch + tree search

- [P1] testNode (the test_pipeline_node tool) ran a deployed node via
  runScriptByPath without `_wmill_skip_asset_dispatch`, so previewing one node
  could fan out to downstream deployed subscribers and run side-effecting
  scripts. Add the skip flag (test is always single-node) + a regression test.
- [P2] Home tree view injected pipeline folders — and rendered their Pipeline
  row — even during a text search, surfacing unrelated pipelines. Gate both the
  TreeViewRoot injection and TreeView's hasPipeline on `!isSearching`, matching
  the list view which hides pipeline rows on a query.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): keep the pipeline prompt after update_user_instructions

rebuildGlobalSystemMessage (called by the update_user_instructions tool)
rebuilt only the base Global prompt, dropping the pipeline-editor section that
configureGlobalMode appends. So after the chat remembered an instruction, the
next GLOBAL turn lost the active /pipeline/<folder> context + direct-draft/
materialize guidance while pipeline tools stayed registered. Re-append the
pipeline section here when a pipeline editor is registered.

Addresses Codex review [P2].

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(home): gate pipeline entries by kind/archived/owner filters

Codex review [P2]: pipeline rows/folders rendered independently of the item
filters, so a pipeline still showed under the Flows/Apps tabs, in the archived
view, and outside a selected owner. Add `visiblePipelineFolders` applying the
same gates the items get (kind ∈ {all, script}, not archived, owner-prefix
match) and route the list rows, tree injection, and empty-state check through
it. Pipelines are always `f/<folder>`, so the user-folder toggle and kind=script
keep including them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline): address review — route folder-switch state, AI node guards, diff identity

claude[bot] [P1]: the route page's in-app folder switcher navigates same-route
(no remount), but nothing reset PipelineEditorState — so folder A's drafts
displayed under B and autosave persisted them into B's bundle, and B never
hydrated. Reset pe on folder change (mirror the session retarget), and guard the
shared hydrateDrafts against a stale folder result landing after a retarget.

codex/claude [P2]: build_pipeline_node (proposeNode) only checked drafts.has —
now rejects a path outside the open folder and one colliding with an existing
deployed node (model should edit_pipeline_node). + 3 regression tests.

codex/claude [P2]: exploded pipeline-node diff rows shared `script/<path>` with a
standalone script draft at the same path, colliding in the {#each} key + value
cache. Add an explicit unique `key` (the distinct bundle-nested path) on DiffRow;
pipeline nodes set/look up by it while `path` stays the real edit target.

claude [P2]: session AI test_pipeline_node now arms the live run badge
(onRunStarted), matching the route page.

nit: pipelineAiHelpers.test uses afterEach(restoreAllMocks) instead of an
unreachable inline mockRestore.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline): harden AI node mutations + close home label-filter / rename gaps

Codex [P1] (AI mutations trust model paths) — fully scoped now:
- editNode validates the open folder too (proposeNode already did), via a shared
  assertInFolder; an edit_pipeline_node for f/other/* no longer persists an
  unrelated script into the current folder's data_pipeline bundle.
- both build_pipeline_node and edit_pipeline_node now require the `// pipeline`
  annotation (assertPipelineAnnotation) so a staged draft is definitionally a
  pipeline member, not a silently-non-member script. + tests.
  (proposeNode's folder + deployed-collision guards landed in the prior commit.)

Codex [P2] home label filter — visiblePipelineFolders ignored labelFilter, so a
label selection still showed every pipeline (and the empty-state fell through to
render pipeline rows). Pipelines carry no labels, so a label filter hides them.

Codex [P2] session rename — PipelineEditorView now wires onScriptRenamed
(repoint selection + refetch), matching the route page; a persisted-script
rename no longer leaves the canvas on the old path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(pipeline-ai): language-specific comment prefix for annotations

Codex [P2]: the tool schema and prompt told the model to write `// pipeline` /
`// on` / `// materialize` regardless of language, and pipeline-base.md grouped
SQL with `#`. A `//` (or `#`) annotation line is invalid in a DuckDB/Postgres
node — it passes the frontend parser (which strips `//`/`--`/`#`) but is a SQL
syntax error at deploy/run. Make the guidance language-specific everywhere:
`--` for SQL (duckdb/postgresql), `#` for python3/bash, `//` for bun/TS — the
`//` in examples is the TS form to translate. Regenerated the prompt outputs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): re-scope Global prompt on folder switch + language-aware base prompt

Codex [P2] x2:
- The route page resets editor state on an in-app folder switch, but the Global
  chat's system message kept the old `/pipeline/<folder>` scope (the helper
  methods read the reactive folder, but the prompt string is only rebuilt on
  Global-mode reconfigure). Rebuild it on folder change so the next turn targets
  the new folder.
- The pre-editor base Global prompt (seen before open_preview/get_instructions)
  still showed TS-only `// pipeline` / `// on`. Make it language-aware (`--` SQL,
  `#` Python/Bash, `//` TS) so the model can't draft invalid DuckDB/Postgres
  nodes before the pipeline tools are registered.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): authoritative new-node probe + SQL-correct eval checklist

Codex [P2] x2:
- build_pipeline_node's collision check relied on the resolved graph, which can
  be empty while the session preview races open_preview (a build could shadow a
  deployed node before the graph loads) and only covered pipeline runnables, not
  a non-pipeline script at the same path. Add an authoritative backend probe
  (ScriptService.getScriptByPath): any deployed script at the path → reject with
  "use edit_pipeline_node". + regression test (empty graph, deployed script).
- The DuckLake eval judgeChecklist required the exact `// pipeline` annotation,
  which would penalize the now-correct `-- pipeline` SQL output (or reward
  invalid DuckDB syntax). Make both cases syntax-aware.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): rebuild Global prompt on session preview folder retarget

Codex [P2]: open_preview(kind="pipeline", path="B") can retarget an existing
pipeline preview from folder A to B without remounting. The retarget effect
resets editor state and the helper methods read the new path, but the
registration effect only depends on isActiveSession, so the Global system
message stayed scoped to /pipeline/A. Mirror the route-page fix: rebuild the
global system message on retarget (gated on isActiveSession — only the active
session's helpers are registered; a hidden session reconfigures when it next
becomes active).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(pipeline-ai): edit_pipeline_node preserves deployed script metadata

Codex [P1]: editNode kept only the deployed script's language and staged a fresh
makePipelineScript draft with empty hash/summary/description/tag/schema/settings.
Deploying that edit from the pane (auto_parent) would update the script while
wiping its metadata, and the route "Save all" path (no parent_hash) could hit
the backend path-conflict branch on the occupied path. Base the draft on the
existing draft's / deployed script object and replace ONLY content (+ inferred
output assets), preserving hash and metadata. + regression test.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-29 20:33:50 +00:00

..

src

feat(pipeline): AI-chat data-pipeline editor (route + in-session) + home surfacing (#9805)

2026-06-29 20:33:50 +00:00

Cargo.toml

feat: ducklake materialization for data pipelines (#9689)

2026-06-20 15:42:03 +02:00