feat: pipeline editor: assets-only view, Fix actions and draft triggers (#11440)

* wip: one sidebar for operators and developers, with a detach mode

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: snap-detach the operator sidebar, and join the handle to the picker

Dragging the sidebar into the left snap zone detaches it for an operator, who
has no icon-only rail to collapse into; the width the drag started from is what
gets persisted, so re-attaching restores the sidebar's size.

The handle's colour pastille slides onto the card's workspace pastille as the
card opens, alongside the workspace name, and both buttons that attach or
detach the sidebar carry a tooltip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: attach and detach from the sidebar header, open the card under its handle

The button that swaps the sidebar between docked and detached moves to the
header, next to the workspace name, so it stays in one place across the swap;
the icon-only rail has no room for it and keeps the footer copy.

The handle itself becomes a component of its own, carries the workspace icon
its picker uses, and opens the card rather than docking — docking is the header
button's job. The card is placed under the handle measured at open time instead
of at a fixed offset.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: one page header across routes, with the workspace picker in it

Every route now wears the same bar above its content: the sidebar toggle, the
workspace picker, and whatever the route registers — an item's breadcrumb and
summary, or a section name — with the route's own buttons at the far end. The
bar's row is level with the sidebar's own top row, which the workspace ⇄
sessions switch now owns, so the two read as one band.

A route fills the bar by mounting PageHeaderContent, which registers a getter
rather than a snapshot: the layout calls it while rendering, so a path or
summary the page edits reaches the bar without an effect keeping two copies in
step. Nothing registers yet; the routes follow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: detail pages fill the page header, which now spans above the sidebar

A script or flow detail page no longer draws a bar of its own: its path becomes
the header's breadcrumb, its summary keeps the rename-and-labels popover next to
it, and its badges, menu and buttons ride along as the header's actions. The
band runs the full width with the sidebar hanging below it.

A snippet rendered in the header initialises its components under the header,
not under the page that wrote it, so anything the page's tree provides is lost —
the detail page's buttons read TriggerContext and threw. ContextBridge carries
the entries a registration names back around the render.

The floating handle is gone with it: the header's own toggle is the one control,
hovering it peeks the sidebar in and clicking pins it. The workspace picker in
the header is a denser variant of the sidebar's, and the card's outside-click
catcher starts at the card rather than the viewport top, so it no longer
swallows clicks meant for the toggle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: one breadcrumb in the page header, and the banners that used to sit under it

The header band now carries a single breadcrumb — workspace, fork, then the item's
path — built from one component, so every segment has the same size, weight and icon
size, and slashes separate them. The item's kind icon rides with its summary, after a
dot, rather than with the path.

The fork and drafts banners move into the band as badges plus a subtle Review button:
ahead/behind counts instead of a bare change total, and the per-kind breakdown stays on
the compare page. The home hero's CLI, MCP and Hub links move into an ellipsis menu,
alongside the toggle that pins the session chat to the page.

The flow editor and the app viewer hand their bar to the band too, and a route can drop
the band's bottom border when its own content draws one — the border stays in the box
with a transparent colour, so contents never shift a pixel across a navigation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the runs page's own header row moves into the page header

The title, the queue counters, the kind and status filters, the schedule and
future-job switches, the search field and the timeframe picker all ride the band
now, and the page starts at the chart — one row of vertical space back.

The two toggle groups appear from 1500px: below that the bar cannot hold them next
to the search field, and the same filtering is reachable from the search field's own
kind and status chips. The search field is what gives width back, and the breadcrumb
yields last, so a page's own name stays whole while the workspace and fork names
truncate.

The queue counters keep their icon form at every width, since the bar has room for
two counters and never for the two labelled blocks. Their counts hang outside their
icons, so the group carries the margins that keep a badge off its neighbour, and the
suspended counter answers a hover at zero rather than sitting there as a dead icon.

The band is 44px rather than 40, and a page can hand the breadcrumb a hint — the
documentation tooltip that belongs with the page's name rather than with its buttons.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: no border under the page header, and the peek card closes on the way out

The band draws no bottom edge and no page asks it to, so `barBorder` goes with it —
a page that needs the separation draws its own line, as the detail pages now do above
the run form and the tab strip. With nothing toggling a border, the band is a flat
44px on every route.

The detached sidebar's card is a hover affordance, so it leaves when the pointer does:
off the handle and off the card both, on the same debounce a hover menu uses, because
the two do not touch and the leave of one fires before the enter of the other. The
mobile drawer is opened deliberately and still stays until it is dismissed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: one band, one workspace part, and the pickers that hang off it

The band is a single element whose placement is styled rather than branched: a page
registering a side-panel inset used to swap it between two `{#if}` arms, destroying the
header and rebuilding it a frame later — a blink of no header with the content sliding
up into the gap.

The workspace and its fork become one breadcrumb part: the name leads home (inert in a
session, where the workspace is the session's scope rather than a place to go) and one
chevron opens the picker, which already expands a family to reach its forks. The
picker's panel hangs from the part's left edge, opens with the active family expanded,
and closes before the switch runs — the switch renames the workspace, which moves the
trigger the panel hangs from, and floating-ui was chasing it across the viewport.

The item's path is a plain label rather than a row of pickers: same separators and
weight, and a click copies it with its own confirmation. The kind icon goes, the page
below says what kind it is.

Around them: a "prod" badge for a root workspace, the session's acting fork beside the
session's name rather than in the breadcrumb, the edit-in-dev affordance as a button in
the header, the drafts count as a sentence, and a fade when the workspace changes under
a page that stays mounted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep a page's own controls when the workspace navigation is hidden

`menuHidden` means "this is an embed, don't show the workspace's navigation" — but
Runs, the detail pages and the flow editor now keep their controls in the band and
nowhere else, so hiding it took the filters, the timeframe and the page's buttons with
it. A session's preview iframe opens exactly that way. The band now renders for an
embed too, without its breadcrumb, so the page keeps its controls and the embed still
gains no way to navigate the host's workspace.

The flow editor's autosave indicator moves into the header's actions with the rest of
its bar: saving state, the autosave toggle and other people's drafts were left behind
in the branch the full-page editor no longer takes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the header carries what it promised, and nothing it does not

The registry's item described pickers the breadcrumb no longer has: nine fields — the
editability flags, the change callbacks, the navigation hook — that nothing read, while
the flow editor passed eight of them and quietly lost its inline rename. The type now
says what the bar does: a path, and either a summary or the page's own control for one.
The flow editor renames through that control, as the detail page already did.

`barPlacement` went the same way: no page registered 'inline', so the layout's guard
never fired.

The sessions page registered an inset of 0 whenever its preview pane never mounted —
the "no session selected" and opted-out arms — and 0 reads as "a page owns the right
edge", floating the band over the page's first line. It registers an inset only once
the pane has a width.

The band's sidebar handle docks through `setDetached` like the card's own toggle, so
docking still suppresses the rail's entry animation and closes the card; a summary
without its breadcrumb no longer leads with a stray separator in an embed; a page's
hint sits with its name whichever way the page named itself; and `Menu.closeInstantly`
and `EditorHeader`'s `inline`/`hideSummary` go, their callers having been replaced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: a deployed app gets the viewport, and the header waits in the corner

An app is the page, not a page inside Windmill's chrome, so the band leaves the flow
and floats above the viewport until the handle in the app's corner calls it down. The
handle is faded until the pointer nears it, hovering peeks the band over the app, and a
click pins it — one flag for every deployed app, in this browser, because someone who
wants the breadcrumb over a dashboard wants it over the next one too.

Peeking overlays rather than pushes: the app is an iframe, and taking 44px off its box
on every pass of the pointer would re-lay-out whatever it draws. Pinned, the band is in
flow like any other route's and the app starts under it.

The sidebar is untouched — docked stays docked, detached stays a peek — since the pin
is about the header alone. A page says it wants the viewport with one registry field,
so the layout keeps asking the registry rather than knowing about apps.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: one corner control for the chrome an app page hides

Hovering the button overlaid on a deployed app brings back whatever that page is
hiding: the band always, and the sidebar card too when the sidebar is detached. The
card hangs from this button rather than from the header's own handle, which is
off-screen with the band.

The handle in the band no longer attaches the sidebar — it reveals it and nothing more.
A control that hides the thing it sits on cannot also be the way to bring it back, so
attaching stays with the toggle in the sidebar's own footer, where detaching happens.

The band's two buttons go in the order the things nest: the sidebar's handle, then the
header's pin. The overlaid button stands where the pin will be once pinned — same 28px
box, and a slot further along when the sidebar's handle takes the first — so pinning
moves nothing. It wears the panel pair the sidebar's toggle uses: solid while the band
is away, dashed while it is shown.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the access point sits in the corner, and the header covers it

The button is in the page's top-left corner — 8px in, or just past the rail when the
sidebar is docked — rather than shifted along to where the header's pin would land.
Hovering brings the header down over it: an access point that the regular view covers
once it arrives, so there is nothing to line up and nothing to move.

Because the header lands on the button, the pointer leaves it the instant it arrives.
The band holds both itself and the sidebar card open while the pointer is anywhere on
it, or the card would retract under a header that stayed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: on an app page the floating button owns both, and they move together

The band drops its sidebar handle there: the floating control in the corner reveals the
sidebar along with the band, and two handles for one gesture is one too many. The two
are then linked — the pointer resting on either holds both, and leaving either collapses
both — so they behave as the single thing the one button revealed.

The card meets the header in a line: it hangs from the header's bottom edge rather than
from whichever control opened it, which had it starting a couple of pixels high. With no
header to sit under — an embed — it still hangs from the control that reached for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the revealed sidebar is a panel under the header, not a card over the page

No rounded corners, no inset from the window: it starts at the header's bottom edge and
runs to the left and bottom edges, with one border down its right side and a shadow to
lift it off the page. A card floating in from nowhere read as a separate thing; this
reads as the sidebar the header sits on top of.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: only an app page gets the flush panel; elsewhere the card still floats

Over an app the sidebar reads as the panel the header sits on — flush left and bottom,
square, one border down its right side. On every other page it is what it was: a card
inset from the window with rounded corners and a border all round.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: an app opens with the header already away

Opening an app showed the header and then slid it out: the page claimed the viewport on
mount, and the band animated from where it had been rendering. The slide belongs to the
hover, not to arriving, so it is armed a frame after the claim — the first hide is a cut.

The claim moves to the route, which mounts before the viewer waiting on the workspace to
resolve. A cold load now paints the band away from the first frame instead of showing a
header that leaves again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: an app's Edit sits with its name

At the far end of the bar it was a journey across the window from the thing it edits —
on a page whose header is only there while the pointer holds it, a journey with a time
limit. It goes beside the app's path instead.

The slot it uses is the one Runs' documentation tooltip already had, renamed from `hint`
to `afterName` and documented for both: what belongs to the page's name, whether that is
a mark or a control that acts on what the name points at.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: developers collapse, operators detach

One control each, rather than three for everyone. A developer switches the sidebar
between icon-only and full width and never detaches it; an operator, who has no
icon-only rail to collapse to, hides it instead — by the button in its footer or by the
same drag that collapses it for a developer.

The way back survives either way: a detached sidebar keeps its "Show sidebar" button for
anyone, so a developer who detached before this was so is not left with a sidebar they
cannot return.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: configure pipeline triggers and outputs as drafts in the editor

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: one card for pipeline nodes, and the pipeline pages in the page header

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: drop a removed node's trigger drafts, and confirm triggers when leaving

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: show pipeline assets red when unused or awaited but never produced

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: route long pipeline edges around nodes in the target column, and flag scripts waiting on unproduced assets

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a page is as tall as it is given, not as tall as the screen

The band takes 44px off the top of every route, so a page sized to the
viewport overflows by exactly that much: h-screen inside a column that
already lost the band's height. These pages measure the box they are in
instead.

CenteredModal gains `inPage` for the five pages that render it inside the
layout rather than over it — the same modal, centred in what is left of
the page rather than in the window.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the band narrows, wraps, and draws its own edge

The band publishes its width, so the pages filling it can answer it. Two
tiers: under 1000px the breadcrumb keeps the workspace disc and its env
badge and drops the name, and under PHONE_BAR (800) the trail stops
yielding at 10rem — that floor is what makes the bar wrap, so a page's
buttons take the line below rather than squeezing the name to an ellipsis.

The band also draws the line between itself and the page, and only while
something is scrolled under it. One capture listener on the window sees
every scroller, since most pages scroll inside their own box and never
move the layout's. An inset shadow, not a border: a border would move
every page down a pixel the moment the line appeared.

DetailPageLayout and InWorkspaceAppViewer drew that first line themselves;
two lines would stack, so theirs go.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the script, app, runs and audit pages fill the band

Each of these kept a header of its own above the page. They register with
the band instead: the breadcrumb names what the page is on, and the page's
buttons sit at the far end of the one row.

They answer the band's width rather than the window's, since the buttons
share that row with the breadcrumb. The script editor folds Diff and
Settings into the menu it already shows when compact, the app editor drops
the panel buttons, Awareness and Export and goes icon-only on Deploy and
the mode switch, the flow editor menus its diff, and runs hides its
counters. Runs also declares `actionsFlexible`: its search is the one
control here that can give width back, and a box that refused to shrink
would make it take its maximum instead.

The run detail page loses its ActionRow, and the audit log its big title —
the breadcrumb says where you are, and the page keeps its tooltip.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: one sidebar column, shown as a rail or as a panel

The rail and the drawer held two copies of the same column — the same
sections, in the same order, in two pieces of markup, so a change to one
silently skipped the other. Both now render `sidebarColumn`, and the
shells keep only what differs: the rail its width, resize handle and
collapse button, the panel its outside-click catcher and its transform.

A phone gets that panel rather than a modal. There was a width test
deciding between two presentations; the card under the band's handle works
at any width, so it goes. What is left of the width test is the choice
between a rail and a panel, plus a `(hover: hover)` test — the old
`innerWidth >= 768` was standing in for "has a pointer", which is what
decides whether the panel closes when the pointer leaves it.

With the card everywhere, the backdrop, the `aria-modal` (nothing dims and
nothing traps focus) and the drawer's close cross all go: the band's
handle opens the panel and a tap outside closes it. The burger row goes
with them. Entering panel width now collapses the rail transiently, so
coming back to a wide window does not land on a rail the page was resized
away from.

The band also stops easing its right edge. That transition belongs to the
rail, which moves the band's left; its right follows a splitter the page
owns, and easing it left the band lying over a session's side panel — a
blank strip across that panel's own header for 200ms after every drag.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the home composer flies to the menu that restores it

Removing the composer puts it away rather than throws it out, and the way
back is one item in the band's menu — easy to miss on a page the composer
has just vacated. A mark flies from the control that removed it to that
menu, and the menu answers with one ring as the mark dissolves, so where
it went is witnessed rather than explained.

The cross and the menu item now read "Remove session chat", one name for
one thing whichever a reader reaches for.

Reduced motion skips both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: give pipeline asset nodes the trigger nodes' background again

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: taller pipeline nodes, with the asset + back on the node's edge

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: judge dbt relations by provenance, keep dynamic_skip on reopen, refresh saved scripts' titles

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: label the pipeline + buttons: add data source, add downstream step on hover

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: name a schedule node's cadence, like "Every day at 12:00"

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: keep the downstream-step pill open while its menu is, without flicker

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: room between pipeline node chips and the node's edge

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: refuse renaming a pipeline draft out of the pipeline's folder

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: title pipeline schedule nodes with the schedule's summary

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* revert: keep operators out of AI sessions

That change is its own PR; this one is about the navigation. `prefersSessionHandoff`
takes the role again and every entry point passes it, the sessions page refuses an
operator arriving by URL, and the layout leaves them on the docked chat.

Two clauses are new rather than restored, because the nav rework is what gives an
operator a sidebar at all: the Workspace/AI Sessions switch and session mode are both
hidden from them, since the page behind them declines. `askAiShown` gains the operator
case for the same reason — they keep the docked pane whatever the beta gate says, so
the sidebar has to offer a way into it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: assets-only view of a pipeline, with producer and trigger chips

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: default schedule handlers, unique trigger draft paths, zoned cadence, valid script names

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: setup banner without icon, button actions, plainer activity rows

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: assets-only trigger chip under the title, editable and selectable

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: await schedule defaults before confirming, refuse colliding trigger renames

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: size assets-only cards to their content

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: pipeline draft sync status beside the folder name in edit mode

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: autosave edits to a deployed pipeline script from the first keystroke

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: show schedule handlers once their defaults load, draft suffix on static chips

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: keep the pipeline folder while the page tears down on leaving

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: discard the open pane's autosaved edits with Discard all

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: pipeline node context menus, add-at-cursor, and assets-only delete

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: borderless pipeline cards on the main surface, like flow steps

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: remember which pipelines are in the assets-only view

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: edit an asset's producing script from its pane and multi-script chip

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: more room between pipeline layers for the bottom + button

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: background on the trigger chip only when it opens an editor

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: assets-only delete in both modes, why it is blocked, and a Show scripts toggle

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: remove the script before its trigger, archive it for non-admins

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: scope assets-only memory by workspace, no delete for operators, flow reason

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: fly in the pipeline node menus

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: size the Show drafts toggle like the other header buttons

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: a No asset node for scripts that build nothing, in the assets-only view

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: describe a pipeline edge's relationship on hover

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: name assets-only edges by what is built from what

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: open an asset's upstream trigger from its pane header

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: reopen the edit-mode pane when coming back from View

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* style: no red border on errored pipeline nodes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: Fix actions on misconfigured pipeline nodes, and trigger toggles on edges

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: folded errors and edge toggles in assets-only view, preset writer wizard

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: flag pipeline scripts nothing starts, with schedule/webhook/remove fixes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: an unwritten subscription is fixed on its script only, by removing it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: hide subscribed-only unwritten assets, review fixes for the edit pane and edge actions

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: an unwritten subscription is the subscriber's error, not the asset's

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a script only started by reads of unwritten assets is flagged too

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: no add-downstream + on assets nothing in the workspace writes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: Save all treats a draft identical to the deployed script as saved

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: remove-from-View opens its dialog, error chip fits, no link to an undrawn asset

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: drag a script's dot onto an asset to subscribe it; unbuilt asset opens its script

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: toggle to hide reads that don't trigger runs in the pipeline graph

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: drag an asset's + onto a script or the canvas; tell reactive assets-only edges apart

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a programmatic on/mute line joins the existing ones in the header

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: address review — live-version save check, bounded writer lookups, script-only trigger writes, dbt sources, hidden pane

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: review — compare every deployed field, passive-read filter only with its toggle, drag cancel, asset escape hatch

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: keep every control the band took over reachable

Four controls the page-header conversion could put out of reach, each found by a
review pass and reproduced in the browser:

- The flow editor's preview buttons. `topbarWidth` is bound only in the branch a
  full-page editor does not render, so `compactTopbar` was permanently false
  there; below the 800px phone bar `phoneTopbar` dropped the buttons inline and
  the graph overlay never took them, leaving FlowPreviewButtons mounted nowhere
  and step testing, whole-flow testing and "Test flow & record" no-ops through
  optional calls. It now measures the header the way ScriptBuilder and
  AppEditorHeader do, and the inline and overlay conditions are exact
  complements.
- The pin on a deployed app. The band lands over the corner handle that called
  it down, so the pointer that arrives there hits the band. The band carries the
  pin itself now, on the same pixels, in place of the sidebar handle it hides.
- The detached sidebar's own menus. They portal to `body`, so moving into one
  left the card and closed it 200ms later, sliding it out from under the menu
  being read. The card holds while the pointer is in a menu and takes the close
  from the leave of that.
- A deployed app's name for a read-only viewer. The breadcrumb was registered
  behind write access, so without it the trail fell back to the section name
  "Apps"; and the Edit button rode `afterName`, which the band drops for an
  embed along with the trail. The name is registered for whoever opened the app,
  and what sits after it survives `navHidden` as the page's own control.

navPageFor gets a table test: it answers both the sidebar links and the
breadcrumb label, and its longest-prefix and first-segment rules are what made
the "Apps" fallback look like a correct answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: drop what the operator revert and this refactor left behind

The operator tour's last step named AI sessions among the pages the sidebar
opens, which operators have no way to reach. The nav handle's comment named the
AI chat layout's menu button as a second way into the card; that button and its
prop are gone, and the corner control on a page that hides the band is what
stands in for the handle now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style: prettier the three files this branch left unformatted

`prettier --check` fails on all three at the previous head. Whitespace only,
apart from one Button tag that fits on a line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: assets-only nodes name their script in a header tab instead of a side chip

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: no loading spinner in the pane while a clicked script loads

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: no subscribe drag dot in view mode

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: opening a draft no longer autosaves; header tab styling, selected edge, taller cards

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* revert: clicking an unbuilt asset opens the asset again, not its script

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: header tab sets card width, edit icons on script header and trigger chip, no pane header shift on open

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: no trash in the pane header; card width ignores the run chip so a click never re-lays the graph

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: taller trigger chip; node menu and guard badge on the card corner, below the header

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: give a page without the band's chrome what the band was carrying

Second review round, three reachable defects plus one decision:

- A public raw app was 150px tall. The bundle's iframe is `h-full`, and dropping
  its `min-h-screen` left nothing to resolve against on the share, embed and
  custom-path routes — none of which has a layout above them. Measured: 150px in
  a 900px viewport. The floor goes back for those routes and stays off inside the
  workspace, where the layout hands a height down and `min-h-screen` would
  overflow it by the band.
- The band retracted out from under its own open menu on a deployed app. Last
  round taught the sidebar card to hold while the pointer is in a portaled menu
  and left its sibling alone, but the band opens menus too — the workspace disc,
  the fork family, every path level. One helper now, used by both closers.
- `bandPeeked` outlived the page it belonged to. Clicking a control in the band
  navigates away, and the next page may not be one the band floats over, so the
  hover handlers that would clear it are gone — leaving the band already down
  when a full-bleed page came back. `beforeNavigate` clears it.
- An embed has no band to carry a control, so the app's Edit floats over the
  canvas there, where it sat before the band existed. Registering it as the
  band's `afterName` under `navHidden` instead would have cost the app 44px of
  the iframe for a row holding one button.

Reading "is the workspace navigation hidden" now has one definition
(`isMenuHidden`), since a page has to answer it too: the flag is sticky inside an
iframe, so the raw query param is the wrong thing to read.

Also inset the band by the rail only when there is a rail — an embed reserved
its width and crowded its own controls into a compact layout early.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: a path that names nothing yet has no breadcrumb leaf

`splitItemPath` documents that it gives up on a path still being typed, but
`f/demo/` is three parts and returned a leaf with an empty name — a blank
segment in the trail. The editor binds the path live, so the breadcrumb sees
every keystroke.

Tested here rather than `navPageFor`'s sort: no two `NAV_PAGES` paths nest, so
the longest-prefix case that test claimed to pin has only one candidate. Renamed
it to what it actually checks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style: prettier the four remaining files this branch left unformatted

All four fail `prettier --check` on the branch and pass at the merge base.
`raw_apps/utils.ts` also fails but already did at the base, so it is left alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: pipelines open in the assets-only view by default

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: a No asset node per script, banner-only when writes start it; no On asset change chip

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: delete a node's script, trigger, or both from its menu; red header for a script with errors

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: an errored script header stays red when selected

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: coloured run badge border; per-script delete when no combined delete; run button centred on the card

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: Add data source pill sits one small gap above the graph

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: Multiple triggers chip picks the trigger to edit; each folded trigger deletable from the node menu

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: add-step menu labels its path field Path

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: iterate NodeLists with Array.from so svelte-check passes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: no trigger delete for flow-built assets; Multiple triggers chip shows the one being edited

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: Archive button for archive-only deletes, single delete doc, inert Multiple triggers label in View

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: the raw app editor fills the band like the other three

It was the one editor route still rendering its own bar under the band, so it
showed two header rows and the band named the app "Apps raw" — `navPageFor`'s
first-segment fallback, because nothing on the route registered an item. Its top
bar now becomes the page header the way the script, flow and low-code app
editors' do: the path and summary are the breadcrumb's, the file-sidebar toggle,
autosave cloud, presence mark, menu, Diff, Jobs, export, AI and Deploy are the
header's actions, and the path is a plain label since this editor renames from
its deploy drawer.

The actions render under the band, outside the subtree `RawAppEditor` declares
the operating workspace in, and `OpenInSessionButton` asks that context who the
acting user is — so the workspace travels with them.
`operatingWorkspaceContextEntry()` hands it over without exporting the key.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: keep the folders a half-typed path already names

The trailing-separator guard gave up on `f/demo/` entirely, which dropped the
whole trail to the kind chip and brought `f/demo` back on the first character of
the name. The levels the path does name come back now; only the leaf waits.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: pin holds the first slot, and the nav table says what it shares

Pinning brings the sidebar's handle back, and from behind it the unpin button sat
one slot over from where the pin had just been clicked — so a second click in the
same place opened the sidebar instead. An operator is detached by default, which
makes that the default state of the feature rather than an edge case.

`NAV_PAGES` claimed one table stops a page being called one thing in the nav and
another in the header, which holds for the seven entries the sidebar's main group
is built from and not for the four grouped separately — `/routes` was "Triggers"
in the header and "HTTP" in the nav. The comment now says which is which, and
that entry is named for the page rather than its group.

`sidebarPageAllowed` gets a table test: it decides what an operator sees, and its
`=== true` is what a future `Option`-style default would silently change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the band names the path Deploy will create, and keeps the summary readable

A brand-new app is parked at a `draft_<uuid>` placeholder, and the band named
that rather than the path its Deploy will create — `newEditedPath` refuses the
placeholder for exactly this reason, and the editor's own bar read it. Both app
editors register it now. Verified: the trail reads `u/admin/gentle_app` while the
URL is still the draft.

The band's summary gets the `title` the editor's own bar had, so one the bar
truncates can still be read on hover.

The wrapper that hid AppExportButton on a phone never hid anything:
the component draws no trigger — it is the export drawer, which portals to
`body` — so the class toggle was inert in both states and the comment read as if
something moved. Dropped from both app editors.

Two comments corrected: `NAV_PAGES`' `icon` does reach the breadcrumb for all
eleven entries (`NavBreadcrumb` renders `routePage.icon`), so only `operatorKey`
goes unread for the four the sidebar groups separately; and the raw editor's
phone note described folding that does not happen — the file-sidebar toggle stays
in the band, Jobs and the export are already in the menu, and AI is what stands
down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: an item the picker files under no kind keeps its name in the band

The merge brought main's agents detail page, which reaches DetailPageHeader with
no `errorHandlerKind` — and the band's item required one. `WorkspaceItemKindIcon`
falls through to the script glyph for anything it does not know, so defaulting
would have labelled an agent a script; the kind is optional now and the trail
names the item without claiming it is one of the three.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the file sidebar's toggle leaves the band

The band is the workspace's header, not the raw editor's layout controls. ⌘B
still folds the file sidebar, and the editor's own bar keeps the button where it
draws one — a session pane.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the scroll edge stops measuring, and the comments describe the code

The band's edge listener captures every scroller in the app — Monaco, the audit
table, a run's logs — about once a frame while any of them moves, and it took two
`getBoundingClientRect()` reads before looking at `scrollTop`. It now answers from
`scrollTop` first and only measures when that answer differs from what the band
draws, which is the transition rather than the steady state. The rect check still
authorizes every change, so a scroller that does not start at the content's top
still cannot claim the edge.

The two agent routes main just added size themselves to the viewport, which is
44px more than the box the band leaves them. They measure 856 either way — the
layout's `min-h-0` column clamps them — so this is the sweep the rest of the
branch already had, making the height follow the box rather than depend on a
parent's clamp.

Six comments described the conversion rather than the code, which AGENTS.md calls
out by name. Each now states the constraint a reader could break: why the summary
keeps its popover here, why the audit hint is `afterName` and not an action, why a
run registers a `section` rather than an item, why the Runs filters are
`actionsFlexible`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "fix: the file sidebar's toggle leaves the band"

This reverts commit 1da8d16740.

A folded file sidebar renders nothing of its own, so the toggle on the bar is the
only way back short of knowing ⌘B.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the editors name their item in the band, and a pen renames it

The band showed each editor's summary as an inline field, which put editing
where the page names what it is working on. The summary is text there now, with
its `title` for one the band truncates, and the pen that opens the path for
editing sits after it — the same popover the editors' own bar carries, at the end
of the name it renames. Every editor already has a field for the summary itself:
the script's settings, the flow's Settings tab, and both app editors' deploy
drawer.

The popover moves into `PathEditPopover` rather than being written out four more
times. It owns the open state and hands back the path as it was when it opened:
the band's trail reads that while the popover is open, so a path growing under
the pointer does not drag the pen — and floating-ui the popover — along with it.
`EditorHeader` renders the same component, so the session pane and the band
cannot drift apart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the pen renames the item and its summary

The pen opened the path alone, which left the summary with nowhere to be edited
from the band. It carries a Summary field above the path now — the item's name
first, the address it lives at below — and the band's text follows it as the user
types.

`TextInput` with a label rather than the bar's inline field: `EditableInput`'s own
docs rule itself out inside a popover, and it wraps this same input in its edit
mode, so the field, the "Add a summary..." placeholder and the trim on commit are
the behaviour that was there, in the shape the deploy drawer's Summary field
already uses. The keydown stops at the field because every editor binds keys of
its own and a summary is prose.

`summaryEditable` carries the hosts' `customUi.topBar.editableSummary` through,
and the editors' own bar passes it false: that bar keeps its inline field, and
`hidePath` would take the pen with it, leaving a condensed session pane no way to
name the item at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the agent editor fills the band too

It was the last editor route drawing its own header row under the band, and the
band named it "Agents" — `navPageFor`'s first-segment fallback, because nothing
on the route registered an item. The agent's path is the breadcrumb now, its
version badge reads after that name, the sentence that was the row's second line
became the hint beside it, and the Chat/Form switch, Settings, Evals, version
history and Deploy are the band's actions.

No trail levels in the registration: this layout opens settings in a drawer and
evals over the page, so unlike the dialog it never stands on a level — `trail` is
its root alone here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the pipeline page fills the band

Found by sweeping every route under the band: the pipeline editor drew its own
row beneath it, 48px of a second header, while the band named only the section.
Its name is the band's now, the folder it is open on sits with that name — the
switcher is how you change pipelines, so it belongs to the name rather than to
the actions — and the mode switch, deploy-all, activity, refresh and overflow
menu are the bar's actions. The arrow back to the list stays: nothing else on the
page points there, and the sidebar does not list pipelines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the app canvas controls stay reachable, and an unnamed item says so

Codex found the app editor's canvas controls unreachable on an ordinary laptop:
the owning route collapsed at 1350px, and a 1440 window with the rail out leaves
1232, so the width, theme and breakpoint toggles were hidden with nothing in the
menu to replace them — and the breakpoint is the only way onto the mobile layout.
Measured on the band, the toggles are 240px beside 334 of other actions and a
~430 trail, so they fit to about 1050: the threshold is 1100 now, and below it
each of the three keeps a menu entry rather than disappearing.

An item with no summary read as a bare `·` with an invisible pen after it, since
the plain text replaced a field that had a placeholder. It now reads "Add a
summary..." in the italic SummaryPathDisplay uses for the same state, and the pen
stops hiding when there is nothing to hover.

Three more from the review:
- `actionsFlexible` says "these actions can give width back", which is true of the
  Runs filter row and not of the pipeline's row of fixed buttons.
- `pathSnapshot ?? (newEditedPath || …)` keeps the `||` chain the five rename
  sites read and the declaration documents; `??` had made the rest unreachable,
  since `newEditedPath` is a string that is never undefined.
- The band bridged `contexts` onto the actions alone, while the pen that decides
  whether a path is the caller's to change renders in `summaryContent`. The
  breadcrumb's `afterName` and the summary are bridged the same way now, so that
  permission check reads the tree that registered it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the widest action rows scroll, and the canvas toggles show only where they fit

Measuring the band instead of estimating it: with the canvas toggles shown the
app editor's actions are 826-911px and the trail with the summary another ~500,
so the row overflowed the bar at every width up to about 1470 — the 1100 I set
from an estimate was wrong, and the 1350 before it was too. The toggles come out
at 1500 now, and the menu entries are what keeps all three reachable below that
rather than merely out of the way.

The two widest rows — this editor's and the pipeline's — scroll inside their own
box, which is the escape the rows they replaced had (`overflow-x-auto
whitespace-nowrap`). That answers pi's finding and clears the last 23-40px of
band overflow at middling widths, which predated the conversion. Measured after:
zero horizontal overflow from 852px of bar to 1392.

From the CI round's other findings:
- The pen is offered when either the summary or the path is editable, not the
  path alone: a host that allowed one and refused the other could no longer edit
  the summary from the band. `PathEditPopover` takes `pathEditable` to match, and
  its label names whichever it is given.
- The four editors' comments called the summary read-only, which stopped being
  true when the pen beside it started editing it.
- The pipeline's "centre group" comment described a layout it no longer has.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: actionsFlexible covers both ways a row gives width back

It read as if a search field were the only way to earn it, which stopped being
true when the app and pipeline editors' rows started scrolling inside their own
box. Both are the same bargain — the box may shrink because its contents can —
and the flag is what the bar reads to allow it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: data pipelines in AI sessions (alpha) (#11453)

* feat: pipelines in AI sessions with several folders open at once

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: deploy and discard a session's pipeline edits as one changes-list row

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: pipelines in the chat's listing and the preview picker; session tab shares the page's asset prefetch

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a masked pipeline preselects only the scripts and triggers of its folder

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: review round — exact pipeline scripts in the fork mask, refuse a failed hand-off flush, deploy guard before awaits

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: review round — failed bundle save fails the deploy, removing a draft records no change

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: a session's pipeline tab opens node details in a modal, as the flow editor does

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: route pipeline graph edges orthogonally with spread ports, separate tracks and lanes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: pipeline graph edges merge by shared end, enter nodes by the side, keep fewest crossings

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: review round — incremental crossing search and cached routes, retry the leftover draft save, drop the pipeline row when its last draft goes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: retry the leftover draft save with what failed, never an empty draft

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* revert: pipeline graph edges back to bezier curves, detours curved too

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: hovering a pipeline graph node fades everything outside its lineage

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: hover fade eases edges too, through a stylesheet instead of new node objects

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: gentler hover fade without flicker on click; implied trigger edges recede in the assets-only view

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs: pipeline nodes ingest straight into DuckLake or data tables from Python/TS

* fix: pipeline tools callable right after open_preview; write_script refuses pipeline nodes in sessions

* feat: delete a data pipeline from the home list; pipeline rows match item rows

* fix: upward detour edges keep their gutter lane

* fix: pipeline delete takes only its own triggers and discards draft-only nodes by storage key

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: AI attaches trigger drafts to the pipeline nodes that declare them; materialize nodes keep no stale output

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: AI gives every pipeline node a succinct summary

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: session pipeline tools survive another tab taking the panel, reopening the editor on call

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: pipeline trigger lookups skip same-path flows; a failed trigger never hides its built node

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: deleting a pipeline clears it from open sessions so autosave cannot restore it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: data upload opens on its inputs alone, works in sessions, with a prominent Upload data button

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: closing the pipeline pane closes its modal instead of leaving it blank

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: a DuckLake asset opens on a Data tab with the database manager

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: inputs-only view infers draft inputs, keeps stored lineage, runs the live body

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: lineage follows the editor that inferred it; upload runs preview unsaved edits of deployed nodes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: readyDataUploadPaths is a ReadonlySet in the pipeline editor

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: selected pipeline nodes get their blue edge in dark mode too

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: no red flash on load while writer lookups and script bodies are in flight

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: deploy an open deployed script's edits, and don't resurrect them after a discard

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* refactor: deploy and discard close the open pane first instead of patching its edits

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: keep the pane closed while deploying, and clear its buffer once its edits are a draft

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a draft edited while it deploys keeps the newer edits, chained on the new version

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: a draft whose summary or labels changed mid-deploy is kept too

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: the app viewer names the page it owns, and stops claiming the viewport

A deployed app opened in an AI session's preview panel renamed the band from
the session to the app, and put the app's Edit beside the session's name. The
viewer registered wherever it was mounted; it takes `ownsPageHeader` now, the
way the detail pages already did, and keeps Edit over its own canvas otherwise.

Both viewers also claimed the whole viewport under a band that takes 44px of
it, so a deployed app scrolled by exactly the height of its own header (the
low-code one measured 900px in an 856px box). The height comes from the host
now: the three standalone routes floor themselves at the viewport and
everything inside them is `h-full`. A raw app reached through `/app_embed` had
no height above it at all and collapsed to nothing; it fills the frame now,
though that route still fetches no bundle secret, so what it shows is blank —
as it is on main.

The agent editor's pen was a second editor of the same path that reported
nothing: an invalid path never reached the deploy gate the settings drawer's
field reports to. It binds to the same slot now. Two things that surfaced
with it: a rename never reached the trail, which read the deployed path; and
typing the saved path back raised "path already used", since the popover's
`initialPath` moves with the rename — `allowedExistingPath` is what says the
item already occupies it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: the band's controls come from the design system, and the row has one copy

`NavBreadcrumbTrigger` was `MeltButton` written again — same `use:melt` on a
plain button, same spread — so the workspace chevron uses the component that
already exists, and the sidebar handle is a `Button`.

`DetailPageHeader` carried its action row twice, once for the band and once
for the row it draws inside a session's preview panel, and the two had already
drifted: different dropdown size, no `unifiedSize` on the buttons, and the
deprecated `variant="contained" color="light" size="xs"` on the schedule
button. Both placements render the same two snippets now, at the size each
needs.

`OperatorPageKey` admitted every `operator_settings` field, including the three
write rights, which `sidebarPageAllowed`'s `=== true` rule answers wrongly: a
write right is granted unless withdrawn, so its absence means "never
configured". It is the ten visibility flags now.

A test over the registry's merge: later registration wins field by field,
actions sort by `actionsOrder` then registration, release gives its fields
back. That rule is why a component registering where it is not the page
renames someone else's band.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the band owns its bottom edge, and one pen renames everywhere

The page header decides whether a separator is drawn, so a page cannot draw a
second line under it. A registration asks for `onScroll` (the default, drawn
only while something passes under the bar), `always` for a page whose content
starts right beneath it, or `none` where the page genuinely draws its own first
line. Detail pages, the agent editor, workspace settings and the pipeline list
take `always`; the raw app editor takes `none`, since its splitpanes already
draw that line.

The pen that opens the rename popover becomes PathEditPen, shared by the
editors and the deployed pages so the two cannot drift, and the deployed pages
gain it. SummaryPathDisplay makes the pen the only trigger rather than the whole
name block, so the name is selectable text again. Renaming an agent goes through
ResourceService with only the fields the popover edits: the backend sets a
column per field it is given, so writing the config back would clobber a
concurrent edit. The popover also stops clearing a real collision verdict by
telling Path which path is its own.

Two Button fixes, both reaching call sites beyond this work:

- `dropdownItems` given as a function was asked for `.length`, which is its
  arity and therefore 0, so a split button built that way rendered no caret.
  Both app editors' Deploy dropdowns were unreachable.
- the anchor branch dropped `title`, so a link button had no tooltip.

The audit log filters move into the header actions at the design system's sizes
and trade absolutely positioned labels for the form layout in
brand-guidelines.md: a label above its input, 4px apart, fields 8px apart.

The deployed app viewer floors itself at `min-h-full` instead of claiming the
viewport, which would overhang the page box by the height of the band, and
carries Edit and a public-url link that fold into a menu on a narrow bar. The
root layout drops the legacy menu block, which rendered a second hidden sidebar
and 54 invisible tab stops, and marks the closed detached sidebar `inert`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: service logs and the audit filters move into the band

Service logs: the page's name is the breadcrumb's, so the `<h1>` is gone and the
search field, the timeframe and the two switches sit in the band beside it, with
the documentation tooltip after the name. ServiceLogsInner registers the band
itself, the way RunsPage does, since that route is its only caller, and the
`title` snippet the page used to hand it goes with the heading.

The search is the widest control there, because searching is what the page is
for, and it is the one that gives width back under pressure. The two switches
fold into a popover below a 1200px bar rather than at a phone's width, so the
field keeps its full width down to a much narrower window — folded rather than
dropped, because the page no longer carries a toolbar they could fall back to.

The timeframe control took no size of its own, so it defaulted to `md` beside a
`sm` field, and sat in a `w-48` box 20px narrower than its content, whose
overflow is visible and so spilled into its neighbours.

No `separator`: the two panes below start under the bar with borders of their
own, so at rest there is nothing for the band to divide.

Audit logs: the filters lose their labels. Each select says what it filters
while unset — "all usernames", "all resources" — and once set the value names
the field by itself. From and To become the TimeframeSelect the runs page uses,
which absorbs the standalone Refresh button, since that control's left half is
the refresh. `auditTimeframes` joins the runs and service-logs presets, with
"All time" first. The bounds stay the filters' own `after`/`before` rather than a
remembered preset label: the timeline's zoom writes them too, and a label could
then contradict the chart.

That takes the row from 1326px to 898px, so it stays inline down to a ~1500px
viewport instead of ~2500. The fold measures the row against the bar less what
the breadcrumb takes, plus a gap the two must never close — the breadcrumb
yields width grudgingly and the row not at all, so sized to just meet they
overlap rather than compress, and mid-resize the tooltip after the page's name
lands on the timeframe button.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: an embedded page keeps its name in the band

A page in a session's preview panel loads in a frame that sets `nomenubar`, so the
layout renders the band with `navHidden`. That flag hid the whole breadcrumb —
which is what carries the page's name — while the actions beside it had no such
guard, leaving a row of controls belonging to nothing.

Hiding the trail is still right: it leads to the host's workspace and offers a
picker that would take the reader out of the session. So the breadcrumb gains a
`nameOnly` mode that drops the workspace part and its picker and keeps the page's
own name, and the band asks for it instead of rendering nothing. The summary line
loses the same guard, so an item in there shows its path and summary.

Only the pages whose own title the band took over are affected. Variables,
Resources, Schedules and Triggers still draw their own `PageHeader` and render no
band when embedded, so nothing is doubled.

The bar's left padding is there to sit a button against — the sidebar handle, or
the workspace disc, both of which carry their own margin. Embedded there is
neither, and the name started as bare text 8px from the edge, so that case takes
the same 16px the bar already keeps on its right.

Workspace settings drops the copy-id button from the band: the General tab names
the workspace ID with a field of its own, which is where someone looks for it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: a session in full screen keeps its tabs in the band

Full screen used to drop the session's name from the breadcrumb and leave the
tab strip inside the panel. The band now carries both: the session's name where
"Sessions" stood, and from the breadcrumb to the right edge one filled surface
holding the tabs and the preview's controls, the bar's full height.

The switch between the two layouts is the panel expanding to the left. The chat
column is resized to nothing rather than unmounted, so it can be seen taking its
place -- and so a toggle no longer tears down every mounted chat and rebuilds it
on the way back. Panes transition their width only while that switch is in
flight, since a splitter drag sets the same inline width and would trail the
pointer.

The band floats over the content in full screen as it already did beside a
panel, so only its right edge moves and the page below never shifts by the
bar's height. Leaving full screen, the panel travels back across ground the
band has already claimed, so it passes one layer above it for the length of the
move: pinned to the width the panel will end at, the bar would otherwise show
through as a white patch over the panel's left edge.

The breadcrumb also gains the acting fork, the way every other page shows it,
which replaces the "Acting on" control the session header carried.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the band's inset survives a resize made in full screen

The panel was remembered as a width, sampled while it stood still. A window
resized during full screen left that width describing a layout that no longer
existed, and the way out pinned the band to it for the length of the move. It
is now remembered as the share of the row the panel takes, which the new row
width turns back into the right inset: leaving full screen at 1200px after
entering it at 1600px, the band starts at 628 rather than at 909.

The pane transition's duration is read from the property the timer is set from,
so the class cannot outlive the movement it describes, and two comments that
described earlier drafts are rewritten: the band's name in full screen comes
from the session's own header once it has mounted, and the tab strip's wrapper
carries a fade, not a measured move.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: an agent's pen keeps the labels the agent already carries

The pen on an agent's page was handed the summary and the path but not the
labels, so the popover offered an empty list over a resource that had some. It
saves every field it shows in one write, and the resource endpoint sets the
column it is given, so adding a label there replaced the agent's own: an agent
labelled `production` and `scheduled` came back from one edit labelled only with
the label just added. A flow's page and a script's page both pass theirs; this
one now does too.

Verified against the running instance: an agent created with
`["production","scheduled"]`, one label added through the pen, reads back
`["production","scheduled","new"]` — and `["new"]` with the prop removed again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the sidebar remembers detaching per role, not per account

Whether the sidebar starts detached depends on the role: an operator gets it out
of the way, a developer gets it docked. That default held only until the control
was pressed once, because the choice went to a single key read before the role.
From then on both roles wore whichever was pressed last, and the one that had
not asked for it could never get its own default back — switch a workspace where
you are an operator for one where you are a developer, and the sidebar stayed
detached.

Each role now keeps its own choice, and falls back to its own default until it
has one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: the handle's comment describes what the handle does

It claimed a click holds the sidebar card open. Nothing distinguishes a click
from a hover there — both call the same opener, and the pointer leaving
schedules the close either way — so on a desktop the card follows the pointer
and goes when it goes. Measured: hovering slides the rail from -213 to 5, the
click moves nothing, and the pointer leaving puts it back at -213.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: a dev workspace's badge wears its fork's own colour

Every surface that names a dev workspace drew the same blue badge, which said
nothing about which workspace it belonged to. One component, DevWorkspaceBadge,
now draws it: a fork with a colour fills the badge with that hue and writes the
label in white, and a fork without one keeps whatever its row had before.

The hue cannot carry white at a fixed lightness — HSL lightness is not perceived
brightness, so yellow and purple at the same L differ by several stops — so
forkAccentStyle solves the badge's lightness by bisection instead, taking the
lightest shade of the hue that still holds white at WCAG AA. Measured across the
wheel the contrast lands between 4.52 and 4.64.

The session picker opens with every family collapsed, since the fork a session
runs in is the session's own and not something that picker should offer to
change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the name is what opens its rename editor, and the trail truncates from its start

The pen beside a name is gone. The name itself is the trigger, with a hover fill
behind it where the click opens an editor rather than going anywhere — the path
segment that only copies keeps no fill, which is what tells the two apart. A bar
with a summary field of its own (EditorHeader's condensed pane) still gets the
pen, since it has no name to hang the trigger on.

Clicking either half of a name opens the editor under that half with the cursor
in the matching field. The band draws the path and the page draws the summary, so
one popover cannot anchor under both: a page hands the band a `pathTrigger`
snippet and the band renders its path segment inside it, passing the classes its
own segment wears. Each editor supplies a second PathEditPopover for it, and
`focusField` names the field to focus — `#path` explicitly, since melt otherwise
lands on the row's first button.

Those snippets render from the band's tree, which outlives the values they read
by a flush, so each is gated on the value it reads. Without that an agent page
whose draft had not loaded threw inside the band and left the breadcrumb empty.

The trail now carries two weights: the segment that ends it is what the page is
about and takes `font-medium text-emphasis`, everything before it is a step
quieter. Which segment that is follows the route — the workspace on home (the
fork alone inside a fork), a list page's own name, an item's summary.

And it truncates instead of switching to icons at a breakpoint. The workspace
part yields first, the family name before the fork, then the path's folders, and
the item's own name last. The bar no longer wraps on a phone: a wrapping row
never squeezes anything, it hands the trail a line of its own and leaves it
whole.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: a page keeps its path segment when it cannot offer a rename

Four things the local reviews caught in the rename work:

A detail page handed the band a `pathTrigger` whether or not it could save one.
Without `onSaved` — every script, flow or agent page the reader cannot write —
`SummaryPathDisplay` fell through to its read-only branch, which ignores the
`label` it was given and draws its own block: the trail lost its path and the
summary appeared twice, two lines deep, in a 44px bar. The page now offers the
trigger only where a rename can be saved, and the component renders what it was
handed rather than its own row.

The band gives up its own segment the moment a page offers a `pathTrigger`, so a
trigger whose own gate is closed left the trail with no path at all — for the
whole of an agent draft's load, not just a flush. Each gated snippet now draws
the path itself in that case.

`openFocus` named the path field as `#path`, which melt resolves with a
document-wide query; `Path` hardcodes that id at every call site, and the
popover is portalled to the body, so it lost document order to any other `Path`
on the page. It is scoped to the popover's own field now.

And `open` was an exported `$bindable(false)` on an optional prop, which the
frontend's own rules ban. No caller bound it, so it is local state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the badge solves its lightness against the colour the browser paints

Three nits from the review round.

The contrast bisection measured the hue at full float precision, but a browser
paints 8-bit channels: rounding them moved the result up to 0.025 off the target,
which took #993388 to 4.478 against white — under the AA floor the comment
claims. Quantizing the channels inside `paintedLuminance` puts the worst case
across 2880 hue and saturation combinations at 4.500, and your own fork's badge
measures 4.53 as painted.

An agent nobody can write still offered the rename behind its path segment: an
editor with two read-only fields, in place of the click that copies the path.
It is gated on `readOnly` now, the way a detail page is gated on `onSaved`. The
script and flow editors take the same gate from the `customUi` flags that
already decide whether their summary offers one — unreachable today, since the
editors own the band only on routes that pass no `customUi`, but the four
editors now answer the question the same way.

And the registry's doc named an `onEditName` field that has never existed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: an operator is not offered a fork of a workspace

A fork does not carry the role. The server writes the forker's membership row in
the new workspace from the parent's `is_admin` and lets `operator` fall to its
`false` default, so an operator who forks lands in a full-rights copy of the
workspace they could only run things in.

`canCreateFork` refuses an operator, which takes the entry out of the workspace
picker, the session family picker and the session changes bar;
`editInForkAllowed` refuses one too, so no editor offers "Edit in fork". The
fork page refuses on its own, since a URL reaches it whatever the pickers show.
And a workspace the user is an operator in is no longer a base the fork form
will accept, so the base picker cannot route around the gate.

This hides the affordance; it does not enforce the rule. `create_fork` asks only
for membership — `require_admin` covers the dev-workspace case alone — so the
API still answers an operator who calls it directly. The server-side fix is to
carry `operator` into that membership row, the way the member-copy path already
does, and it belongs in a backend change rather than here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* style: the fork refusal uses the design system's alert

The refusal panel was hand-built from a div and two paragraphs with literal
colors, copied from the Runs page's own operator refusal. The frontend's rules
say to use the design-system component, and `Alert` already draws exactly this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: frontend type errors failing npm_check

The dev workspace badge accepts the nullable label the API returns, the
session strip transitions take the node Svelte passes them, and an unused
import is dropped.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: the frontend type-checks again

`check frontend build` has been red on every head for a day, through four review
rounds that each came back clean — none of the reviewers type-check, and they
say so in their own coverage sections. Ten errors:

The dev badge's `label` and `color` were typed `string | undefined`, but the
workspace rows carry `string | null`, so all seven of its call sites failed.
`forkAccentStyle` took the same widening.

A Svelte transition is called with the node it animates, and `stripArrive` and
`stripLeave` were declared with no parameters at all.

And `DevWorkspaceSetting` still imported `Badge` after its badge moved into the
shared component.

`svelte-check` now reports 0 errors. Note that it only agrees with CI once
`src/lib/gen` has been regenerated — the gitignored client goes stale against
main and invents errors CI never sees.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the home bar carries the CLI and hub buttons while it has the room

A menu is where a thing goes when the bar cannot hold it, not where it belongs.
Above 1024px of bar, "CLI / MCP" and "Hub" stand as their own buttons; below it
they fold back into the ellipsis, which is still where they live on a phone.

The threshold reads `pageHeader.barWidth`, not the viewport, so it tracks what
the sidebar actually leaves behind. Home's trail is ~250px and its other
registered actions ~400, so the two labelled buttons fit from about 850 up;
1024 leaves margin before the trail would start truncating, and it does not
truncate at any width between 760 and 1600.

The ellipsis hides itself when it would open an empty menu, which happens only
once both side trips are in the bar and the composer is already on the page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: a fork's band counts its drafts beside its drift, and every count opens what it names

A fork that had drifted stopped saying anything about its drafts: the banner
printed the draft count only while the fork was in sync, and the drift badges
replaced it the moment there was anything to deploy. A fork can be 86 ahead, 39
behind and holding 128 drafts; it now says so.

All four counts open the page that holds them — ahead the deploy direction,
behind `?dir=update`, conflicts the side they are resolved from before
deploying, drafts the drafts half. The button beside them still chooses a
direction on the reader's behalf; a badge means the one thing it names.

The colours and the icons are the compare page's, so the counts in the bar and
the rows behind them say the same thing the same way: ahead green, behind blue,
conflicts orange. Drafts share blue with behind and are told apart by a pencil
and by naming what they count, rather than by an arrow that would read as a
third direction. Both arrows are straight, here and on the compare page.

"Review changes" replaces "Review & Deploy Changes", and it renders in every
state. What the comparison has to say — still counting, or failed — is a mark
beside the button rather than a branch that replaces it: a fork whose count is
unfinished is exactly when a reader wants the page that counts properly.

The drafts banner outside a fork drops "This workspace has", so the count reads
the same wherever it appears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the home band keeps its hero's own cross, and the side trips stay quiet

The CLI and hub buttons take the shape the hero gave them before the bar did —
`xs`, hint-coloured, the hub keeping its outbound mark — so a side trip does not
read as loudly as the thing the page is for.

The ellipsis is gone while the hero is on the page: the hero's cross already
puts it away, so a menu whose only item repeats that is a button that does
nothing new. It comes back when it has something of its own to carry, which is
the side trips on a narrow bar and the way back to a hero that has been put
away. That entry is now "Pin session hero to homepage", and "Remove session
chat" survives only as the cross's own words, where one action keeps one name.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: a fork's counts and the way to review them are one control

Four badges beside a button read as four things that happen to sit together.
They are one thing — every segment opens the same page on the part it names — so
they now share a box: one outline, square corners inside it, and the rounding on
the box alone, which means no segment has to know where in the row it sits.

The only rule in the strip is the one before the end cap: everything left of it
is a reading, that is an action.

With room every count says what it counts; below 1280px of bar they keep their
icon and their number, which is the part a reader scans for, and the tooltip
names them either way. The end cap keeps its words at every width — the counts
have icons to fall back on and it has none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the compare page asks once, says it is working, and knows its own name

The page re-requested the same summaries and on-behalf-of emails forever. Two
faults compounding:

`fetchOnBehalfOfInfo` skipped a key it had already answered with
`onBehalfOfInfo[k] !== undefined`, but stored `undefined` on failure — so
"asked, and there was nothing" could not be told from "never asked". An item
that is only ahead does not exist in the parent, so that side always fails:
a permanently un-cacheable set, re-requested on every pass.

And the effect that drives both fetchers depended on what they write. Each reads
its cache synchronously before its first `await`, which registers that cache as
a dependency; the write that follows re-runs the effect. Write, re-run, write.

Both now record what they asked for in a plain Set, marked before the request
rather than after the answer, and the effect is untracked so it depends on the
comparison alone. The caches are cleared when the compared pair changes, since a
summary's key carries no workspace and those Sets would otherwise never ask
again.

While the comparison was in flight the page printed "No changes between this
workspace and X" — an empty list and an unanswered one look alike from the
layout, and that is a claim with no answer behind it. The page tracks the
request now, and the layout spins instead.

The breadcrumb called the page "Forks", which is what `navPageFor` falls back to
when a route names itself nowhere: the first segment, capitalised. It registers
"Compare" instead — in the page rather than in NAV_PAGES, which is what the
sidebar renders from.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "feat: a fork's counts and the way to review them are one control"

This reverts commit e4504d2843. The counts go back to separate badges and
"Review changes" to a button of its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the compare page follows the fork picked in the trail

Switching a fork from the breadcrumb left the page comparing the pair it opened
with. Two reasons, neither reactive: the page seeds `currentWorkspaceId` from
`?workspace_id` once at mount and never reads it again, and the switch rewrites
`?workspace`, a different param that this page does not read.

It follows `$workspaceStore` now — picking a fork in the trail means comparing
that fork. The param is rewritten so a reload keeps the pair on screen, and an
arbitrary `?target=` is dropped, since it named a destination for the workspace
just left.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: the band is a step taller, and the runs row sizes with it

The bar goes from h-11 to h-12, taken from one constant beside the top
padding a floating band owes the page under it, so the two cannot drift.

The runs page's controls now share the sm height the rest of a header's
actions use: FilterSearchbar learns the size, dropping the top margin
that only centres text at md and taking the shared leading instead, and
sm buttons get px-3 so a label reads as a button. The actions sit in
their own row with a wider gap, and the chart below meets the header
with no gap of its own.

The search field holds one width through typing. As flex-1 it was
whatever the row had left over, so committing a chip grew it from 226 to
288 and took those pixels off the breadcrumb.

A fork's count badges keep their labels while the bar is at least 1280
wide and fall back to icon and number below that.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the trail holds a path's place instead of naming its kind

A page registers its kind when it mounts and its path once it has loaded
one. In between the trail printed "script" or "flow", which reads as the
answer and then jumps when the real path lands. A skeleton of a
plausible path's width holds the place, with aria-busy and a label
naming what is loading.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the compare page reads the workspace it was linked with

The effect that follows the trail's fork also ran on mount, so a
`?workspace_id` that differs from the navigation workspace was thrown
away. That difference is the contract: a session's Review button passes
the fork it committed to and leaves the navigation workspace alone, as
does the prod -> dev link in UpdateDevWorkspaceModal. Both opened on the
sidebar's workspace instead, comparing the wrong pair or none at all.

Only a change in the store now retargets the page, measured against what
it held when the page opened.

The URL rewriting goes with it. The picker's items are links that carry
every other param forward, so their navigation lands after the effect
and put back the `workspace_id` it had just rewritten: switching fork
and reloading left the trail naming one workspace and the page comparing
another. The page reads `?workspace=` ahead of `?workspace_id=` instead
— the param the picker writes and the root layout already applies to the
store — and `?target=` lapses while the page is on another workspace
rather than being deleted from a URL someone else is about to replace.

Two 44px literals left over from the h-11 band: the detached sidebar
card's measured-top fallback and the rail morph's end state, both now
from BAR_HEIGHT_PX beside the class the band wears.

The runs search field keeps its 18rem without growing, but as `shrink-0`
nothing in that row could give width back, so a short bar had no answer
but the trail's truncation and an overlong row would have run past the
edge. `min-w-0` restores the yield without restoring the growth: a flex
item's floor is its min-content width, which the chips inside the field
raise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the compare page's target picker works, and a switch survives reload

`?target=` was gated on the page still being on the workspace the URL
named. Opened from "Merge into another workspace" in the workspace
settings the URL names no workspace at all, so that comparison was
`<id> === undefined` and the param was ignored for good: picking a
target left the page on the "no parent workspace" prompt, which is the
only place the migration can be started from. Read the param as it was
read before.

The follow effect writes `?workspace=` again. A plain click in the
picker preventDefaults its own link and switches the store directly, so
nothing navigates afterwards and nothing overwrites the write — while
`fixupUrlAfterWorkspaceSwitch` rewrites the param only once it is
already there, which on a page opened with `workspace_id` alone it never
is. Without it, switching fork and reloading left the trail naming one
workspace and the page comparing another.

Exercised on a fork pair: the settings entry's picker loads the
comparison it names, a plain-click switch survives a reload, and a
`workspace_id` deep link still outranks the navigation workspace.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: the compare page reads its pair instead of following it

Net effect count: three removed, none added.

The page derives the workspace it compares from the URL and the store
rather than keeping it in state and running an effect to follow a
switch. The workspace-switch link is what moves it: the picker's items
carry every param forward, so a page-side rewrite raced that navigation
and lost — the trace showed the fixup's replaceState landing first and
the link's pushState putting the old params straight back. The href now
swaps `workspace_id` alongside `workspace` and drops `target`, and
`fixupUrlAfterWorkspaceSwitch` does the same for the paths that do not
navigate, so both writers agree and the order stops mattering.

Dropping `target` on a switch also closes the case where switching onto
the workspace that was the target left the page comparing it with
itself.

CompareWorkspaces remounts on either side of the pair changing, not just
the target: two forks of one parent shared a mount, since the target is
the parent they have in common, and with it the selection, the deploy
direction, the summaries, the pinned rows, the deploy permissions and
the CI results. The cache-clearing effect that stood in for that is
gone. Its summary loops outlive their mount, so they read the pair once
at the start and stop writing when the instance is destroyed.

The app viewer asked for a public secret on every app view to decide
whether to draw a button; it reads the mode off the app it has already
loaded and fetches the secret when the button is used.

The summary popover seeds its fields on the open transition rather than
from an effect that also read the item: a save landing while it was open
reseeded the fields over what was being typed.

Exercised against a running instance, 18 cases: idle issues nothing, a
switch moves page, trail and URL together and survives a reload, a deep
`workspace_id` link still outranks the navigation workspace, four fast
switches settle with no foreign row, and the migration target picker
loads the pair it names.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the public url opens the tab the click itself opened

The secret is fetched when the button is used, and the tab was opened
once it arrived — after an await, where Safari no longer counts the
click as the gesture that opened it and blocks the popup without saying
so. Open the tab synchronously and point it at the url once the secret
is known, closing it and saying why when there is none.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the public url button survives a sandboxed app

A sandboxed low-code app loads inside the opaque /app_embed iframe, so
`loadApp` never runs on this page and `app` stays undefined — taking the
button and its compact-menu entry with it, for the owner too. The mode
now comes from the lite read that already runs for the Edit button,
which is there for exactly this reason: it is the one description of the
app this page has whether or not the app itself loads here.

Exercised on four apps: anonymous sandboxed low-code, anonymous
unsandboxed low-code and an anonymous raw app all show the button and
fetch the secret only on click; a publisher app shows neither and asks
nothing. At 640px the button folds into the menu, which carries it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor: the removed hero leaves a blue flash on the ellipsis

The mark that flew from the cross to the menu is gone, with its two
element refs, its flight state and timer, and the `data-home-ai-menu`
hook that existed only as its destination. What is left is the thing it
was there to say: the ellipsis holds the way back, so it flashes blue as
it appears.

Written as a keyframe animation, not a transition. A transition needs
the element painted at its start value before the end value is set, and
this mounts in the same flush that collapses the hero, so the frame that
would have shown it at rest is the frame it is already told to leave —
one `requestAnimationFrame` is not reliably past that paint, and the
flash was never drawn at all. `animate-fade-out` already exists for this
(AssetsDropdownButton flashes a background the same way); mounting it
only while pulsing gets a fresh run per removal without that component's
reflow juggling.

The flash rides the ellipsis rather than the row: with the CLI and hub
buttons beside it the row is three controls wide, and marking all three
names none of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the flash rests invisible, and its comment describes what is there

`animate-fade-out` carries no fill mode, so at the end of the run the
element returns to its own opacity. At the default 1 that is the flash
coming back at full strength for as long as the unmount timer lags the
animation — which the hero's collapse, happening right there, is well
placed to cause. `opacity-0` at rest makes the revert land on invisible.

The comment above it still described the mark that used to fly to the
menu, and made its point three times over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the compare page asks only about the pair it is showing

Switching workspace on this page fetched the previous pair's item paths
from the new workspace, where none of them exist: 97 of 174 requests
404ing on one switch. The comparison was plain state, and the card is
remounted when the pair changes — in the same render that changes it,
before the effect that cleared the comparison could run, so the new
mount started from the old pair's diffs. It now carries the pair it
describes and is read only while that matches, which leaves nothing to
clear.

`computeFullScan` had the same shape: it read the target again after its
await, so a switch mid-scan invalidated another pair's cache, named the
wrong workspace in its toast, and left the scan button spinning on a
pair that was never scanned. It captures the pair up front, re-compares
only while the page is still on it, and reports it as its own.

Each diff says which side holds the item, so the summary and
on-behalf-of loops stop asking the side that does not: an item that is
only ahead cannot answer in the parent, and asking was a guaranteed 404
and a console error per one-sided row.

Those two loops also read the same object — a summary and an identity
are fields of one payload — so every row was fetched twice per
workspace. One read per item per workspace now serves both. A failed
read is not kept: held, it would read as "this item has no identity",
which is what decides whether a deploy offers to preserve one, and an
item deployed without it silently becomes the deployer's.

The header names the workspace being compared when that is not the one
the app is pointed at, which is how a session's Review button and the
prod -> dev link open this page. The trail said the parent while the
page compared the fork.

Measured on a 90-item pair, before -> after: a cold load 247 -> 170
requests and 167 -> 90 item reads; a workspace switch 174 requests with
97 404s -> 71 with none. Sixteen cases over cold load, idle, switching
in both directions, four switches with no time to settle, a cross-family
switch, reload, deep link, the migration picker and leaving an arbitrary
target: no request in any of them names a pair the page has left.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: the scan toast reports the pair it scanned

Off the scanned pair the re-compare is skipped, so there is nothing to
report as compared — but the success branch said "Compared N items"
anyway, and the error it checked first belonged to whichever pair the
page had moved to. Each outcome now says what happened: the seed alone
when the page has moved on, and the comparison's result only where one
was asked for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Guilhem Lemouel <guilhemlemouel@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
This commit is contained in:
Diego Imbert
2026-10-08 14:23:29 +02:00
committed by GitHub
co-authored by Claude Opus 5 Guilhem Lemouel Ruben Fiszel
parent a85d0b76cc
commit 3dae553576
137 changed files with 11099 additions and 2864 deletions
+40 -2
View File
@@ -7915,10 +7915,48 @@ A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates step
A pipeline node that produces a table should almost always be a **\`duckdb\`** node that materializes its output into a **DuckLake** table with \`-- materialize ducklake://<name>/<table>\` (in a DuckDB node the annotation uses SQL \`--\` comment syntax; write the body as a bare \`SELECT\` and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else:
- \`postgresql\` / data tables — only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live).
- \`bun\` / \`python3\` — only for non-tabular work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node still produces tabular data for downstream steps, land it in DuckLake (write it with the wmill SDK / ducklake helpers) rather than inventing a parallel store.
- \`bun\` / \`python3\` — only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it **straight into DuckLake** (or a data table) from the same script — see "Ingesting from Python / TypeScript" below — rather than inventing a parallel store.
Do not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal.
## Ingesting from Python / TypeScript
A node that fetches data in code (an API, a SaaS export, a scraper) writes the rows **directly** into their DuckLake table with \`wmill.ducklake()\`, or into a data table with \`wmill.datatable()\`. DuckLake already stores every table as parquet files in the workspace object storage, so **never stage the raw payload first** — no intermediate JSON / CSV / parquet file in S3 and no second "load the raw file" node. One ingestion node, one table.
Pass the rows as a query argument (a list of dicts / array of objects is sent as JSON) and let SQL unnest them; declare the column types so the table gets a real schema:
\`\`\`python
# pipeline
# on schedule
import wmill
def main():
rows = fetch_charges() # list[dict]
lake = wmill.ducklake()
lake.query("""
CREATE OR REPLACE TABLE stripe_charges AS
SELECT unnest(from_json($rows, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true)
""", rows=rows).fetch()
\`\`\`
\`\`\`ts
// pipeline
// on schedule
import * as wmill from "windmill-client"
export async function main() {
const rows = await fetchCharges()
const lake = wmill.ducklake()
await lake\`CREATE OR REPLACE TABLE stripe_charges AS
SELECT unnest(from_json(\${rows}, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true)\`.fetch()
}
\`\`\`
- Use \`INSERT INTO <table> SELECT ...\` instead of \`CREATE OR REPLACE\` to append (an incremental pull), with a one-time \`CREATE TABLE IF NOT EXISTS\` first.
- Assign the client to a variable (\`lake = wmill.ducklake()\`) and keep the SQL a **string literal** (Python) or a template without \`sql.raw\` (TS): that is what lets the graph record the node's write to \`ducklake://main/<table>\`. A chained \`wmill.ducklake().query(...)\` or SQL built in a variable records no output, so the downstream edge never forms.
- \`wmill.ducklake("<name>")\` targets a non-default catalog; \`wmill.datatable("<name>")\` writes a Postgres data table the same way (Python \`db.query(sql, *args)\` with \`$1\` placeholders, TS the same template form) — use it only when the data belongs in Postgres (see above).
- The downstream node triggers on the table itself: \`-- on ducklake://main/stripe_charges\`.
## Storage prerequisites
A DuckLake pipeline only runs once the workspace has **object storage** (S3 / Azure Blob / GCS) **and a DuckLake catalog** configured — DuckLake tables and \`s3://\` assets can't be materialized or read without it. Check which DuckLake catalogs the workspace has before you build.
@@ -7962,7 +8000,7 @@ The key must be a **string literal** — the graph parser is static and cannot f
## Materialize (the managed output)
> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake://<name>/<table>\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the output via the SDK (\`wmill.writeS3File(...)\`, ducklake helpers, …) and let it be inferred. Use \`duckdb\` when a node should materialize a DuckLake table.
> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake://<name>/<table>\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the table itself with \`wmill.ducklake()\` (see "Ingesting from Python / TypeScript") and let the output be inferred. Use \`duckdb\` when a node should materialize a DuckLake table.
>
> The one target ANY language may declare (except a dbt script, whose writes come from its manifest) is a **warehouse relation**: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, where \`<warehouse>\` is a warehouse the workspace configures under Settings → dbt. \`manual\` is the only mode it has — nothing generates warehouse DDL, so the node issues its own write (a postgresql \`CREATE TABLE\` / \`INSERT\`, an SDK load, …) and the annotation records the outcome. Use it on an ingestion node whose output a dbt project reads as a \`source\`: the declared relation and the dbt model land on ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when the ingestion node completes.
@@ -142,7 +142,13 @@
let hintTimer: ReturnType<typeof setTimeout> | undefined
const kindLabel = $derived(
itemKind === 'flow' ? 'flow' : itemKind === 'app' || itemKind === 'raw_app' ? 'app' : 'script'
itemKind === 'flow'
? 'flow'
: itemKind === 'app' || itemKind === 'raw_app'
? 'app'
: itemKind === 'data_pipeline'
? 'pipeline'
: 'script'
)
// Each false → true transition fires a hint (others > loaded). Seeded to
@@ -15,7 +15,7 @@
contentClasses="p-4"
>
{#snippet trigger()}
<Button variant="accent" size="xs" nonCaptureEvent={true} startIcon={{ icon: Clock }}>
<Button variant="default" unifiedSize="sm" nonCaptureEvent={true} startIcon={{ icon: Clock }}>
Cron builder
</Button>
{/snippet}
@@ -17,6 +17,9 @@
selectInputClass?: string
class?: string
onClear?: () => void
/** Size the input to its content, like Path's folder picker. */
useContentEditable?: boolean
RightIcon?: any
}
let {
@@ -27,7 +30,9 @@
placeholder = undefined,
selectInputClass = '',
class: className = '',
onClear = undefined
onClear = undefined,
useContentEditable = false,
RightIcon = undefined
}: Props = $props()
let datatables = usePromise(() =>
@@ -47,6 +52,8 @@
{placeholder}
inputClass={selectInputClass}
{onClear}
{useContentEditable}
{RightIcon}
/>
{#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'datatable', path: value })}
<ExploreAssetButton class="mt-1 w-fit" asset={{ kind: 'datatable', path: value }} />
@@ -17,6 +17,9 @@
selectInputClass?: string
class?: string
onClear?: () => void
/** Size the input to its content, like Path's folder picker. */
useContentEditable?: boolean
RightIcon?: any
}
let {
@@ -27,7 +30,9 @@
placeholder = undefined,
selectInputClass = '',
class: className = '',
onClear = undefined
onClear = undefined,
useContentEditable = false,
RightIcon = undefined
}: Props = $props()
let ducklakes = usePromise(() =>
@@ -45,6 +50,8 @@
{placeholder}
inputClass={selectInputClass}
{onClear}
{useContentEditable}
{RightIcon}
/>
{#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'ducklake', path: value })}
<ExploreAssetButton class="mt-1 w-fit" asset={{ kind: 'ducklake', path: value }} />
@@ -1523,6 +1523,9 @@
} catch (e) {
console.log('error initializing services', e)
}
// Unmounted while the services loaded: the container is detached, and
// creating the editor on it throws.
if (destroyed || !divEl?.isConnected) return
// vscode.languages.registerDefinitionProvider('*', {
// provideDefinition(document, position, token) {
+34 -37
View File
@@ -2101,43 +2101,15 @@
</div>
{/if}
{#if customUi?.previewPanel?.hideArgs}
<!-- Compact preview layout used by the pipeline editor:
no args column (the script is known to take no
inputs), LogPanel takes the full width, with a
small Test/Cancel button at the top-left of the
preview band. The earlier `-translate-y-1/2`
"float onto the editor" version was clipped in
Firefox (transform + overflow: visible interaction
on the parent splitpane Pane), so the button is
now positioned inside the panel — `top-1` keeps
it visually pinned to the top edge without
relying on cross-browser overflow behaviour. -->
<div class="relative h-full pt-9 flex flex-col">
{#if testJob?.id && testJob.type === 'CompletedJob' && opWs}
<!-- Right-side affordances when we're displaying a *completed*
job (either the user just ran a test, or the on-mount
last-run loader populated the panel). The job-id link
opens the full run page in a new tab; the dispatch
button mounts a popover that shows what downstream
jobs this run triggered (rendered only when the run
actually dispatched anything). -->
<div class="absolute top-1 right-2 z-10 flex items-center gap-2">
<a
class="text-3xs text-blue-600 hover:underline font-mono"
href={`${base}/run/${testJob.id}?workspace=${opWs}`}
target="_blank"
rel="noopener noreferrer"
title="Open this run"
>
{testJob.id.slice(0, 8)}… ↗
</a>
<DispatchEventsButton
workspace={testJob.workspace_id ?? opWs}
jobId={testJob.id}
/>
</div>
{/if}
<div class="absolute top-1 left-2 z-10">
<!-- Compact preview layout used by the pipeline editor: no args
column (most pipeline scripts take no inputs) and a full-width
LogPanel. The controls sit in normal flow rather than floating
onto the editor band, which Firefox clipped (transform +
overflow: visible on the parent splitpane Pane). -->
<div class="h-full flex flex-col">
<!-- Test sits at the top right, the last run's link under it. -->
<div class="shrink-0 flex flex-col items-end gap-1 px-2 pt-2 pb-1">
<div>
{#if testIsLoading}
{@render cancelTestButton('sm', 'shadow-md')}
{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0 || customUi?.previewPanel?.onBoundedRun}
@@ -2271,6 +2243,31 @@
{@render runTestButton('sm', 'shadow-md')}
{/if}
</div>
{#if testJob?.id && testJob.type === 'CompletedJob' && opWs}
<!-- Right-side affordances when we're displaying a *completed*
job (either the user just ran a test, or the on-mount
last-run loader populated the panel). The job-id link
opens the full run page in a new tab; the dispatch
button mounts a popover that shows what downstream
jobs this run triggered (rendered only when the run
actually dispatched anything). -->
<div class="flex items-center gap-2">
<a
class="text-3xs text-blue-600 hover:underline font-mono"
href={`${base}/run/${testJob.id}?workspace=${opWs}`}
target="_blank"
rel="noopener noreferrer"
title="Open this run"
>
{testJob.id.slice(0, 8)}… ↗
</a>
<DispatchEventsButton
workspace={testJob.workspace_id ?? opWs}
jobId={testJob.id}
/>
</div>
{/if}
</div>
{#if customUi?.previewPanel?.argsAboveLogs && schema?.properties && Object.keys(schema.properties).length > 0}
<div
class="shrink-0 overflow-auto max-h-[45%] border-b border-gray-200 dark:border-gray-700 px-4 py-3"
@@ -31,6 +31,13 @@
/** What is being renamed. An agent is a resource, so the path field is scoped to one and
* the write goes through the agent branch of the rename manager. */
kind?: 'flow' | 'script' | 'agent'
/** Saves instead of the API, for an item that exists only locally (a draft).
* Returns an error message to keep the popover open. */
saveOverride?: (next: {
path: string
summary: string
labels: string[] | undefined
}) => string | undefined
/** Header variant: no path line (the host shows it) and a lighter summary. */
compact?: boolean
/** Rendered inside the trigger in place of the summary, for a host that hangs a second copy
@@ -51,6 +58,7 @@
editable = false,
onSaved,
kind = 'flow',
saveOverride = undefined,
compact = false,
label,
triggerClass = 'block min-w-0 max-w-full px-1 py-0.5 rounded text-left cursor-pointer hover:bg-surface-hover transition-colors',
@@ -67,7 +75,9 @@
let popoverOpen = $state(false)
// Resolved before the popover opens, not on opening it: the path field only exists for an owner,
// and `openFocus` looks for it in the same flush the popover renders in.
const own = $derived(isOwner(path ?? '', actingUser, $operatingWorkspace))
const own = $derived(
saveOverride !== undefined || isOwner(path ?? '', actingUser, $operatingWorkspace)
)
let onBehalfOfEmail = $state<string | undefined>(undefined)
let summaryInput: ReturnType<typeof TextInput> | undefined = $state()
let labelsDirty = $state(false)
@@ -101,6 +111,17 @@
async function save(close: () => void) {
const initialPath = path ?? ''
const newPath = own ? editPath : initialPath
if (saveOverride) {
const error = saveOverride({ path: newPath, summary: editSummary, labels })
if (error) {
sendUserToast(error, true)
return
}
labelsDirty = false
close()
onSaved?.(newPath)
return
}
try {
await updateItemPathAndSummary({
@@ -0,0 +1,150 @@
<!--
@component
The menu that adds a pipeline script started by a data source (a schedule, a
webhook, a queue…): the add node's, and the canvas's right-click one.
-->
<script lang="ts">
import type { Snippet } from 'svelte'
import PipelineInsertMenu, {
type PipelineInsertOptions,
type PipelineInsertPick
} from './PipelineInsertMenu.svelte'
import {
Clock,
Webhook,
Mail,
Zap,
Radio,
MessageSquare,
Database,
Send,
CloudCog,
Upload
} from 'lucide-svelte'
import type { ScriptLang } from '$lib/gen'
import type { NativeTriggerKind } from './types'
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
import type { PipelineOutputKind } from './pipelineTemplates'
interface Props {
onAddPipelineScript: (
language: ScriptLang,
path: string,
source: { kind: NativeTriggerKind; path: string | undefined },
outputKind: PipelineOutputKind,
aiPrompt?: string,
options?: PipelineInsertOptions
) => void
pathPrefix: string
defaultPathSuffix: string
trigger: Snippet<[{ open: boolean }]>
/** Opens the menu each time it changes. */
openSignal?: number
}
let {
onAddPipelineScript,
pathPrefix,
defaultPathSuffix,
trigger,
openSignal
}: Props = $props()
function handlePick(pick: PipelineInsertPick) {
if (!pick.language || !pick.path) return
// Native trigger annotation is marker-only — the binding lives on the
// trigger row's own `script_path`, which is created separately.
onAddPipelineScript(
pick.language as ScriptLang,
pick.path,
{ kind: pick.kindId as NativeTriggerKind, path: undefined },
(pick.outputKind ?? 'none') as PipelineOutputKind,
pick.aiPrompt,
{ schedule: pick.schedule, outputAsset: pick.outputAsset }
)
}
</script>
<PipelineInsertMenu
kinds={[
{
id: 'data_upload',
label: 'On data upload',
description: 'UI-first: run by uploading a file via the S3 picker',
icon: Upload
},
{
id: 'schedule',
label: 'On schedule',
description: 'Triggered by a schedule you create',
icon: Clock
},
{
id: 'webhook',
label: 'On webhook',
description: 'Triggered by an HTTP webhook',
icon: Webhook
},
{
id: 'email',
configuredAfterCreate: true,
label: 'On email',
description: 'Triggered by incoming email',
icon: Mail
},
{
id: 'kafka',
configuredAfterCreate: true,
label: 'On Kafka',
description: 'Triggered by a Kafka message',
icon: Zap
},
{
id: 'mqtt',
configuredAfterCreate: true,
label: 'On MQTT',
description: 'Triggered by an MQTT message',
icon: Radio
},
{
id: 'amqp',
configuredAfterCreate: true,
label: 'On AMQP',
description: 'Triggered by an AMQP (RabbitMQ) message',
icon: Radio
},
{
id: 'nats',
configuredAfterCreate: true,
label: 'On NATS',
description: 'Triggered by a NATS message',
icon: MessageSquare
},
{
id: 'postgres',
configuredAfterCreate: true,
label: 'On Postgres',
description: 'Triggered by a Postgres event',
icon: Database
},
{
id: 'sqs',
configuredAfterCreate: true,
label: 'On SQS',
description: 'Triggered by an SQS message',
icon: Send
},
{
id: 'gcp',
configuredAfterCreate: true,
label: 'On GCP Pub/Sub',
description: 'Triggered by a Pub/Sub message',
icon: CloudCog
}
]}
languages={PIPELINE_LANGUAGES as any}
{pathPrefix}
{defaultPathSuffix}
onPick={handlePick}
{openSignal}
{trigger}
/>
@@ -0,0 +1,73 @@
<!--
@component
The menu adding a pipeline script downstream of an asset: opened from the asset's
"+", or where that "+" was dropped on the blank canvas.
-->
<script lang="ts" module>
import type { ScriptLang } from '$lib/gen'
import type { AssetKind } from '$lib/components/assets/lib'
import type { PipelineInsertOptions } from './PipelineInsertMenu.svelte'
import type { PipelineOutputKind } from './pipelineTemplates'
export type AddScriptForAsset = (
asset: { kind: AssetKind; path: string },
language: ScriptLang,
scriptPath: string,
outputKind: PipelineOutputKind,
aiPrompt?: string,
options?: PipelineInsertOptions
) => void
</script>
<script lang="ts">
import { Code2 } from 'lucide-svelte'
import type { Snippet } from 'svelte'
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
let {
asset,
onAddScript,
pathPrefix = '',
defaultPathSuffix = '',
openSignal,
trigger
}: {
asset: { kind: AssetKind; path: string }
onAddScript: AddScriptForAsset
pathPrefix?: string
defaultPathSuffix?: string
openSignal?: number
trigger: Snippet<[{ open: boolean }]>
} = $props()
function handlePick(pick: PipelineInsertPick) {
if (pick.kindId === 'pipeline_script' && pick.language && pick.path) {
onAddScript(
asset,
pick.language as ScriptLang,
pick.path,
(pick.outputKind ?? 'none') as PipelineOutputKind,
pick.aiPrompt,
{ outputAsset: pick.outputAsset }
)
}
}
</script>
<PipelineInsertMenu
kinds={[
{
id: 'pipeline_script',
label: 'Add downstream pipeline script',
description: 'Triggered when this asset changes',
icon: Code2
}
]}
languages={PIPELINE_LANGUAGES as any}
{pathPrefix}
{defaultPathSuffix}
{openSignal}
onPick={handlePick}
{trigger}
/>
@@ -0,0 +1,23 @@
<!--
@component
The "+" on an asset's bottom edge: a circle at rest, widened into a labelled pill
while its menu is open or while it is being dragged. The label animates its
max-width (width can't transition to auto) and never wraps.
-->
<script lang="ts">
import { Plus } from 'lucide-svelte'
let { expanded }: { expanded: boolean } = $props()
</script>
<span
class="group/add h-6 min-w-6 rounded-full flex items-center justify-center bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:text-primary hover:border-gray-500 shadow-sm leading-none overflow-hidden transition-[padding] duration-200 ease-out data-[open=true]:pl-1"
data-open={expanded}
>
<Plus size={16} strokeWidth={2.5} class="shrink-0" />
<span
class="max-w-0 opacity-0 whitespace-nowrap text-3xs font-normal transition-[max-width,opacity,margin] duration-200 ease-out group-data-[open=true]/add:max-w-32 group-data-[open=true]/add:opacity-100 group-data-[open=true]/add:ml-1 group-data-[open=true]/add:mr-2"
>
Add downstream step
</span>
</span>
@@ -1,145 +1,36 @@
<script lang="ts">
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
import {
Plus,
Clock,
Webhook,
Mail,
Zap,
Radio,
MessageSquare,
Database,
Send,
CloudCog,
Upload
} from 'lucide-svelte'
import type { ScriptLang } from '$lib/gen'
import type { NativeTriggerKind } from './types'
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
import type { PipelineOutputKind } from './pipelineTemplates'
// Each left-column kind is just "pipeline script triggered by <trigger
// source>". id === the SCRIPT_TRIGGER_KIND value, so the handler can
// dispatch on it uniformly. Asset-triggered scripts are not in this
// menu; those live under the per-asset + inside the graph.
type KindId = NativeTriggerKind
import AddDataSourceMenu from './AddDataSourceMenu.svelte'
import { Plus } from 'lucide-svelte'
import { NODE } from '$lib/components/graph/util'
interface Props {
data: {
onAddPipelineScript: (
language: ScriptLang,
path: string,
source: { kind: NativeTriggerKind; path: string | undefined },
outputKind: PipelineOutputKind,
aiPrompt?: string
) => void
pathPrefix: string
defaultPathSuffix: string
}
data: Pick<
import('svelte').ComponentProps<typeof AddDataSourceMenu>,
'onAddPipelineScript' | 'pathPrefix' | 'defaultPathSuffix'
>
}
let { data }: Props = $props()
function handlePick(pick: PipelineInsertPick) {
if (!pick.language || !pick.path) return
const kindId = pick.kindId as KindId
const outputKind = (pick.outputKind ?? 'none') as PipelineOutputKind
// Native trigger annotation is marker-only — the binding lives on
// the trigger row's own `script_path`, which the user creates
// separately. Seed with `path: undefined`.
data.onAddPipelineScript(
pick.language as ScriptLang,
pick.path,
{ kind: kindId, path: undefined },
outputKind,
pick.aiPrompt
)
}
</script>
<PipelineInsertMenu
kinds={[
{
id: 'data_upload',
label: 'On data upload',
description: 'UI-first: run by uploading a file via the S3 picker',
icon: Upload
},
{
id: 'schedule',
label: 'On schedule',
description: 'Triggered by a schedule you create',
icon: Clock
},
{
id: 'webhook',
label: 'On webhook',
description: 'Triggered by an HTTP webhook',
icon: Webhook
},
{
id: 'email',
label: 'On email',
description: 'Triggered by incoming email',
icon: Mail
},
{
id: 'kafka',
label: 'On Kafka',
description: 'Triggered by a Kafka message',
icon: Zap
},
{
id: 'mqtt',
label: 'On MQTT',
description: 'Triggered by an MQTT message',
icon: Radio
},
{
id: 'amqp',
label: 'On AMQP',
description: 'Triggered by an AMQP (RabbitMQ) message',
icon: Radio
},
{
id: 'nats',
label: 'On NATS',
description: 'Triggered by a NATS message',
icon: MessageSquare
},
{
id: 'postgres',
label: 'On Postgres',
description: 'Triggered by a Postgres event',
icon: Database
},
{
id: 'sqs',
label: 'On SQS',
description: 'Triggered by an SQS message',
icon: Send
},
{
id: 'gcp',
label: 'On GCP Pub/Sub',
description: 'Triggered by a Pub/Sub message',
icon: CloudCog
}
]}
languages={PIPELINE_LANGUAGES as any}
pathPrefix={data.pathPrefix}
defaultPathSuffix={data.defaultPathSuffix}
onPick={handlePick}
>
{#snippet trigger()}
<!-- Quiet insert affordance, mirroring the flow editor's inline +
buttons (bg-surface + gray border + secondary text) — a filled
accent circle outweighed every real node on the canvas. -->
<button
type="button"
class="w-8 h-8 rounded-full flex items-center justify-center bg-surface border border-gray-400 dark:border-gray-600 text-secondary shadow-sm hover:bg-surface-hover transition-colors cursor-pointer"
title="Add to pipeline"
>
<Plus size={16} />
</button>
{/snippet}
</PipelineInsertMenu>
<!-- The layout gives this node a full node-width slot; centering the pill in
it lines it up with the nodes below whatever its label's width. -->
<div class="flex justify-center" style="width: {NODE.width}px;">
<AddDataSourceMenu
onAddPipelineScript={data.onAddPipelineScript}
pathPrefix={data.pathPrefix}
defaultPathSuffix={data.defaultPathSuffix}
>
{#snippet trigger()}
<!-- Quiet insert affordance, mirroring the flow editor's inline +
buttons (bg-surface + gray border + secondary text) — a filled
accent pill would outweigh every real node on the canvas. -->
<button
type="button"
class="h-8 px-3 rounded-full flex items-center gap-1.5 whitespace-nowrap bg-surface border border-gray-400 dark:border-gray-600 text-xs font-normal text-secondary shadow-sm hover:bg-surface-hover transition-colors cursor-pointer"
>
<Plus size={16} class="shrink-0" />
Add data source
</button>
{/snippet}
</AddDataSourceMenu>
</div>
File diff suppressed because it is too large Load Diff
@@ -20,10 +20,17 @@
SquareFunction,
Trash2,
X,
Pencil
Pencil,
ChevronDown,
Zap
} from 'lucide-svelte'
import Popover from '$lib/components/meltComponents/Popover.svelte'
import { tick } from 'svelte'
import DropdownV2 from '$lib/components/DropdownV2.svelte'
import {
upstreamScriptItems,
upstreamTriggerItems,
type UpstreamScript,
type UpstreamTriggerAction
} from './upstreamScriptItems'
import { inferArgs } from '$lib/infer'
import { emptySchema, sendUserToast } from '$lib/utils'
import type { Schema } from '$lib/common'
@@ -41,6 +48,7 @@
import { extractDraftMacros } from './resolveGraph'
import { assetColumnNodes, type ColumnLineageGraph } from './columnLineageGraph'
import SummaryPathDisplay from '$lib/components/SummaryPathDisplay.svelte'
import Badge from '$lib/components/common/badge/Badge.svelte'
import S3FilePreview from '$lib/components/S3FilePreview.svelte'
import DataTablePreview from './DataTablePreview.svelte'
import DucklakeAssetPanel from './DucklakeAssetPanel.svelte'
@@ -117,7 +125,9 @@
// (`onDraftPersist`). `onDraftPersist` stays the authoritative commit on
// navigate-away; this is the continuous feed that keeps the draft's
// server copy current while the user is still typing.
onContentChange?: (scriptPath: string | undefined, content: string) => void
// `base`: the deployed copy being edited, so the parent can autosave unsaved
// edits to it before they are promoted to a draft. Unset for a draft.
onContentChange?: (scriptPath: string | undefined, content: string, base?: Script) => void
// Fires when the user navigates away from a draft (selects another
// node, closes the pane, etc.) — the parent must persist the new
// content + write outputs into its drafts Map. Without this,
@@ -133,7 +143,8 @@
writes: { kind: AssetWithAltAccessType['kind']; path: string }[]
// Body-inferred reads, so the parent's draft keeps its input
// lineage while inactive (no live inference runs for it).
reads: { kind: AssetWithAltAccessType['kind']; path: string }[]
// Undefined when nothing was inferred: keep what the draft has.
reads?: { kind: AssetWithAltAccessType['kind']; path: string }[]
// Full edited script — set when the source is a persisted
// script (needed to seed a brand-new draft entry).
script?: Script
@@ -156,6 +167,12 @@
path: string
unsaved?: boolean
}>
/** The selected asset's producing scripts, for its "Edit script" action,
* which calls `onEditScript` with the one picked. */
producerScripts?: UpstreamScript[]
onEditScript?: (path: string) => void
/** The selected asset's upstream triggers that can be opened from here. */
producerTriggers?: UpstreamTriggerAction[]
// Pipeline-wide column-lineage graph (built by the parent page from the
// resolved graph). Drives the transitive column-lineage trace shown for a
// selected materialized asset.
@@ -215,15 +232,18 @@
// the edges. Counter rather than boolean so back-to-back runs
// re-fire the effect even if no other prop changed.
requestRunSignal?: number
// Folder-scoped non-editable prefix shown next to the suffix
// editor when the user renames a draft (e.g. `f/<folder>/`). The
// new path = pathPrefix + suffix.
pathPrefix?: string
// Called when the user renames a draft — the parent reseats the
// path key in its drafts map and updates activeDraftPath. Returns
// true on success; false on collision/validation failure so the
// popover can keep itself open and surface the error inline.
onDraftPathChange?: (oldPath: string, newPath: string) => boolean | string
onDraftMetaChange?: (
path: string,
meta: { summary: string; labels: string[] | undefined }
) => void
// The pipeline's folder prefix (`f/<folder>/`). A draft renamed outside it
// would be deployed out of the pipeline, so the rename is refused.
pathPrefix?: string
// Bumped by the parent (e.g. from the runnable-node action menu) to
// auto-open the archive/delete confirmation modal for the currently
// loaded persisted script. No-ops while the pane is showing a draft
@@ -280,6 +300,19 @@
// can persist a data-upload entry's staged input (drives node readiness).
// `isValid` is the full-schema validity, not just "file present".
onRunFormArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
// Show only the open script's inputs and runs (a data-upload click), not
// its editor. `onRunInputs` runs it with them, drafts included, and
// `onShowScript` leaves this view for the script.
inputsOnly?: boolean
/** `live` is the pane's buffer when it differs from what is stored: always
* for a draft, and for a deployed script edited before switching to the
* inputs view. A run with it previews that body. */
onRunInputs?: (
path: string,
args: Record<string, any>,
live?: { content: string; language: ScriptLang }
) => Promise<string | undefined>
onShowScript?: () => void
}
let {
selection,
@@ -301,6 +334,9 @@
onScriptRenamed,
onScriptRemoved,
selectionProducers = [],
producerScripts = [],
onEditScript,
producerTriggers = [],
selectionColumnGraph,
selectionColumnLoading = false,
selectionColumnTruncated = false,
@@ -315,8 +351,9 @@
activeRunnable,
onTestStateChange,
requestRunSignal,
pathPrefix = '',
onDraftPathChange,
onDraftMetaChange,
pathPrefix = '',
requestRemoveSignal,
downstreamSubscribers = 0,
onStartBoundedRun,
@@ -328,10 +365,15 @@
onRunByPath,
onRunCascadeByPath,
runFormInitialArgs,
onRunFormArgsChange
onRunFormArgsChange,
inputsOnly = false,
onRunInputs,
onShowScript
}: Props = $props()
let readOnly = $derived(mode !== 'edit')
// Someone who cannot switch the pipeline to edit mode only gets to read it.
let editScriptLabel = $derived(readOnly && !onRequestEdit ? 'Open script' : 'Edit script')
// Root element of the pane — used to scope the S3-input lookup for the
// data_upload focus effect.
@@ -408,7 +450,7 @@
// PipelineScriptView renders instead of the test panel.
let viewReadyForTarget = $derived.by(
() =>
readOnly &&
(readOnly || inputsOnly) &&
runTargetPath !== undefined &&
script?.path === runTargetPath &&
(isDraft || (!scriptRes.loading && scriptRes.current?.path === runTargetPath))
@@ -484,6 +526,12 @@
// alongside `liveBodyAssets` and forwarded so the live graph can show
// inferred column lineage on the edited script before it deploys.
let liveColumnLineage = $state<ColumnLineage[] | undefined>(undefined)
// The script whose editor last bound `liveBodyAssets` (undefined there means it
// inferred nothing). They stay bound after that editor unmounts, so a script
// that never had one (the inputs-only view) must not report or persist them as
// its lineage. Set by an effect declared after the save-on-switch one, whose
// cleanup still needs the previous owner when the selection moves.
let liveAssetsOwner: string | undefined = undefined
// Bumped when the runs panel reports a watched job has reached a
// terminal state. Drives S3FilePreview's refreshKey so the preview
@@ -549,7 +597,9 @@
let args = $state<Record<string, any>>({})
let argsSeedPath: string | undefined = undefined
$effect.pre(() => {
const p = script?.path
// Re-seeded on leaving the inputs-only view too, so the file picked there
// shows in the editor's test form.
const p = script?.path === undefined ? undefined : `${script.path}|${inputsOnly}`
if (p === argsSeedPath) return
argsSeedPath = p
args = runFormInitialArgs ? structuredClone($state.snapshot(runFormInitialArgs)) : {}
@@ -559,7 +609,8 @@
// required field (not just the S3 file) is satisfied.
let runFormIsValid = $state(true)
$effect(() => {
if (readOnly || !script) return
// Inputs-only mounts no ScriptEditor: PipelineScriptView reports its own args.
if (readOnly || inputsOnly || !script) return
onRunFormArgsChange?.(script.path, $state.snapshot(args), runFormIsValid)
})
@@ -596,18 +647,25 @@
b: Array<{ kind: string; path: string }>
) => a.length === b.length && a.every((x, i) => x.kind === b[i]?.kind && x.path === b[i]?.path)
return () => {
const writes = (liveBodyAssets ?? [])
// No editor inferred this script (the inputs-only view): keep its stored
// lineage rather than persisting it as empty.
const inferred = liveAssetsOwner === captured.path
const writes = !inferred
? (writesAtRegister ?? [])
: (liveBodyAssets ?? [])
.filter((a) => {
const t = a.access_type ?? a.alt_access_type
return t === 'w' || t === 'rw'
})
.map((a) => ({ kind: a.kind, path: a.path }))
const reads = (liveBodyAssets ?? [])
.filter((a) => {
const t = a.access_type ?? a.alt_access_type
return t === 'r' || t === 'rw'
})
.map((a) => ({ kind: a.kind, path: a.path }))
const reads = !inferred
? readsAtRegister
: (liveBodyAssets ?? [])
.filter((a) => {
const t = a.access_type ?? a.alt_access_type
return t === 'r' || t === 'rw'
})
.map((a) => ({ kind: a.kind, path: a.path }))
// Draft runs: emit only when content or lineage changed in THIS clone.
// An unconditional emit ping-pongs forever when the entry is rewritten
// externally while the pane stays mounted (rename rekey, AI edit):
@@ -617,6 +675,7 @@
(captured.content ?? '') === contentAtRegister &&
refsEq(writesAtRegister ?? [], writes) &&
readsAtRegister != undefined &&
reads != undefined &&
refsEq(readsAtRegister, reads)
)
return
@@ -651,12 +710,42 @@
}
})
$effect(() => {
if (script && !readOnly && !inputsOnly) liveAssetsOwner = script.path
})
let saving = $state(false)
// What this pane last deployed, so the persist-back cleanup can tell "the
// buffer equals the new deployed head" from real unsaved edits (plain
// variable: only read inside the untracked cleanup).
let deployedFromPane: { path: string; content: string } | undefined = undefined
let isDraft = $derived(draftScript != undefined)
function liveBody(): { content: string; language: ScriptLang } | undefined {
if (!script) return undefined
const live = { content: script.content ?? '', language: script.language as ScriptLang }
if (isDraft) return live
const orig = scriptRes.current
return orig?.path === script.path && (orig.content ?? '') !== live.content ? live : undefined
}
// A draft's stored schema is empty until an editor infers it, and the inputs
// view mounts none: infer from the body here so its fields (the file picker)
// show up.
const inputsSchema = resource(
[() => (inputsOnly && isDraft ? script?.content : undefined), () => script?.language],
async ([content, language]) => {
if (content === undefined || !language) return undefined
const schema = emptySchema()
try {
await inferArgs(language, content, schema as Schema)
} catch {
return undefined
}
return schema
}
)
let inputsScript = $derived(
script && inputsSchema.current ? { ...script, schema: inputsSchema.current } : undefined
)
// Single trash-bin button opens one modal that exposes both Archive
// (always available) and Delete permanently (admin-only). Archive is
@@ -763,12 +852,17 @@
onAnnotationsChange?.(script?.path, liveAnnotations)
})
$effect(() => {
if (readOnly) return
if (readOnly || (inputsOnly && liveAssetsOwner !== script?.path)) return
onAssetsChange?.(script?.path, liveBodyAssets ?? [], liveColumnLineage)
})
$effect(() => {
if (readOnly) return
onContentChange?.(script?.path, script?.content ?? '')
const base = draftScript ? undefined : scriptRes.current
onContentChange?.(
script?.path,
script?.content ?? '',
base && base.path === script?.path ? base : undefined
)
})
async function save() {
@@ -910,49 +1004,6 @@
JSON.stringify(script.schema ?? null) === JSON.stringify(orig.schema ?? null)
)
})
// Suffix editor for the draft-path popover. Seeded from the current
// path each time the popover opens so the user starts with what they
// see, not stale state from an earlier rename.
let draftPathSuffix = $state('')
let draftPathError = $state<string | undefined>(undefined)
let draftPathInput: HTMLInputElement | undefined = $state(undefined)
function suffixOf(fullPath: string): string {
return fullPath.startsWith(pathPrefix) ? fullPath.slice(pathPrefix.length) : fullPath
}
async function openDraftPathEditor() {
draftPathSuffix = script ? suffixOf(script.path) : ''
draftPathError = undefined
await tick()
draftPathInput?.focus()
draftPathInput?.select()
}
function confirmDraftPath(close: () => void) {
if (!script) return
const suffix = draftPathSuffix.trim()
if (!suffix) {
draftPathError = 'Path cannot be empty'
return
}
const newPath = pathPrefix + suffix
if (newPath === script.path) {
close()
return
}
const result = onDraftPathChange?.(script.path, newPath)
if (result === true || result === undefined) {
script.path = newPath
draftPathError = undefined
close()
} else if (typeof result === 'string') {
draftPathError = result
} else {
draftPathError = 'Path already in use'
}
}
</script>
<div class="flex flex-col h-full bg-surface" bind:this={paneEl}>
@@ -964,80 +1015,34 @@
{@const draftScriptPath = script.path}
<Code2 size={16} class="shrink-0 text-tertiary" />
{#if onDraftPathChange && !readOnly}
<!-- Inline rename popover for drafts. The persisted-script
branch uses SummaryPathDisplay which round-trips through
updateItemPathAndSummary; drafts have no server row yet,
so we just rekey the parent's drafts map locally. -->
<Popover
placement="bottom-start"
contentClasses="p-3"
usePointerDownOutside
on:openChange={(e) => {
if (e.detail) openDraftPathEditor()
<!-- Same summary/path popover as a deployed script, saved into the
drafts map instead of through the API: a draft has no row yet. -->
<SummaryPathDisplay
summary={script.summary ?? ''}
path={draftScriptPath}
labels={script.labels ?? []}
kind="script"
onSaved={() => {}}
saveOverride={({ path, summary, labels }) => {
if (pathPrefix && (!path.startsWith(pathPrefix) || path === pathPrefix)) {
return `A pipeline script stays in ${pathPrefix}: pick a name inside that folder`
}
// Before the rename, which carries the draft entry over as it is.
onDraftMetaChange?.(draftScriptPath, { summary, labels })
if (script) {
script.summary = summary
script.labels = labels
}
if (path === draftScriptPath) return undefined
const result = onDraftPathChange(draftScriptPath, path)
if (result !== true && result !== undefined) {
return typeof result === 'string' ? result : 'Path already in use'
}
if (script) script.path = path
return undefined
}}
>
{#snippet trigger()}
<button
type="button"
class="flex flex-col min-w-0 text-left px-2 py-1 rounded-md hover:bg-surface-hover transition-colors group"
title="Edit draft path"
>
<span
class="text-3xs uppercase tracking-wide text-tertiary flex items-center gap-1"
>
Draft pipeline script
<Pencil size={9} class="opacity-0 group-hover:opacity-60 transition-opacity" />
</span>
<span class="text-xs font-mono truncate" title={draftScriptPath}
>{draftScriptPath}</span
>
</button>
{/snippet}
{#snippet content({ close })}
<div class="flex flex-col gap-2 w-[420px]">
<span class="text-2xs font-normal text-secondary">Path</span>
<div
class="flex items-stretch border rounded-md bg-surface overflow-hidden focus-within:ring-2 focus-within:ring-blue-400"
>
{#if pathPrefix}
<span
class="flex items-center px-2 bg-surface-secondary text-tertiary text-sm font-mono border-r select-none"
>
{pathPrefix}
</span>
{/if}
<input
bind:this={draftPathInput}
bind:value={draftPathSuffix}
onkeydown={(e) => {
if (e.key === 'Enter') {
e.preventDefault()
confirmDraftPath(close)
} else if (e.key === 'Escape') {
e.preventDefault()
close()
}
}}
class="flex-1 min-w-0 px-2 py-1.5 text-sm font-mono bg-transparent focus:outline-none"
placeholder="my_script"
/>
</div>
{#if draftPathError}
<span class="text-2xs text-red-500">{draftPathError}</span>
{/if}
<div class="flex justify-end">
<Button
variant="accent"
unifiedSize="sm"
disabled={!draftPathSuffix.trim()}
onClick={() => confirmDraftPath(close)}
>
Rename
</Button>
</div>
</div>
{/snippet}
</Popover>
/>
<Badge color="gray">Draft</Badge>
{:else}
<div class="flex flex-col min-w-0">
<span class="text-3xs uppercase tracking-wide text-tertiary">
@@ -1097,6 +1102,66 @@
{/if}
</div>
<div class="flex items-center gap-1 shrink-0">
{#if selection?.kind === 'asset' && onEditScript && producerScripts.length === 1}
<Button
variant="default"
unifiedSize="sm"
startIcon={{ icon: Pencil }}
onclick={() => onEditScript(producerScripts[0].path)}
title={`Open ${producerScripts[0].path}, which builds this asset`}
>
{editScriptLabel}
</Button>
{:else if selection?.kind === 'asset' && onEditScript && producerScripts.length > 1}
<DropdownV2
items={upstreamScriptItems(producerScripts, onEditScript)}
placement="bottom-end"
enableFlyTransition
>
{#snippet buttonReplacement()}
<Button
nonCaptureEvent
variant="default"
unifiedSize="sm"
startIcon={{ icon: Pencil }}
endIcon={{ icon: ChevronDown }}
title={`${producerScripts.length} scripts build this asset`}
>
{editScriptLabel}
</Button>
{/snippet}
</DropdownV2>
{/if}
{#if selection?.kind === 'asset' && producerTriggers.length === 1}
<Button
variant="default"
unifiedSize="sm"
startIcon={{ icon: Zap }}
onclick={() => producerTriggers[0].onOpen()}
title={producerTriggers[0].detail}
>
{producerTriggers[0].label}
</Button>
{:else if selection?.kind === 'asset' && producerTriggers.length > 1}
<DropdownV2
items={upstreamTriggerItems(producerTriggers)}
placement="bottom-end"
enableFlyTransition
>
{#snippet buttonReplacement()}
<Button
nonCaptureEvent
variant="default"
unifiedSize="sm"
startIcon={{ icon: Zap }}
endIcon={{ icon: ChevronDown }}
title={`${producerTriggers.length} triggers start the scripts that build this asset`}
>
Triggers
</Button>
{/snippet}
</DropdownV2>
{/if}
{#if readOnly && onRequestEdit}
<Button
variant="accent"
@@ -1118,21 +1183,10 @@
title="Discard draft"
/>
{/if}
<!-- Action order, left → right: trash, external link, save, close.
Save sits closest to Close so the primary commit action is
anchored at the right edge of the bar; trash lives at the
far left so destructive ops are visually separated from
navigation/commit. Mirrors the draft Discard placement. -->
{#if !readOnly && !isDraft && isScriptView && script?.hash}
<Button
variant="subtle"
unifiedSize="sm"
startIcon={{ icon: Trash2 }}
onclick={() => (removeOpen = true)}
iconOnly
title="Archive or delete"
/>
{/if}
<!-- Action order, left → right: external link, save, close. Save sits
closest to Close so the primary commit action is anchored at the
right edge of the bar. Removing a script goes through its node's
menu. -->
{#if !readOnly && !isDraft && selection?.kind === 'runnable'}
<Button
variant="subtle"
@@ -1338,14 +1392,31 @@
{/if}
</div>
{:else if !isDraft && scriptRes.loading && !script}
<div class="absolute inset-0 flex items-center justify-center gap-2 text-tertiary">
<Loader2 size={16} class="animate-spin" />
<span class="text-xs">Loading script…</span>
</div>
<!-- Blank while the script loads: the fetch is short, and a spinner only flashes. -->
{:else if !isDraft && scriptRes.error}
<div class="p-3 text-xs text-red-500">
Failed to load: {scriptRes.error.message}
</div>
{:else if script && inputsOnly}
{#key script.path}
<PipelineScriptView
{isDraft}
canRun
runsDrafts
inputsOnly
{onShowScript}
onRun={onRunInputs && ((path, args) => onRunInputs(path, args, liveBody()))}
script={inputsScript ?? script}
{runsRefreshKey}
{runsPendingJobId}
initialArgs={runFormInitialArgs}
onArgsChange={onRunFormArgsChange}
onRunCompleted={() => {
previewRefreshKey += 1
onRunCompleted?.()
}}
/>
{/key}
{:else if script && readOnly}
<!-- Read-only modes: no Monaco/ScriptEditor (operators are
backend-blocked from previews anyway) — highlighted source,
@@ -1,6 +1,7 @@
<script lang="ts">
import { BaseEdge, getBezierPath, type EdgeProps } from '@xyflow/svelte'
import { NODE } from '$lib/components/graph/util'
import { PIPELINE_NODE_HEIGHT } from './assetGraphLayout'
import { FlaskConical, Columns3, SquareFunction, BellOff } from 'lucide-svelte'
import type { ColumnLineage, DataTest } from './parsePipelineAnnotations'
@@ -103,33 +104,42 @@
// straight under the nodes in between — the child-and-grandchild-of-the-
// same-node case, where the tidy-tree layout puts all three on one
// column. Detour those through the gutter beside the column.
const SKIP_DY = NODE.gap.vertical * 2 + NODE.height
const SKIP_DY = NODE.gap.vertical * 2 + PIPELINE_NODE_HEIGHT
const NEAR_VERTICAL_DX = NODE.width / 4
// Lane offset: half a node plus a margin that stays inside the
// inter-column gutter (gap.horizontal / 2).
const LANE = NODE.width / 2 + NODE.gap.horizontal / 2
const CORNER = 14
// Rounded-orthogonal detour: drop out of the source, swing into the
// gutter lane at `midX`, run it down past the intermediate rows, swing
// back into the target.
// Detour through the gutter lane at `midX`, drawn with the same curves as
// every other edge: curve out of the source into the lane within the gap
// below it, run down the lane past the rows in between, curve into the
// target within the gap above it.
function gutterPath(midX: number): string {
const yTop = sourceY + 24
const yBot = targetY - 24
const outDir = midX > sourceX ? 1 : -1
const inDir = targetX > midX ? 1 : -1
return [
`M ${sourceX} ${sourceY}`,
`L ${sourceX} ${yTop - CORNER}`,
`Q ${sourceX} ${yTop} ${sourceX + outDir * CORNER} ${yTop}`,
`L ${midX - outDir * CORNER} ${yTop}`,
`Q ${midX} ${yTop} ${midX} ${yTop + CORNER}`,
`L ${midX} ${yBot - CORNER}`,
`Q ${midX} ${yBot} ${midX + inDir * CORNER} ${yBot}`,
`L ${targetX - inDir * CORNER} ${yBot}`,
`Q ${targetX} ${yBot} ${targetX} ${yBot + CORNER}`,
`L ${targetX} ${targetY}`
].join(' ')
const bend = NODE.gap.vertical
const yTop = sourceY + bend
// Clamp only downward edges: an upward (feedback) edge must still run up the
// lane, or its last curve cuts across the node the detour was clearing.
const yBot = targetY > sourceY ? Math.max(yTop, targetY - bend) : targetY - bend
const [out] = getBezierPath({
sourceX,
sourceY,
sourcePosition,
targetX: midX,
targetY: yTop,
targetPosition,
curvature: 0.25
})
const [into] = getBezierPath({
sourceX: midX,
sourceY: yBot,
sourcePosition,
targetX,
targetY,
targetPosition,
curvature: 0.25
})
// Continue from the lane rather than lifting the pen.
return `${out} L${midX},${yBot} ${into.replace(/^M/, 'L')}`
}
// Long edges bend out of the source within the layer gap and descend in
@@ -178,7 +188,7 @@
path={edgePath}
{markerEnd}
{style}
interactionWidth={0}
interactionWidth={12}
label={undefined}
labelStyle={undefined}
/>
@@ -2,9 +2,14 @@
import { Handle, Position } from '@xyflow/svelte'
import { twMerge } from 'tailwind-merge'
import AssetGenericIcon from '$lib/components/icons/AssetGenericIcon.svelte'
import { formatShortAssetPath, type AssetKind } from '$lib/components/assets/lib'
import { NODE } from '$lib/components/graph/util'
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
import { formatAssetKind, formatShortAssetPath, type AssetKind } from '$lib/components/assets/lib'
import PipelineNodeCard from './PipelineNodeCard.svelte'
import NodeActionsMenu from './NodeActionsMenu.svelte'
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
import NodeOnDot from './NodeOnDot.svelte'
import { PIPELINE_NODE_HEADER } from './assetGraphLayout'
import AddDownstreamMenu from './AddDownstreamMenu.svelte'
import AddDownstreamPill from './AddDownstreamPill.svelte'
import {
ArrowUpRight,
Code2,
@@ -12,19 +17,28 @@
History,
Play,
Loader2,
Plus,
ShieldCheck,
ShieldAlert,
CheckCircle2,
ChevronDown,
Pencil,
CircleSlash,
Trash2,
Upload,
XCircle
} from 'lucide-svelte'
import { Button } from '$lib/components/common'
import type { ScriptLang } from '$lib/gen'
import { enterpriseLicense } from '$lib/stores'
import { sendUserToast } from '$lib/utils'
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
import { sendUserToast, type Item } from '$lib/utils'
import type { PipelineOutputKind } from './pipelineTemplates'
import type { DbtAssetProvenance } from './types'
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
import RunStateChip from './RunStateChip.svelte'
import DropdownV2 from '$lib/components/DropdownV2.svelte'
import { upstreamScriptItems } from './upstreamScriptItems'
import type { RunnableRunState } from './activeRunnables.svelte'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
const operatingWorkspace = useOperatingWorkspace()
@@ -33,6 +47,49 @@
// `content` / `language` so the page-level run handler can dispatch to
// `runScriptPreview` instead of `runScriptByPath` (which 404s for
// non-deployed scripts).
/** What starts a producing script, as its chip shows it. */
export type TriggerChipData = {
label: string
/** The native trigger kind; a data upload renders as a call to action. */
kind?: string
/** No chip: the node's incoming edges already show what starts it. */
hidden?: boolean
missing?: boolean
draft?: boolean
/** Opens the trigger's editor; unset for what has none (manual, asset change). */
onOpen?: () => void
selected?: boolean
}
export type AssetUpstreamChips =
| { none: true }
| {
multiple: true
/** The producing scripts; one when only its triggers are ambiguous. */
scripts: Array<{
runnableId: string
path: string
summary?: string
language?: ScriptLang
unsaved?: boolean
onOpen: () => void
}>
/** The scripts' own triggers, each named with its script. */
triggers: Array<TriggerChipData & { script: string }>
selected?: boolean
}
| {
runnableId: string
path: string
summary?: string
language?: ScriptLang
unsaved?: boolean
runState?: RunnableRunState
trigger: TriggerChipData
onOpen: () => void
selected?: boolean
}
export type AssetProducer = {
kind: 'script' | 'flow'
path: string
@@ -51,6 +108,8 @@
// "current view of <dim>" marker so it reads as a derived node, not an
// unrelated table.
derived_from?: string
// Why the node is misconfigured, when it is: renders it red.
error?: string
// dbt provenance when this warehouse table is a dbt node: which model
// it is, how dbt materializes it, its tags and its generic tests.
dbt?: DbtAssetProvenance
@@ -65,10 +124,14 @@
language: ScriptLang,
scriptPath: string,
outputKind: PipelineOutputKind,
aiPrompt?: string
aiPrompt?: string,
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
) => void
pathPrefix?: string
defaultPathSuffix?: string
/** Starts dragging the "+" onto a script (subscribing it to this asset)
* or onto the blank canvas (opening the add menu there). */
onStartFeedDrag?: (e: PointerEvent) => void
// Producer scripts/flows that write to this asset, supplied by
// the canvas from the graph's write/rw edges. Drives the on-hover
// "Run" button. Includes unsaved/draft producers so the button
@@ -99,6 +162,41 @@
// instead of only settling once the job ends.
runStatus?: 'running' | 'materialized' | 'failed'
runRowCount?: number | null
/** Set in the assets-only view, where the producing script and its
* trigger are folded into the asset: `none` for an asset nothing in the
* pipeline writes, `multiple` when the producer or its trigger is
* ambiguous. */
upstream?: AssetUpstreamChips
/** What is wrong with the node, and how to fix it: replaces the "+". */
fix?: NodeFix
/** Assets-only: errors of the scripts and triggers folded into this node. */
foldedFixes?: NodeFix[]
/** Assets-only: starts dragging the top dot onto an asset, subscribing the
* scripts that build this one to it. */
onStartOnDrag?: (e: PointerEvent) => void
/** The assets-only node for the scripts that build no asset. */
noAsset?: boolean
/** The No-asset node's script, by its file name. */
noAssetName?: string
/** Assets-only: the node is only its script's banner. */
bannerOnly?: boolean
/** Assets-only: a script folded into the node has an error. */
scriptError?: boolean
/** No script in the workspace writes it; a flow's write fires no asset trigger. */
neverWritten?: boolean
/** Card width in the assets-only view, sized to its text. */
width?: number
/** Assets-only view: the producing script and trigger no other asset
* shares, which the menu offers to delete one by one or together. */
upstreamDelete?: import('./assetsOnlyView').AssetUpstreamDelete
/** When no combined delete is possible: the folded scripts, each deletable
* on its own. */
scriptDeletes?: Array<{ path: string; unsaved: boolean }>
triggerDeletes?: Array<NonNullable<import('./assetsOnlyView').AssetUpstreamDelete['trigger']>>
onDeleteUpstream?: (target: import('./assetsOnlyView').AssetUpstreamDelete) => void
deleteVerb?: 'Delete' | 'Archive'
/** Why nothing can be deleted, when the producer is shared. */
deleteBlocked?: string
}
// SvelteFlow injects this on the node component when the user clicks
// the node. Combined with our own `hovered` state to drive the
@@ -140,19 +238,94 @@
}
}
function handlePick(pick: PipelineInsertPick) {
if (pick.kindId === 'pipeline_script' && pick.language && pick.path) {
data.onAddScript?.(
{ kind: data.asset_kind, path: data.path },
pick.language as ScriptLang,
pick.path,
(pick.outputKind ?? 'none') as PipelineOutputKind,
pick.aiPrompt
)
}
}
let showAdd = $derived(data.onAddScript != undefined)
// A red asset needs fixing where its error comes from, not a downstream step;
// and a step downstream of what nothing writes would never run on its writes.
let showAdd = $derived(data.onAddScript != undefined && !data.error && !data.neverWritten)
let menuItems: Item[] = $derived(
data.onDeleteUpstream && data.upstreamDelete
? deleteItems(data.upstreamDelete, data.onDeleteUpstream)
: data.onDeleteUpstream && (data.scriptDeletes?.length || data.triggerDeletes?.length)
? [
...(data.scriptDeletes ?? []).map((script) => ({ script })),
...(data.triggerDeletes ?? []).map((trigger) => ({ trigger }))
].flatMap((t) => deleteItems(t, data.onDeleteUpstream!))
: data.deleteBlocked
? [
{
displayName: `Can't delete: ${data.deleteBlocked}`,
icon: Trash2,
disabled: true
}
]
: []
)
const lastSegment = (p: string) => p.split('/').pop() ?? p
// A script is archived rather than deleted when the user may not delete it;
// a trigger is always deleted.
function deleteItems(
t: import('./assetsOnlyView').AssetUpstreamDelete,
run: (target: import('./assetsOnlyView').AssetUpstreamDelete) => void
): Item[] {
const items: Item[] = []
if (t.script) {
const script = t.script
items.push({
displayName: `${data.deleteVerb ?? 'Delete'} script ${lastSegment(script.path)}`,
icon: Trash2,
type: 'delete' as const,
action: () => run({ script })
})
}
if (t.trigger) {
const trigger = t.trigger
items.push({
displayName: `Delete ${trigger.kind} trigger ${lastSegment(trigger.path)}`,
icon: Trash2,
type: 'delete' as const,
action: () => run({ trigger })
})
}
if (t.script && t.trigger) {
items.push({
displayName: `${data.deleteVerb === 'Archive' ? 'Remove' : 'Delete'} all 2 items`,
icon: Trash2,
type: 'delete' as const,
action: () => run(t)
})
}
return items
}
let upstream = $derived(data.upstream)
let HEADER_SELECTED = $derived(
data.scriptError
? 'bg-red-200 dark:bg-red-600 hover:bg-red-200 dark:hover:bg-red-600'
: 'bg-surface-accent-selected hover:bg-surface-accent-selected text-accent'
)
const CHIP_SELECTED =
'bg-surface-accent-selected border-border-selected hover:bg-surface-accent-selected text-accent'
let producer = $derived(upstream && 'path' in upstream ? upstream : undefined)
let coProducers = $derived(upstream && 'scripts' in upstream ? upstream.scripts : [])
let hasHeader = $derived(!!producer || coProducers.length > 0)
// The corner controls sit on the card's corner, below the header tab:
// -top-2 plus PIPELINE_NODE_HEADER (22px).
const CARD_CORNER_TOP = 'top-[14px]'
let coSelected = $derived(!!(upstream && 'scripts' in upstream && upstream.selected))
let coProducerItems = $derived(
upstreamScriptItems(coProducers, (path) => coProducers.find((s) => s.path === path)?.onOpen())
)
let scriptMenuOpen = $state(false)
let coTriggers = $derived(upstream && 'triggers' in upstream ? upstream.triggers : [])
let coTriggerItems: Item[] = $derived(
coTriggers.map((t) => ({
displayName: `${t.label}${t.draft ? ' · draft' : ''}`,
description: coProducers.length > 1 ? t.script : undefined,
disabled: !t.onOpen,
action: () => t.onOpen?.()
}))
)
let triggerMenuOpen = $state(false)
// dbt badge. `materialized` is dbt's own word rather than the Windmill
// strategy because `view` and `ephemeral` have no strategy, and showing the
@@ -220,6 +393,185 @@
)
</script>
{#snippet scriptHeader()}
<!-- The script building this asset, folded into it: a strip across the top
naming it, which opens it. Several scripts open a menu to pick one. -->
{@const one = producer ?? (coProducers.length === 1 ? coProducers[0] : undefined)}
{@const headerClass = twMerge(
'flex items-center gap-1.5 w-full min-w-0 px-2 text-3xs font-normal leading-none text-secondary hover:bg-surface-hover hover:text-primary',
data.scriptError &&
'bg-red-100 dark:bg-red-700 text-red-700 dark:text-red-200 hover:bg-red-200 dark:hover:bg-red-600 hover:text-red-800'
)}
{#if one}
{@const isSelected = producer ? !!producer.selected : coSelected}
<button
type="button"
class={twMerge(headerClass, isSelected && HEADER_SELECTED)}
title={`Open ${one.summary ? `${one.summary} (${one.path})` : one.path}${one.unsaved ? ' · draft' : ''}`}
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
onclick={(e) => {
e.stopPropagation()
one.onOpen()
}}
>
{#if one.language}
<LanguageIcon lang={one.language} width={11} height={11} />
{:else}
<Code2 size={11} class="shrink-0" />
{/if}
<span class="truncate">{one.summary || one.path}</span>
{#if one.unsaved}<span class="shrink-0 text-tertiary">draft</span>{/if}
{#if producer?.runState}
<RunStateChip runState={producer.runState} class="ml-auto h-4 px-1 rounded" />
{/if}
<Pencil size={10} class={twMerge('shrink-0 opacity-70', !producer?.runState && 'ml-auto')} />
</button>
{:else if coProducers.length > 1}
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
class="w-full min-w-0 flex"
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
>
<DropdownV2
items={coProducerItems}
placement="bottom-start"
bind:open={scriptMenuOpen}
fixedHeight={false}
usePointerDownOutside
enableFlyTransition
class="w-full"
>
{#snippet buttonReplacement()}
<span
class={twMerge(headerClass, 'h-full', (coSelected || scriptMenuOpen) && HEADER_SELECTED)}
title={`Built by ${coProducers.length} scripts: pick one to open`}
>
{#each coProducers.slice(0, 2) as sc (sc.path)}
{#if sc.language}
<LanguageIcon lang={sc.language} width={11} height={11} />
{:else}
<Code2 size={11} class="shrink-0" />
{/if}
{/each}
<span class="truncate">{coProducers.length} scripts</span>
<ChevronDown size={11} class="shrink-0 ml-auto" />
</span>
{/snippet}
</DropdownV2>
</div>
{/if}
{/snippet}
{#snippet upstreamRow()}
{#if data.foldedFixes?.length}
<!-- A folded script or trigger is misconfigured: its Fix, one at a time. -->
<NodeFixButton
chip
fix={data.foldedFixes[0]}
label={`${data.foldedFixes.length} error${data.foldedFixes.length === 1 ? '' : 's'} · Fix`}
/>
{:else if upstream && coTriggers.length > 1 && !coTriggers.some((t) => t.onOpen)}
<!-- Several triggers, none editable here (View mode): a plain label. -->
<span
class="truncate rounded-md border px-1.5 py-1 text-3xs leading-none font-normal border-gray-300 dark:border-gray-600 text-secondary"
>Multiple triggers</span
>
{:else if upstream && coTriggers.length > 1}
<!-- Several triggers start what builds this asset: one chip, a menu to pick
the one to edit. -->
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
class="min-w-0 flex"
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
>
<DropdownV2
items={coTriggerItems}
placement="bottom-start"
bind:open={triggerMenuOpen}
fixedHeight={false}
usePointerDownOutside
enableFlyTransition
>
{#snippet buttonReplacement()}
<span
class={twMerge(
'flex items-center gap-1 min-w-0 rounded-md border px-1.5 py-1 text-3xs leading-none font-normal border-gray-300 dark:border-gray-600 text-secondary bg-surface hover:bg-surface-hover cursor-pointer',
(triggerMenuOpen || coTriggers.some((t) => t.selected)) && CHIP_SELECTED
)}
title="Started by several triggers: pick one to edit"
>
<span class="truncate">Multiple triggers</span>
<ChevronDown size={9} class="shrink-0 opacity-70" />
</span>
{/snippet}
</DropdownV2>
</div>
{:else if upstream}
{@const trigger = producer?.trigger ?? coTriggers[0]}
{@const triggerScript = producer?.path ?? coTriggers[0]?.script}
{@const chipClass = twMerge(
'truncate rounded-md border px-1.5 py-1 text-3xs leading-none font-normal',
trigger?.missing
? 'border-red-300 dark:border-red-600 text-red-700 dark:text-red-300'
: 'border-gray-300 dark:border-gray-600 text-secondary',
trigger?.draft && 'border-dashed',
trigger?.onOpen && 'bg-surface hover:bg-surface-hover cursor-pointer',
trigger?.selected && CHIP_SELECTED
)}
{#if trigger?.kind === 'data_upload' && trigger.onOpen}
<!-- The one trigger that needs the user: nothing runs until a file is
provided, so it reads as the node's main action rather than a chip. -->
<!-- svelte-ignore a11y_no_static_element_interactions -->
<span
class="flex min-w-0"
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
>
<Button
variant="accent"
unifiedSize="sm"
startIcon={{ icon: Upload }}
title={`Upload the data ${triggerScript} runs on`}
onClick={(e) => {
e?.stopPropagation()
trigger.onOpen?.()
}}
>
Upload data
</Button>
</span>
{:else if trigger?.onOpen}
<button
type="button"
class={twMerge(chipClass, 'flex items-center gap-1 min-w-0')}
title={trigger.missing
? `No ${trigger.label} trigger targets ${triggerScript} yet: click to create one`
: `Edit the ${trigger.label} trigger`}
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
onclick={(e) => {
e.stopPropagation()
trigger.onOpen?.()
}}
>
<span class="truncate">{trigger.label}{trigger.draft ? ' · draft' : ''}</span>
{#if !trigger.missing}<Pencil size={9} class="shrink-0 opacity-70" />{/if}
</button>
{:else}
<span class={chipClass}>
{#if 'none' in upstream}
External source
{:else if trigger}
{trigger.label}{trigger.draft ? ' · draft' : ''}
{/if}
</span>
{/if}
{/if}
{/snippet}
<!-- onmouseenter/leave on the wrapper (not the inner card) so the run
button — which floats outside the card — keeps the hover state alive
when the cursor moves between the card and the button. -->
@@ -236,123 +588,169 @@
<div class="wm-recompute-flash pointer-events-none absolute inset-0 z-10 rounded-md"></div>
{/key}
{/if}
<!-- Mirrors the flow editor's asset pill: quiet surface + gray border at
rest, accent reserved for the selected state. -->
<div
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden border',
'bg-surface-secondary border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500 transition-colors',
selected && 'bg-surface-accent-selected border-border-selected'
<NodeActionsMenu
items={menuItems}
hover={hovered}
kebabClass={twMerge(
showGuardBadge && '-right-8',
hasHeader && !data.bannerOnly && CARD_CORNER_TOP
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={data.path}
>
<!-- Data identity carries the accent (luminance blue), pairing with
the blue write edges that produce these assets — scripts stay
neutral, so script vs data reads at a glance. -->
<AssetGenericIcon
assetKind={data.asset_kind}
class={`shrink-0 ml-2 mr-2 ${selected ? 'text-accent' : 'text-blue-600 dark:text-blue-400'}`}
size="14px"
/>
{#if data.runStatus}
<!-- The run in view, per relation: a spinner while its producer is
building it, then its outcome. Left of the name so the eye finds
the moving nodes first on a wide graph. -->
<span
class="shrink-0 mr-1 {data.runStatus === 'failed'
? 'text-red-600 dark:text-red-400'
: data.runStatus === 'materialized'
? 'text-green-600 dark:text-green-400'
: 'text-blue-600 dark:text-blue-400'}"
title={data.runStatus}
{#if data.bannerOnly}
<!-- A script that builds nothing and runs only on writes to what it reads:
the incoming edges say how it starts, so its banner is the whole node. -->
<div
class={twMerge(
'flex items-stretch min-w-0 rounded-md overflow-hidden bg-surface border border-gray-200 dark:border-gray-700 drop-shadow-sm',
(producer ? producer.selected : coSelected) &&
'border-border-selected dark:border-border-selected'
)}
style="width: {data.width ?? 160}px; height: {PIPELINE_NODE_HEADER}px;"
>
{#if data.runStatus === 'running'}
<Loader2 size={11} class="animate-spin" />
{:else if data.runStatus === 'failed'}
<XCircle size={11} />
{@render scriptHeader()}
</div>
{:else}
<PipelineNodeCard
kindLabel={data.noAsset ? 'No asset' : formatAssetKind(asset)}
title={data.noAsset ? (data.noAssetName ?? '') : formatShortAssetPath(asset)}
tooltip={data.noAsset
? 'This script builds no asset'
: data.error
? `${data.path}: ${data.error}`
: data.path}
{selected}
tone={data.error ? 'error' : undefined}
subtitle={data.foldedFixes?.length ||
(upstream &&
!producer?.trigger?.hidden &&
!('triggers' in upstream && upstream.triggers.length === 0))
? upstreamRow
: undefined}
header={hasHeader ? scriptHeader : undefined}
headerTone={data.scriptError ? 'error' : undefined}
headerSelected={producer ? !!producer.selected : coSelected}
width={data.width}
surface={upstream ? 'primary' : 'secondary'}
>
{#snippet icon()}
<!-- Data identity carries the accent (luminance blue), pairing with
the blue write edges that produce these assets — scripts stay
neutral, so script vs data reads at a glance. -->
{#if data.noAsset}
<CircleSlash size={14} class="text-tertiary" />
{:else}
<CheckCircle2 size={11} />
<AssetGenericIcon
assetKind={data.asset_kind}
class={data.error
? 'text-red-600 dark:text-red-200'
: selected
? 'text-accent'
: 'text-blue-600 dark:text-blue-400'}
size="14px"
/>
{/if}
</span>
<!-- Rows the run wrote. Recorded per relation already, and the cheapest
answer to "did this model actually produce anything" — a model that
built green but emitted 0 rows is the failure that looks like a
success. Only once settled: mid-build the number is not yet real. -->
{#if data.runStatus === 'materialized' && data.runRowCount != undefined}
<span
class="shrink-0 mr-1 text-3xs tabular-nums text-tertiary"
title="{data.runRowCount} rows written by this run"
>
{Intl.NumberFormat().format(data.runRowCount)}
</span>
{/if}
{/if}
<span class="flex-1 min-w-0 pr-1 py-0.5 text-2xs font-mono text-emphasis truncate">
{formatShortAssetPath(asset)}
</span>
<!-- Fork data-environment chip: in a fork every asset shares its parent's
name, so the env it resolves to must read at a glance. Labeled + tinted
(amber "parent" = deferred read of the parent's current table via a
view; emerald "fork" = the fork's own materialized copy) rather than a
bare icon, which was too easy to miss. The title carries the detail. -->
{#if data.fork_materialization === 'deferred'}
<span
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300 border border-amber-300 dark:border-amber-700"
title="Deferred to parent workspace: reads the parent's current data. Materialize it in this fork to iterate on it."
>
<ArrowUpRight size={10} />
parent
</span>
{:else if data.fork_materialization === 'fork'}
<span
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border border-emerald-300 dark:border-emerald-700"
title="Materialized in this fork: reads and writes use the fork's isolated copy."
>
<GitFork size={10} />
fork
</span>
{/if}
<!-- dbt chip: names the materialization dbt declares, plus a test count
when the model carries generic tests. Orange keeps it visually
separate from the fork/SCD2 chips, which describe Windmill state.
Where the project node is on the graph it also stands in for the edge
to it: hovering lights that node up, clicking selects it. Where it is
not — the run page, and a pipeline holding a dbt project — there is
nothing to point at, so the chip renders inert. -->
{#if data.dbt}
<button
type="button"
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold tracking-wide bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300 border border-orange-300 dark:border-orange-700 {data.onDbtSelect
? 'hover:brightness-95 cursor-pointer'
: 'cursor-default'}"
title={dbtTitle}
onmouseenter={() => data.onDbtHover?.(true)}
onmouseleave={() => data.onDbtHover?.(false)}
onclick={(e) => {
e.stopPropagation()
data.onDbtSelect?.()
}}
>
<DbtIcon width={9} height={9} />
{dbtLabel}
{#if data.dbt.data_tests?.length}
<span class="opacity-70">&middot; {data.dbt.data_tests.length}T</span>
{/snippet}
{#snippet leading()}
{#if data.runStatus}
<!-- The run in view, per relation: a spinner while its producer is
building it, then its outcome. Left of the name so the eye finds
the moving nodes first on a wide graph. -->
<span
class="shrink-0 mr-1 {data.runStatus === 'failed'
? 'text-red-600 dark:text-red-400'
: data.runStatus === 'materialized'
? 'text-green-600 dark:text-green-400'
: 'text-blue-600 dark:text-blue-400'}"
title={data.runStatus}
>
{#if data.runStatus === 'running'}
<Loader2 size={11} class="animate-spin" />
{:else if data.runStatus === 'failed'}
<XCircle size={11} />
{:else}
<CheckCircle2 size={11} />
{/if}
</span>
<!-- Rows the run wrote. Recorded per relation already, and the cheapest
answer to "did this model actually produce anything" — a model that
built green but emitted 0 rows is the failure that looks like a
success. Only once settled: mid-build the number is not yet real. -->
{#if data.runStatus === 'materialized' && data.runRowCount != undefined}
<span
class="shrink-0 mr-1 text-3xs tabular-nums text-tertiary"
title="{data.runRowCount} rows written by this run"
>
{Intl.NumberFormat().format(data.runRowCount)}
</span>
{/if}
{/if}
</button>
{/snippet}
{#snippet trailing()}
<!-- Fork data-environment chip: in a fork every asset shares its parent's
name, so the env it resolves to must read at a glance. Labeled + tinted
(amber "parent" = deferred read of the parent's current table via a
view; emerald "fork" = the fork's own materialized copy) rather than a
bare icon, which was too easy to miss. The title carries the detail. -->
{#if data.fork_materialization === 'deferred'}
<span
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300 border border-amber-300 dark:border-amber-700"
title="Deferred to parent workspace: reads the parent's current data. Materialize it in this fork to iterate on it."
>
<ArrowUpRight size={10} />
parent
</span>
{:else if data.fork_materialization === 'fork'}
<span
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border border-emerald-300 dark:border-emerald-700"
title="Materialized in this fork: reads and writes use the fork's isolated copy."
>
<GitFork size={10} />
fork
</span>
{/if}
<!-- dbt chip: names the materialization dbt declares, plus a test count
when the model carries generic tests. Orange keeps it visually
separate from the fork/SCD2 chips, which describe Windmill state.
Where the project node is on the graph it also stands in for the edge
to it: hovering lights that node up, clicking selects it. Where it is
not — the run page, and a pipeline holding a dbt project — there is
nothing to point at, so the chip renders inert. -->
{#if data.dbt}
<button
type="button"
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold tracking-wide bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300 border border-orange-300 dark:border-orange-700 {data.onDbtSelect
? 'hover:brightness-95 cursor-pointer'
: 'cursor-default'}"
title={dbtTitle}
onmouseenter={() => data.onDbtHover?.(true)}
onmouseleave={() => data.onDbtHover?.(false)}
onclick={(e) => {
e.stopPropagation()
data.onDbtSelect?.()
}}
>
<DbtIcon width={9} height={9} />
{dbtLabel}
{#if data.dbt.data_tests?.length}
<span class="opacity-70">&middot; {data.dbt.data_tests.length}T</span>
{/if}
</button>
{/if}
<!-- SCD2 companion marker: this node is the `<dim>_current` "latest row
per key" view its producer maintains alongside the base dimension.
Icon-only (the pill already truncates); the title names the base. -->
{#if data.derived_from}
<span
class="shrink-0 mr-1.5 text-violet-600 dark:text-violet-400"
title={`SCD2 current view of ${data.derived_from}: latest row per key, maintained by the same producer as the base dimension.`}
>
<History size={12} />
</span>
{/if}
{/snippet}
</PipelineNodeCard>
{/if}
<!-- SCD2 companion marker: this node is the `<dim>_current` "latest row
per key" view its producer maintains alongside the base dimension.
Icon-only (the pill already truncates); the title names the base. -->
{#if data.derived_from}
<span
class="shrink-0 mr-1.5 text-violet-600 dark:text-violet-400"
title={`SCD2 current view of ${data.derived_from}: latest row per key, maintained by the same producer as the base dimension.`}
>
<History size={12} />
</span>
{/if}
</div>
</NodeActionsMenu>
{#if showGuardBadge}
<!-- Data-test outcome badge. Floats off the TOP-RIGHT corner (opposite the
left-edge run button and the bottom + inserter) so it never collides
@@ -361,6 +759,7 @@
<div
class={twMerge(
'absolute -top-2 -right-2 z-10 rounded-full w-5 h-5 grid place-items-center border shadow-sm',
hasHeader && CARD_CORNER_TOP,
guardClass
)}
title={guardTitle}
@@ -376,7 +775,13 @@
cards. Drafts are runnable too via runScriptPreview, so no
greyed-out state — the page-supplied callback handles the
dispatch. -->
<div class="absolute -left-3 top-1/2 -translate-y-1/2 z-10">
<!-- Centred on the card, below the header tab when there is one. -->
<div
class={twMerge(
'absolute -left-3 top-1/2 -translate-y-1/2 z-10',
hasHeader && !data.bannerOnly && 'top-[calc(50%+11px)]'
)}
>
<button
type="button"
onclick={runProducers}
@@ -398,53 +803,47 @@
</button>
</div>
{/if}
{#if showAdd}
<!-- Always-visible + below the asset for downstream pipeline-script
creation. Half-overlapping the bottom edge so it visually attaches
to the node like the flow editor's between-step inserter. -->
{#if data.onStartOnDrag}
<NodeOnDot onStart={data.onStartOnDrag} visible={hovered} />
{/if}
{#if data.fix}
<!-- A red node's way out replaces its "+": adding downstream of a broken
node is not what it needs. -->
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
<PipelineInsertMenu
kinds={[
{
id: 'pipeline_script',
label: 'Add downstream pipeline script',
description: 'Triggered when this asset changes',
icon: Code2
}
]}
languages={PIPELINE_LANGUAGES as any}
pathPrefix={data.pathPrefix ?? ''}
defaultPathSuffix={data.defaultPathSuffix ?? ''}
onPick={handlePick}
<NodeFixButton fix={data.fix} />
</div>
{:else if showAdd}
<!-- Always-visible + on the asset's bottom edge for downstream pipeline-
script creation, half-overlapping it like the flow editor's
between-step inserter. -->
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
<AddDownstreamMenu
asset={{ kind: data.asset_kind, path: data.path }}
onAddScript={data.onAddScript!}
pathPrefix={data.pathPrefix}
defaultPathSuffix={data.defaultPathSuffix}
>
{#snippet trigger()}
<!--
Sizing notes for the round + button:
- w-6/h-6 (24px) chosen so that with border-2 (2px each
side) the inner area is exactly 20px — divisible by
the 16px icon to leave a 2px gap on every side. At
20px / 12px before, the gap was 4px-each but the
odd rounding interacted badly with svelte-flow's
fractional zoom transforms and the icon drifted
half a pixel off-center on certain zoom levels.
- `grid place-items-center` instead of flex centering:
flex's baseline alignment introduces a sub-pixel
nudge on small elements that grid avoids.
- `leading-none` strips the default line-height
contribution that the SVG inherits via the parent's
text rendering, otherwise the icon is shifted
downward by ~0.5px at fractional zooms.
-->
{#snippet trigger({ open })}
<!-- A click opens the menu; a drag goes to the canvas, which
tells a real drag from a click by how far the pointer moves. -->
<button
type="button"
class="nopan nodrag block"
onclick={(e) => e.stopPropagation()}
class="bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:bg-surface-hover rounded-full w-6 h-6 grid place-items-center shadow-sm leading-none"
title="Add downstream pipeline script"
onpointerdown={(e) => {
if (e.button !== 0) return
e.preventDefault()
data.onStartFeedDrag?.(e)
}}
aria-label="Add downstream step"
title={data.onStartFeedDrag
? 'Click to add a downstream step, or drag onto a script to run it after each write'
: undefined}
>
<Plus size={16} strokeWidth={2.5} />
<AddDownstreamPill expanded={open} />
</button>
{/snippet}
</PipelineInsertMenu>
</AddDownstreamMenu>
</div>
{/if}
</div>
@@ -11,7 +11,8 @@
// can't be read), so a selectable version always exists in the table.
import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte'
import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte'
import { Table2, History, Columns3 } from 'lucide-svelte'
import { Table2, History, Columns3, Database } from 'lucide-svelte'
import DBManagerContent from '$lib/components/DBManagerContent.svelte'
import { Pane, Splitpanes } from 'svelte-splitpanes'
import { resource } from 'runed'
import { fetchDucklakeSnapshots } from '$lib/components/dbOps'
@@ -31,7 +32,7 @@
}
let { path, workspace, schemaCanEvolve = true }: Props = $props()
let tab = $state<'partitions' | 'history' | 'schema'>('partitions')
let tab = $state<'data' | 'partitions' | 'history' | 'schema'>('data')
// The user's explicit snapshot pick (undefined until they click a row).
let selectedVersion = $state<number | undefined>(undefined)
@@ -72,24 +73,28 @@
</script>
<div class="flex flex-col h-full">
{#if !qualifiedTable}
<!-- Catalog-level ducklake node (no table segment, e.g. `ducklake://main`):
snapshot history / time-travel are per-table, so only the partition
grid applies here. -->
<PartitionStatusGrid {path} {workspace} />
{:else}
<div class="flex items-center gap-2 px-3 py-2 border-b shrink-0">
<ToggleButtonGroup selected={tab} on:selected={(e) => (tab = e.detail)}>
{#snippet children({ item })}
<ToggleButton size="sm" value="partitions" label="Partitions" icon={Table2} {item} />
<div class="flex items-center gap-2 px-3 py-2 border-b shrink-0">
<ToggleButtonGroup selected={tab} on:selected={(e) => (tab = e.detail)}>
{#snippet children({ item })}
<ToggleButton size="sm" value="data" label="Data" icon={Database} {item} />
<ToggleButton size="sm" value="partitions" label="Partitions" icon={Table2} {item} />
<!-- Snapshot history and time-travel are per table: a catalog-level node
(`ducklake://main`, no table segment) has only its data and partitions. -->
{#if qualifiedTable}
<ToggleButton size="sm" value="schema" label="Schema" icon={Columns3} {item} />
<ToggleButton size="sm" value="history" label="History" icon={History} {item} />
{/snippet}
</ToggleButtonGroup>
</div>
{/if}
{/snippet}
</ToggleButtonGroup>
</div>
<div class="flex-1 min-h-0">
{#if tab === 'partitions'}
<div class="flex-1 min-h-0">
{#if tab === 'data' && parsed}
<!-- Keyed on the asset: the manager opens its table once, on mount. -->
{#key path}
<DBManagerContent input={parsed} {workspace} />
{/key}
{:else if tab === 'partitions' || !qualifiedTable}
<PartitionStatusGrid {path} {workspace} />
{:else if tab === 'schema'}
<SchemaHistoryPanel {path} {workspace} canEvolve={schemaCanEvolve} />
@@ -117,7 +122,6 @@
</Pane>
</Splitpanes>
</div>
{/if}
</div>
{/if}
{/if}
</div>
</div>
@@ -25,6 +25,8 @@
if (!nodesInitialized.current) return
if (!nodes.some((n) => n.type !== 'add')) return
fittedFor = fitKey
void fitView({ padding: 0.1, maxZoom: 1 })
// A re-fit over nodes that keep their ids finds them already measured, one
// render before their new positions reach the flow.
requestAnimationFrame(() => void fitView({ padding: 0.1, maxZoom: 1 }))
})
</script>
@@ -0,0 +1,68 @@
<!--
@component
A pipeline node's actions, reachable two ways: the kebab revealed on hover at the
node's top-right corner, and a right-click menu over the node carrying the same
items. Place it inside the node's `relative` wrapper, around the node's card.
-->
<script lang="ts">
import type { Snippet } from 'svelte'
import { twMerge } from 'tailwind-merge'
import { EllipsisVertical } from 'lucide-svelte'
import { stopPropagation, preventDefault } from 'svelte/legacy'
import DropdownV2 from '$lib/components/DropdownV2.svelte'
import ContextMenu from '$lib/components/common/contextmenu/ContextMenu.svelte'
import { contextMenuItemsFromMenu } from '$lib/components/common/contextmenu/fromMenuItems'
import type { Item } from '$lib/utils'
interface Props {
items: Item[]
/** The pointer is over the node: shows the kebab. */
hover: boolean
/** Moves the kebab, for a node with its own badge on that corner. */
kebabClass?: string
children: Snippet
}
let { items, hover, kebabClass, children }: Props = $props()
let menuOpen = $state(false)
let contextItems = $derived(contextMenuItemsFromMenu(items))
</script>
{#if items.length > 0}
<ContextMenu items={contextItems}>
{@render children()}
</ContextMenu>
<!-- Rendered only on hover or while open, so the canvas stays clean at rest.
`pointerdown` is stopped so svelte-flow doesn't select the node when the
user reaches for the menu. -->
<div
class={twMerge('absolute -top-2 -right-2 h-7 p-1 min-w-7', kebabClass)}
style="will-change: transform;"
>
<DropdownV2
{items}
placement="bottom-end"
bind:open={menuOpen}
fixedHeight={false}
usePointerDownOutside
enableFlyTransition
>
{#snippet buttonReplacement()}
<button
class={twMerge(
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
hover || menuOpen ? 'block' : '!hidden',
'shadow-md rounded-md'
)}
onpointerdown={stopPropagation(preventDefault(() => {}))}
title="Actions"
>
<EllipsisVertical size={12} />
</button>
{/snippet}
</DropdownV2>
</div>
{:else}
{@render children()}
{/if}
@@ -0,0 +1,124 @@
<!--
@component
The "Fix" pill on the bottom edge of a misconfigured (red) pipeline node: what is
wrong, and the actions that fix it. Styled like the node's "+" pill; the node's
red fill already says something is wrong.
-->
<script lang="ts" module>
export type NodeFixAction = {
label: string
/** The side effect, spelled out: what changes, where. */
detail?: string
/** `anchor`: where the Fix pill sits, for an action opening a menu there. */
run: (anchor: { x: number; y: number }) => void
}
/** Prose, with the entities it names as links that select them. */
export type FixExplainer = Array<string | { text: string; onClick: () => void }>
export type NodeFix = { explainer: FixExplainer; actions: NodeFixAction[] }
/** A fix as its supplier declares it: an action, or opening the schedule
* wizard (which the canvas owns). */
export type NodeFixSpec = {
explainer: FixExplainer
actions: Array<{
label: string
detail?: string
run?: () => void
/** Opens the schedule wizard for this script; its confirm calls `onSchedule`. */
scheduleFor?: {
script: string
onSchedule: (s: import('./PipelineInsertMenu.svelte').PipelineInsertSchedule) => void
}
}>
}
</script>
<script lang="ts">
import Popover from '$lib/components/meltComponents/Popover.svelte'
import { Button } from '$lib/components/common'
import { Wrench } from 'lucide-svelte'
let {
fix,
label = 'Fix',
chip = false
}: {
fix: NodeFix
label?: string
/** A small chip in a node's text, rather than the pill on its edge. */
chip?: boolean
} = $props()
let open = $state(false)
let anchorEl: HTMLElement | undefined = $state()
</script>
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
bind:this={anchorEl}
onpointerdown={(e) => e.stopPropagation()}
onkeydown={(e) => e.stopPropagation()}
onclick={(e) => e.stopPropagation()}
>
<Popover
bind:isOpen={open}
placement="bottom"
enableFlyTransition
usePointerDownOutside
contentClasses="p-3 w-96"
>
{#snippet trigger()}
{#if chip}
<span
class="flex items-center gap-1 rounded-md border px-1.5 py-1 text-3xs leading-none font-normal whitespace-nowrap bg-surface border-red-300 dark:border-red-600 text-red-700 dark:text-red-300 hover:bg-surface-hover"
>
<Wrench size={10} class="shrink-0" />
{label}
</span>
{:else}
<span
class="h-6 px-2 rounded-full flex items-center gap-1 bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:text-primary hover:border-gray-500 shadow-sm text-2xs font-normal leading-none whitespace-nowrap"
>
<Wrench size={12} class="shrink-0" />
{label}
</span>
{/if}
{/snippet}
{#snippet content()}
<div class="flex flex-col gap-3">
<p class="text-xs text-primary">
{#each fix.explainer as part}
{#if typeof part === 'string'}{part}{:else}<button
type="button"
class="text-accent hover:underline"
onclick={() => {
open = false
part.onClick()
}}>{part.text}</button
>{/if}
{/each}
</p>
<div class="flex flex-col gap-2">
{#each fix.actions as action, i (action.label)}
<div class="flex flex-col gap-0.5">
<Button
variant={i === 0 ? 'accent' : 'default'}
unifiedSize="sm"
onclick={() => {
const r = anchorEl?.getBoundingClientRect()
open = false
action.run({ x: r ? r.left + r.width / 2 : 0, y: r ? r.bottom : 0 })
}}
>
{action.label}
</Button>
{#if action.detail}
<span class="text-2xs text-secondary">{action.detail}</span>
{/if}
</div>
{/each}
</div>
</div>
{/snippet}
</Popover>
</div>
@@ -0,0 +1,23 @@
<!--
@component
The dot on a pipeline node's top edge: dragged onto an asset, it makes the node's
script run after each write to that asset.
-->
<script lang="ts">
let { onStart, visible }: { onStart: (e: PointerEvent) => void; visible: boolean } = $props()
</script>
<button
type="button"
class="absolute -top-1.5 left-1/2 -translate-x-1/2 z-10 w-3 h-3 rounded-full bg-surface border-2 border-blue-500 cursor-crosshair transition-opacity {visible
? 'opacity-100'
: 'opacity-0'}"
title="Drag onto an asset to run after each write to it"
aria-label="Drag onto an asset to run after each write to it"
onpointerdown={(e) => {
e.stopPropagation()
e.preventDefault()
onStart(e)
}}
onclick={(e) => e.stopPropagation()}
></button>
@@ -2,6 +2,7 @@
import { useSvelteFlow, type Node } from '@xyflow/svelte'
import { untrack } from 'svelte'
import { NODE } from '$lib/components/graph/util'
import { PIPELINE_NODE_HEIGHT } from './assetGraphLayout'
// Smoothly pans the viewport so a freshly-created node lands in the middle,
// keeping the current zoom. Lives inside <SvelteFlow> so useSvelteFlow() has
@@ -27,8 +28,8 @@
untrack(() => {
const node = ns.find((n) => n.id === id)
if (!node) return // not laid out yet — re-runs when `nodes` updates
const cx = node.position.x + NODE.width / 2
const cy = node.position.y + NODE.height / 2
const cx = node.position.x + (node.measured?.width ?? NODE.width) / 2
const cy = node.position.y + PIPELINE_NODE_HEIGHT / 2
const key = `${id}:${cx}:${cy}`
if (key === lastKey) return
lastKey = key
@@ -416,7 +416,7 @@
onmouseenter={() => onHoverRun?.([e.path])}
onmouseleave={() => onHoverRun?.(undefined)}
class={twMerge(
'w-full flex items-center gap-2 px-2 py-1 rounded-sm hover:bg-surface-hover text-xs text-left',
'w-full flex items-center gap-2 px-2 py-1.5 rounded-sm font-normal hover:bg-surface-hover text-xs text-left',
isMember && 'pl-7',
expanded && 'bg-surface-selected'
)}
@@ -445,7 +445,7 @@
{:else}
<Code2 size={12} class="shrink-0 text-tertiary" />
{/if}
<span class="flex-1 min-w-0 truncate font-mono text-2xs" title={e.path}>
<span class="flex-1 min-w-0 truncate text-xs" title={e.path}>
{e.path}
</span>
{#if showSource}
@@ -533,7 +533,7 @@
onclick={() => toggleGroup(g.key)}
onmouseenter={() => onHoverRun?.(g.members.map((m) => m.path))}
onmouseleave={() => onHoverRun?.(undefined)}
class="w-full flex items-center gap-2 px-2 py-1 rounded-sm hover:bg-surface-hover text-xs text-left"
class="w-full flex items-center gap-2 px-2 py-1.5 rounded-sm font-normal hover:bg-surface-hover text-xs text-left"
title={collapsed ? 'Expand cascade runs' : 'Collapse cascade runs'}
>
<span class="shrink-0 text-tertiary">
@@ -549,7 +549,7 @@
{/if}
</span>
<Workflow size={12} class="shrink-0 text-indigo-500" />
<span class="flex-1 min-w-0 truncate font-mono text-2xs" title={groupTitle(g)}>
<span class="flex-1 min-w-0 truncate text-xs" title={groupTitle(g)}>
{groupTitle(g)}
</span>
<span
@@ -1,40 +0,0 @@
<script lang="ts">
import { base } from '$lib/base'
import Alert from '$lib/components/common/alert/Alert.svelte'
import Button from '$lib/components/common/button/Button.svelte'
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
import { BookOpen } from 'lucide-svelte'
const DBT_DOCS = 'https://www.windmill.dev/docs/getting_started/scripts_quickstart/dbt'
</script>
<Alert type="info" title="Prefer using dbt? Windmill also runs dbt" size="xs">
<div class="flex flex-col gap-2">
<p>
A pipeline is Windmill's own abstraction; an existing dbt project stays a dbt project, and
runs here unchanged as its own kind of script, with dbt still owning its models, refs and
tests. All it needs is a warehouse configured under
<a class="underline" href="{base}/workspace_settings?tab=dbt">Settings → dbt</a>. Its models
show up in the asset graph as <span class="font-mono">dbt://</span> nodes, visible next to whatever
pipelines you do build, so you can adopt pipelines later, or never.
</p>
<div class="flex flex-row flex-wrap items-center gap-2">
<!-- The dbt mark goes in as a child rather than `startIcon`: that slot
sizes its icon with lucide's `size`, which DbtIcon has no prop for. -->
<Button variant="default" unifiedSize="sm" href="{base}/scripts/add?lang=dbt">
<DbtIcon width={14} height={14} />
New dbt script
</Button>
<Button
variant="subtle"
unifiedSize="sm"
href={DBT_DOCS}
target="_blank"
rel="noreferrer"
startIcon={{ icon: BookOpen }}
>
dbt documentation
</Button>
</div>
</div>
</Alert>
@@ -0,0 +1,49 @@
<script lang="ts">
import { AlertTriangle } from 'lucide-svelte'
import Popover from '$lib/components/meltComponents/Popover.svelte'
// Which drafts failed to deploy and why, anchored next to the deploy button so
// the reasons are one click away without leaving the editor. Deployed drafts
// leave the map; what is listed is still unresolved.
let {
errors,
open = $bindable()
}: {
errors: ReadonlyMap<string, string>
open?: boolean
} = $props()
</script>
<Popover
placement="bottom-end"
contentClasses="p-3 max-w-[480px]"
usePointerDownOutside
enableFlyTransition
bind:isOpen={open}
>
{#snippet trigger()}
<button
type="button"
class="flex items-center gap-1.5 px-2 py-1 rounded-md text-red-600 dark:text-red-400 bg-red-50 dark:bg-red-900/30 hover:bg-red-100 dark:hover:bg-red-900/50 transition-colors text-xs font-medium"
title="View save errors"
>
<AlertTriangle size={14} />
<span>{errors.size} failed</span>
</button>
{/snippet}
{#snippet content()}
<div class="flex flex-col gap-2">
<span class="text-xs font-semibold text-emphasis">Save errors</span>
<div class="flex flex-col gap-2 max-h-72 overflow-y-auto">
{#each [...errors.entries()] as [path, message] (path)}
<div class="flex flex-col gap-0.5 border-l-2 border-red-400 pl-2">
<span class="text-2xs font-mono text-emphasis">{path}</span>
<span class="text-2xs text-red-600 dark:text-red-400 break-words">
{message}
</span>
</div>
{/each}
</div>
</div>
{/snippet}
</Popover>
@@ -0,0 +1,47 @@
<script lang="ts">
import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte'
import type { PipelineTriggerDraft } from './types'
// Triggers start running once deployed, so deploying a pipeline's trigger
// drafts asks first, listing what will start.
let {
open,
triggerDrafts,
onConfirmed,
onCanceled
}: {
open: boolean
triggerDrafts: ReadonlyMap<string, PipelineTriggerDraft>
onConfirmed: () => void
onCanceled: () => void
} = $props()
</script>
<ConfirmationModal
{open}
title="Deploy triggers?"
confirmationText="Deploy"
type="info"
{onConfirmed}
{onCanceled}
>
<div class="flex flex-col gap-2 text-xs text-secondary">
<p>
{triggerDrafts.size === 1
? 'Saving the pipeline also creates this trigger. It starts running its script once deployed.'
: `Saving the pipeline also creates these ${triggerDrafts.size} triggers. They start running their scripts once deployed.`}
</p>
<ul class="flex flex-col gap-1">
{#each [...triggerDrafts] as [key, d] (key)}
<li class="flex flex-col">
<span class="font-mono text-2xs text-emphasis truncate">{d.config.path}</span>
<span class="text-2xs text-hint truncate">
{d.kind}{#if d.kind === 'schedule'}
· <span class="font-mono">{d.config.schedule}</span> ({d.config.timezone}){/if}
→ {d.config.script_path}
</span>
</li>
{/each}
</ul>
</div>
</ConfirmationModal>
@@ -18,11 +18,18 @@
mode?: 'view' | 'edit'
// Called right before navigating — lets a wrapping modal close itself.
onOpen?: () => void
// Only the folder picker, for hosts that don't list pipelines.
hideExisting?: boolean
}
let { currentFolder = undefined, mode = 'view', onOpen = undefined }: Props = $props()
let {
currentFolder = undefined,
mode = 'view',
onOpen = undefined,
hideExisting = false
}: Props = $props()
let pipelines = resource(
() => $workspaceStore,
() => (hideExisting ? undefined : $workspaceStore),
async (ws) => {
if (!ws) return [] as ListPipelineFoldersResponse
return await AssetService.listPipelineFolders({ workspace: ws })
@@ -49,50 +56,52 @@
</script>
<div class="flex flex-col gap-6 w-full min-w-0">
<section class="flex flex-col gap-2">
<h3 class="text-xs font-semibold text-secondary uppercase tracking-wide">
Existing pipelines
</h3>
{#if pipelines.loading && !pipelines.current}
<div class="text-tertiary text-sm flex items-center gap-2">
<Loader2 size={14} class="animate-spin" />
Loading…
</div>
{:else if pipelines.error}
<div class="text-red-500 text-sm">Failed: {pipelines.error.message}</div>
{:else if visiblePipelines.length === 0}
<div class="text-tertiary text-sm">
{currentFolder
? 'No other pipelines in this workspace.'
: 'No pipelines yet. A pipeline is any folder whose scripts carry pipeline annotations.'}
</div>
{:else}
<div class="flex flex-col border rounded-md overflow-hidden max-h-64 overflow-y-auto">
{#each visiblePipelines as p (p.folder)}
<button
type="button"
class="flex items-center justify-between px-3 py-2 border-b last:border-b-0 bg-surface hover:bg-surface-hover transition-colors text-left"
onclick={() => openExistingPipeline(p.folder)}
>
<span class="font-mono text-sm">f/{p.folder}</span>
<span class="text-2xs text-tertiary">
{p.script_count}
{p.script_count === 1 ? 'script' : 'scripts'}
</span>
</button>
{/each}
</div>
{/if}
</section>
{#if !hideExisting}
<section class="flex flex-col gap-2">
<h3 class="text-sm font-semibold text-emphasis">Existing pipelines</h3>
{#if pipelines.loading && !pipelines.current}
<div class="text-tertiary text-sm flex items-center gap-2">
<Loader2 size={14} class="animate-spin" />
Loading…
</div>
{:else if pipelines.error}
<div class="text-red-500 text-sm">Failed: {pipelines.error.message}</div>
{:else if visiblePipelines.length === 0}
<div class="text-tertiary text-sm">
{currentFolder
? 'No other pipelines in this workspace.'
: 'No pipelines yet. A pipeline is any folder whose scripts carry pipeline annotations.'}
</div>
{:else}
<div class="flex flex-col border rounded-md overflow-hidden max-h-64 overflow-y-auto">
{#each visiblePipelines as p (p.folder)}
<button
type="button"
class="flex items-center justify-between px-3 py-2 border-b last:border-b-0 bg-surface hover:bg-surface-hover transition-colors text-left"
onclick={() => openExistingPipeline(p.folder)}
>
<span class="font-mono text-sm">f/{p.folder}</span>
<span class="text-2xs text-tertiary">
{p.script_count}
{p.script_count === 1 ? 'script' : 'scripts'}
</span>
</button>
{/each}
</div>
{/if}
</section>
{/if}
{#if !$userStore?.operator}
<!-- Operators only view existing pipelines — opening an arbitrary
folder is a build-a-new-pipeline affordance. -->
<section class="flex flex-col gap-2">
<h3 class="text-xs font-semibold text-secondary uppercase tracking-wide">
Pick or create a folder
</h3>
<div class="flex items-center gap-2">
<section class="flex flex-col gap-1">
<h3 class="text-sm font-semibold text-emphasis">Pick or create a folder</h3>
<p class="text-xs text-secondary">
Each folder holds one pipeline, made of its pipeline scripts. Pick a folder to open its
pipeline, or create one to start a new pipeline.
</p>
<div class="flex items-center gap-2 mt-1">
<div class="flex-1 min-w-0">
<FolderPicker bind:folderName={pickedFolder} />
</div>
@@ -1,6 +1,15 @@
<script lang="ts">
import { untrack, type Snippet } from 'svelte'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
import { MousePointerClick } from 'lucide-svelte'
import Disposable from '$lib/components/common/drawer/Disposable.svelte'
import Portal from '$lib/components/Portal.svelte'
import { getOverlayHost } from '$lib/components/common/overlayHost.svelte'
import { useFlowPanelMode } from '$lib/components/flows/flowPanelMode.svelte'
import {
PIPELINE_DRAFT_KIND,
pipelineBundlePath,
pipelineLocalMirrorKey
} from '$lib/pipelinePaths'
import { Loader2 } from 'lucide-svelte'
import { Pane, Splitpanes } from 'svelte-splitpanes'
import { DraftService } from '$lib/gen'
@@ -11,6 +20,11 @@
import type { ColumnLineageGraph } from './columnLineageGraph'
import HideButton from '$lib/components/apps/editor/settingsPanel/HideButton.svelte'
import AssetGraphCanvas from './AssetGraphCanvas.svelte'
import { TRIGGER_NODE_STYLE } from './TriggerNode.svelte'
import { describeSchedule } from '$lib/utils/describeCron'
import type { UpstreamTriggerAction } from './upstreamScriptItems'
import type { AssetUpstreamDelete } from './assetsOnlyView'
import type { NodeFixSpec } from './NodeFixButton.svelte'
import AssetGraphDetailsPane from './AssetGraphDetailsPane.svelte'
import PipelineEventLog from './PipelineEventLog.svelte'
import type {
@@ -18,8 +32,10 @@
AssetGraphSelection,
NativeTriggerKind,
PipelineMode,
PipelineTriggerDraft,
DbtAssetProvenance
} from './types'
import { isDraftableTriggerKind, triggerDraftKey } from './pipelineTriggerDrafts'
import type { AssetKind, Script, ScriptLang } from '$lib/gen'
import type { RunnableRunState, PipelineEvent } from './activeRunnables.svelte'
import type { PipelineOutputKind } from './pipelineTemplates'
@@ -41,6 +57,7 @@
defaultPathSuffix = 'new_pipeline_script',
panelHidden = false,
onTogglePanelHidden,
modalPanel = false,
prefetchingAssets = false,
hoveredPaths = [],
selectedRunPaths = [],
@@ -65,6 +82,11 @@
onAddScriptForAsset,
onAddPipelineScript,
onRunnableMenuRemove,
onDeleteAssetUpstream,
nodeFixes,
assetDeleteVerb,
onEdgeAction,
onAddAssetTrigger,
onRunProducer,
idlePane,
onRequestEdit,
@@ -73,6 +95,9 @@
onRunCascadeByPath,
runFormInitialArgs,
onRunFormArgsChange,
inputsOnly = false,
onRunInputs,
onShowScript,
readyDataUploadPaths,
resolveLocalScript,
localScriptsVersion,
@@ -126,6 +151,9 @@
pathPrefix: string
defaultPathSuffix?: string
panelHidden?: boolean
/** Let the details pane open as a modal over a full-width graph when the editor
* is too narrow to dock it — the flow editor's step panel rule. */
modalPanel?: boolean
onTogglePanelHidden?: () => void
prefetchingAssets?: boolean
hoveredPaths?: string[]
@@ -153,16 +181,29 @@
language: ScriptLang,
scriptPath: string,
outputKind: PipelineOutputKind,
aiPrompt?: string
aiPrompt?: string,
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
) => void
onAddPipelineScript?: (
language: ScriptLang,
path: string,
source: { kind: NativeTriggerKind; path: string | undefined },
outputKind: PipelineOutputKind,
aiPrompt?: string
aiPrompt?: string,
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
) => void
onRunnableMenuRemove?: (...args: any[]) => void
onDeleteAssetUpstream?: (target: AssetUpstreamDelete) => void
nodeFixes?: ReadonlyMap<string, NodeFixSpec>
assetDeleteVerb?: 'Delete' | 'Archive'
onEdgeAction?: (a: {
action: 'listen' | 'stop'
targets: Array<{ script: string; asset: { kind: AssetKind; path: string }; reads: boolean }>
}) => void
onAddAssetTrigger?: (a: {
scripts: Array<{ script: string; reads: boolean }>
asset: { kind: AssetKind; path: string }
}) => void
onRunProducer?: (producer: RunProducer) => Promise<string | undefined>
idlePane?: Snippet
onRequestEdit?: () => void
@@ -175,8 +216,16 @@
// callback to persist them as they change — see the page's dataUploadArgs.
runFormInitialArgs?: Record<string, any>
onRunFormArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
/** See AssetGraphDetailsPane's `inputsOnly`. */
inputsOnly?: boolean
onRunInputs?: (
path: string,
args: Record<string, any>,
live?: { content: string; language: ScriptLang }
) => Promise<string | undefined>
onShowScript?: () => void
// Data-upload entry scripts whose staged upload is ready (green node).
readyDataUploadPaths?: Set<string>
readyDataUploadPaths?: ReadonlySet<string>
/** Local-dev (`/pipeline_dev`): resolve a node to its working-tree content
* so the details pane skips the (nonexistent) deployed-script fetch. May
* be async (to infer the args schema for the run form). */
@@ -247,18 +296,81 @@
? (editor.selection != undefined || editor.activeDraftPath != undefined) && !panelHidden
: !panelHidden
)
// Docked beside the graph, or in a modal over it (modal mode). In modal mode the
// details only show once asked for: a double-click on a node, a click on the
// node already selected, or a script's open button — never on a selection the
// editor makes itself (an AI-built node), which would pop a modal unasked.
const panelController = useFlowPanelMode({ enabled: () => modalPanel })
$effect(() => panelController.measure(containerWidth))
const panelMode = $derived(panelController.mode)
let panelModalOpen = $state(false)
const dockedPaneOpen = $derived(detailsPaneOpen && panelMode === 'docked')
$effect(() => {
if (untrack(() => panelModalOpen) && (panelMode === 'docked' || !detailsPaneOpen)) {
panelModalOpen = false
}
})
let panelDisposable: Disposable | undefined = $state(undefined)
// Disposable joins the overlay stack through its methods, not by watching `open`.
$effect(() => {
panelModalOpen
untrack(() => {
panelModalOpen ? panelDisposable?.openDrawer() : panelDisposable?.closeDrawer()
})
})
const overlayHost = getOverlayHost()
const modalHost = $derived(overlayHost?.el())
function selectableNodeAt(e: MouseEvent): HTMLElement | null {
return (e.target as HTMLElement | null)?.closest('.svelte-flow__node.selectable') ?? null
}
// Read in the capture phase: once the click bubbles, the canvas has applied
// its selection and a first click looks like a click on the selected node.
let clickStartedOnSelected = false
function noteSelectionBeforeClick(e: MouseEvent) {
clickStartedOnSelected = !!selectableNodeAt(e)?.classList.contains('selected')
}
function openPanelModalIfReselected(e: MouseEvent) {
if (clickStartedOnSelected && selectableNodeAt(e)) panelModalOpen = true
}
function openPanelModalFromGraph(e: MouseEvent) {
if (selectableNodeAt(e)) panelModalOpen = true
}
// A button in the pane that ends it closes the modal in the same update: left to
// the effect above, the modal would first render the pane without its selection.
function closePanelModal() {
if (panelMode === 'modal') panelModalOpen = false
}
function handleCanvasSelect(s: AssetGraphSelection | undefined, opts?: { open?: boolean }) {
onSelect(s)
if (opts?.open && s && panelMode === 'modal') panelModalOpen = true
}
// A data-upload node's click asks for its run form; in modal mode nothing shows
// it unless the modal opens too.
$effect(() => {
if (focusUploadSignal > 0 && untrack(() => panelMode) === 'modal') {
untrack(() => (panelModalOpen = true))
}
})
const showNodeHint = $derived(panelMode === 'modal' && !panelModalOpen)
const nodeHintText = $derived(
effectiveSelection
? 'Click the selected node to open it'
: 'Double click a node to open it'
)
let idleView = $derived(
mode !== 'edit' && editor.selection == undefined && editor.activeDraftPath == undefined
)
// Wrap the writes (and the rightPaneSize read in the else branch) in untrack so
// the effect tracks only `detailsPaneOpen` / `storedRightPaneSize` — without it,
// the effect tracks only `dockedPaneOpen` / `storedRightPaneSize` — without it,
// the Pane `bind:size` feedback loops the effect and pegs the main thread.
$effect(() => {
// Stacked (vertical) splits give the details pane more room — the run
// form + result need height more than the graph needs it.
const fallback = stacked ? 55 : 40
if (detailsPaneOpen) {
if (dockedPaneOpen) {
const restore = storedRightPaneSize
untrack(() => {
rightPaneSize = restore > 0 ? restore : fallback
@@ -272,6 +384,67 @@
}
})
// The selected asset's producing scripts, named and iconed from the graph, for
// the pane's "Edit script".
let producerScripts = $derived(
selectionProducers
.filter((p) => p.kind === 'script')
.map((p) => {
const r = displayGraph.runnables.find((x) => x.usage_kind === 'script' && x.path === p.path)
return { path: p.path, summary: r?.summary, language: r?.language }
})
)
// The selected asset's upstream triggers that can be opened from here, each
// with the same action its trigger node has: edit (edit mode), create a
// missing one (edit mode), or the webhook / data-upload drawers.
let producerTriggers = $derived.by((): UpstreamTriggerAction[] => {
const scripts = new Set(producerScripts.map((p) => p.path))
const seen = new Set<string>()
const out: UpstreamTriggerAction[] = []
for (const t of displayGraph.triggers ?? []) {
if (t.trigger_kind === 'asset' || !scripts.has(t.runnable_path)) continue
const kind = t.trigger_kind
const rowless = kind === 'webhook' || kind === 'data_upload'
const key = `${kind}:${rowless || t.missing ? t.runnable_path : t.path}`
if (seen.has(key)) continue
seen.add(key)
const label = TRIGGER_NODE_STYLE[kind]?.label ?? kind
const script = t.runnable_path
let action: UpstreamTriggerAction | undefined
if (kind === 'webhook') {
if (onOpenWebhook)
action = { label: 'Open webhook', detail: script, onOpen: () => onOpenWebhook(script) }
} else if (kind === 'data_upload') {
if (onOpenDataUpload)
action = { label: 'Upload data', detail: script, onOpen: () => onOpenDataUpload(script) }
} else if (t.missing && !rowless) {
if (onCreateMissingTrigger)
action = {
label: `Create ${label} trigger`,
detail: script,
onOpen: () => onCreateMissingTrigger(kind, script)
}
} else if (t.path && onEditTrigger) {
const path = t.path
action = {
label: kind === 'schedule' ? 'Edit schedule' : `Edit ${label} trigger`,
detail:
kind === 'schedule' && t.schedule
? (t.summary ?? describeSchedule(t.schedule, t.timezone) ?? path)
: path,
onOpen: () => onEditTrigger(kind, path, script)
}
}
if (action) out.push(action)
}
return out
})
function editProducerScript(path: string) {
if (mode !== 'edit') onRequestEdit?.()
onSelect({ kind: 'runnable', runnable_kind: 'script', path })
}
// ===================== Draft autosave (persistDrafts only) =====================
// All of this folder's in-flight drafts live in ONE per-user DB draft (typ
// `data_pipeline`) keyed at the folder, syncing across devices + surfacing in
@@ -279,12 +452,21 @@
// for the one-time migration below; the DB is the source of truth on load.
// FlowBuilder's autosave analogue — gated by `persistDrafts`.
let pipelineDraftPath = $derived(pipelineBundlePath(folder))
let storageKey = $derived(`pipeline-${folder}`)
type PipelineDraftBundle = { drafts: Array<[string, PipelineDraft]>; activeDraftPath?: string }
let storageKey = $derived(pipelineLocalMirrorKey(folder))
type PipelineDraftBundle = {
drafts: Array<[string, PipelineDraft]>
activeDraftPath?: string
triggerDrafts?: PipelineTriggerDraft[]
}
// Hydration is tracked on the editor instance (`editor.hydratedFromDb`), not a
// component flag: the in-session preview reuses one instance across editor
// hide/show, and it must hydrate ONCE per instance, not on every remount.
let lastPersistedBundle: string | undefined = undefined
// What decides whether the bundle needs saving: its drafts, not which one is
// open. Opening a draft is not an edit, so it rides along with the next real
// change instead of saving (and spinning the sync indicator) on a click.
const persistKey = (b: PipelineDraftBundle | undefined) =>
b ? JSON.stringify({ drafts: b.drafts, triggerDrafts: b.triggerDrafts }) : undefined
function restoreBundle(bundle: PipelineDraftBundle) {
if (Array.isArray(bundle.drafts)) {
@@ -300,6 +482,18 @@
if (loaded.size > 0) editor.drafts = loaded
}
if (typeof bundle.activeDraftPath === 'string') editor.activeDraftPath = bundle.activeDraftPath
if (Array.isArray(bundle.triggerDrafts)) {
editor.triggerDrafts = new Map(
bundle.triggerDrafts
.filter(
(d) =>
isDraftableTriggerKind(d?.kind) &&
typeof d.config?.path === 'string' &&
typeof d.config?.script_path === 'string'
)
.map((d) => [triggerDraftKey(d.kind, d.config.path), d])
)
}
}
function readLocalBundle(): PipelineDraftBundle | undefined {
@@ -311,7 +505,8 @@
if (s && (Array.isArray(s.drafts) || typeof s.activeDraftPath === 'string')) {
return {
drafts: Array.isArray(s.drafts) ? s.drafts : [],
activeDraftPath: typeof s.activeDraftPath === 'string' ? s.activeDraftPath : undefined
activeDraftPath: typeof s.activeDraftPath === 'string' ? s.activeDraftPath : undefined,
triggerDrafts: Array.isArray(s.triggerDrafts) ? s.triggerDrafts : undefined
}
}
} catch (e) {
@@ -339,8 +534,8 @@
fromServer = true
}
}
// A folder retarget during the await (the route-page header switcher, or a
// session retarget) changed the target and reset hydratedFromDb. Don't
// A folder switch during the await (the route page's header switcher)
// changed the target and reset hydratedFromDb. Don't
// apply this now-stale folder's bundle or mark the new folder hydrated —
// otherwise the stale result blocks the new folder's hydrate and its
// drafts bleed across folders.
@@ -359,7 +554,7 @@
{ workspace: ws ?? '', itemKind: PIPELINE_DRAFT_KIND, path },
migratedFromLocal ? undefined : serverSavedAt
)
if (!migratedFromLocal) lastPersistedBundle = bundle ? JSON.stringify(bundle) : undefined
if (!migratedFromLocal) lastPersistedBundle = persistKey(bundle)
} catch (e) {
console.warn('failed to load pipeline drafts', e)
} finally {
@@ -412,30 +607,52 @@
return [p, d] as [string, PipelineDraft]
return [p, { ...d, script, outputAssets, inputAssets }] as [string, PipelineDraft]
})
// A deployed script being edited is only promoted to a draft when its pane
// closes; until then its buffer is saved as the draft it will become.
const { scriptPath: livePath, content: liveBody, base } = editor.liveContent
if (
livePath != undefined &&
base &&
!editor.drafts.has(livePath) &&
liveBody !== (base.content ?? '')
) {
const live = editor.liveBodyAssets.scriptPath === livePath
const writes = live ? extractWrites(editor.liveBodyAssets.assets) : []
serialized.push([
livePath,
{
localId: editor.promotedDraftLocalId(livePath),
script: { ...base, content: liveBody },
outputAssets: writes.length > 0 ? writes : undefined,
inputAssets: live ? extractReads(editor.liveBodyAssets.assets) : undefined
}
])
}
const activePath = editor.activeDraftPath
const triggerDrafts = [...editor.triggerDrafts.values()]
const key = storageKey
const ws = workspace
const path = pipelineDraftPath
const hydrated = editor.hydratedFromDb
untrack(() => {
if (!hydrated) return
const isEmpty = serialized.length === 0 && !activePath
const isEmpty = serialized.length === 0 && !activePath && triggerDrafts.length === 0
const bundle: PipelineDraftBundle | undefined = isEmpty
? undefined
: { drafts: serialized, activeDraftPath: activePath }
: {
drafts: serialized,
activeDraftPath: activePath,
...(triggerDrafts.length > 0 ? { triggerDrafts } : {})
}
try {
if (typeof localStorage !== 'undefined') {
if (isEmpty) localStorage.removeItem(key)
else
localStorage.setItem(
key,
encodeState({ drafts: serialized, activeDraftPath: activePath })
)
else localStorage.setItem(key, encodeState(bundle))
}
} catch (e) {
console.warn('failed to mirror pipeline state', e)
}
const serializedBundle = bundle ? JSON.stringify(bundle) : undefined
const serializedBundle = persistKey(bundle)
if (serializedBundle === lastPersistedBundle) return
lastPersistedBundle = serializedBundle
if (!ws) return
@@ -450,10 +667,90 @@
})
</script>
<div class="h-full w-full" bind:clientWidth={containerWidth}>
{#snippet detailsBody(ws: string)}
{#if idleView && idlePane}
{@render idlePane()}
{:else}
<AssetGraphDetailsPane
{mode}
{onRequestEdit}
{canRunByPath}
{onRunByPath}
{onRunCascadeByPath}
{runFormInitialArgs}
{onRunFormArgsChange}
{inputsOnly}
{onRunInputs}
{onShowScript}
{resolveLocalScript}
{localScriptsVersion}
selection={activeDraft ? undefined : editor.selection}
selectionProducers={activeDraft ? [] : selectionProducers}
{producerScripts}
onEditScript={editProducerScript}
{producerTriggers}
{selectionColumnGraph}
{selectionColumnLoading}
{selectionColumnTruncated}
{selectionColumnFailed}
{selectionDbt}
{schemaCanEvolve}
{selectionForkMaterialization}
{schemaContractContext}
{runsRefreshKey}
{runsPendingJobId}
{activeRunnable}
{downstreamSubscribers}
onStartBoundedRun={canBoundedRunOpenScript &&
editor.openScriptPath &&
onStartBoundedRunForOpen
? () => onStartBoundedRunForOpen?.(editor.openScriptPath!)
: undefined}
{onRunCompleted}
{onTestStateChange}
{requestRemoveSignal}
{requestRunSignal}
{requestRunCascadeSignal}
{focusUploadSignal}
draftScript={activeDraft?.script}
draftOutputAssets={activeDraft?.outputAssets}
draftInputAssets={activeDraft?.inputAssets}
{onDraftPathChange}
onDraftMetaChange={editor.setDraftMeta}
{pathPrefix}
workspace={ws}
onAnnotationsChange={editor.handleAnnotationsChange}
onAssetsChange={editor.handleAssetsChange}
onContentChange={editor.handleContentChange}
onDraftPersist={editor.handleDraftPersist}
onclose={() => {
closePanelModal()
onClose()
}}
onHide={onTogglePanelHidden &&
(() => {
closePanelModal()
onTogglePanelHidden()
})}
{onDiscard}
{onDraftSaved}
{onPersistedSaved}
{onScriptRenamed}
{onScriptRemoved}
/>
{/if}
{/snippet}
<div class="relative h-full w-full" bind:clientWidth={containerWidth}>
<Splitpanes class="!h-full" horizontal={stacked}>
<Pane bind:size={leftPaneSize}>
<div class="relative h-full">
<!-- svelte-ignore a11y_click_events_have_key_events, a11y_no_static_element_interactions -->
<div
class="relative h-full"
ondblclick={panelMode === 'modal' ? openPanelModalFromGraph : undefined}
onpointerdowncapture={panelMode === 'modal' ? noteSelectionBeforeClick : undefined}
onclick={panelMode === 'modal' ? openPanelModalIfReselected : undefined}
>
<AssetGraphCanvas
graph={displayGraph}
selection={effectiveSelection}
@@ -470,10 +767,15 @@
{onOpenWebhook}
{onOpenDataUpload}
{readyDataUploadPaths}
onselect={onSelect}
onselect={handleCanvasSelect}
{onAddScriptForAsset}
{onAddPipelineScript}
{onRunnableMenuRemove}
{onDeleteAssetUpstream}
{nodeFixes}
{assetDeleteVerb}
{onEdgeAction}
{onAddAssetTrigger}
{onRunProducer}
{validStartPaths}
{onStartBoundedRun}
@@ -482,6 +784,8 @@
{panToNodeId}
showMinimap={!stacked}
viewportFitKey={viewportFitKey ?? folder}
assetsOnlyToggle
assetsOnlyFolder={workspace ? `${workspace}/${folder}` : undefined}
/>
{#if boundBar}{@render boundBar()}{/if}
{#if mode === 'edit'}
@@ -509,66 +813,47 @@
{/if}
</div></Pane
>
{#if detailsPaneOpen && workspace}
{#if dockedPaneOpen && workspace}
<Pane bind:size={rightPaneSize} minSize={25}>
{#if idleView && idlePane}
{@render idlePane()}
{:else}
<AssetGraphDetailsPane
{mode}
{onRequestEdit}
{canRunByPath}
{onRunByPath}
{onRunCascadeByPath}
{runFormInitialArgs}
{onRunFormArgsChange}
{resolveLocalScript}
{localScriptsVersion}
selection={activeDraft ? undefined : editor.selection}
selectionProducers={activeDraft ? [] : selectionProducers}
{selectionColumnGraph}
{selectionColumnLoading}
{selectionColumnTruncated}
{selectionColumnFailed}
{selectionDbt}
{schemaCanEvolve}
{selectionForkMaterialization}
{schemaContractContext}
{runsRefreshKey}
{runsPendingJobId}
{activeRunnable}
{downstreamSubscribers}
onStartBoundedRun={canBoundedRunOpenScript &&
editor.openScriptPath &&
onStartBoundedRunForOpen
? () => onStartBoundedRunForOpen?.(editor.openScriptPath!)
: undefined}
{onRunCompleted}
{onTestStateChange}
{requestRemoveSignal}
{requestRunSignal}
{requestRunCascadeSignal}
{focusUploadSignal}
draftScript={activeDraft?.script}
draftOutputAssets={activeDraft?.outputAssets}
draftInputAssets={activeDraft?.inputAssets}
{pathPrefix}
{onDraftPathChange}
{workspace}
onAnnotationsChange={editor.handleAnnotationsChange}
onAssetsChange={editor.handleAssetsChange}
onContentChange={editor.handleContentChange}
onDraftPersist={editor.handleDraftPersist}
onclose={onClose}
onHide={onTogglePanelHidden}
{onDiscard}
{onDraftSaved}
{onPersistedSaved}
{onScriptRenamed}
{onScriptRemoved}
/>
{/if}
{@render detailsBody(workspace)}
</Pane>
{/if}
</Splitpanes>
{#if showNodeHint}
<!-- Above the event log's bar, which owns the canvas's bottom edge. -->
<div
class="pointer-events-none absolute bottom-10 left-3 z-30 flex items-center gap-1.5 text-xs text-hint"
>
<MousePointerClick size={13} />
{nodeHintText}
</div>
{/if}
</div>
<!-- Portalled to the host's overlay anchor (a session preview tab) so the modal covers
that tab rather than the whole page; Disposable owns its place in the overlay stack
and its Escape. -->
<Disposable bind:open={panelModalOpen} bind:this={panelDisposable}>
{#snippet children({ zIndex })}
{#if panelMode === 'modal' && panelModalOpen && workspace}
<Portal target={modalHost ?? 'body'} class="contents">
<!-- svelte-ignore a11y_click_events_have_key_events -->
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
class="{modalHost ? 'absolute' : 'fixed'} inset-0 flex justify-center px-2 py-6"
style="z-index: {zIndex}"
role="dialog"
>
<div class="absolute inset-0 bg-black/20" onclick={() => (panelModalOpen = false)}></div>
<div
class="relative flex w-full max-w-4xl flex-col overflow-hidden rounded-md border bg-surface shadow-xl"
>
<div class="min-h-0 flex-1 overflow-auto">
{@render detailsBody(workspace)}
</div>
</div>
</div>
</Portal>
{/if}
{/snippet}
</Disposable>
@@ -1,6 +1,7 @@
<script lang="ts" module>
import type { ComponentType } from 'svelte'
import type { SupportedLanguage } from '$lib/common'
import type { AssetKind, NewSchedule } from '$lib/gen'
export type PipelineInsertKind = {
// Machine-readable id; drives which right-column panel renders.
@@ -8,6 +9,9 @@
label: string
description?: string
icon?: ComponentType
// Its trigger is configured in its own editor right after the script is
// created, so the wizard's last button reads "Configure".
configuredAfterCreate?: boolean
}
export type PipelineInsertPick = {
@@ -22,7 +26,18 @@
// from this prompt via AI (using language + outputKind + the
// upstream input as context) instead of using the seeded template.
aiPrompt?: string
// Set by the schedule step when the picked kind is `schedule`.
schedule?: PipelineInsertSchedule
// Set by the asset step for the table output kinds, replacing the
// generated name.
outputAsset?: { kind: AssetKind; path: string }
}
export type PipelineInsertOptions = Pick<PipelineInsertPick, 'schedule' | 'outputAsset'>
// Everything but the target, which is the script being created. An empty
// `path` means the default `<script>_schedule`.
export type PipelineInsertSchedule = Omit<NewSchedule, 'script_path'>
</script>
<script lang="ts">
@@ -36,10 +51,28 @@
PIPELINE_OUTPUT_KINDS,
type PipelineOutputKind
} from './pipelineTemplates'
import CronInput from '$lib/components/CronInput.svelte'
import type { ScriptLang } from '$lib/gen'
import Label from '$lib/components/Label.svelte'
import TextInput from '$lib/components/text_input/TextInput.svelte'
import { CornerDownLeft, Sparkles } from 'lucide-svelte'
import TextInput, {
inputBaseClass,
inputBorderClass,
inputSizeClasses
} from '$lib/components/text_input/TextInput.svelte'
import DucklakePicker from '$lib/components/DucklakePicker.svelte'
import DatatablePicker from '$lib/components/DatatablePicker.svelte'
import Path from '$lib/components/Path.svelte'
import Section from '$lib/components/Section.svelte'
import Toggle from '$lib/components/Toggle.svelte'
import DateTimeInput from '$lib/components/DateTimeInput.svelte'
import ScheduleAdvancedOptions, {
emptyScheduleAdvanced,
loadDefaultScheduleAdvanced,
scheduleAdvancedCfg
} from '$lib/components/triggers/schedules/ScheduleAdvancedOptions.svelte'
import { workspaceStore } from '$lib/stores'
import { ArrowLeft, ChevronDown, CornerDownLeft, Loader2, Sparkles } from 'lucide-svelte'
import { tick, untrack } from 'svelte'
import { arrowTabNav } from '$lib/attachments/arrowTabNav'
import { selectAndAdvanceTo } from '$lib/attachments/selectAndAdvanceTo'
@@ -49,8 +82,15 @@
pathPrefix?: string
defaultPathSuffix?: string
onPick: (pick: PipelineInsertPick) => void
trigger: import('svelte').Snippet
/** Receives whether the menu is open, e.g. to keep a hover state while it is. */
trigger: import('svelte').Snippet<[{ open: boolean }]>
placement?: 'bottom' | 'top' | 'left' | 'right'
/** Opens the menu each time it changes, for an entry point other than the trigger. */
openSignal?: number
/** With `openSignal`: only the schedule step, for this existing script; its
* confirm hands the schedule to `onSchedule` instead of creating a script. */
scheduleFor?: string
onSchedule?: (schedule: PipelineInsertSchedule) => void
}
let {
@@ -60,7 +100,10 @@
trigger: triggerSnippet,
placement = 'bottom',
defaultPathSuffix,
onPick
onPick,
openSignal,
scheduleFor,
onSchedule
}: Props = $props()
// When there's only one trigger kind, hide the Trigger column entirely
@@ -75,6 +118,107 @@
aiPrompt: undefined as undefined | string
})
let selected = $state(buildEmptySelected())
let menuOpen = $state(false)
$effect(() => {
if (!openSignal) return
untrack(() => {
if (scheduleFor) {
resetWizard()
selected.triggerId = 'schedule'
step = 1
config.schedulePath = `${scheduleFor}_schedule`
loadAdvancedDefaults()
void focusCurrentStep()
}
menuOpen = true
})
})
let selectedKind = $derived(kinds.find((k) => k.id === selected.triggerId))
// Steps after the first, each configuring something the picks call for.
type ConfigStep = 'schedule' | 'asset'
const NAMED_OUTPUT_KINDS: PipelineOutputKind[] = ['materialize', 'datatable', 'ducklake']
let configSteps = $derived.by<ConfigStep[]>(() => {
const steps: ConfigStep[] = []
if (selected.triggerId === 'schedule') steps.push('schedule')
if (selected.outputId && NAMED_OUTPUT_KINDS.includes(selected.outputId)) steps.push('asset')
return steps
})
// 0 is the trigger/language/output/path step; `n` is `configSteps[n - 1]`.
let step = $state(0)
let currentConfigStep = $derived(step > 0 ? configSteps[step - 1] : undefined)
let isLastStep = $derived(step >= configSteps.length)
const buildEmptyConfig = () => ({
schedule: '0 0 12 * *',
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
cronVersion: 'v2',
validCron: true,
// Follows the script path until the Path field is edited.
schedulePath: '',
schedulePathDirty: false,
schedulePathError: '',
summary: '',
description: '',
pauseUntil: false,
pausedUntil: undefined as string | undefined,
advanced: emptyScheduleAdvanced(),
// The workspace's default handlers, loaded when the schedule step opens.
advancedDefaults: 'unloaded' as 'unloaded' | 'loading' | 'loaded',
// `store` is the data table or DuckLake catalog name. The table name
// follows the script name until the user types in it.
asset: undefined as
| undefined
| { kind: 'ducklake' | 'datatable'; store: string | undefined; table: string },
tableEdited: false
})
let config = $state(buildEmptyConfig())
// A new schedule starts from the workspace's default handlers, as in the schedule
// drawer. The step cannot be confirmed until they are in.
function loadAdvancedDefaults() {
const ws = $workspaceStore
if (!ws || config.advancedDefaults !== 'unloaded') return
const target = config
target.advancedDefaults = 'loading'
loadDefaultScheduleAdvanced(ws)
.then((d) => (target.advanced = d))
.catch(() => {})
.finally(() => (target.advancedDefaults = 'loaded'))
}
function resetWizard() {
selected = buildEmptySelected()
config = buildEmptyConfig()
step = 0
}
const TABLE_NAME_RE = /^[^/\s]+$/
// The path grammar of `Path.svelte`: word segments, no empty or trailing ones.
const SCRIPT_NAME_RE = /^[\w-]+(\/[\w-]+)*$/
let scriptNameValid = $derived(SCRIPT_NAME_RE.test(selected.scriptPath.trim()))
let assetValid = $derived(!!config.asset?.store && TABLE_NAME_RE.test(config.asset.table.trim()))
let stepValid = $derived(
currentConfigStep === 'schedule'
? config.validCron &&
!!config.schedule.trim() &&
!config.schedulePathError &&
config.advancedDefaults === 'loaded'
: currentConfigStep === 'asset'
? assetValid
: scriptNameValid && !!selected.triggerId && !!selected.language && !!selected.outputId
)
function enterAssetStep() {
const kind = selected.outputId === 'datatable' ? 'datatable' : 'ducklake'
if (config.asset?.kind !== kind) {
config.asset = { kind, store: 'main', table: '' }
config.tableEdited = false
}
if (!config.tableEdited && config.asset) {
config.asset.table = selected.scriptPath.trim().split('/').pop() ?? ''
}
}
// Refs to the column containers so Enter inside one column can hand
// focus off to the first tabbable in the next column.
@@ -104,16 +248,73 @@
return out
}
// A step change re-renders the popover body, dropping focus to <body>, where
// keystrokes (Enter, Cmd+Enter) no longer reach the step's key handlers.
let contentEl: HTMLElement | undefined = $state()
async function focusCurrentStep() {
await tick()
const target =
step === 0
? pathEl?.querySelector('input')
: contentEl?.querySelector<HTMLElement>('[data-autofocus] input')
target?.focus()
}
function goBack() {
step -= 1
void focusCurrentStep()
}
function scheduleConfig(): PipelineInsertSchedule {
return {
path: config.schedulePath,
schedule: config.schedule.trim(),
timezone: config.timezone,
cron_version: config.cronVersion,
summary: config.summary.trim() || undefined,
description: config.description,
paused_until: config.pauseUntil ? config.pausedUntil : undefined,
is_flow: false,
args: {},
...scheduleAdvancedCfg($state.snapshot(config.advanced))
}
}
function confirm(close: () => void) {
if (scheduleFor) {
if (!stepValid) return
onSchedule?.(scheduleConfig())
close()
return
}
const suffix = selected.scriptPath.trim()
if (!suffix || !selected.triggerId || !selected.language || !selected.outputId) return
if (!stepValid) return
if (!isLastStep) {
step += 1
if (configSteps[step - 1] === 'asset') enterAssetStep()
if (configSteps[step - 1] === 'schedule') {
loadAdvancedDefaults()
if (!config.schedulePathDirty) config.schedulePath = `${pathPrefix}${suffix}_schedule`
}
void focusCurrentStep()
return
}
const trimmedPrompt = selected.aiPrompt?.trim()
onPick({
kindId: selected.triggerId,
language: selected.language,
path: pathPrefix + suffix,
outputKind: selected.outputId,
aiPrompt: trimmedPrompt && trimmedPrompt.length > 0 ? trimmedPrompt : undefined
aiPrompt: trimmedPrompt && trimmedPrompt.length > 0 ? trimmedPrompt : undefined,
schedule: configSteps.includes('schedule') ? scheduleConfig() : undefined,
outputAsset:
configSteps.includes('asset') && config.asset
? {
kind: config.asset.kind as AssetKind,
path: `${config.asset.store}/${config.asset.table.trim()}`
}
: undefined
})
close()
}
@@ -123,9 +324,18 @@
enableFlyTransition
contentClasses={twMerge(
'p-0 bg-surface overflow-hidden relative transition-height',
showBottomPanel ? 'h-[26rem]' : 'h-[22rem]'
// The first step's columns (w-56 + w-48 + w-80), held across steps.
singleKind ? 'w-[32rem]' : 'w-[46rem]',
currentConfigStep === 'schedule'
? 'h-[27rem]'
: currentConfigStep === 'asset'
? 'h-[14rem]'
: showBottomPanel
? 'h-[26rem]'
: 'h-[22rem]'
)}
class="inline-block"
bind:isOpen={menuOpen}
usePointerDownOutside
floatingConfig={{
placement,
@@ -136,24 +346,30 @@
fitViewport: true,
overlap: false
}}
onClose={() => (selected = buildEmptySelected())}
onClose={resetWizard}
>
{#snippet trigger()}
{@render triggerSnippet?.()}
{@render triggerSnippet?.({ open: menuOpen })}
{/snippet}
{#snippet content({ close })}
<div class="flex flex-col h-full">
<div class={'flex flex-row transition-height divide-x overflow-y-scroll'}>
{@render topSection()}
</div>
<div
class={twMerge(
'flex flex-col gap-5 grow transition-height px-4 border-t',
showBottomPanel ? 'h-[14rem] py-4' : 'h-0'
)}
>
{@render bottomSection(close)}
</div>
<div class="h-full" bind:this={contentEl}>
{#if currentConfigStep}
{@render configStepSection(currentConfigStep, close)}
{:else}
<div class="flex flex-col h-full">
<div class={'flex flex-row transition-height divide-x overflow-y-scroll'}>
{@render topSection()}
</div>
<div
class={twMerge(
'flex flex-col gap-5 grow transition-height px-4 border-t',
showBottomPanel ? 'h-[14rem] py-4' : 'h-0'
)}
>
{@render bottomSection(close)}
</div>
</div>
{/if}
</div>
{/snippet}
</Popover>
@@ -170,7 +386,10 @@
<Button
variant="subtle"
btnClasses={'text-left'}
onClick={() => (selected.triggerId = k.id)}
onClick={() => {
selected.triggerId = k.id
if (k.id === 'schedule') loadAdvancedDefaults()
}}
selected={isSelected}
>
{#if k.icon}
@@ -264,14 +483,30 @@
{#snippet bottomSection(close: () => void)}
<Label label="Path">
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div bind:this={pathEl} class="flex" onkeydown={selectAndAdvanceTo(() => aiPromptEl ?? saveEl)}>
<div
bind:this={pathEl}
class="flex"
onkeydown={(e) =>
(e.metaKey || e.ctrlKey) && e.key === 'Enter'
? (e.preventDefault(), e.stopPropagation(), confirm(close))
: selectAndAdvanceTo(() => aiPromptEl ?? saveEl)(e)}
>
<div
class="border rounded-md rounded-r-none border-r-0 text-xs w-fit shrink-0 whitespace-nowrap flex items-center px-2 text-secondary bg-surface-input"
>
{pathPrefix}
</div>
<TextInput bind:value={selected.scriptPath} class="rounded-l-none" />
<TextInput
bind:value={selected.scriptPath}
class="rounded-l-none"
error={!scriptNameValid && !!selected.scriptPath.trim()}
/>
</div>
{#if !scriptNameValid && selected.scriptPath.trim()}
<span class="text-2xs text-red-600 dark:text-red-400">
Letters, digits, <code>_</code> and <code>-</code>, in <code>/</code>-separated segments
</span>
{/if}
</Label>
{#if !$copilotInfo.workspaceDisabled}
<Label label="AI Prompt (optional)">
@@ -292,15 +527,217 @@
</div>
</Label>
{/if}
{@const hasAiPrompt = !!selected.aiPrompt?.trim()}
<div class="ml-auto" bind:this={saveEl}>
<Button
variant="accent"
btnClasses="w-fit"
disabled={!selected.scriptPath.trim()}
onClick={() => confirm(close)}
startIcon={hasAiPrompt ? { icon: Sparkles } : undefined}
shortCut={{ Icon: CornerDownLeft }}>{hasAiPrompt ? 'Generate' : 'Create'}</Button
>
{@render confirmButton(close)}
</div>
{/snippet}
{#snippet confirmButton(close: () => void)}
{@const hasAiPrompt = !!selected.aiPrompt?.trim()}
<Button
variant="accent"
btnClasses="w-fit"
disabled={!stepValid}
onClick={() => confirm(close)}
startIcon={isLastStep && hasAiPrompt ? { icon: Sparkles } : undefined}
shortCut={{ Icon: CornerDownLeft }}
>{scheduleFor
? 'Create schedule'
: !isLastStep
? step === 0
? 'Configure'
: 'Next'
: hasAiPrompt
? 'Generate'
: selectedKind?.configuredAfterCreate
? 'Configure'
: 'Create'}</Button
>
{/snippet}
{#snippet configStepSection(configStep: ConfigStep, close: () => void)}
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div
class="flex flex-col h-full"
onkeydown={(e) => {
if (e.key !== 'Enter') return
const target = e.target as HTMLElement | null
// Cmd/Ctrl+Enter confirms from anywhere; bare Enter only from the step's own
// inputs, not from the Advanced fields (a textarea, pickers with their own Enter).
const bareEnterSubmits = target?.tagName === 'INPUT' && !target.closest('[data-advanced]')
if (e.metaKey || e.ctrlKey || bareEnterSubmits) {
e.preventDefault()
e.stopPropagation()
confirm(close)
}
}}
>
<div class="flex items-center gap-2 px-2 py-2 border-b">
{#if !scheduleFor}
<Button
variant="subtle"
unifiedSize="sm"
iconOnly
startIcon={{ icon: ArrowLeft }}
onClick={goBack}
title="Back"
/>
{/if}
<span class="text-xs font-semibold text-emphasis {scheduleFor ? 'pl-2' : ''}">
{scheduleFor
? `Schedule for ${scheduleFor}`
: configStep === 'schedule'
? 'Schedule'
: 'Output asset'}
</span>
{#if !scheduleFor}
<span class="text-2xs text-hint ml-auto pr-2"
>Step {step + 1} of {configSteps.length + 1}</span
>
{/if}
</div>
<div class="flex flex-col gap-2 grow overflow-auto p-4">
{#if configStep === 'schedule'}
<span class="text-2xs text-secondary">
Saved as a draft schedule and deployed along with the pipeline.
</span>
<div data-autofocus class="contents">
<CronInput
bind:schedule={config.schedule}
bind:timezone={config.timezone}
bind:validCRON={config.validCron}
bind:cronVersion={config.cronVersion}
/>
</div>
<div data-advanced class="mt-4">
<Section label="Advanced" collapsable initiallyCollapsed>
<div class="flex flex-col gap-6">
<Label label="Summary">
<TextInput
bind:value={config.summary}
inputProps={{ placeholder: 'Short summary to be displayed when listed' }}
/>
</Label>
<Label label="Path">
<Path
bind:path={config.schedulePath}
bind:dirty={config.schedulePathDirty}
bind:error={config.schedulePathError}
initialPath=""
checkInitialPathExistence
namePlaceholder="schedule"
kind="schedule"
/>
</Label>
<Label label="Description">
<TextInput
underlyingInputEl="textarea"
bind:value={config.description}
inputProps={{ placeholder: 'What this schedule does and how to use it' }}
/>
</Label>
<div class="flex flex-col gap-1">
<Toggle
options={{
right: 'Pause schedule until...',
rightTooltip:
'Pausing the schedule will program the next job to run as if the schedule starts at the time the pause is lifted, instead of now.'
}}
bind:checked={config.pauseUntil}
/>
{#if config.pauseUntil}
<DateTimeInput bind:value={config.pausedUntil} />
{/if}
</div>
<!-- Rendered once the workspace defaults are in, so they never
overwrite a handler picked while they load. -->
{#if config.advancedDefaults === 'loaded'}
<ScheduleAdvancedOptions
wsId={$workspaceStore}
itemKind="script"
canWrite
bind:errorHandlerSelected={config.advanced.errorHandlerSelected}
bind:errorHandlerPath={config.advanced.errorHandlerPath}
bind:errorHandleritemKind={config.advanced.errorHandleritemKind}
bind:errorHandlerExtraArgs={config.advanced.errorHandlerExtraArgs}
bind:wsErrorHandlerMuted={config.advanced.wsErrorHandlerMuted}
bind:failedTimes={config.advanced.failedTimes}
bind:failedExact={config.advanced.failedExact}
bind:recoveryHandlerSelected={config.advanced.recoveryHandlerSelected}
bind:recoveryHandlerPath={config.advanced.recoveryHandlerPath}
bind:recoveryHandlerItemKind={config.advanced.recoveryHandlerItemKind}
bind:recoveryHandlerExtraArgs={config.advanced.recoveryHandlerExtraArgs}
bind:recoveredTimes={config.advanced.recoveredTimes}
bind:successHandlerSelected={config.advanced.successHandlerSelected}
bind:successHandlerPath={config.advanced.successHandlerPath}
bind:successHandlerItemKind={config.advanced.successHandlerItemKind}
bind:successHandlerExtraArgs={config.advanced.successHandlerExtraArgs}
bind:retry={config.advanced.retry}
bind:dynamicSkipPath={config.advanced.dynamicSkipPath}
bind:tag={config.advanced.tag}
/>
{:else}
<div class="flex items-center gap-1.5 text-xs text-secondary">
<Loader2 size={12} class="animate-spin" />
Loading the workspace's default handlers…
</div>
{/if}
</div>
</Section>
</div>
{:else if configStep === 'asset' && config.asset}
<Label label="Name">
<div
class={twMerge(
inputBaseClass,
inputBorderClass({ error: !assetValid }),
inputSizeClasses.md,
'flex flex-row items-center gap-0 py-0 pl-2 pr-0'
)}
>
<span class="text-xs text-secondary shrink-0">{config.asset.kind}://</span>
{#if config.asset.kind === 'datatable'}
<DatatablePicker
bind:value={config.asset.store}
class="shrink-0"
selectInputClass="!border-none"
placeholder="data table"
useContentEditable
RightIcon={ChevronDown}
/>
{:else}
<DucklakePicker
bind:value={config.asset.store}
class="shrink-0"
selectInputClass="!border-none"
placeholder="catalog"
useContentEditable
RightIcon={ChevronDown}
/>
{/if}
<span class="text-sm text-secondary">/</span>
<div data-autofocus class="grow flex">
<TextInput
bind:value={config.asset.table}
class="!border-none grow"
inputProps={{
placeholder: 'table',
oninput: () => (config.tableEdited = true)
}}
/>
</div>
</div>
</Label>
<span class="text-2xs text-hint">
{config.asset.kind === 'datatable' ? 'Data table' : 'DuckLake catalog'}, then the table
name
</span>
{/if}
</div>
<div class="flex px-4 py-3 border-t">
<div class="ml-auto">
{@render confirmButton(close)}
</div>
</div>
</div>
{/snippet}
@@ -0,0 +1,174 @@
<!--
@component
The body every pipeline graph node shares: icon, a kind line ("Script · draft", "Data table"),
a title, and trailing chips. Node components add only what is theirs — the icon, the chips,
and the controls hung around the card.
-->
<script lang="ts">
import { NODE } from '$lib/components/graph/util'
import {
PIPELINE_NODE_EXTRA_ROW,
PIPELINE_NODE_HEADER,
PIPELINE_NODE_HEADER_CARD_PAD,
PIPELINE_NODE_HEIGHT
} from './assetGraphLayout'
import { twMerge } from 'tailwind-merge'
import type { Snippet } from 'svelte'
interface Props {
icon: Snippet
kindLabel: string
title: string
/** Hover text; the path, when the title is a summary. */
tooltip?: string
/** Scripts sit on the main surface, assets and triggers a step down; in the
* assets-only view, which has no scripts, assets take the main surface. */
surface?: 'primary' | 'secondary'
selected?: boolean
/** Not deployed yet: dashed edge. */
draft?: boolean
/** Feedback states, which recolor the whole card. `error` is a misconfigured
* node, styled like a failed flow step. */
tone?: 'danger' | 'error' | 'success' | 'running'
/** Makes the whole card a button. */
onclick?: (e: MouseEvent) => void
/** Between the icon and the text: a live status the eye should find first. */
leading?: Snippet
trailing?: Snippet
/** Defaults to `NODE.width`. */
width?: number
/** A line under the title, in the text column. */
subtitle?: Snippet
/** A tab on the card's top edge. Only on a card without `onclick`, since it
* may hold buttons of its own. */
header?: Snippet
/** The header's script is selected: its edge takes the selected colour. */
headerSelected?: boolean
/** The header's script has an error: its edge goes red. */
headerTone?: 'error'
}
let {
icon,
kindLabel,
title,
tooltip,
surface = 'secondary',
selected = false,
draft = false,
tone,
onclick,
leading,
trailing,
subtitle,
header,
headerSelected = false,
headerTone,
width = NODE.width
}: Props = $props()
const cardClass = $derived(
twMerge(
'flex items-center w-full text-left font-normal rounded-md drop-shadow-sm overflow-hidden border transition-colors',
surface === 'primary' ? 'bg-surface' : 'bg-surface-secondary',
'border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500',
// Like a flow step: a card with a background needs no edge, only its shadow,
// until one of the states below draws one.
surface === 'primary' &&
!draft &&
!selected &&
!tone &&
'border-transparent dark:border-transparent hover:border-transparent dark:hover:border-transparent',
draft && 'border-dashed border-gray-400 dark:border-gray-500',
// Like a selected flow step: the fill and the edge colour change, the edge's
// width and dashes stay.
// Each dark: variant too: the base edge sets its own dark colour, which a plain
// border class does not override.
selected &&
'bg-surface-accent-selected border-border-selected dark:border-border-selected hover:border-border-selected dark:hover:border-border-selected',
tone === 'danger' &&
'bg-red-50 dark:bg-red-900/30 border-dashed border-red-400 dark:border-red-500 hover:bg-red-100 dark:hover:bg-red-900/40',
tone === 'error' &&
(selected
? 'bg-red-200 dark:bg-red-600 border-red-500 dark:border-red-500 hover:border-red-500 dark:hover:border-red-500'
: 'bg-red-100 dark:bg-red-700 border-transparent dark:border-transparent hover:border-transparent dark:hover:border-transparent'),
tone === 'success' &&
'bg-green-50 dark:bg-green-900/30 border-green-500 dark:border-green-600',
tone === 'running' &&
'bg-amber-50 dark:bg-amber-900/30 border-amber-400 dark:border-amber-600 animate-pulse'
)
)
const minHeight = $derived(PIPELINE_NODE_HEIGHT + (subtitle ? PIPELINE_NODE_EXTRA_ROW : 0))
const toneText = $derived(
tone === 'danger'
? 'text-red-700 dark:text-red-400'
: tone === 'error'
? 'text-red-700 dark:text-red-200'
: tone === 'success'
? 'text-green-700 dark:text-green-400'
: undefined
)
</script>
{#snippet body()}
<span class="shrink-0 ml-3 mr-2.5 flex items-center">{@render icon()}</span>
{@render leading?.()}
<span class="flex flex-col min-w-0 flex-1 pr-1 py-1 leading-tight">
<span class={twMerge('text-2xs truncate text-secondary', toneText)}>{kindLabel}</span>
<span class={twMerge('text-xs truncate text-emphasis', selected && 'text-accent', toneText)}
>{title}</span
>
{#if subtitle}
<span class="flex items-center min-w-0 mt-1">{@render subtitle()}</span>
{/if}
</span>
{#if trailing}
<span class="flex items-center shrink-0 pr-2">{@render trailing()}</span>
{/if}
{/snippet}
{#if onclick}
<button
type="button"
{onclick}
class={cardClass}
style="width: {width}px; min-height: {minHeight}px;"
title={tooltip}
>
{@render body()}
</button>
{:else if header}
<!-- The header is a tab on the card's top edge, inset from its sides. -->
<div class="flex flex-col" style="width: {width}px;">
<div
class={twMerge(
'mx-2 flex items-stretch min-w-0 rounded-t-md overflow-hidden bg-surface border border-b-0 border-gray-200 dark:border-gray-700',
// Like an errored card: red, and a stronger red when selected.
headerTone === 'error'
? headerSelected
? 'border-red-500 dark:border-red-500'
: 'border-red-300 dark:border-red-600'
: headerSelected && 'border-border-selected dark:border-border-selected'
)}
style="height: {PIPELINE_NODE_HEADER}px;"
>
{@render header()}
</div>
<div
class={twMerge(cardClass, !selected && !tone && 'bg-surface-tertiary')}
style="min-height: {minHeight + PIPELINE_NODE_HEADER_CARD_PAD}px; padding-block: {PIPELINE_NODE_HEADER_CARD_PAD /
2}px;"
title={tooltip}
>
{@render body()}
</div>
</div>
{:else}
<div
class={cardClass}
style="width: {width}px; min-height: {minHeight}px;"
title={tooltip}
>
{@render body()}
</div>
{/if}
@@ -1,6 +1,6 @@
<script lang="ts">
import type { AssetKind, Script } from '$lib/gen'
import { CalendarClock, Loader2, Play, Upload, Zap } from 'lucide-svelte'
import { CalendarClock, Code2, Loader2, Play, Upload, Zap } from 'lucide-svelte'
import Button from '$lib/components/common/button/Button.svelte'
import HighlightCode from '$lib/components/HighlightCode.svelte'
import PipelineRunForm from './PipelineRunForm.svelte'
@@ -44,6 +44,12 @@
// "green / ready" state and seeds the whole-pipeline run. `isValid` is the
// full-schema validity (all required fields, not just the S3 file).
onArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
// Just the run form and the runs, without the source: what a data-upload
// click asks for. `onShowScript` switches to the script itself.
inputsOnly?: boolean
onShowScript?: () => void
// `onRun` previews a draft's own content, so a draft can run too.
runsDrafts?: boolean
}
let {
@@ -57,8 +63,12 @@
runsPendingJobId,
onRunCompleted,
initialArgs,
onArgsChange
onArgsChange,
inputsOnly = false,
onShowScript,
runsDrafts = false
}: Props = $props()
let runBlockedByDraft = $derived(isDraft && !runsDrafts)
// Seed once from the persisted args (see initialArgs). Cloned so the run
// form mutates its own copy, not the page's stored snapshot.
@@ -139,7 +149,7 @@
</script>
<div class="flex flex-col h-full">
{#if isDraft}
{#if runBlockedByDraft}
<div
class="shrink-0 flex items-center gap-2 px-3 py-1.5 text-2xs bg-amber-50 dark:bg-amber-900/30 border-b border-amber-200 dark:border-amber-900/60 text-amber-700 dark:text-amber-400"
>
@@ -168,13 +178,24 @@
{/if}
</span>
<div class="flex items-center gap-1.5">
{#if inputsOnly && onShowScript}
<Button
variant="subtle"
unifiedSize="sm"
startIcon={{ icon: Code2 }}
onclick={onShowScript}
title="Show the script instead of its inputs"
>
Open script
</Button>
{/if}
{#if hasCascade}
<Button
variant="accent-secondary"
unifiedSize="sm"
startIcon={{ icon: running ? Loader2 : Zap }}
onclick={() => run(true)}
disabled={isDraft || running || !runValid}
disabled={runBlockedByDraft || running || !runValid}
title={`Run this script with these inputs, then run its ${downstreamCount} downstream pipeline script${downstreamCount === 1 ? '' : 's'} in order`}
>
Run + downstream
@@ -185,8 +206,8 @@
unifiedSize="sm"
startIcon={{ icon: running ? Loader2 : Play }}
onclick={() => run(false)}
disabled={isDraft || running || !runValid || !onRun}
title={isDraft
disabled={runBlockedByDraft || running || !runValid || !onRun}
title={runBlockedByDraft
? 'Deploy this draft to run it for real'
: hasCascade
? 'Run just this step — downstream scripts are not triggered'
@@ -210,9 +231,11 @@
{/key}
{/if}
</div>
<div class="flex-1 min-h-0 overflow-auto text-xs p-3">
<HighlightCode code={script.content ?? ''} language={script.language} />
</div>
{#if !inputsOnly}
<div class="flex-1 min-h-0 overflow-auto text-xs p-3">
<HighlightCode code={script.content ?? ''} language={script.language} />
</div>
{/if}
</div>
</Pane>
<Pane size={45} minSize={20}>
@@ -3,6 +3,8 @@
import { WorkspaceService } from '$lib/gen'
import { base } from '$lib/base'
import { CircleCheck, CircleAlert, Database, HardDrive, ArrowRight } from 'lucide-svelte'
import Alert from '$lib/components/common/alert/Alert.svelte'
import { Button } from '$lib/components/common'
interface Props {
workspace: string
@@ -73,48 +75,43 @@
</script>
{#if show}
<div
class="flex flex-col gap-3 rounded-lg border border-amber-300 dark:border-amber-900/60 bg-amber-50 dark:bg-amber-950/30 p-4"
>
<div class="flex items-start gap-2">
<CircleAlert size={18} class="text-amber-600 dark:text-amber-400 mt-0.5 shrink-0" />
<div class="flex flex-col gap-0.5">
<h3 class="text-sm font-semibold text-emphasis">Finish setting up pipelines</h3>
<p class="text-xs text-tertiary">
Pipelines materialize data into DuckLake tables backed by object storage. Configure the
following before your first pipeline can run.
</p>
</div>
<Alert type="warning" title="Finish setting up pipelines" hideIcon>
<div class="flex flex-col gap-3">
<p>
Pipelines materialize data into DuckLake tables backed by object storage. Configure the
following before your first pipeline can run.
</p>
<ul class="flex flex-col gap-2">
{#each steps as step (step.title)}
{@const Icon = step.icon}
<li class="flex items-center gap-3 rounded-md border bg-surface-tertiary px-3 py-2">
{#if step.done === true}
<CircleCheck size={16} class="text-green-600 dark:text-green-400 shrink-0" />
{:else if step.done === false}
<CircleAlert size={16} class="text-yellow-600 dark:text-yellow-400 shrink-0" />
{:else}
<Icon size={16} class="text-secondary shrink-0" />
{/if}
<div class="flex flex-col min-w-0 flex-1">
<span class="text-xs font-semibold text-emphasis">{step.title}</span>
<span class="text-2xs text-secondary">{step.description}</span>
</div>
{#if step.done !== true}
<Button
variant="default"
unifiedSize="sm"
href={step.href}
endIcon={{ icon: ArrowRight }}
wrapperClasses="shrink-0"
>
{step.cta}
</Button>
{:else}
<span class="shrink-0 text-2xs text-green-700 dark:text-green-400">Configured</span>
{/if}
</li>
{/each}
</ul>
</div>
<ul class="flex flex-col gap-2">
{#each steps as step (step.title)}
{@const Icon = step.icon}
<li class="flex items-center gap-3 rounded-md border bg-surface px-3 py-2">
{#if step.done === true}
<CircleCheck size={16} class="text-green-600 dark:text-green-400 shrink-0" />
{:else if step.done === false}
<CircleAlert size={16} class="text-amber-600 dark:text-amber-400 shrink-0" />
{:else}
<Icon size={16} class="text-tertiary shrink-0" />
{/if}
<div class="flex flex-col min-w-0 flex-1">
<span class="text-xs font-semibold text-primary">{step.title}</span>
<span class="text-2xs text-tertiary">{step.description}</span>
</div>
{#if step.done !== true}
<a
href={step.href}
class="shrink-0 inline-flex items-center gap-1 text-xs text-blue-500 hover:underline whitespace-nowrap"
>
{step.cta}
<ArrowRight size={12} />
</a>
{:else}
<span class="shrink-0 text-2xs text-green-700 dark:text-green-400">Configured</span>
{/if}
</li>
{/each}
</ul>
</div>
</Alert>
{/if}
@@ -1,18 +1,10 @@
<script lang="ts">
import { sendUserToast } from '$lib/utils'
import { workspaceStore } from '$lib/stores'
import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte'
import type { NativeTriggerKind } from './types'
import {
EmailTriggerService,
GcpTriggerService,
KafkaTriggerService,
MqttTriggerService,
AmqpTriggerService,
NatsTriggerService,
PostgresTriggerService,
ScheduleService,
SqsTriggerService
} from '$lib/gen'
import type { NativeTriggerKind, PipelineTriggerDraftKind } from './types'
import type { Schedule } from '$lib/gen'
import { deleteTriggerRow } from './pipelineTriggerDraftDeploy'
import KafkaTriggerEditor from '$lib/components/triggers/kafka/KafkaTriggerEditor.svelte'
import MqttTriggerEditor from '$lib/components/triggers/mqtt/MqttTriggerEditor.svelte'
import AmqpTriggerEditor from '$lib/components/triggers/amqp/AmqpTriggerEditor.svelte'
@@ -22,6 +14,10 @@
import GcpTriggerEditor from '$lib/components/triggers/gcp/GcpTriggerEditor.svelte'
import EmailTriggerEditor from '$lib/components/triggers/email/EmailTriggerEditor.svelte'
import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte'
import {
loadDefaultScheduleAdvanced,
scheduleAdvancedCfg
} from '$lib/components/triggers/schedules/ScheduleAdvancedOptions.svelte'
import WebhookEditor from '$lib/components/triggers/webhook/WebhookEditor.svelte'
import { setOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
@@ -110,6 +106,56 @@
}
}
/** The kind's full editor over a pipeline-local draft; Save hands the config back,
* and the drawer stays open when `onSave` refuses it.
* `saved` is the draft being edited (absent for a new one), applied over `defaults`. */
export function openTriggerDraft(
kind: PipelineTriggerDraftKind,
scriptPath: string,
defaults: Record<string, any>,
saved: Record<string, any> | undefined,
onSave: (cfg: Record<string, any>) => boolean
) {
switch (kind) {
case 'schedule': {
// A new draft starts from the workspace's default handlers, like a new
// schedule does; the drawer only applies them when it opens without one.
const open = (advanced: Record<string, any>) =>
scheduleEditor?.openDraft(
{
is_flow: false,
args: {},
...advanced,
...defaults,
...saved,
script_path: scriptPath
} as Schedule,
onSave
)
if (saved || !$workspaceStore) return open({})
return loadDefaultScheduleAdvanced($workspaceStore)
.then((a) => open(scheduleAdvancedCfg(a)))
.catch(() => open({}))
}
case 'kafka':
return kafkaEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'mqtt':
return mqttEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'amqp':
return amqpEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'nats':
return natsEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'postgres':
return postgresEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'sqs':
return sqsEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'gcp':
return gcpEditor?.openDraft(scriptPath, defaults, saved, onSave)
case 'email':
return emailEditor?.openDraft(scriptPath, defaults, saved, onSave)
}
}
// Open the webhook drawer (endpoint URLs + token flow). Caller owns the
// draft guard.
export function openWebhook(scriptPath: string) {
@@ -133,37 +179,7 @@
const { kind, path: triggerPath } = triggerDeleteTarget
triggerDeleteLoading = true
try {
switch (kind) {
case 'schedule':
await ScheduleService.deleteSchedule({ workspace, path: triggerPath })
break
case 'kafka':
await KafkaTriggerService.deleteKafkaTrigger({ workspace, path: triggerPath })
break
case 'mqtt':
await MqttTriggerService.deleteMqttTrigger({ workspace, path: triggerPath })
break
case 'amqp':
await AmqpTriggerService.deleteAmqpTrigger({ workspace, path: triggerPath })
break
case 'nats':
await NatsTriggerService.deleteNatsTrigger({ workspace, path: triggerPath })
break
case 'postgres':
await PostgresTriggerService.deletePostgresTrigger({ workspace, path: triggerPath })
break
case 'sqs':
await SqsTriggerService.deleteSqsTrigger({ workspace, path: triggerPath })
break
case 'gcp':
await GcpTriggerService.deleteGcpTrigger({ workspace, path: triggerPath })
break
case 'email':
await EmailTriggerService.deleteEmailTrigger({ workspace, path: triggerPath })
break
default:
return
}
if (!(await deleteTriggerRow(kind, triggerPath, workspace))) return
sendUserToast(`Deleted ${kind} trigger "${triggerPath}"`)
triggerDeleteTarget = undefined
onUpdate()
@@ -0,0 +1,37 @@
<script lang="ts">
import { twMerge } from 'tailwind-merge'
import { CheckCircle2, Loader2, XCircle } from 'lucide-svelte'
import type { RunnableRunState } from './activeRunnables.svelte'
let { runState: rs, class: className }: { runState: RunnableRunState; class?: string } = $props()
</script>
<div
class={twMerge(
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 rounded-sm border',
rs.status === 'running'
? 'bg-blue-100 dark:bg-blue-900/40 text-blue-700 dark:text-blue-300 border-blue-300 dark:border-blue-700'
: rs.status === 'success'
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border-emerald-300 dark:border-emerald-700'
: 'bg-red-100 dark:bg-red-900/40 text-red-700 dark:text-red-300 border-red-300 dark:border-red-700',
className
)}
title={`${
rs.status === 'running'
? 'Running now'
: rs.status === 'success'
? 'Last run succeeded'
: 'Last run failed'
}${rs.runs > 0 ? ` — ran ${rs.runs}× this session` : ''}`}
>
{#if rs.status === 'running'}
<Loader2 size={10} class="animate-spin" />
{:else if rs.status === 'success'}
<CheckCircle2 size={10} />
{:else}
<XCircle size={10} />
{/if}
{#if rs.runs > 0}
<span class="text-3xs leading-none tabular-nums">×{rs.runs}</span>
{/if}
</div>
@@ -1,10 +1,8 @@
<script lang="ts">
import { Handle, Position } from '@xyflow/svelte'
import {
CheckCircle2,
ChevronDown,
Code2,
EllipsisVertical,
GitBranch,
Layers,
Loader2,
@@ -15,17 +13,20 @@
Target,
Timer,
Trash2,
XCircle,
Zap
} from 'lucide-svelte'
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
import { twMerge } from 'tailwind-merge'
import { preventDefault, stopPropagation } from 'svelte/legacy'
import type { GraphUsageKind } from './types'
import type { RunnableRunState } from './activeRunnables.svelte'
import { parseDurationSecs } from './parsePipelineAnnotations'
import { NODE } from '$lib/components/graph/util'
import DropdownV2 from '$lib/components/DropdownV2.svelte'
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
import PipelineNodeCard from './PipelineNodeCard.svelte'
import NodeActionsMenu from './NodeActionsMenu.svelte'
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
import NodeOnDot from './NodeOnDot.svelte'
import RunStateChip from './RunStateChip.svelte'
import type { ScriptLang } from '$lib/gen'
import Popover from '$lib/components/meltComponents/Popover.svelte'
import type { Item } from '$lib/utils'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
@@ -35,8 +36,16 @@
interface Props {
data: {
/** What is wrong with the node, and how to fix it: shows the Fix pill. */
fix?: NodeFix
/** Starts dragging the top dot onto an asset to subscribe the script to it. */
onStartOnDrag?: (e: PointerEvent) => void
runnable_kind: GraphUsageKind
path: string
summary?: string
language?: ScriptLang
// Why the script is misconfigured, when it is: renders it red.
error?: string
in_pipeline?: boolean
partition_kind?: 'daily' | 'hourly' | 'weekly' | 'monthly' | 'dynamic'
freshness?: string
@@ -101,10 +110,11 @@
}
let { data, selected = false }: Props = $props()
// The icon alone conveys "script" vs "flow"; the uppercase kind label was
// visually noisy and redundant. Tooltip on hover surfaces the path in
// full when truncated.
let Icon = $derived(data.runnable_kind === 'flow' ? GitBranch : Code2)
let kindLabel = $derived(
data.runnable_kind === 'flow' ? 'Flow' : data.dbt ? 'dbt project' : 'Script'
)
// The title shows the summary when there is one, so the tooltip keeps the path.
let nodeTooltip = $derived(
data.unsaved
? `${data.path} (unsaved draft)`
@@ -114,7 +124,6 @@
)
let hover = $state(false)
let menuOpen = $state(false)
let running = $state(false)
// Amber "computing now" surface, gated so only the replay player lights it up.
@@ -214,131 +223,106 @@
<!--
Mirrors the flow editor's step styling (getNodeColorClasses): muted
surface-tertiary fill with a quiet gray border, accent only for the
selected state, dashes for unsaved drafts. The 2px dashed stroke is
necessary for the dash gaps to be readable; 1px disappears into the
fill.
selected state, dashes for unsaved drafts — 1px, like the trigger
nodes' dashed outline, so every draft node reads the same.
-->
<div
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden border transition-colors',
'bg-surface border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500',
selected && 'bg-surface-accent-selected border-border-selected',
data.unsaved && 'border-2 border-dashed border-gray-400 dark:border-gray-500',
computingNow &&
'bg-amber-50 dark:bg-amber-900/30 border-amber-400 dark:border-amber-600 animate-pulse'
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={nodeTooltip}
>
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${selected ? 'text-accent' : 'text-secondary'}`} />
<span class="flex-1 min-w-0 pr-1 py-0.5 text-2xs font-mono text-emphasis truncate">
{data.path}
</span>
<!-- Annotation chips share one neutral treatment — the icon carries
the meaning (colors are reserved for feedback, per the brand
guidelines). Only the freshness chip (when it has a verdict)
and the run-state chip below use semantic colors. -->
{#if data.partition_kind}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// partitioned ${data.partition_kind}`}
>
<Layers size={10} />
<span class="text-3xs leading-none">{data.partition_kind}</span>
</div>
{/if}
<!-- Freshness is the one annotation chip that carries feedback (a
fresh/stale verdict against real run history), so like the
run-state chip it uses semantic colors: emerald = within window,
amber = stale. Neutral when there's no verdict (drafts, bad
window value). -->
{#if data.freshness}
<div
class={twMerge(
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
freshnessState === 'fresh'
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
: freshnessState === 'stale'
? 'bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300'
: 'bg-surface-secondary text-secondary'
)}
title={freshnessTooltip}
>
<Timer size={10} />
<span class="text-3xs leading-none">{data.freshness}</span>
</div>
{/if}
{#if data.tag}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// tag ${data.tag}`}
>
<Tag size={10} />
<span class="text-3xs leading-none">{data.tag}</span>
</div>
{/if}
{#if data.retry}
{@const r = data.retry}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// retry ${r.count}${r.delay ? ` ${r.delay}` : ''}`}
>
<RotateCw size={10} />
<span class="text-3xs leading-none">×{r.count}</span>
</div>
{/if}
{#if data.macros && data.macros.length > 0}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// macros — defines ${data.macros.length} macro${data.macros.length > 1 ? 's' : ''}:\n${data.macros
.map((m) => `• ${m.name}(${m.params})${m.is_table ? ' → table' : ''}`)
.join('\n')}`}
>
<SquareFunction size={10} />
<span class="text-3xs leading-none">×{data.macros.length}</span>
</div>
{/if}
{#if data.dbt}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300"
title={`dbt project — materializes ${data.dbt.model_count} model${data.dbt.model_count === 1 ? '' : 's'}`}
>
<DbtIcon width={10} height={10} />
<span class="text-3xs leading-none">×{data.dbt.model_count}</span>
</div>
{/if}
{#if data.runState}
{@const rs = data.runState}
<div
class={twMerge(
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
rs.status === 'running'
? 'bg-blue-100 dark:bg-blue-900/40 text-blue-700 dark:text-blue-300'
: rs.status === 'success'
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
: 'bg-red-100 dark:bg-red-900/40 text-red-700 dark:text-red-300'
)}
title={`${
rs.status === 'running'
? 'Running now'
: rs.status === 'success'
? 'Last run succeeded'
: 'Last run failed'
}${rs.runs > 0 ? ` — ran ${rs.runs}× this session` : ''}`}
>
{#if rs.status === 'running'}
<Loader2 size={10} class="animate-spin" />
{:else if rs.status === 'success'}
<CheckCircle2 size={10} />
<NodeActionsMenu items={menuItems} {hover}>
<PipelineNodeCard
surface="primary"
kindLabel="{kindLabel}{data.unsaved ? ' · draft' : ''}"
title={data.summary || data.path}
tooltip={data.error ? `${data.path}: ${data.error}` : nodeTooltip}
{selected}
draft={data.unsaved}
tone={computingNow ? 'running' : data.error ? 'error' : undefined}
>
{#snippet icon()}
{#if data.language && data.runnable_kind === 'script'}
<LanguageIcon lang={data.language} width={14} height={14} />
{:else}
<XCircle size={10} />
<Icon size={14} class={selected ? 'text-accent' : 'text-secondary'} />
{/if}
{#if rs.runs > 0}
<span class="text-3xs leading-none tabular-nums">×{rs.runs}</span>
{/snippet}
{#snippet trailing()}
<!-- Annotation chips share one neutral treatment — the icon carries
the meaning (colors are reserved for feedback, per the brand
guidelines). Only the freshness chip (when it has a verdict)
and the run-state chip below use semantic colors. -->
{#if data.partition_kind}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// partitioned ${data.partition_kind}`}
>
<Layers size={10} />
<span class="text-3xs leading-none">{data.partition_kind}</span>
</div>
{/if}
</div>
{/if}
</div>
<!-- Freshness is the one annotation chip that carries feedback (a
fresh/stale verdict against real run history), so like the
run-state chip it uses semantic colors: emerald = within window,
amber = stale. Neutral when there's no verdict (drafts, bad
window value). -->
{#if data.freshness}
<div
class={twMerge(
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
freshnessState === 'fresh'
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
: freshnessState === 'stale'
? 'bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300'
: 'bg-surface-secondary text-secondary'
)}
title={freshnessTooltip}
>
<Timer size={10} />
<span class="text-3xs leading-none">{data.freshness}</span>
</div>
{/if}
{#if data.tag}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// tag ${data.tag}`}
>
<Tag size={10} />
<span class="text-3xs leading-none">{data.tag}</span>
</div>
{/if}
{#if data.retry}
{@const r = data.retry}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// retry ${r.count}${r.delay ? ` ${r.delay}` : ''}`}
>
<RotateCw size={10} />
<span class="text-3xs leading-none">×{r.count}</span>
</div>
{/if}
{#if data.macros && data.macros.length > 0}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
title={`// macros — defines ${data.macros.length} macro${data.macros.length > 1 ? 's' : ''}:\n${data.macros
.map((m) => `• ${m.name}(${m.params})${m.is_table ? ' → table' : ''}`)
.join('\n')}`}
>
<SquareFunction size={10} />
<span class="text-3xs leading-none">×{data.macros.length}</span>
</div>
{/if}
{#if data.dbt}
<div
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300"
title={`dbt project — materializes ${data.dbt.model_count} model${data.dbt.model_count === 1 ? '' : 's'}`}
>
<DbtIcon width={10} height={10} />
<span class="text-3xs leading-none">×{data.dbt.model_count}</span>
</div>
{/if}
{#if data.runState}
<RunStateChip runState={data.runState} class="mr-1" />
{/if}
{/snippet}
</PipelineNodeCard>
</NodeActionsMenu>
{#if showRun}
<!-- Run button revealed on hover/select. Matches the placement, size,
and behaviour of AssetNode's run button so both nodes feel
@@ -463,38 +447,12 @@
{/if}
</div>
{/if}
{#if menuItems.length > 0}
<!--
Hover-revealed action menu, top-right of the node. Mirrors the
FlowModuleSchemaItem pattern: position the trigger button just
outside the node frame, only render it on hover (or while the
menu is open) so the canvas isn't visually cluttered when idle.
The pointerdown stop+preventDefault keeps xyflow from selecting
the node when the user is reaching for the menu.
-->
<div class="absolute -top-2 -right-2 h-7 p-1 min-w-7" style="will-change: transform;">
<DropdownV2
items={menuItems}
placement="bottom-end"
bind:open={menuOpen}
fixedHeight={false}
usePointerDownOutside
>
{#snippet buttonReplacement()}
<button
class={twMerge(
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
hover || menuOpen ? 'block' : '!hidden',
'shadow-md rounded-md'
)}
onpointerdown={stopPropagation(preventDefault(() => {}))}
title="Actions"
>
<EllipsisVertical size={12} />
</button>
{/snippet}
</DropdownV2>
{#if data.onStartOnDrag}
<NodeOnDot onStart={data.onStartOnDrag} visible={hover} />
{/if}
{#if data.fix}
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
<NodeFixButton fix={data.fix} />
</div>
{/if}
</div>
@@ -24,46 +24,36 @@
type Presentation = {
icon: ComponentType
label: string
// Tailwind class fragments for bg + border + text accent.
bg: string
border: string
borderUnsaved: string
iconText: string
}
// One muted treatment for every source kind — colors are meaningful, not
// One muted icon color for every source kind — colors are meaningful, not
// decorative (brand guidelines), so the kind is carried by the icon and
// label while bg/border stay on the surface/gray scale like the flow
// editor's nodes. Red stays reserved for the missing-trigger state.
const MUTED = {
bg: 'bg-surface-secondary',
border: 'outline-gray-400 dark:outline-gray-600',
borderUnsaved: 'outline-dashed outline-gray-400 dark:outline-gray-500',
iconText: 'text-secondary'
}
// label. Red stays reserved for the missing-trigger state.
const MUTED = { iconText: 'text-secondary' }
export const TRIGGER_NODE_STYLE: Record<TriggerNodeKind, Presentation> = {
schedule: { icon: Clock, label: 'schedule', ...MUTED },
webhook: { icon: Webhook, label: 'webhook', ...MUTED },
email: { icon: Mail, label: 'email', ...MUTED },
kafka: { icon: Zap, label: 'kafka', ...MUTED },
mqtt: { icon: Radio, label: 'mqtt', ...MUTED },
amqp: { icon: Radio, label: 'amqp', ...MUTED },
nats: { icon: MessageSquare, label: 'nats', ...MUTED },
postgres: { icon: Database, label: 'postgres', ...MUTED },
sqs: { icon: Send, label: 'sqs', ...MUTED },
gcp: { icon: CloudCog, label: 'gcp', ...MUTED },
data_upload: { icon: Upload, label: 'data upload', ...MUTED }
schedule: { icon: Clock, label: 'Schedule', ...MUTED },
webhook: { icon: Webhook, label: 'Webhook', ...MUTED },
email: { icon: Mail, label: 'Email', ...MUTED },
kafka: { icon: Zap, label: 'Kafka', ...MUTED },
mqtt: { icon: Radio, label: 'MQTT', ...MUTED },
amqp: { icon: Radio, label: 'AMQP', ...MUTED },
nats: { icon: MessageSquare, label: 'NATS', ...MUTED },
postgres: { icon: Database, label: 'Postgres', ...MUTED },
sqs: { icon: Send, label: 'SQS', ...MUTED },
gcp: { icon: CloudCog, label: 'GCP Pub/Sub', ...MUTED },
data_upload: { icon: Upload, label: 'Data upload', ...MUTED }
}
</script>
<script lang="ts">
import { Handle, Position } from '@xyflow/svelte'
import { NODE } from '$lib/components/graph/util'
import { twMerge } from 'tailwind-merge'
import { AlertTriangle, CheckCircle2, EllipsisVertical, Target, Trash2 } from 'lucide-svelte'
import DropdownV2 from '$lib/components/DropdownV2.svelte'
import { stopPropagation, preventDefault } from 'svelte/legacy'
import PipelineNodeCard from './PipelineNodeCard.svelte'
import NodeActionsMenu from './NodeActionsMenu.svelte'
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
import { describeSchedule } from '$lib/utils/describeCron'
import { AlertTriangle, CheckCircle2, Target, Trash2 } from 'lucide-svelte'
import type { Item } from '$lib/utils'
interface Props {
@@ -74,16 +64,22 @@
// owning script is in `runnable_path` (used by the title and the
// "+ Create trigger" drawer hook passed by the page).
data: {
/** What is wrong with the node, and how to fix it: shows the Fix pill. */
fix?: NodeFix
kind: TriggerNodeKind
ref: string
unsaved?: boolean
missing?: boolean
// A pipeline-local schedule draft: `ref` is the path it deploys to.
// Edit/delete go through the same callbacks, which the page routes
// to the draft instead of a trigger row.
draft?: boolean
// The cron, for a schedule: the node names its cadence instead of "Schedule".
schedule?: string
timezone?: string
// Shown as the title over the trigger's path, when the trigger has one.
summary?: string
runnable_path?: string
// True iff the target script is still a draft (no DB row yet).
// Drives the "(after draft save)" hint on the missing
// placeholder, since the page-level handler refuses to open the
// create drawer until the script is deployed.
runnable_unsaved?: boolean
// Page-supplied dispatcher that opens the matching native
// trigger drawer with `script_path` pre-filled. When absent
// (e.g. schedule, or a kind without an editor) the placeholder is
@@ -128,7 +124,6 @@
let { data }: Props = $props()
let hover = $state(false)
let menuOpen = $state(false)
let style = $derived(TRIGGER_NODE_STYLE[data.kind])
// Webhooks are never genuinely "missing" — every deployed runnable has an
@@ -143,7 +138,7 @@
let Icon = $derived(displayMissing ? AlertTriangle : style.icon)
let missingTitle = $derived(
displayMissing
? `Missing ${style.label} trigger: ${data.runnable_path ?? ''} declares \`// on ${style.label}\` but no ${style.label} trigger targets it. Click to create one, or remove the annotation.`
? `Missing ${style.label} trigger: ${data.runnable_path ?? ''} declares \`// on ${data.kind}\` but no ${style.label} trigger targets it. Click to create one, or remove the annotation.`
: undefined
)
// Webhook gets a drawer (URLs + webhook-specific token creation) rather
@@ -200,7 +195,7 @@
...(canDelete
? [
{
displayName: 'Delete…',
displayName: data.draft ? 'Discard draft' : 'Delete…',
icon: Trash2,
type: 'delete' as const,
action: () => {
@@ -212,6 +207,75 @@
: [])
])
// A schedule names its cadence ("Every day at 4:00", or the cron itself
// when it has no plain reading); every other kind names its kind.
let kindName = $derived(
data.kind === 'schedule' && data.schedule
? (describeSchedule(data.schedule, data.timezone) ?? `Schedule (${data.schedule})`)
: style.label
)
// One card for every state: which label, title and click the node gets.
let card = $derived.by(
(): {
kindLabel: string
title: string
tooltip?: string
draft: boolean
tone?: 'danger' | 'success'
onclick?: () => void
} => {
const unsaved = !!data.unsaved
if (canCreate)
return {
kindLabel: `${style.label} · missing`,
title: 'Click to create',
tooltip: missingTitle,
draft: false,
tone: 'danger',
onclick: handleMissingClick
}
if (canEdit)
return {
kindLabel: `${kindName}${data.draft ? ' · draft' : unsaved ? ' · unsaved' : ''}`,
title: data.summary || data.ref,
tooltip: data.draft
? `Draft ${style.label}: ${data.ref} — created when you save the pipeline. Click to edit.`
: `Edit ${style.label} trigger: ${data.ref}`,
draft: unsaved,
onclick: handleEditClick
}
if (canOpenWebhook)
return {
kindLabel: style.label,
title: 'URLs & token',
tooltip: `Webhook endpoint for ${data.runnable_path ?? ''} — click to view URLs and create a token`,
draft: unsaved,
onclick: handleWebhookClick
}
if (canOpenDataUpload)
return {
kindLabel: `${style.label}${dataUploadReady ? ' · ready' : ''}`,
title: dataUploadReady ? 'File staged' : 'Upload & run',
tooltip: dataUploadReady
? `Data upload for ${data.runnable_path ?? ''} — a file is staged; the pipeline is ready to run. Click to change it.`
: `Data upload for ${data.runnable_path ?? ''} — click to open the run form and upload a file`,
draft: !dataUploadReady && unsaved,
tone: dataUploadReady ? 'success' : undefined,
onclick: handleDataUploadClick
}
return {
kindLabel: `${displayMissing ? style.label : kindName}${displayMissing ? ' · missing' : unsaved ? ' · unsaved' : ''}`,
title: displayMissing ? 'no trigger row' : data.summary || data.ref,
tooltip:
missingTitle ??
(unsaved ? `Unsaved ${style.label}: ${data.ref}` : `${style.label}: ${data.ref}`),
draft: unsaved && !displayMissing,
tone: displayMissing ? 'danger' : undefined
}
}
)
function handleMissingClick() {
if (!canCreate || !data.runnable_path || !data.onCreateMissingTrigger) return
data.onCreateMissingTrigger(data.kind as NativeTriggerKind, data.runnable_path)
@@ -235,195 +299,33 @@
<!-- svelte-ignore a11y_no_static_element_interactions -->
<div class="relative" onmouseenter={() => (hover = true)} onmouseleave={() => (hover = false)}>
{#if canCreate}
<!-- Same affordance as the asset's downstream + button: render the
whole node as a button so the cursor + hover + click is obvious
without depending on svelte-flow's wrapper-level handlers. -->
<button
type="button"
onclick={handleMissingClick}
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
'bg-red-50 dark:bg-red-900/30 outline-dashed outline-red-400 dark:outline-red-500',
'hover:bg-red-100 dark:hover:bg-red-900/40 transition-colors'
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={missingTitle}
<NodeActionsMenu items={menuItems} {hover}>
<PipelineNodeCard
kindLabel={card.kindLabel}
title={card.title}
tooltip={card.tooltip}
draft={card.draft}
tone={card.tone}
onclick={card.onclick}
>
<Icon size={14} class="shrink-0 ml-2 mr-2 text-red-600 dark:text-red-400" />
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
<span class="text-3xs uppercase tracking-wide truncate text-red-700 dark:text-red-400">
{style.label} · missing
</span>
<span class="text-2xs font-mono truncate text-red-700 dark:text-red-400">
{data.runnable_unsaved ? 'Click to create (after draft save)' : 'Click to create'}
</span>
</div>
</button>
{:else if canEdit}
<!-- Mirrors the missing-trigger pattern: render the whole node as a
button so clicks open the editor drawer reliably (don't rely on
svelte-flow's onnodeclick wiring). -->
<button
type="button"
onclick={handleEditClick}
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
style.bg,
data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={`Edit ${style.label} trigger: ${data.ref}`}
>
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${style.iconText}`} />
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
<span class="text-3xs uppercase tracking-wide truncate text-tertiary">
{style.label}{data.unsaved ? ' · unsaved' : ''}
</span>
<span class="text-2xs font-mono truncate text-emphasis">
{data.ref}
</span>
</div>
</button>
{:else if canOpenWebhook}
<!-- Webhook endpoint: implicit (no trigger row), so the node opens a
drawer with the URLs + webhook-specific token creation instead of
an editor. Styled like an attached trigger, never the red
"missing" state. -->
<button
type="button"
onclick={handleWebhookClick}
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
style.bg,
data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={`Webhook endpoint for ${data.runnable_path ?? ''} — click to view URLs and create a token`}
>
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${style.iconText}`} />
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
<span class="text-3xs uppercase tracking-wide truncate text-tertiary">
{style.label}
</span>
<span class="text-2xs font-mono truncate text-emphasis"> URLs & token </span>
</div>
</button>
{:else if canOpenDataUpload}
<!-- Data upload: UI-first entry point (no trigger row). Clicking the
node opens the target script's run form, where the auto-generated
S3 picker lets the user upload a file and run the pipeline. Goes
green once a file is staged (ready), so "Run pipeline" can proceed;
until then it stays neutral with an "upload a file" prompt. Never
the red "missing" state. -->
<button
type="button"
onclick={handleDataUploadClick}
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
dataUploadReady
? 'bg-green-50 dark:bg-green-900/30 outline-green-500 dark:outline-green-600'
: style.bg,
dataUploadReady ? '' : data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={dataUploadReady
? `Data upload for ${data.runnable_path ?? ''} — a file is staged; the pipeline is ready to run. Click to change it.`
: `Data upload for ${data.runnable_path ?? ''} — click to open the run form and upload a file`}
>
<DataUploadIcon
size={14}
class={`shrink-0 ml-2 mr-2 ${dataUploadReady ? 'text-green-600 dark:text-green-400' : style.iconText}`}
/>
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
<span
class={twMerge(
'text-3xs uppercase tracking-wide truncate',
dataUploadReady ? 'text-green-700 dark:text-green-400' : 'text-tertiary'
)}
>
{style.label}{dataUploadReady ? ' · ready' : ''}
</span>
<span
class={twMerge(
'text-2xs font-mono truncate',
dataUploadReady ? 'text-green-700 dark:text-green-400' : 'text-emphasis'
)}
>
{dataUploadReady ? 'File staged' : 'Upload & run'}
</span>
</div>
</button>
{:else}
<div
class={twMerge(
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1',
displayMissing
? 'bg-red-50 dark:bg-red-900/30 outline-dashed outline-red-400 dark:outline-red-500'
: style.bg,
displayMissing ? '' : data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border
)}
style="width: {NODE.width}px; min-height: {NODE.height}px;"
title={missingTitle ??
(data.unsaved ? `Unsaved ${style.label}: ${data.ref}` : `${style.label}: ${data.ref}`)}
>
<Icon
size={14}
class={`shrink-0 ml-2 mr-2 ${displayMissing ? 'text-red-600 dark:text-red-400' : style.iconText}`}
/>
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
<span
class={twMerge(
'text-3xs uppercase tracking-wide truncate',
displayMissing ? 'text-red-700 dark:text-red-400' : 'text-tertiary'
)}
>
{style.label}{displayMissing ? ' · missing' : data.unsaved ? ' · unsaved' : ''}
</span>
<span
class={twMerge(
'text-2xs font-mono truncate',
displayMissing ? 'text-red-700 dark:text-red-400' : 'text-emphasis'
)}
>
{displayMissing ? 'no trigger row' : data.ref}
</span>
</div>
</div>
{/if}
{#if menuItems.length > 0}
<!-- Hover-revealed kebab menu (Delete only for now). Mirrors the
RunnableNode pattern: positioned just outside the top-right of
the node, rendered only on hover or while the menu is open so
the canvas stays clean at rest. `pointerdown` is stopped so
svelte-flow doesn't kick off node selection / drag when the
user reaches for the menu. -->
<div class="absolute -top-2 -right-2 h-7 p-1 min-w-7" style="will-change: transform;">
<DropdownV2
items={menuItems}
placement="bottom-end"
bind:open={menuOpen}
fixedHeight={false}
usePointerDownOutside
>
{#snippet buttonReplacement()}
<button
class={twMerge(
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
hover || menuOpen ? 'block' : '!hidden',
'shadow-md rounded-md'
)}
onpointerdown={stopPropagation(preventDefault(() => {}))}
title="Actions"
>
<EllipsisVertical size={12} />
</button>
{/snippet}
</DropdownV2>
{#snippet icon()}
{#if canOpenDataUpload}
<DataUploadIcon
size={14}
class={dataUploadReady ? 'text-green-600 dark:text-green-400' : style.iconText}
/>
{:else}
<Icon
size={14}
class={displayMissing || canCreate ? 'text-red-600 dark:text-red-400' : style.iconText}
/>
{/if}
{/snippet}
</PipelineNodeCard>
</NodeActionsMenu>
{#if data.fix}
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
<NodeFixButton fix={data.fix} />
</div>
{/if}
</div>
@@ -147,4 +147,18 @@ describe('layoutAssetGraph (tidy-tree with join breaks)', () => {
expect(pos.get(id)!.y).toBeGreaterThan(anchor.y)
}
})
it('packs narrow nodes by their own width', () => {
const pos = layoutAssetGraph({
nodes: [
{ ...n('p'), width: 100 },
{ ...n('l'), width: 100 },
{ ...n('r'), width: 180 }
],
edges: [e('p', 'l'), e('p', 'r')]
})
const gap = pos.get('r')!.x - 90 - (pos.get('l')!.x + 50)
expect(gap).toBe(NODE.gap.horizontal)
expect(pos.get('l')!.x).toBe(50)
})
})
@@ -7,7 +7,15 @@ import { NODE } from '$lib/components/graph/util'
// breathing room.
const NODE_WIDTH = NODE.width
const NODE_HEIGHT = NODE.height + 30
const LAYER_GAP = NODE.gap.vertical
/** Rendered height of a pipeline node: a kind label over a title. Fits inside
* the `NODE_HEIGHT` row the layout reserves. */
export const PIPELINE_NODE_HEIGHT = 48
// A step more than the flow editor's gap: the + on a node's bottom edge sits in it.
const LAYER_GAP = NODE.gap.vertical + 16
/** Height of the "Add data source" pill the anchor draws, and its gap to the
* graph: tighter than LAYER_GAP, since no edge runs through it. */
const ANCHOR_HEIGHT = 32
const ANCHOR_GAP = 40
const SIBLING_GAP = NODE.gap.horizontal
// Horizontal gutter between two disjoint subgraphs. Wider than the
// within-subgraph sibling gap so visually-unrelated components (e.g. two
@@ -15,10 +23,16 @@ const SIBLING_GAP = NODE.gap.horizontal
// into each other's columns — but only ~2×, so they don't drift far apart.
const COMPONENT_GAP = SIBLING_GAP * 2
const LAYER_H = NODE_HEIGHT + LAYER_GAP
/** Extra height of an assets-only node: the trigger chip under its title. */
export const PIPELINE_NODE_EXTRA_ROW = 20
/** Height of the header strip naming an assets-only node's producing script. */
export const PIPELINE_NODE_HEADER = 22
/** Extra vertical padding of a card under a header, top plus bottom. */
export const PIPELINE_NODE_HEADER_CARD_PAD = 12
interface GraphInput {
nodes: Array<{ id: string; data: AssetGraphNodeData }>
/** `width` defaults to `NODE.width`. */
nodes: Array<{ id: string; data: AssetGraphNodeData; width?: number }>
edges: Array<{ source: string; target: string }>
}
@@ -52,15 +66,18 @@ interface Band {
// intersects theirs.
//
// y comes from longest-path layering (same top-down orientation as before:
// producers above, assets in the middle, consumers below). Returns positions
// (band centers) normalized so the component's min x,y = 0. Cyclic input is
// handled by dropping feedback edges (see the Kahn step below).
// producers above, assets in the middle, consumers below). Returns node
// centers, normalized so the component's leftmost edge and top are at 0.
// Cyclic input is handled by dropping feedback edges (see the Kahn step below).
function layoutComponent(
nodes: GraphInput['nodes'],
edges: GraphInput['edges']
edges: GraphInput['edges'],
rowH: number
): Map<string, Positioned> {
const layerH = rowH + LAYER_GAP
const out = new Map<string, Positioned>()
const ids = new Set(nodes.map((n) => n.id))
const nodeW = new Map(nodes.map((n) => [n.id, n.width ?? NODE_WIDTH]))
const parents = new Map<string, string[]>()
const children = new Map<string, string[]>()
@@ -137,7 +154,7 @@ function layoutComponent(
const kids = treeChildren.get(id)!
const kidsW =
kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * Math.max(0, kids.length - 1)
W.set(id, Math.max(NODE_WIDTH, kidsW))
W.set(id, Math.max(nodeW.get(id)!, kidsW))
}
// Vertical (pixel) extent of a subtree, for band collision checks.
@@ -152,14 +169,14 @@ function layoutComponent(
if (l > hi) hi = l
for (const k of treeChildren.get(cur)!) stack.push(k)
}
return { top: lo * LAYER_H, bottom: hi * LAYER_H + NODE_HEIGHT }
return { top: lo * layerH, bottom: hi * layerH + rowH }
}
// Recursive placement: node centered over its band, children packed
// side-by-side and centered within it.
function placeTree(id: string, left: number) {
const w = W.get(id)!
out.set(id, { x: left + w / 2, y: layer.get(id)! * LAYER_H })
out.set(id, { x: left + w / 2, y: layer.get(id)! * layerH })
const kids = treeChildren.get(id)!
if (kids.length === 0) return
const kidsW = kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * (kids.length - 1)
@@ -211,11 +228,11 @@ function layoutComponent(
placeAndRecord(id, center - w / 2)
}
// Normalize so the component's min x,y = 0.
// Normalize so the component's leftmost edge and top are at 0.
let minX = Infinity
let minY = Infinity
for (const p of out.values()) {
if (p.x < minX) minX = p.x
for (const [id, p] of out) {
if (p.x - nodeW.get(id)! / 2 < minX) minX = p.x - nodeW.get(id)! / 2
if (p.y < minY) minY = p.y
}
if (isFinite(minX) && isFinite(minY)) {
@@ -243,7 +260,13 @@ function layoutComponent(
//
// Falls back to a stable grid if the component layout throws (defensive —
// cycles are already absorbed by feedback-edge dropping in layoutComponent).
export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<string, Positioned> {
export function layoutAssetGraph(
graph: GraphInput,
anchorId?: string,
extraRowHeight = 0
): Map<string, Positioned> {
const rowH = NODE_HEIGHT + extraRowHeight
const layerH = rowH + LAYER_GAP
const byId = new Map<string, Positioned>()
if (graph.nodes.length === 0) return byId
@@ -297,19 +320,20 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
// 3. Lay out each component and pack side-by-side. xOffset advances by
// the laid-out width of each component (max node center + a node width,
// since positions are node centers) plus the gutter.
const widthOf = new Map(nodes.map((n) => [n.id, n.width ?? NODE_WIDTH]))
let xOffset = 0
for (let c = 0; c < nComp; c++) {
const positions = layoutComponent(compNodes[c], compEdges[c])
let maxX = 0
for (const p of positions.values()) if (p.x > maxX) maxX = p.x
const positions = layoutComponent(compNodes[c], compEdges[c], rowH)
let maxRight = 0
for (const [id, p] of positions) {
maxRight = Math.max(maxRight, p.x + widthOf.get(id)! / 2)
byId.set(id, { x: p.x + xOffset, y: p.y })
}
xOffset += maxX + NODE_WIDTH + COMPONENT_GAP
xOffset += maxRight + COMPONENT_GAP
}
// 4. Re-place the anchor centered horizontally over the whole packed
// graph, one layer above it; then renormalize so the anchor sits at the
// graph, just above it; then renormalize so the anchor sits at the
// top (y = 0), pushing the components down a layer to make room (mirrors
// the previous "anchor is the parent of every root" behaviour).
if (anchorId && graph.nodes.some((n) => n.id === anchorId)) {
@@ -324,7 +348,9 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
if (p.x > maxX) maxX = p.x
if (p.y < minY) minY = p.y
}
byId.set(anchorId, { x: (minX + maxX) / 2, y: minY - LAYER_H })
// One gap above the graph, sized to the pill rather than a full node row:
// a row tall enough for an assets-only card leaves it floating high.
byId.set(anchorId, { x: (minX + maxX) / 2, y: minY - ANCHOR_HEIGHT - ANCHOR_GAP })
let nMinY = Infinity
for (const p of byId.values()) if (p.y < nMinY) nMinY = p.y
for (const p of byId.values()) p.y -= nMinY
@@ -336,7 +362,7 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
graph.nodes.forEach((n, i) => {
byId.set(n.id, {
x: (i % cols) * (NODE_WIDTH + SIBLING_GAP),
y: Math.floor(i / cols) * LAYER_H
y: Math.floor(i / cols) * layerH
})
})
return byId
@@ -0,0 +1,66 @@
import { NODE } from '$lib/components/graph/util'
const MIN_WIDTH = 160
let ctx: CanvasRenderingContext2D | null | undefined
/** Rendered width of `text` at `rem` in the page font, or a rough guess without a DOM. */
function textWidth(text: string, rem: number): number {
if (typeof document === 'undefined') return text.length * rem * 16 * 0.6
ctx ??= document.createElement('canvas').getContext('2d')
const root = parseFloat(getComputedStyle(document.documentElement).fontSize) || 16
if (!ctx) return text.length * rem * root * 0.6
const body = getComputedStyle(document.body)
ctx.font = `${rem * root}px ${body.fontFamily}`
const spacing = parseFloat(body.letterSpacing) || 0
return ctx.measureText(text).width + spacing * text.length
}
/**
* Width of an assets-only asset card, from the pieces `PipelineNodeCard` and
* `AssetNode` lay out: icon and the widest of kind / title / trigger chip, or the
* header naming the producing script, if wider. The numbers mirror their Tailwind spacing; change one
* and the other drifts. Clamped to the regular node width, past which the title
* truncates as usual.
*/
export function assetsOnlyNodeWidth(n: {
kind: string
title: string
chip: string
/** The chip leads with an icon (the folded-errors Fix chip). */
chipIcon?: boolean
/** The chip ends in the edit icon (an editable trigger). */
chipEdit?: boolean
/** The header tab: its text (a script's summary or path, "N scripts"), icon
* count, and whether it ends in the menu chevron (else the edit icon) or has a "draft" mark;
* undefined without a header. */
header?: { label: string; icons: number; chevron?: boolean; draft?: boolean }
}): number {
const icon = 12 + 14 + 10
const text = Math.max(
textWidth(n.kind, 0.7),
textWidth(n.title, 0.75),
textWidth(n.chip, 0.65) + 12 + 2 + (n.chipIcon ? 14 : 0) + (n.chipEdit ? 4 + 9 : 0)
)
const body = Math.ceil(icon + text + 4 + 8 + 2 + 8)
// Header tab: mx-2 and a 1px border around px-2, then gap-1.5 between its
// items: the icons, the label, a "draft" mark, and the chevron or edit icon.
// The run chip is left out: it comes and goes with runs, and a width that
// followed it would re-lay the graph on a click; the label truncates instead.
const h = n.header
const header = h
? Math.ceil(
16 +
2 +
16 +
h.icons * (11 + 6) +
textWidth(h.label, 0.65) +
(h.draft ? 6 + textWidth('draft', 0.65) : 0) +
// The menu chevron, or the edit icon of a single script.
6 +
(h.chevron ? 11 : 10) +
4
)
: 0
return Math.min(NODE.width, Math.max(MIN_WIDTH, body, header))
}
@@ -0,0 +1,25 @@
// Pipelines open in the assets-only view. This remembers the ones last left
// showing their scripts and triggers, as `<workspace>/<folder>`, so reopening one
// shows it the same way. Browser-local: a missing or unreadable list reads as
// "none", and storage that throws (private mode, blocked site data) is ignored.
const KEY = 'pipeline-full-view-folders'
function read(): string[] {
try {
const parsed = JSON.parse(localStorage.getItem(KEY) ?? '[]')
return Array.isArray(parsed) ? parsed.filter((f) => typeof f === 'string') : []
} catch {
return []
}
}
export function isAssetsOnlyFolder(folder: string): boolean {
return !read().includes(folder)
}
export function setAssetsOnlyFolder(folder: string, on: boolean): void {
const rest = read().filter((f) => f !== folder)
try {
localStorage.setItem(KEY, JSON.stringify(on ? rest : [...rest, folder]))
} catch {}
}
@@ -0,0 +1,162 @@
import { describe, expect, it } from 'vitest'
import {
assetsOnlyView,
upstreamDeletion,
noAssetNodeId,
withoutPassiveReads,
type AssetUpstream
} from './assetsOnlyView'
const node = (id: string, type: string, data: any = {}) => ({ id, type, data })
const edge = (source: string, target: string, kind: string) => ({
id: `${kind}:${source}->${target}`,
source,
target,
kind
})
describe('assetsOnlyView', () => {
const nodes = [
node('asset:ducklake:raw', 'asset'),
node('asset:ducklake:clean', 'asset'),
node('asset:ducklake:report', 'asset'),
node('script:f/p/clean', 'runnable'),
node('script:f/p/report', 'runnable'),
node('script:f/p/other', 'runnable'),
node('trigger:schedule:f/p/daily', 'trigger', { kind: 'schedule', schedule: '0 0 4 * * *' })
]
const edges = [
edge('trigger:schedule:f/p/daily', 'script:f/p/clean', 'trigger-native'),
edge('asset:ducklake:raw', 'script:f/p/clean', 'lineage-read'),
edge('script:f/p/clean', 'asset:ducklake:clean', 'lineage-write'),
// read-modify-write: no self edge
edge('asset:ducklake:clean', 'script:f/p/clean', 'lineage-read'),
edge('asset:ducklake:clean', 'script:f/p/report', 'trigger-asset'),
edge('asset:ducklake:clean', 'script:f/p/report', 'lineage-read'),
edge('script:f/p/report', 'asset:ducklake:report', 'lineage-write'),
edge('script:f/p/other', 'asset:ducklake:report', 'lineage-write')
]
const v = assetsOnlyView(nodes, edges)
it('joins each script input to its outputs, once', () => {
expect(v.edges.map((e) => `${e.source} -> ${e.target}`).sort()).toEqual([
'asset:ducklake:clean -> asset:ducklake:report',
'asset:ducklake:raw -> asset:ducklake:clean'
])
expect([...v.nodeIds].sort()).toEqual([
'asset:ducklake:clean',
'asset:ducklake:raw',
'asset:ducklake:report'
])
})
it('names the single producer and its trigger, or marks the asset ambiguous', () => {
expect(v.upstream.get('asset:ducklake:raw')).toBeUndefined()
expect(v.upstream.get('asset:ducklake:clean')).toMatchObject({
multiple: false,
runnableId: 'script:f/p/clean',
trigger: { kind: 'schedule', data: { schedule: '0 0 4 * * *' } }
})
expect(v.upstream.get('asset:ducklake:report')).toEqual({
multiple: true,
runnableIds: ['script:f/p/report', 'script:f/p/other']
})
})
})
describe('the No asset nodes', () => {
it('give each script that builds nothing its own node, fed by what it reads', () => {
const v = assetsOnlyView(
[
node('asset:ducklake:a', 'asset'),
node('asset:ducklake:b', 'asset'),
node('script:f/p/on_a', 'runnable'),
node('script:f/p/on_b', 'runnable'),
node('script:f/p/dbt', 'runnable', { dbt: { model_count: 3 } }),
node('script:f/p/macros', 'runnable', { macros: [{ name: 'm' }] })
],
[
edge('asset:ducklake:a', 'script:f/p/on_a', 'trigger-asset'),
edge('asset:ducklake:b', 'script:f/p/on_b', 'trigger-asset')
]
)
const onA = noAssetNodeId('script:f/p/on_a')
const onB = noAssetNodeId('script:f/p/on_b')
expect(v.noAssetNodeIds).toEqual([onA, onB])
expect(v.upstream.get(onA)).toMatchObject({ runnableId: 'script:f/p/on_a' })
expect(v.edges.map((e) => `${e.source} -> ${e.target}`)).toEqual([
`asset:ducklake:a -> ${onA}`,
`asset:ducklake:b -> ${onB}`
])
})
})
describe('upstreamDeletion', () => {
const script = { runnable_kind: 'script', path: 'f/p/a' }
const schedule = (targets: string[]) => ({
kind: 'schedule',
nodeId: 'trigger:schedule:f/p/a_schedule',
data: { ref: 'f/p/a_schedule', runnable_paths: targets }
})
const own = (trigger?: any): AssetUpstream & { multiple: false } => ({
multiple: false,
runnableId: 'script:f/p/a',
trigger
})
it('deletes the script and a trigger that fires only it', () => {
const u = own(schedule(['f/p/a']))
expect(upstreamDeletion(u, script, new Map([['asset:x', u]]))).toEqual({
script: { path: 'f/p/a', unsaved: false },
trigger: { kind: 'schedule', path: 'f/p/a_schedule', draft: false }
})
})
it('keeps a trigger that fires other scripts', () => {
const u = own(schedule(['f/p/a', 'f/p/b']))
expect(upstreamDeletion(u, script, new Map([['asset:x', u]]))?.trigger).toBeUndefined()
})
it('deletes nothing when the script also co-produces another asset', () => {
const u = own(schedule(['f/p/a']))
const all = new Map<string, AssetUpstream>([
['asset:x', u],
['asset:y', { multiple: true, runnableIds: ['script:f/p/a', 'script:f/p/b'] }]
])
expect(upstreamDeletion(u, script, all)).toBeUndefined()
})
it('deletes nothing when the script builds another asset too', () => {
const u = own(schedule(['f/p/a']))
const all = new Map([
['asset:x', u],
['asset:y', u]
])
expect(upstreamDeletion(u, script, all)).toBeUndefined()
})
})
describe('withoutPassiveReads', () => {
it('drops reads that trigger nothing, and the assets only they drew', () => {
const nodes = [
node('asset:src', 'asset'),
node('asset:lookup', 'asset'),
node('asset:out', 'asset'),
node('script:a', 'runnable')
]
const edges = [
edge('asset:src', 'script:a', 'lineage-read'),
edge('asset:src', 'script:a', 'trigger-asset'),
edge('asset:lookup', 'script:a', 'lineage-read'),
edge('asset:out', 'script:a', 'lineage-read'),
edge('script:a', 'asset:out', 'lineage-write')
]
const r = withoutPassiveReads(nodes, edges)
expect(r.nodes.map((n) => n.id)).toEqual(['asset:src', 'asset:out', 'script:a'])
expect(r.edges.map((e) => e.id)).toEqual([
'lineage-read:asset:src->script:a',
'trigger-asset:asset:src->script:a',
'lineage-write:script:a->asset:out'
])
})
})
@@ -0,0 +1,234 @@
import type { NativeTriggerKind } from './types'
/** A node or edge of the canvas model, reduced to what the assets-only view reads. */
export type ViewNode = { id: string; type: string; data: any }
export type ViewEdge = {
id: string
source: string
target: string
kind: string
unsaved?: boolean
/** Assets-only edges: the folded runnables the data goes through. */
via?: string[]
/** Assets-only edges: a write to the source reruns one of `via`. */
reactive?: boolean
}
/** The assets-only node standing for a script that builds no asset: one per
* script, so scripts reacting to different inputs stay apart. */
export const noAssetNodeId = (runnableId: string) => `no-asset:${runnableId}`
export const isNoAssetNode = (id: string) => id.startsWith('no-asset:')
/** What an assets-only delete removes: the script, its trigger, or both. */
export type AssetUpstreamDelete = {
script?: { path: string; unsaved: boolean }
trigger?: { kind: NativeTriggerKind; path: string; draft: boolean }
}
/** What an asset shows of the script and trigger that produce it, in the assets-only view. */
export type AssetUpstream =
/** Several producing scripts, or one with several triggers. */
| { multiple: true; runnableIds: string[] }
| {
multiple: false
/** Canvas id of the producing runnable (`script:<path>`). */
runnableId: string
/** The producer's trigger; undefined when nothing triggers it (run by hand). */
trigger?: { nodeId?: string; kind: string; data?: any }
}
/**
* The asset → asset view of a pipeline graph: runnables and triggers disappear,
* and every read (or asset trigger) of a script is joined to each of its writes,
* so an asset points at the assets computed from it. Each produced asset keeps a
* reference to its single producer and that producer's single trigger, or is
* marked `multiple` when either is ambiguous.
*/
export function assetsOnlyView<N extends ViewNode, E extends ViewEdge>(
nodes: N[],
edges: E[]
): {
nodeIds: Set<string>
edges: Array<ViewEdge>
upstream: Map<string, AssetUpstream>
/** A node per script that builds no asset (`noAssetNodeId`), its upstream
* under that id. */
noAssetNodeIds: string[]
} {
const nodeById = new Map(nodes.map((n) => [n.id, n]))
const inputsOf = new Map<string, E[]>()
const triggersOf = new Map<string, E[]>()
const producersOf = new Map<string, E[]>()
const push = (m: Map<string, E[]>, k: string, e: E) => m.set(k, [...(m.get(k) ?? []), e])
for (const e of edges) {
if (e.kind === 'lineage-write') push(producersOf, e.target, e)
else if (e.kind === 'lineage-read' || e.kind === 'trigger-asset') {
push(inputsOf, e.target, e)
if (e.kind === 'trigger-asset') push(triggersOf, e.target, e)
} else if (e.kind === 'trigger-native') push(triggersOf, e.target, e)
}
const triggered = new Set(
edges.filter((e) => e.kind === 'trigger-asset').map((e) => `${e.source}\n${e.target}`)
)
const out = new Map<string, ViewEdge>()
for (const [assetId, writes] of producersOf) {
for (const w of writes) {
for (const r of inputsOf.get(w.source) ?? []) {
// A read-modify-write of the same table is not a dependency on itself.
if (r.source === assetId) continue
const id = `flow:${r.source}->${assetId}`
const prev = out.get(id)
const unsaved = !!(w.unsaved || r.unsaved)
const reactive = triggered.has(`${r.source}\n${w.source}`)
if (prev) {
prev.unsaved = prev.unsaved && unsaved
prev.reactive ||= reactive
if (!prev.via?.includes(w.source)) prev.via = [...(prev.via ?? []), w.source]
} else {
out.set(id, {
id,
source: r.source,
target: assetId,
kind: 'asset-flow',
unsaved,
reactive,
via: [w.source]
})
}
}
}
}
for (const e of edges) if (e.kind === 'dbt-ref') out.set(e.id, e)
function describe(runnableIds: string[]): AssetUpstream {
if (runnableIds.length !== 1) return { multiple: true, runnableIds }
const runnableId = runnableIds[0]
// Every asset trigger of a script is one cause, "an input changed".
const triggers = [
...new Map(
(triggersOf.get(runnableId) ?? []).map((t) => [
t.kind === 'trigger-asset' ? 'asset' : t.source,
t
])
).values()
]
if (triggers.length > 1) return { multiple: true, runnableIds }
const t = triggers[0]
const tNode = t && nodeById.get(t.source)
return {
multiple: false,
runnableId,
trigger: !t
? undefined
: t.kind === 'trigger-asset'
? { kind: 'asset' }
: { nodeId: t.source, kind: tNode?.data?.kind, data: tNode?.data }
}
}
const upstream = new Map<string, AssetUpstream>()
for (const [assetId, writes] of producersOf) {
upstream.set(assetId, describe([...new Set(writes.map((w) => w.source))]))
}
// Scripts that build no asset have nowhere to fold into, so each gets a "No
// asset" node, fed by what it reads. dbt projects (whose writes are not drawn)
// and macro libraries (which build nothing by design) get none.
const writers = new Set([...producersOf.values()].flat().map((w) => w.source))
const orphans = nodes
.filter(
(n) => n.type === 'runnable' && !writers.has(n.id) && !n.data?.dbt && !n.data?.macros?.length
)
.map((n) => n.id)
for (const id of orphans) {
const nodeId = noAssetNodeId(id)
upstream.set(nodeId, describe([id]))
for (const r of inputsOf.get(id) ?? []) {
const eid = `flow:${r.source}->${nodeId}`
const reactive = triggered.has(`${r.source}\n${id}`)
const prev = out.get(eid)
if (prev) {
prev.unsaved = prev.unsaved && !!r.unsaved
prev.reactive ||= reactive
} else {
out.set(eid, {
id: eid,
source: r.source,
target: nodeId,
kind: 'asset-flow',
unsaved: !!r.unsaved,
reactive,
via: [id]
})
}
}
}
const nodeIds = new Set(nodes.filter((n) => n.type === 'asset').map((n) => n.id))
return {
nodeIds,
edges: [...out.values()],
upstream,
noAssetNodeIds: orphans.map(noAssetNodeId)
}
}
/** What deleting an asset's upstream removes: the script unless another asset
* is also built by it, and with it the trigger unless that fires other scripts.
* Undefined when nothing can go. */
export function upstreamDeletion(
u: AssetUpstream & { multiple: false },
r: { runnable_kind: string; path: string; unsaved?: boolean },
all: Map<string, AssetUpstream>
): AssetUpstreamDelete | undefined {
if (r.runnable_kind !== 'script') return undefined
const producesOthers =
[...all.values()].filter((o) =>
o.multiple ? o.runnableIds.includes(u.runnableId) : o.runnableId === u.runnableId
).length > 1
if (producesOthers) return undefined
const t = u.trigger
const kind = t?.kind as NativeTriggerKind | undefined
const targets: string[] = t?.data?.runnable_paths ?? []
const ownTrigger =
kind &&
kind !== 'webhook' &&
kind !== 'data_upload' &&
t?.data?.ref &&
!t.data.missing &&
targets.every((p) => p === r.path)
return {
script: { path: r.path, unsaved: !!r.unsaved },
trigger: ownTrigger ? { kind, path: t.data.ref, draft: !!t.data.draft } : undefined
}
}
/**
* The graph without its reads that start nothing: a read edge with no asset
* trigger along it, and the assets only such reads drew.
*/
export function withoutPassiveReads<N extends ViewNode, E extends ViewEdge>(
nodes: N[],
edges: E[]
): { nodes: N[]; edges: E[]; dropped: number } {
const triggers = new Set(
edges.filter((e) => e.kind === 'trigger-asset').map((e) => `${e.source}\n${e.target}`)
)
const passive = (e: E) =>
e.kind === 'lineage-read' && !triggers.has(`${e.source}\n${e.target}`)
const kept = edges.filter((e) => !passive(e))
const dropped = edges.length - kept.length
if (dropped === 0) return { nodes, edges, dropped }
const linked = new Set(
kept.filter((e) => e.kind !== 'add-anchor').flatMap((e) => [e.source, e.target])
)
const readOnly = new Set(
edges.filter(passive).map((e) => e.source).filter((id) => !linked.has(id))
)
return {
nodes: nodes.filter((n) => !readOnly.has(n.id)),
edges: kept.filter((e) => !readOnly.has(e.source) && !readOnly.has(e.target)),
dropped
}
}
@@ -0,0 +1,51 @@
import { describe, expect, it } from 'vitest'
import { describeEdge } from './edgeDescription'
const names: Record<string, string> = {
'asset:ducklake:main/raw': 'raw',
'asset:ducklake:main/clean': 'clean',
'script:f/p/clean': 'Clean',
'trigger:schedule:f/p/daily': 'Daily'
}
const nameOf = (id: string) => names[id] ?? id
const dataOf = (id: string) =>
id === 'trigger:schedule:f/p/daily'
? { kind: 'schedule', schedule: '0 0 12 * * *', timezone: 'UTC' }
: undefined
describe('describeEdge', () => {
it('names an asset trigger by what runs it: any write, changed or not', () => {
expect(
describeEdge(
{ source: 'asset:ducklake:main/raw', target: 'script:f/p/clean', kind: 'trigger-asset' },
nameOf,
dataOf
)
).toEqual({ subject: 'Clean', phrase: 'runs after each write to', object: 'raw' })
})
it("gives a schedule's cadence", () => {
expect(
describeEdge(
{ source: 'trigger:schedule:f/p/daily', target: 'script:f/p/clean', kind: 'trigger-native' },
nameOf,
dataOf
)?.phrase
).toBe('runs on schedule: Every day at 12:00 UTC')
})
it('names the folded script on an assets-only edge', () => {
expect(
describeEdge(
{
source: 'asset:ducklake:main/raw',
target: 'asset:ducklake:main/clean',
kind: 'asset-flow',
via: ['script:f/p/clean']
},
nameOf,
dataOf
)
).toEqual({ subject: 'clean', phrase: 'is built from', object: 'raw', note: 'By Clean. A write to raw does not rerun it.' })
})
})
@@ -0,0 +1,110 @@
import { describeSchedule } from '$lib/utils/describeCron'
import { isNoAssetNode } from './assetsOnlyView'
/** What a hovered edge says: "`subject` `phrase` `object`", then a note. */
export type EdgeDescription = { subject: string; phrase: string; object?: string; note?: string }
/** The edge fields the description reads (the canvas model's edge). */
export type DescribedEdge = {
source: string
target: string
kind: string
unsaved?: boolean
muted?: boolean
missing?: boolean
via?: string[]
reactive?: boolean
macro_names?: string[]
via_use?: boolean
}
// What starts a script, per trigger kind: "runs on <event>".
const TRIGGER_EVENT: Record<string, string> = {
kafka: 'Kafka message',
mqtt: 'MQTT message',
amqp: 'AMQP message',
nats: 'NATS message',
sqs: 'SQS message',
gcp: 'Pub/Sub message',
email: 'incoming email',
postgres: 'Postgres event',
webhook: 'webhook call',
data_upload: 'data upload'
}
/**
* The relationship an edge draws, as a sentence about its two ends. `nameOf`
* names a node, `dataOf` gives its data (a trigger's kind and cadence).
*/
export function describeEdge(
e: DescribedEdge,
nameOf: (id: string) => string,
dataOf: (id: string) => any
): EdgeDescription | undefined {
const src = nameOf(e.source)
const tgt = nameOf(e.target)
const draft = e.unsaved ? 'Not deployed yet.' : undefined
switch (e.kind) {
case 'lineage-write':
return { subject: src, phrase: 'writes to', object: tgt, note: draft }
case 'lineage-read':
return {
subject: tgt,
phrase: 'reads from',
object: src,
note: e.muted ? 'A write to it does not run the script (muted).' : draft
}
case 'trigger-asset':
return { subject: tgt, phrase: 'runs after each write to', object: src, note: draft }
case 'trigger-native': {
const t = dataOf(e.source) ?? {}
if (e.missing) {
return {
subject: tgt,
phrase: `declares \`// on ${t.kind}\`, but no trigger of that kind targets it`
}
}
const phrase =
t.kind === 'schedule' && t.schedule
? `runs on schedule: ${describeSchedule(t.schedule, t.timezone) ?? t.schedule}`
: `runs on ${TRIGGER_EVENT[t.kind] ?? `${t.kind} trigger`}`
return { subject: tgt, phrase, note: t.draft ? 'Draft trigger, created on save.' : draft }
}
case 'asset-flow': {
const via = (e.via ?? []).map(nameOf).join(', ')
const rerun = e.reactive
? `Each write to ${src} reruns it.`
: `A write to ${src} does not rerun it.`
// The scripts that build nothing only read: name them, not the node.
if (isNoAssetNode(e.target)) {
return { subject: src, phrase: 'is read by', object: via || tgt, note: rerun }
}
return {
subject: tgt,
phrase: 'is built from',
object: src,
note: via ? `By ${via}. ${rerun}` : rerun
}
}
case 'dbt-ref':
return { subject: tgt, phrase: 'references', object: src, note: 'dbt ref().' }
case 'macro':
return {
subject: tgt,
phrase: e.via_use ? 'uses the macro library' : 'uses macros from',
object: src,
note: e.macro_names?.length ? e.macro_names.join(', ') : undefined
}
case 'test-dependency':
return {
subject: src,
phrase: 'must run before',
object: tgt,
note: 'Its data test reads what the first one builds.'
}
case 'data-test':
return { subject: tgt, phrase: 'tests', object: src }
default:
return undefined
}
}
@@ -1,6 +1,8 @@
import { describe, expect, it } from 'vitest'
import type { AssetGraphResponse } from './types'
import {
hoverLineage,
transitivelyImpliedEdges,
buildDownstreamMap,
computeDownstreamClosure,
computeInducedSchedule
@@ -225,3 +227,44 @@ describe('computeInducedSchedule', () => {
expect(sched.cyclic.sort()).toEqual(['b', 'c'])
})
})
describe('hoverLineage', () => {
// a → b → c → d, with a side branch x → c and c → y; z is unrelated to b.
const edges = [
{ source: 'a', target: 'b' },
{ source: 'b', target: 'c' },
{ source: 'x', target: 'c' },
{ source: 'c', target: 'd' },
{ source: 'c', target: 'y' },
{ source: 'z', target: 'y' }
]
it('reaches everything upstream and downstream, recursively', () => {
const l = hoverLineage(edges, 'b')
expect([...l.upstream].sort()).toEqual(['a', 'b'])
expect([...l.downstream].sort()).toEqual(['b', 'c', 'd', 'y'])
})
it('keeps only the edges on a path through the node', () => {
const l = hoverLineage(edges, 'b')
expect(edges.filter(l.hasEdge)).toEqual([
{ source: 'a', target: 'b' },
{ source: 'b', target: 'c' },
{ source: 'c', target: 'd' },
{ source: 'c', target: 'y' }
])
})
})
describe('transitivelyImpliedEdges', () => {
it('flags an edge another path already implies, and only that one', () => {
const edges = [
{ id: 'ab', source: 'a', target: 'b' },
{ id: 'bc', source: 'b', target: 'c' },
{ id: 'ac', source: 'a', target: 'c' },
{ id: 'ad', source: 'a', target: 'd' },
{ id: 'ad2', source: 'a', target: 'd' }
]
expect([...transitivelyImpliedEdges(edges)]).toEqual(['ac'])
})
})
@@ -226,3 +226,76 @@ export function computeInducedSchedule(
const roots = ordered.filter((n) => (cleanIndegree.get(n) ?? 0) === 0)
return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, roots, cyclic }
}
/** What is linked to a node through the drawn graph: everything it depends on
* (upstream) and everything depending on it (downstream), both recursive, the
* node itself in each. An edge lies on a path through the node exactly when
* both its ends are upstream, or both downstream. */
export type HoverLineage = {
upstream: Set<string>
downstream: Set<string>
hasEdge: (e: { source: string; target: string }) => boolean
}
export function hoverLineage(
edges: ReadonlyArray<{ source: string; target: string }>,
id: string
): HoverLineage {
const walk = (from: 'source' | 'target', to: 'source' | 'target') => {
const next = new Map<string, string[]>()
for (const e of edges) next.set(e[from], [...(next.get(e[from]) ?? []), e[to]])
const seen = new Set([id])
const stack = [id]
while (stack.length) {
for (const n of next.get(stack.pop()!) ?? []) {
if (!seen.has(n)) {
seen.add(n)
stack.push(n)
}
}
}
return seen
}
const upstream = walk('target', 'source')
const downstream = walk('source', 'target')
return {
upstream,
downstream,
hasEdge: (e) =>
(upstream.has(e.source) && upstream.has(e.target)) ||
(downstream.has(e.source) && downstream.has(e.target))
}
}
/** Ids of the edges another path already implies: an edge A → C when C can be
* reached from A through at least one other node (A → B → … → C). Edges joining
* the same two nodes don't make each other redundant. */
export function transitivelyImpliedEdges(
edges: ReadonlyArray<{ id: string; source: string; target: string }>
): Set<string> {
const next = new Map<string, Set<string>>()
for (const e of edges) {
if (e.source === e.target) continue
next.set(e.source, (next.get(e.source) ?? new Set()).add(e.target))
}
// Whether `to` is reachable from `from` without taking the direct step.
const reachableAround = (from: string, to: string) => {
const seen = new Set<string>([from])
const stack = [...(next.get(from) ?? [])].filter((n) => n !== to)
for (const n of stack) seen.add(n)
while (stack.length) {
const cur = stack.pop()!
for (const n of next.get(cur) ?? []) {
if (n === to) return true
if (!seen.has(n)) {
seen.add(n)
stack.push(n)
}
}
}
return false
}
const implied = new Set<string>()
for (const e of edges) if (e.source !== e.target && reachableAround(e.source, e.target)) implied.add(e.id)
return implied
}
@@ -0,0 +1,22 @@
import { describe, expect, it } from 'vitest'
import { onDirectiveTargets } from './onDirectiveTargets'
describe('onDirectiveTargets', () => {
it('leaves out what the script feeds, directly or not, and what already triggers it', () => {
const nodes = [
{ id: 'script:a', type: 'runnable' },
{ id: 'script:b', type: 'runnable' },
{ id: 'asset:out', type: 'asset' },
{ id: 'asset:later', type: 'asset' },
{ id: 'asset:in', type: 'asset' },
{ id: 'asset:free', type: 'asset' }
]
const edges = [
{ source: 'script:a', target: 'asset:out', kind: 'lineage-write' },
{ source: 'asset:out', target: 'script:b', kind: 'trigger-asset' },
{ source: 'script:b', target: 'asset:later', kind: 'lineage-write' },
{ source: 'asset:in', target: 'script:a', kind: 'trigger-asset' }
]
expect([...onDirectiveTargets(nodes, edges, 'script:a')]).toEqual(['asset:free'])
})
})
@@ -0,0 +1,34 @@
/**
* The asset nodes a script can be subscribed to (`// on <asset>`) by dragging:
* every asset except those downstream of it, whose writes it would then run on
* in a loop, and those that already trigger it.
*/
export function onDirectiveTargets(
nodes: Array<{ id: string; type: string }>,
edges: Array<{ source: string; target: string; kind: string }>,
scriptId: string
): Set<string> {
const next = new Map<string, string[]>()
for (const e of edges) {
if (e.kind === 'add-anchor') continue
next.set(e.source, [...(next.get(e.source) ?? []), e.target])
}
const downstream = new Set<string>([scriptId])
const stack = [scriptId]
while (stack.length) {
for (const n of next.get(stack.pop()!) ?? []) {
if (!downstream.has(n)) {
downstream.add(n)
stack.push(n)
}
}
}
const triggering = new Set(
edges.filter((e) => e.kind === 'trigger-asset' && e.target === scriptId).map((e) => e.source)
)
return new Set(
nodes
.filter((n) => n.type === 'asset' && !downstream.has(n.id) && !triggering.has(n.id))
.map((n) => n.id)
)
}
@@ -8,7 +8,8 @@ import type { AssetGraphResponse } from './types'
function makeHandle(
initial: Array<[string, PipelineDraft]> = [],
runnables: Array<{ path: string }> = [],
folder = 'x'
folder = 'x',
triggers: unknown[] = []
) {
let drafts = new Map(initial)
let forgotten: string[] = []
@@ -16,11 +17,13 @@ function makeHandle(
getFolder: () => folder,
getWorkspace: () => 'w',
getResolvedGraph: () =>
({ assets: [], runnables, edges: [], triggers: [] }) as unknown as AssetGraphResponse,
({ assets: [], runnables, edges: [], triggers }) as unknown as AssetGraphResponse,
getDrafts: () => drafts,
setDrafts: (next) => (drafts = next),
newDraftLocalId: () => 'id',
onForgetPath: (p) => forgotten.push(p)
onForgetPath: (p) => forgotten.push(p),
getTriggerDrafts: () => new Map(),
setTriggerDraft: () => true
})
return { handle, drafts: () => drafts, forgotten: () => forgotten }
}
@@ -155,6 +158,30 @@ describe('pipeline AI direct-draft helpers', () => {
expect(res.detectedWrites).toEqual(['ducklake://main/out'])
})
// The canvas draws an inactive draft's stored outputs beside its materialize
// target, so a seeded default left on a materialize node shows a phantom asset.
it('keeps no seeded or stale output on a node with a materialize target', async () => {
vi.spyOn(ScriptService, 'getScriptByPath').mockRejectedValue(new Error('404'))
const { handle, drafts } = makeHandle()
await handle.proposeNode({
path: 'f/x/mat',
language: 'duckdb' as any,
content: '-- pipeline\n-- materialize ducklake://main/abc\nSELECT 1',
outputKind: 'ducklake' as any
})
expect(drafts().get('f/x/mat')?.outputAssets).toBeUndefined()
await handle.proposeNode({
path: 'f/x/later',
language: 'duckdb' as any,
content: '-- pipeline\nSELECT 1',
outputKind: 'ducklake' as any
})
expect(drafts().get('f/x/later')?.outputAssets?.length).toBeGreaterThan(0)
await handle.editNode('f/x/later', '-- pipeline\n-- materialize ducklake://main/abc\nSELECT 1')
expect(drafts().get('f/x/later')?.outputAssets).toBeUndefined()
})
it('editNode rejects a path outside the open folder', async () => {
const { handle, drafts } = makeHandle()
await expect(handle.editNode('f/other/foo', '-- pipeline')).rejects.toThrow(/open folder/)
@@ -181,4 +208,24 @@ describe('pipeline AI direct-draft helpers', () => {
expect(d?.script.tag).toBe('custom')
expect(d?.script.content).toBe('-- pipeline\nSELECT 2')
})
it("does not take a flow's schedule at the same path for the script's", async () => {
const flowSchedule = {
trigger_kind: 'schedule',
runnable_kind: 'flow',
runnable_path: 'f/x/ingest',
path: 'f/x/flow_schedule'
}
const content = '# pipeline\n# on schedule\ndef main():\n return 1'
const { handle } = makeHandle(
[['f/x/ingest', draft({ script: { content, language: 'python3' } as any })]],
[],
'x',
[flowSchedule]
)
expect(await handle.unconfiguredTriggers('f/x/ingest')).toEqual(['schedule'])
await expect(
handle.setNodeTrigger('f/x/ingest', 'schedule', { schedule: '0 0 6 * * *', timezone: 'UTC' })
).resolves.toMatchObject({ path: 'f/x/ingest_schedule' })
})
})
@@ -9,7 +9,12 @@ import {
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
import { assetUri, autoOutputAsset, type PipelineOutputKind } from './pipelineTemplates'
import { parsePipelineAnnotations, scd2CurrentTargetPath } from './parsePipelineAnnotations'
import type { AssetGraphResponse } from './types'
import type {
AssetGraphResponse,
PipelineTriggerDraft,
PipelineTriggerDraftKind
} from './types'
import { defaultTriggerPath, isDraftableTriggerKind, triggerDraftKey } from './pipelineTriggerDrafts'
import type {
PipelineAIChatHelpers,
PipelineContext,
@@ -62,20 +67,26 @@ export type PipelineAiHelperDeps = {
onForgetPath?: (path: string) => void
/** Notify the caller a test run started so it can light up its run UI. */
onRunStarted?: (jobId: string, path: string) => void
/** Whether the folder has trigger drafts, which live beside the script drafts. */
hasTriggerDrafts?: () => boolean
getTriggerDrafts: () => Map<string, PipelineTriggerDraft>
/** False when another trigger draft already holds the config's path. */
setTriggerDraft: (d: PipelineTriggerDraft, previousKey?: string) => boolean
}
export function makePipelineScript(
language: ScriptLang,
scriptPath: string,
content: string,
createdAt: string
createdAt: string,
summary = ''
): Script {
// Cast through unknown: a local draft only needs path/language/content/schema;
// the many readonly deployment fields on Script don't matter until createScript.
return {
hash: '',
path: scriptPath,
summary: '',
summary,
description: '',
content,
schema: emptySchema(),
@@ -199,7 +210,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
path: r.path,
language: draft?.script.language,
unsaved: r.unsaved ?? false,
summary: draft?.script.summary || undefined,
summary: (draft ? draft.script.summary : r.summary) || undefined,
writes: [...new Set(writes)],
reads: [...new Set(reads)],
triggers: [...new Set(triggers)]
@@ -213,7 +224,36 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
}
}
// Declared `on <kind>` bindings of a node that a trigger row or draft fills.
// Webhook and data-upload need none, so they never count as unconfigured.
function unconfiguredTriggers(path: string, content: string): PipelineTriggerDraftKind[] {
const declared = parsePipelineAnnotations(content)
.nativeTriggers.map((n) => n.kind)
.filter(isDraftableTriggerKind)
const drafted = new Set(
[...deps.getTriggerDrafts().values()]
.filter((d) => d.config.script_path === path)
.map((d) => d.kind)
)
const deployed = new Set(
deps
.getResolvedGraph()
.triggers.filter(
(t) =>
t.trigger_kind !== 'asset' &&
t.runnable_kind === 'script' &&
t.runnable_path === path &&
t.path &&
!t.missing
)
.map((t) => t.trigger_kind)
)
return [...new Set(declared)].filter((k) => !drafted.has(k) && !deployed.has(k))
}
const helpers: PipelineAIChatHelpers = {
getFolder: folderName,
hasDrafts: () => deps.getDrafts().size > 0 || (deps.hasTriggerDrafts?.() ?? false),
getPipelineContext: buildContext,
getNodeBody: async (path) => {
const draft = deps.getDrafts().get(path)
@@ -227,7 +267,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
return undefined
}
},
proposeNode: async ({ path, language, content, outputKind }) => {
proposeNode: async ({ path, language, content, outputKind, summary }) => {
deps.ensureEditable?.()
// build_pipeline_node creates a NEW node in the OPEN folder. Reject a path
// outside the folder (it would silently stage into this folder's bundle)
@@ -267,10 +307,13 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
}
const inferred = await inferDraftAssets(language, content)
// Fall back to a seeded output (from the declared output_kind) when the
// body doesn't yet write anything inferable.
// body doesn't yet write anything inferable. A `materialize` target is the
// output already (the canvas adds it from the annotation): a seed beside it
// would show as a second, default-named output.
const materialized = materializeWrites(content).length > 0
const seeded =
inferred.writes[0] ??
(outputKind
(outputKind && !materialized
? autoOutputAsset(outputKind as PipelineOutputKind, folderName(), language)
: undefined)
// Effective outputs = what actually becomes an output edge on the canvas:
@@ -280,7 +323,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
const next = new Map(drafts)
next.set(path, {
localId: deps.newDraftLocalId(),
script: makePipelineScript(language, path, content, new Date().toISOString()),
script: makePipelineScript(language, path, content, new Date().toISOString(), summary),
outputAssets,
inputAssets: inferred.reads
})
@@ -298,7 +341,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
)
}
},
editNode: async (path, content) => {
editNode: async (path, content, summary) => {
deps.ensureEditable?.()
assertInFolder(path)
assertPipelineAnnotation(content)
@@ -319,11 +362,19 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
baseScript = await ScriptService.getScriptByPath({ workspace, path })
}
const inferred = await inferDraftAssets(baseScript.language, content)
const outputAssets = inferred.writes.length > 0 ? inferred.writes : existing?.outputAssets
// The stored outputs only stand in for a body that writes nothing inferable;
// with a `materialize` target the annotation is the output, so stale ones
// (a seeded default name, the target before a rename) must not linger.
const outputAssets =
inferred.writes.length > 0
? inferred.writes
: materializeWrites(content).length > 0
? undefined
: existing?.outputAssets
const next = new Map(drafts)
next.set(path, {
localId: existing?.localId ?? deps.newDraftLocalId(),
script: { ...baseScript, content },
script: { ...baseScript, content, ...(summary !== undefined ? { summary } : {}) },
outputAssets,
inputAssets: inferred.reads
})
@@ -332,12 +383,61 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
deps.onProposeNode?.(path)
// Deployable lineage only (see proposeNode): body writes + materialize target.
return {
summary: summary ?? baseScript.summary ?? '',
detectedReads: inferred.reads.map(assetUri),
detectedWrites: dedupeAssets([...inferred.writes, ...materializeWrites(content)]).map(
assetUri
)
}
},
unconfiguredTriggers: async (path) => {
const body = await helpers.getNodeBody(path)
return body ? unconfiguredTriggers(path, body.content) : []
},
setNodeTrigger: async (path, kind, config) => {
deps.ensureEditable?.()
assertInFolder(path)
const body = await helpers.getNodeBody(path)
if (!body) throw new Error(`No pipeline node at '${path}'. Build it first.`)
const prefix = body.language === 'python3' || body.language === 'bash' ? '#' : body.language === 'duckdb' || body.language === 'postgresql' ? '--' : '//'
if (!parsePipelineAnnotations(body.content).nativeTriggers.some((n) => n.kind === kind)) {
throw new Error(
`'${path}' does not declare \`${prefix} on ${kind}\`. Add that annotation line with edit_pipeline_node first, so the trigger is wired to the node.`
)
}
const row = deps
.getResolvedGraph()
.triggers.find(
(t) =>
t.trigger_kind === kind &&
t.runnable_kind === 'script' &&
t.runnable_path === path &&
t.path &&
!t.missing &&
!(t as { draft?: boolean }).draft
)
if (row && 'path' in row) {
throw new Error(
`'${path}' already has a deployed ${kind} trigger at '${row.path}'. Change that one instead of adding a second.`
)
}
const existing = [...deps.getTriggerDrafts().values()].find(
(d) => d.kind === kind && d.config.script_path === path
)
const triggerPath =
(typeof config.path === 'string' && config.path) ||
existing?.config.path ||
defaultTriggerPath(path, kind)
const saved = deps.setTriggerDraft(
{ kind, config: { ...config, path: triggerPath, script_path: path, is_flow: false } },
existing ? triggerDraftKey(kind, existing.config.path) : undefined
)
if (!saved) {
throw new Error(`Another draft trigger already uses the path '${triggerPath}'. Pass a different config.path.`)
}
deps.onShowDrafts?.()
return { path: triggerPath, replaced: !!existing }
},
removeProposedNode: async (path) => {
if (!deps.getDrafts().has(path)) {
throw new Error(`No unsaved draft at '${path}' to discard.`)
@@ -0,0 +1,66 @@
import { describe, expect, it } from 'vitest'
import {
addTriggerDirective,
listenToAsset,
removeTriggerDirective,
stopListeningToAsset
} from './pipelineAnnotationEdits'
const table = { kind: 'datatable' as const, path: 'main/orders' }
const lake = { kind: 'ducklake' as const, path: 'main/orders' }
describe('pipeline annotation edits', () => {
it('adds `on` at the end of the header, in its comment style, leaving the body', () => {
const sql = '-- pipeline\n-- materialize ducklake://main/x\nSELECT 1; -- on nothing'
expect(listenToAsset(sql, table, false)).toBe(
'-- pipeline\n-- materialize ducklake://main/x\n-- on datatable://main/orders\nSELECT 1; -- on nothing'
)
})
it('adds `on` next to the existing `on` lines', () => {
const sql = '-- pipeline\n-- on schedule\n-- materialize ducklake://main/x\nSELECT 1;'
expect(listenToAsset(sql, table, false)).toBe(
'-- pipeline\n-- on schedule\n-- on datatable://main/orders\n-- materialize ducklake://main/x\nSELECT 1;'
)
})
it('removes an `on` line however it is spelled, options included', () => {
const py = '# pipeline\n# on datatable://main/orders debounce=60s\nprint(1)'
expect(stopListeningToAsset(py, table, true)).toBe('# pipeline\nprint(1)')
})
it('unmutes an auto-triggering read rather than adding `on`, and mutes it to stop', () => {
const muted = '// pipeline\n// mute ducklake://main/orders\nexport function main() {}'
expect(listenToAsset(muted, lake, true)).toBe('// pipeline\nexport function main() {}')
expect(stopListeningToAsset('// pipeline\nexport function main() {}', lake, true)).toBe(
'// pipeline\n// mute ducklake://main/orders\nexport function main() {}'
)
})
it('adds `on` for an auto-triggering kind the script does not read', () => {
expect(listenToAsset('-- pipeline\nSELECT 1;', lake, false)).toBe(
'-- pipeline\n-- on ducklake://main/orders\nSELECT 1;'
)
})
it('mutes only an auto-triggering asset the script reads', () => {
const sub = '-- pipeline\n-- on ducklake://main/orders\nSELECT 1;'
expect(stopListeningToAsset(sub, lake, false)).toBe('-- pipeline\nSELECT 1;')
})
it('adds `on` for an auto-triggering kind when every read is muted', () => {
const all = '// pipeline\n// mute all\nexport function main() {}'
expect(listenToAsset(all, lake, true)).toBe(
'// pipeline\n// mute all\n// on ducklake://main/orders\nexport function main() {}'
)
})
it('adds a trigger kind once, and drops it', () => {
expect(addTriggerDirective('// pipeline\n// on webhook\nx()', 'webhook')).toBe(
'// pipeline\n// on webhook\nx()'
)
expect(removeTriggerDirective('// pipeline\n// on kafka\nx()', 'kafka')).toBe(
'// pipeline\nx()'
)
})
})
@@ -0,0 +1,149 @@
import type { AssetKind } from '$lib/gen'
import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations'
// Edits to a pipeline script's leading annotation header (`// on …`, `// mute …`),
// the comment block `parsePipelineAnnotations` reads. Lines are kept in the
// script's own comment style, and nothing past the header is touched.
/** Asset kind → the prefix of its `// on <ref>` spelling. Mirrors ASSET_KINDS in
* backend/parsers/windmill-parser/src/asset_parser.rs. */
export const ASSET_REF_PREFIX: Record<AssetKind, string> = {
s3object: 's3://',
resource: '$res:',
ducklake: 'ducklake://',
datatable: 'datatable://',
volume: 'volume://',
dbt: 'dbt://'
}
// A `// pipeline` script reading one of these kinds runs after each write to it
// unless muted, with no `// on` line (see AUTO_TRIGGER_KINDS in resolveGraph).
const AUTO_TRIGGER_KINDS: ReadonlySet<AssetKind> = new Set(['ducklake', 's3object'])
export function assetRef(asset: { kind: AssetKind; path: string }): string {
return `${ASSET_REF_PREFIX[asset.kind]}${asset.path}`
}
const PREFIXES = ['//', '--', '#']
type HeaderLine = { index: number; prefix: string; inner: string }
/** The header: leading comment lines (blank lines allowed), up to the first code. */
function header(lines: string[]): HeaderLine[] {
const out: HeaderLine[] = []
for (let i = 0; i < lines.length; i++) {
const t = lines[i].trimStart()
if (t === '') continue
const prefix = PREFIXES.find((p) => t.startsWith(p))
if (!prefix) break
out.push({ index: i, prefix, inner: t.slice(prefix.length).trim() })
}
return out
}
// A header line matched the way the parser reads it, so `on <ref> debounce=60s`,
// `res://` vs `$res:` spellings and dbt path quoting all count as the same line.
type LineMatch = (parsed: PipelineAnnotations) => boolean
const onAsset =
(asset: { kind: AssetKind; path: string }): LineMatch =>
(p) =>
p.triggerAssets.some((a) => a.kind === asset.kind && a.path === asset.path)
const muteAsset =
(asset: { kind: AssetKind; path: string }): LineMatch =>
(p) =>
p.muteAssets.some((a) => a.kind === asset.kind && a.path === asset.path)
const muteAll: LineMatch = (p) => p.muteAll
const onTrigger =
(kind: string): LineMatch =>
(p) =>
p.nativeTriggers.some((t) => t.kind === kind)
function matches(line: HeaderLine, match: LineMatch): boolean {
return match(parsePipelineAnnotations(`${line.prefix} ${line.inner}`))
}
function hasDirective(content: string, match: LineMatch): boolean {
return header(content.split('\n')).some((l) => matches(l, match))
}
/** Adds `<prefix> <directive>` in the header's comment style: after the last line
* of the same directive (`on`, `mute`), so they stay grouped, else at its end. */
function addDirective(content: string, directive: string): string {
const lines = content.split('\n')
const h = header(lines)
const keyword = directive.split(/\s/)[0]
const sameKind = h.filter((l) => l.inner.split(/\s/)[0] === keyword)
const after = sameKind[sameKind.length - 1] ?? h[h.length - 1]
const line = `${after?.prefix ?? '//'} ${directive}`
lines.splice(after ? after.index + 1 : 0, 0, line)
return lines.join('\n')
}
function removeDirective(content: string, match: LineMatch): string {
const lines = content.split('\n')
const drop = new Set(
header(lines)
.filter((l) => matches(l, match))
.map((l) => l.index)
)
return lines.filter((_, i) => !drop.has(i)).join('\n')
}
/** The script runs after each write to the asset: `// on <ref>`, and any
* `// mute <ref>` lifted. An auto-triggering kind it `reads` needs no `// on`
* once unmuted, unless `// mute all` still silences it. */
export function listenToAsset(
content: string,
asset: { kind: AssetKind; path: string },
reads: boolean
): string {
const ref = assetRef(asset)
let out = removeDirective(content, muteAsset(asset))
const auto = reads && AUTO_TRIGGER_KINDS.has(asset.kind) && !hasDirective(out, muteAll)
if (!auto && !hasDirective(out, onAsset(asset))) out = addDirective(out, `on ${ref}`)
return out
}
/** The script no longer runs after writes to the asset: its `// on <ref>` goes,
* and an auto-triggering kind it `reads` is muted, since the read alone would
* still trigger it. */
export function stopListeningToAsset(
content: string,
asset: { kind: AssetKind; path: string },
reads: boolean
): string {
const ref = assetRef(asset)
let out = removeDirective(content, onAsset(asset))
if (reads && AUTO_TRIGGER_KINDS.has(asset.kind) && !hasDirective(out, muteAsset(asset))) {
out = addDirective(out, `mute ${ref}`)
}
return out
}
/** Adds a `// on <trigger kind>` line (`schedule`, `webhook`, …) unless it has one. */
export function addTriggerDirective(content: string, kind: string): string {
return hasDirective(content, onTrigger(kind)) ? content : addDirective(content, `on ${kind}`)
}
/** Drops a `// on <trigger kind>` line (`kafka`, `schedule`, …). */
export function removeTriggerDirective(content: string, kind: string): string {
return removeDirective(content, onTrigger(kind))
}
/** What each edit says it changes, for the button that applies it. */
export function listenChangeText(
asset: { kind: AssetKind; path: string },
reads: boolean
): string {
return reads && AUTO_TRIGGER_KINDS.has(asset.kind)
? `Removes any \`mute ${assetRef(asset)}\` line (adds \`on ${assetRef(asset)}\` if all reads are muted)`
: `Adds \`on ${assetRef(asset)}\``
}
export function stopListeningChangeText(
asset: { kind: AssetKind; path: string },
reads: boolean
): string {
return reads && AUTO_TRIGGER_KINDS.has(asset.kind)
? `Removes \`on ${assetRef(asset)}\` and adds \`mute ${assetRef(asset)}\``
: `Removes \`on ${assetRef(asset)}\``
}
@@ -0,0 +1,164 @@
import { untrack } from 'svelte'
import { ScriptService, type AssetKind, type ScriptLang } from '$lib/gen'
import { inferAssets } from '$lib/infer'
import {
extractReads,
extractWrites,
type AssetWithAltAccessType
} from '$lib/components/assets/lib'
import { parsePipelineAnnotations } from './parsePipelineAnnotations'
import type { AssetGraphResponse, NativeTriggerKind } from './types'
import type { PipelineEditorState } from './pipelineEditorState.svelte'
type AssetRef = { kind: AssetKind; path: string }
/**
* Reads the body of every deployed script of a pipeline's graph and infers its
* asset reads/writes and `// on <native trigger>` annotations, so the canvas
* draws the edges of scripts the user never opened. The open script's live
* buffer wins over its cached body.
*
* The caches are only-add and keyed by path: the derived maps iterate the
* current graph's scripts, so a renamed or deleted script drops out when the
* graph refetches. A stale value under a live path does not, so a save through
* the editor must `forget` its path for the sweep to read it again.
*/
export function usePipelineAssetPrefetch(deps: {
getWorkspace: () => string | undefined
getGraph: () => AssetGraphResponse | undefined
editor: PipelineEditorState
}) {
const pe = deps.editor
let bodies = $state<Map<string, string>>(new Map())
// Deployed summary + language from the same fetch, for node titles and icons.
let scriptMeta = $state<Map<string, { summary?: string; language: ScriptLang }>>(new Map())
let inferredAssets = $state<Map<string, AssetWithAltAccessType[]>>(new Map())
let prefetching = $state(false)
// Bumped by every sweep so one started for an older graph stops writing.
let gen = 0
$effect(() => {
const ws = deps.getWorkspace()
const g = deps.getGraph()
if (!ws || !g) return
const sweep = ++gen
const targets = untrack(() =>
g.runnables
.filter((r) => r.usage_kind === 'script')
.map((r) => r.path)
.filter((p) => !pe.drafts.has(p) && !bodies.has(p))
)
if (targets.length === 0) return
let i = 0
const POOL = 6
const worker = async () => {
while (i < targets.length && sweep === gen) {
const path = targets[i++]
try {
const s = await ScriptService.getScriptByPath({ workspace: ws, path })
if (sweep !== gen) return
const content = s.content ?? ''
const res = await inferAssets(s.language, content)
if (sweep !== gen) return
const inferred = (res?.assets ?? []) as AssetWithAltAccessType[]
untrack(() => {
if (!bodies.has(path)) bodies = new Map(bodies).set(path, content)
if (!scriptMeta.has(path)) {
scriptMeta = new Map(scriptMeta).set(path, {
summary: s.summary || undefined,
language: s.language
})
}
if (!inferredAssets.has(path)) {
inferredAssets = new Map(inferredAssets).set(path, inferred)
}
})
} catch {
// Skip — that node just falls back to base-graph edges.
}
}
}
prefetching = true
const pool = Array.from({ length: Math.min(POOL, targets.length) }, () => worker())
void Promise.all(pool).then(() => {
if (sweep === gen) prefetching = false
})
})
// Both edge maps in one pass over the graph's scripts; they only differ by
// extractWrites vs extractReads over the same source.
const edges = $derived.by(() => {
const writes = new Map<string, AssetRef[]>()
const reads = new Map<string, AssetRef[]>()
const g = deps.getGraph()
if (!g) return { writes, reads }
for (const r of g.runnables) {
if (r.usage_kind !== 'script') continue
const assets =
pe.liveBodyAssets.scriptPath === r.path
? pe.liveBodyAssets.assets
: inferredAssets.get(r.path)
if (!assets) continue
const w = extractWrites(assets)
if (w.length > 0) writes.set(r.path, w)
const rd = extractReads(assets)
if (rd.length > 0) reads.set(r.path, rd)
}
return { writes, reads }
})
const annotatedNativeKinds = $derived.by(() => {
const out = new Map<string, Set<NativeTriggerKind>>()
const g = deps.getGraph()
if (!g) return out
const livePath = pe.liveAnnotations.scriptPath
for (const r of g.runnables) {
if (r.usage_kind !== 'script') continue
let kinds: Set<NativeTriggerKind>
if (r.path === livePath) {
kinds = new Set(pe.liveAnnotations.annotations.nativeTriggers.map((n) => n.kind))
} else {
const body = bodies.get(r.path)
if (!body) continue
kinds = new Set(parsePipelineAnnotations(body).nativeTriggers.map((n) => n.kind))
}
if (kinds.size > 0) out.set(r.path, kinds)
}
return out
})
return {
get bodies(): ReadonlyMap<string, string> {
return bodies
},
get scriptMeta(): ReadonlyMap<string, { summary?: string; language: ScriptLang }> {
return scriptMeta
},
get inferredWritesByPath() {
return edges.writes
},
get inferredReadsByPath() {
return edges.reads
},
get annotatedNativeKindsByPath() {
return annotatedNativeKinds
},
get prefetching() {
return prefetching
},
/** Drop cached paths so the next sweep re-reads them (after a save). */
forget(...paths: string[]) {
const nextBodies = new Map(bodies)
const nextMeta = new Map(scriptMeta)
const nextInferred = new Map(inferredAssets)
for (const p of paths) {
nextBodies.delete(p)
nextMeta.delete(p)
nextInferred.delete(p)
}
bodies = nextBodies
scriptMeta = nextMeta
inferredAssets = nextInferred
}
}
}
@@ -0,0 +1,87 @@
import { untrack } from 'svelte'
import type { AssetGraphResponse } from './types'
type Staged = { args: Record<string, any>; valid: boolean }
// An S3Object-shaped value (the file picker writes `{ s3: '<path>' }`).
function isS3Object(v: any): boolean {
return !!v && typeof v === 'object' && !Array.isArray(v) && 's3' in v
}
// Whether a staged value carries actual data. Covers the two shapes a
// data-upload entry takes: an S3Object (file picker → `{ s3: '<path>' }`) and
// a plain required input (e.g. a JSON array pasted into the run form).
function hasMeaningfulValue(v: any): boolean {
if (v == null) return false
if (typeof v === 'string') return v.length > 0
if (Array.isArray(v)) return v.length > 0
if (typeof v === 'object')
return isS3Object(v) ? typeof v.s3 === 'string' && v.s3.length > 0 : Object.keys(v).length > 0
return true // numbers / booleans count as provided
}
/**
* Staged run-form input for the graph's data-upload entry scripts, keyed by path.
* Lifted out of the (transient, per-selection) run form so the uploaded/entered
* data persists across selection changes: it drives each entry node's green
* "ready" state and seeds a run with that input. `valid` is the run form's
* full-schema validity (all required fields satisfied).
*/
export function usePipelineDataUploads(getGraph: () => AssetGraphResponse) {
let staged = $state<Record<string, Staged>>({})
// Scripts that are data-upload entry points (a `data_upload` trigger in the
// displayed graph). They can't auto-run — they need their data first.
const entryPaths = $derived(
new Set(
getGraph()
.triggers.filter((t) => t.trigger_kind === 'data_upload' && t.runnable_kind === 'script')
.map((t) => t.runnable_path)
)
)
// Ready once the user actually provided its data, not just opened the form. Two
// guards: the form's own full-schema `valid` (every required field — including
// non-file ones — is satisfied), AND that any declared S3Object file field
// actually carries a file (the picker can leave an empty `{ s3: '' }` on a
// non-required file field, which `valid` alone wouldn't catch).
function isReady(path: string): boolean {
const s = staged[path]
if (!s || !s.valid) return false
const values = Object.values(s.args)
const s3s = values.filter(isS3Object)
if (s3s.length > 0) return s3s.every(hasMeaningfulValue)
return values.some(hasMeaningfulValue)
}
const readyPaths = $derived(new Set([...entryPaths].filter((p) => isReady(p))))
return {
get entryPaths(): ReadonlySet<string> {
return entryPaths
},
/** Entry scripts with their data provided — drives the green node treatment. */
get readyPaths(): ReadonlySet<string> {
return readyPaths
},
isReady,
argsFor: (path: string): Record<string, any> | undefined => staged[path]?.args,
// Persist the run form's args + validity, but only for data-upload entries —
// other run forms (partitioned producers) run their own way and must not be
// mistaken for a staged upload. Idempotent: the run form re-emits on every
// keystroke/validation pass, so bail when the value is unchanged — otherwise
// each emit would reassign `staged`, giving `readyPaths` a fresh Set identity
// that re-syncs the canvas and re-fires the form's emit effect
// (effect_update_depth_exceeded).
// Called from the run forms' own effects, so its reads are untracked: a form
// must not re-run because the store it just wrote to changed.
stage: (path: string, args: Record<string, any>, valid: boolean) =>
untrack(() => {
if (!entryPaths.has(path)) return
const prev = staged[path]
if (prev && prev.valid === valid && JSON.stringify(prev.args) === JSON.stringify(args))
return
staged = { ...staged, [path]: { args, valid } }
})
}
}
@@ -0,0 +1,52 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { DraftService, ScriptService } from '$lib/gen'
import { planPipelineDelete } from './pipelineDelete'
afterEach(() => vi.restoreAllMocks())
describe('planPipelineDelete', () => {
it("takes only the pipeline's own triggers and addresses draft-only nodes by storage key", async () => {
vi.spyOn(ScriptService, 'listRunnables').mockResolvedValue({
items: [
{ type: 'script', path: 'f/sales/ingest', auto_kind: 'pipeline' },
{ type: 'script', path: 'f/sales/report' },
{
type: 'script',
path: 'u/alice/draft_abc',
draft_only: true,
draft_path: 'f/sales/unpublished',
auto_kind: 'pipeline'
}
]
} as any)
vi.spyOn(DraftService, 'getOwnDraft').mockResolvedValue(undefined as any)
const trigger = (runnable_kind: string, runnable_path: string, path: string) => ({
trigger_kind: 'schedule',
runnable_kind,
runnable_path,
path
})
vi.stubGlobal(
'fetch',
vi.fn().mockResolvedValue({
ok: true,
json: async () => ({
triggers: [
trigger('script', 'f/sales/ingest', 'f/sales/ingest_schedule'),
trigger('script', 'f/sales/report', 'f/sales/report_schedule'),
trigger('flow', 'f/sales/ingest', 'f/sales/flow_schedule')
]
})
})
)
const plan = await planPipelineDelete('w', 'sales')
expect(plan.triggers).toEqual([{ kind: 'schedule', path: 'f/sales/ingest_schedule' }])
expect(plan.scripts).toEqual([
{ path: 'f/sales/ingest', displayPath: 'f/sales/ingest', draftOnly: false },
{ path: 'u/alice/draft_abc', displayPath: 'f/sales/unpublished', draftOnly: true }
])
vi.unstubAllGlobals()
})
})
@@ -0,0 +1,135 @@
import { DraftService, OpenAPI, ScriptService, type RunnableItem } from '$lib/gen'
import { discardDraft } from '$lib/utils_draft_deploy'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath, pipelineLocalMirrorKey } from '$lib/pipelinePaths'
import { deleteTriggerRow } from './pipelineTriggerDraftDeploy'
import type { AssetGraphResponse, NativeTriggerKind } from './types'
export type PipelineDeleteResult = { removed: number; failures: string[] }
/** The folder's pipeline-member scripts, deployed and draft-only alike: every one of
* them makes the home list show the pipeline, so one left behind brings it back. */
async function listPipelineScripts(workspace: string, folder: string): Promise<RunnableItem[]> {
const out: RunnableItem[] = []
let cursor: string | undefined
do {
const res = await ScriptService.listRunnables({
workspace,
kinds: 'script',
pathStart: `f/${folder}/`,
includeDraftOnly: true,
perPage: 100,
cursor
})
out.push(...res.items.filter((it) => it.auto_kind === 'pipeline'))
cursor = res.next_cursor ?? undefined
} while (cursor)
return out
}
/** Trigger rows starting one of `scripts`. The folder graph also carries the
* triggers of the folder's ordinary scripts and flows, which are not the pipeline's. */
async function listNativeTriggers(workspace: string, folder: string, scripts: ReadonlySet<string>) {
const res = await fetch(
`${OpenAPI.BASE ?? ''}/w/${workspace}/assets/graph?${new URLSearchParams({ folder })}`,
{ credentials: 'include' }
)
if (!res.ok) throw new Error(`GET /assets/graph → ${res.status}`)
const graph = (await res.json()) as AssetGraphResponse
return graph.triggers.flatMap((t) =>
t.trigger_kind !== 'asset' &&
t.runnable_kind === 'script' &&
scripts.has(t.runnable_path) &&
t.trigger_kind !== 'webhook' &&
t.trigger_kind !== 'data_upload' &&
t.path &&
!t.unsaved
? [{ kind: t.trigger_kind, path: t.path }]
: []
)
}
export type PipelineDeletePlan = {
/** `path` is what the API addresses (a draft-only node's storage key);
* `displayPath` is the path the user typed for it. */
scripts: { path: string; displayPath: string; draftOnly: boolean }[]
triggers: { kind: NativeTriggerKind; path: string }[]
/** Whether the caller has an unsaved pipeline draft for this folder. */
hasDraft: boolean
}
/** Everything `deletePipeline` would remove, so it can be listed before it is. */
export async function planPipelineDelete(
workspace: string,
folder: string
): Promise<PipelineDeletePlan> {
const scripts = await listPipelineScripts(workspace, folder)
const deployed = new Set(scripts.filter((s) => !s.draft_only).map((s) => s.path))
const [triggers, draft] = await Promise.all([
listNativeTriggers(workspace, folder, deployed),
DraftService.getOwnDraft({
workspace,
kind: PIPELINE_DRAFT_KIND,
path: pipelineBundlePath(folder)
}).catch(() => undefined)
])
return {
scripts: scripts.map((s) => ({
path: s.path,
displayPath: (s.draft_only && s.draft_path) || s.path,
draftOnly: !!s.draft_only
})),
triggers,
hasDraft: !!draft
}
}
/** Removes what `plan` lists: the trigger rows, the scripts (deleted when
* `hardDelete`, which the backend allows admins only, archived otherwise; a
* draft-only node is discarded) and the caller's pipeline draft. The data the
* nodes wrote (DuckLake tables, S3 objects) is left alone. Triggers go first: a
* script refused afterwards loses its trigger, but no trigger outlives the script
* it starts. */
export async function deletePipeline(
workspace: string,
folder: string,
plan: PipelineDeletePlan,
hardDelete: boolean
): Promise<PipelineDeleteResult> {
const failures: string[] = []
let removed = 0
const fail = (what: string, e: any) =>
failures.push(`${what}: ${e?.body ?? e?.message ?? String(e)}`)
for (const t of plan.triggers) {
try {
if (await deleteTriggerRow(t.kind, t.path, workspace)) removed++
} catch (e) {
fail(`${t.kind} trigger ${t.path}`, e)
}
}
for (const s of plan.scripts) {
try {
if (s.draftOnly) {
const r = await discardDraft('script', s.path, workspace, true, false, false)
if (!r.success) throw new Error(r.error)
} else if (hardDelete) {
await ScriptService.deleteScriptByPath({ workspace, path: s.path })
} else {
await ScriptService.archiveScriptByPath({ workspace, path: s.path })
}
removed++
} catch (e) {
fail(s.displayPath, e)
}
}
if (plan.hasDraft) {
const bundle = await discardDraft(PIPELINE_DRAFT_KIND, pipelineBundlePath(folder), workspace, true)
if (!bundle.success) failures.push(`pipeline draft: ${bundle.error}`)
}
try {
localStorage.removeItem(pipelineLocalMirrorKey(folder))
} catch {
// Storage unavailable: nothing was mirrored either.
}
return { removed, failures }
}
@@ -0,0 +1,230 @@
import { DraftService, ScriptService, type Script } from '$lib/gen'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
import type { PipelineTriggerDraft } from './types'
import type { Schema } from '$lib/common'
import { emptySchema } from '$lib/utils'
import { inferArgs, inferAssets } from '$lib/infer'
import type { AssetWithAltAccessType } from '$lib/components/assets/lib'
import type { PipelineDraft } from './pipelineAiHelpers'
import type { PipelineEditorState } from './pipelineEditorState.svelte'
import { deployTriggerDraft } from './pipelineTriggerDraftDeploy'
// Deploying a pipeline's drafts: the scripts first, then the trigger drafts whose
// script is now deployed. Shared by the pipeline page's "Save all" and the AI
// session's "Deploy pipeline", so both deploy exactly the same way.
/** Deploy one pipeline script draft; resolves to the new version's hash. */
export async function deployPipelineScript(
draft: PipelineDraft,
workspace: string
): Promise<string> {
const script = structuredClone($state.snapshot(draft.script) as Script)
script.schema = script.schema ?? emptySchema()
try {
const result = await inferArgs(script.language, script.content, script.schema as Schema)
;(script as any).auto_kind = result?.auto_kind || undefined
script.has_preprocessor = result?.has_preprocessor ?? false
} catch {
// Inference failures don't block deploys (the same fallback the per-pane
// save uses): createScript is the real validation gate.
}
// Re-infer the lineage from the CURRENT body. `script.assets` is a snapshot
// that isn't refreshed on edit, so a renamed/removed output would otherwise be
// re-deployed as a phantom write edge.
let assets: AssetWithAltAccessType[] = []
try {
const inferred = await inferAssets(script.language, script.content)
if (inferred?.status !== 'error') assets = (inferred?.assets ?? []) as AssetWithAltAccessType[]
} catch {
// An unparsable body deploys with no lineage rather than the stale snapshot.
}
return await ScriptService.createScript({
workspace,
requestBody: {
...script,
language: script.language,
description: script.description ?? '',
// A brand-new draft has an empty hash (the backend rejects an empty hex
// string); a draft of a deployed script chains off its hash, since
// createScript rejects a parentless deploy to an occupied path.
parent_hash: script.hash ? String(script.hash) : undefined,
is_template: false,
tag: script.tag,
kind: script.kind as Script['kind'] | undefined,
lock: undefined,
assets: assets as any
}
})
}
// Every field a deploy sends: a draft differing from the live script in any of
// them still has something to deploy.
const DEPLOYED_FIELDS = [
'content',
'language',
'summary',
'description',
'tag',
'kind',
'labels'
] as const
/** Whether two versions of a script agree on everything a deploy sends. */
function sameDeployedFields(a: Partial<Script>, b: Partial<Script>): boolean {
const norm = (v: unknown) => JSON.stringify(Array.isArray(v) && v.length === 0 ? null : (v ?? null))
return DEPLOYED_FIELDS.every(
(f) =>
norm(f === 'description' ? (a[f] ?? '') : a[f]) ===
norm(f === 'description' ? (b[f] ?? '') : b[f])
)
}
/** Whether a draft equals the live script at its path, i.e. has nothing to deploy. */
export async function matchesDeployedScript(
path: string,
d: PipelineDraft,
workspace: string
): Promise<boolean> {
try {
const live = await ScriptService.getScriptByPath({ workspace, path })
return sameDeployedFields(live, d.script)
} catch {
return false
}
}
/**
* A folder's draft bundle against what is deployed, one entry per node (and the
* trigger drafts), for a diff viewer: the unit a session reviews and deploys.
*/
export async function loadPipelineDraftDiff(
workspace: string,
folder: string
): Promise<{ before: unknown; after: unknown }> {
const row = await DraftService.getOwnDraft({
workspace,
kind: PIPELINE_DRAFT_KIND,
path: pipelineBundlePath(folder)
}).catch(() => undefined)
const bundle = (row?.value ?? {}) as {
drafts?: Array<[string, PipelineDraft]>
triggerDrafts?: PipelineTriggerDraft[]
}
const drafts = Array.isArray(bundle.drafts) ? bundle.drafts : []
const before: Record<string, string> = {}
const after: Record<string, string> = {}
await Promise.all(
drafts.map(async ([path, d]) => {
after[path] = d.script?.content ?? ''
const live = await ScriptService.getScriptByPath({ workspace, path }).catch(() => undefined)
if (live) before[path] = live.content
})
)
const triggers = (bundle.triggerDrafts ?? []).map((t) => ({
kind: t.kind,
path: t.config.path,
script_path: t.config.script_path
}))
return {
before: { nodes: before },
after: { nodes: after, ...(triggers.length > 0 ? { new_triggers: triggers } : {}) }
}
}
export type PipelineDeployOutcome = {
savedPaths: string[]
/** Keys of the trigger drafts that deployed. */
savedTriggers: string[]
/** Failures by script or trigger path. */
errors: Map<string, string>
}
/**
* Deploy every draft of `editor` into `workspace`, dropping from it what deployed:
* failed drafts stay so they can be fixed and retried. The open pane's unsaved
* keystrokes are deployed with their draft.
*/
export async function deployPipelineDrafts(
editor: PipelineEditorState,
workspace: string
): Promise<PipelineDeployOutcome> {
// Closing the pane lands an open deployed script's edits as its draft, so they
// deploy too. It reopens on that script once the deploy is done, unless another
// one was opened meanwhile.
const reopen =
editor.liveEditPath != undefined || editor.activeDraftPath != undefined
? await editor.closePane()
: undefined
const entries = [...editor.drafts.entries()]
const errors = new Map<string, string>()
const savedPaths: string[] = []
const hashes = new Map<string, string>()
// Parallel: every createScript is independent, and one bad body must not block
// the others.
const results = await Promise.allSettled(
entries.map(([, d]) => deployPipelineScript(d, workspace))
)
for (let i = 0; i < results.length; i++) {
const r = results[i]
const [path, d] = entries[i]
if (r.status === 'fulfilled') {
savedPaths.push(path)
if (r.value) hashes.set(path, r.value)
continue
}
// A refused draft that equals the deployed script has nothing left to deploy.
// A duplicate refusal alone does not prove it: it is checked against every
// past version, not just the live one.
if (await matchesDeployedScript(path, d, workspace)) {
savedPaths.push(path)
continue
}
const msg = String((r.reason as any)?.body ?? (r.reason as any)?.message ?? r.reason)
errors.set(
path,
/same hash/i.test(msg)
? 'This is the content of an earlier version, which cannot be deployed again as is. Change anything in it (a comment will do) to deploy it.'
: msg
)
}
// A trigger's `script_path` must already exist, so triggers whose script is
// still an undeployed draft stay drafts.
const ready = [...editor.triggerDrafts].filter(
([, d]) => !editor.drafts.has(d.config.script_path) || savedPaths.includes(d.config.script_path)
)
const triggerResults = await Promise.all(ready.map(([, d]) => deployTriggerDraft(d, workspace)))
const savedTriggers: string[] = []
triggerResults.forEach((ok, i) => {
const [key, d] = ready[i]
if (ok) savedTriggers.push(key)
else errors.set(d.config.path, `Could not create the ${d.kind} trigger, see the notification`)
})
for (const key of savedTriggers) editor.discardTriggerDraft(key)
if (savedPaths.length > 0) {
// A draft edited again while it deployed (its node reopened from the canvas)
// keeps the newer edits, now on top of the version just deployed; only a draft
// still holding exactly what was sent is done.
const sent = new Map(entries.map(([p, d]) => [p, d.script]))
const next = new Map<string, PipelineDraft>()
for (const [k, d] of editor.drafts) {
if (!savedPaths.includes(k)) {
next.set(k, d)
continue
}
const open = editor.liveContent.scriptPath === k ? editor.liveContent.content : undefined
const now = open === undefined ? d.script : { ...d.script, content: open }
const was = sent.get(k)
if (was && sameDeployedFields(now, was)) continue
const hash = hashes.get(k)
next.set(k, hash ? { ...d, script: { ...d.script, hash } } : d)
}
editor.drafts = next
}
if (reopen && editor.activeDraftPath === undefined && editor.selection === undefined) {
if (editor.drafts.has(reopen)) editor.activeDraftPath = reopen
else editor.selection = { kind: 'runnable', runnable_kind: 'script', path: reopen }
}
return { savedPaths, savedTriggers, errors }
}
@@ -0,0 +1,56 @@
import { describe, expect, it, vi } from 'vitest'
const createScript = vi.fn()
vi.mock('$lib/gen', () => ({
ScriptService: { createScript: (a: unknown) => createScript(a) },
DraftService: {}
}))
vi.mock('$lib/infer', () => ({
inferArgs: async () => undefined,
inferAssets: async () => ({ status: 'ok', assets: [] })
}))
vi.mock('./pipelineTriggerDraftDeploy', () => ({ deployTriggerDraft: async () => true }))
const { deployPipelineDrafts } = await import('./pipelineDeploy.svelte')
const { PipelineEditorState } = await import('./pipelineEditorState.svelte')
const draft = (content: string) => ({
localId: 'pe-1',
script: { path: 'f/x/n', language: 'duckdb', content, hash: 'aaaa' } as any
})
describe('deployPipelineDrafts', () => {
it('drops a draft once what it holds is deployed', async () => {
const pe = new PipelineEditorState()
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
createScript.mockResolvedValueOnce('bbbb')
await deployPipelineDrafts(pe, 'ws')
expect(pe.drafts.has('f/x/n')).toBe(false)
})
it('keeps edits made while the deploy was in flight, on top of the new version', async () => {
const pe = new PipelineEditorState()
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
createScript.mockImplementationOnce(async () => {
pe.drafts = new Map([['f/x/n', draft('SELECT 3')]])
return 'bbbb'
})
await deployPipelineDrafts(pe, 'ws')
expect(pe.drafts.get('f/x/n')?.script.content).toBe('SELECT 3')
expect(pe.drafts.get('f/x/n')?.script.hash).toBe('bbbb')
})
})
describe('deployPipelineDrafts metadata', () => {
it('keeps a summary edited while the deploy was in flight', async () => {
const pe = new PipelineEditorState()
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
createScript.mockImplementationOnce(async () => {
const d = draft('SELECT 2')
pe.drafts = new Map([['f/x/n', { ...d, script: { ...d.script, summary: 'Renamed' } }]])
return 'bbbb'
})
await deployPipelineDrafts(pe, 'ws')
expect(pe.drafts.get('f/x/n')?.script.summary).toBe('Renamed')
})
})
@@ -1,6 +1,8 @@
import { tick } from 'svelte'
import type { AssetKind, Script } from '$lib/gen'
import type { AssetWithAltAccessType } from '$lib/components/assets/lib'
import type { AssetGraphSelection } from './types'
import type { AssetGraphSelection, PipelineTriggerDraft } from './types'
import { defaultTriggerPath, triggerDraftKey } from './pipelineTriggerDrafts'
import {
parsePipelineAnnotations,
type ColumnLineage,
@@ -26,11 +28,18 @@ type LiveBodyAssets = {
assets: AssetWithAltAccessType[]
columnLineage?: ColumnLineage[]
}
type LiveContent = { scriptPath: string | undefined; content: string }
type LiveContent = {
scriptPath: string | undefined
content: string
/** The deployed script the buffer was opened from; unset for a draft. */
base?: Script
}
export class PipelineEditorState {
/** In-flight drafts keyed by script path (manual + AI-staged). */
drafts = $state<Map<string, PipelineDraft>>(new Map())
/** Undeployed triggers keyed by `triggerDraftKey`, persisted in the same bundle. */
triggerDrafts = $state<Map<string, PipelineTriggerDraft>>(new Map())
/** Draft open in the details pane (mutually exclusive with `selection`). */
activeDraftPath = $state<string | undefined>(undefined)
/** The persisted node/asset selected on the canvas. */
@@ -51,23 +60,22 @@ export class PipelineEditorState {
* autosave hydrate when persistence is enabled. */
loadedFromDbDraft = $state(false)
/** Folder this state is scoped to. Used by the in-session preview (where one
* instance is reused across editor hide/show) to detect a retarget to a
* different folder and reset, so stale drafts don't bleed across folders. */
/** Folder this state is scoped to. The route page compares it on a folder
* switch and resets, so one folder's drafts don't bleed into the next. */
folder = $state<string | undefined>(undefined)
/** True once the DB draft bundle for the current folder has been hydrated
* into this instance. Gated per-instance (not per component mount) so the
* in-session preview hydrates ONCE when its runtime is fresh and then keeps
* the in-memory drafts across editor hide/show — re-reading the DB on every
* remount would race a not-yet-flushed autosave and drop a just-staged draft.
* Reset to false on a folder retarget so the new folder re-hydrates. */
* into this instance. Gated per-instance (not per component mount) so an AI
* session's folder hydrates ONCE and then keeps its in-memory drafts across
* editor remounts — re-reading the DB on every remount would race a
* not-yet-flushed autosave and drop a just-staged draft. `reset` clears it
* so a new folder re-hydrates. */
hydratedFromDb = $state(false)
/** Clear all in-flight state. Used when the session preview retargets a
* different pipeline folder (a same-folder remount keeps the drafts). */
/** Clear all in-flight state, for the route page switching folders. */
reset = () => {
this.drafts = new Map()
this.triggerDrafts = new Map()
this.activeDraftPath = undefined
this.selection = undefined
this.clearLiveOverlays()
@@ -84,6 +92,18 @@ export class PipelineEditorState {
return `pe-${this.#nextDraftLocalId}`
}
// A deployed script with unsaved edits is autosaved as a draft before the pane
// promotes it, so both must give it the same id.
#promotedLocalIds = new Map<string, string>()
promotedDraftLocalId = (path: string): string => {
let id = this.#promotedLocalIds.get(path)
if (!id) {
id = this.newDraftLocalId()
this.#promotedLocalIds.set(path, id)
}
return id
}
handleAnnotationsChange = (scriptPath: string | undefined, annotations: PipelineAnnotations) => {
this.liveAnnotations = { scriptPath, annotations }
}
@@ -94,8 +114,34 @@ export class PipelineEditorState {
) => {
this.liveBodyAssets = { scriptPath, assets, columnLineage }
}
handleContentChange = (scriptPath: string | undefined, content: string) => {
this.liveContent = { scriptPath, content }
handleContentChange = (scriptPath: string | undefined, content: string, base?: Script) => {
this.liveContent = { scriptPath, content, base }
}
/** The open deployed script's unsaved edits, which live only in `liveContent`
* until its pane closes: the path, or undefined when there are none. */
get liveEditPath(): string | undefined {
const { scriptPath, content, base } = this.liveContent
if (scriptPath == undefined || !base || this.drafts.has(scriptPath)) return undefined
return content !== (base.content ?? '') ? scriptPath : undefined
}
/** Closes the open pane and waits for its edits to land as a draft, as closing
* it by hand does: what is deployed or discarded next then includes them, and
* no pane is left holding a stale copy. Returns the path the pane was on. */
closePane = async (): Promise<string | undefined> => {
const path =
this.activeDraftPath ?? (this.selection?.kind === 'runnable' ? this.selection.path : undefined)
if (this.activeDraftPath === undefined && this.selection === undefined) return undefined
this.activeDraftPath = undefined
this.selection = undefined
// The pane unmounts on the next flush; its save-back commits a microtask later.
await tick()
await new Promise<void>((resolve) => queueMicrotask(resolve))
// Its buffer now lives in the draft; left here, autosave would serialize it
// again once that draft is deployed or discarded.
this.clearLiveOverlays()
return path
}
clearLiveOverlays = () => {
@@ -125,6 +171,72 @@ export class PipelineEditorState {
next.delete(path)
this.drafts = next
this.forgetPath(path)
this.discardTriggerDraftsFor(path)
}
/** Summary and labels of a script draft, edited from the pane header. */
setDraftMeta = (path: string, meta: { summary: string; labels: string[] | undefined }) => {
const d = this.drafts.get(path)
if (!d) return
this.drafts = new Map(this.drafts).set(path, {
...d,
script: { ...d.script, summary: meta.summary, labels: meta.labels }
})
}
/** Add or replace a trigger draft; `previousKey` is the entry being edited, which
* a rename drops. False, and nothing changes, when the path is another draft's:
* both would deploy to one trigger. */
setTriggerDraft = (d: PipelineTriggerDraft, previousKey?: string): boolean => {
const key = triggerDraftKey(d.kind, d.config.path)
if (key !== previousKey && this.triggerDrafts.has(key)) return false
const next = new Map(this.triggerDrafts)
if (previousKey) next.delete(previousKey)
next.set(key, d)
this.triggerDrafts = next
return true
}
discardTriggerDraft = (key: string) => {
if (!this.triggerDrafts.has(key)) return
const next = new Map(this.triggerDrafts)
next.delete(key)
this.triggerDrafts = next
}
/** A trigger draft cannot outlive the script draft it was configured with:
* deploying it would target a path nothing is deployed at. */
discardTriggerDraftsFor = (scriptPath: string) => {
const next = new Map(
[...this.triggerDrafts].filter(([, d]) => d.config.script_path !== scriptPath)
)
if (next.size !== this.triggerDrafts.size) this.triggerDrafts = next
}
/** Follow a script draft rename, keeping default `<script>_<kind>` names in step. */
/** Returns the trigger path another draft already holds, and changes nothing,
* when a renamed default name would land on it. */
retargetTriggerDrafts = (oldScriptPath: string, newScriptPath: string): string | undefined => {
const moved = new Map<string, PipelineTriggerDraft>()
const kept = new Map<string, PipelineTriggerDraft>()
for (const [k, d] of this.triggerDrafts) {
if (d.config.script_path !== oldScriptPath) {
kept.set(k, d)
continue
}
const path =
d.config.path === defaultTriggerPath(oldScriptPath, d.kind)
? defaultTriggerPath(newScriptPath, d.kind)
: d.config.path
moved.set(triggerDraftKey(d.kind, path), {
...d,
config: { ...d.config, path, script_path: newScriptPath }
})
}
if (moved.size === 0) return undefined
for (const [k, d] of moved) if (kept.has(k)) return d.config.path
this.triggerDrafts = new Map([...kept, ...moved])
return undefined
}
/** Commit body edits + inferred outputs back into the drafts Map on pane
@@ -147,7 +259,7 @@ export class PipelineEditorState {
if (!snapshot.script) return
const next = new Map(this.drafts)
next.set(p, {
localId: this.newDraftLocalId(),
localId: this.promotedDraftLocalId(p),
script: snapshot.script,
outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined,
inputAssets: snapshot.reads
@@ -104,3 +104,72 @@ describe('PipelineEditorState.handleDraftPersist — read capture', () => {
expect(pe.drafts.get('f/x/n')?.inputAssets).toBeUndefined()
})
})
describe('PipelineEditorState trigger drafts', () => {
it('follow their script draft: renamed with it, discarded with it', () => {
const pe = new PipelineEditorState()
pe.drafts = new Map([['f/x/n', draft('SELECT 1')]])
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/n_kafka', script_path: 'f/x/n' } })
pe.setTriggerDraft({ kind: 'schedule', config: { path: 'f/x/nightly', script_path: 'f/x/n' } })
pe.retargetTriggerDrafts('f/x/n', 'f/x/m')
// A default `<script>_<kind>` name follows the script; a chosen one stays.
expect([...pe.triggerDrafts.keys()].sort()).toEqual([
'kafka:f/x/m_kafka',
'schedule:f/x/nightly'
])
expect([...pe.triggerDrafts.values()].every((d) => d.config.script_path === 'f/x/m')).toBe(true)
pe.drafts = new Map([['f/x/m', draft('SELECT 1')]])
pe.discardDraft('f/x/m')
expect(pe.triggerDrafts.size).toBe(0)
})
it('refuse a path another draft holds, but let a draft keep its own', () => {
const pe = new PipelineEditorState()
const a = { kind: 'kafka' as const, config: { path: 'f/x/k', script_path: 'f/x/a' } }
expect(pe.setTriggerDraft(a)).toBe(true)
expect(
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/k', script_path: 'f/x/b' } })
).toBe(false)
expect(pe.triggerDrafts.get('kafka:f/x/k')?.config.script_path).toBe('f/x/a')
expect(pe.setTriggerDraft({ ...a, config: { ...a.config, topics: ['t'] } }, 'kafka:f/x/k')).toBe(
true
)
})
it('refuse a script rename that moves a default name onto another draft', () => {
const pe = new PipelineEditorState()
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/a_kafka', script_path: 'f/x/a' } })
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/b_kafka', script_path: 'f/x/c' } })
expect(pe.retargetTriggerDrafts('f/x/a', 'f/x/b')).toBe('f/x/b_kafka')
expect([...pe.triggerDrafts.keys()].sort()).toEqual(['kafka:f/x/a_kafka', 'kafka:f/x/b_kafka'])
})
})
describe('PipelineEditorState.liveEditPath', () => {
const base = { path: 'f/x/n', language: 'duckdb', content: 'SELECT 1' } as PipelineDraft['script']
it('counts an open deployed script with unsaved edits, until it has a draft', () => {
const pe = new PipelineEditorState()
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 1', base }
expect(pe.liveEditPath).toBeUndefined()
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 2', base }
expect(pe.liveEditPath).toBe('f/x/n')
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
expect(pe.liveEditPath).toBeUndefined()
})
})
describe('PipelineEditorState.closePane', () => {
it("leaves no open buffer behind for autosave to bring back once the draft is gone", async () => {
const pe = new PipelineEditorState()
const base = { path: 'f/x/n', language: 'duckdb', content: 'SELECT 1' } as PipelineDraft['script']
pe.selection = { kind: 'runnable', runnable_kind: 'script', path: 'f/x/n' }
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 2', base }
expect(await pe.closePane()).toBe('f/x/n')
expect(pe.selection).toBeUndefined()
expect(pe.liveEditPath).toBeUndefined()
expect(pe.liveContent.scriptPath).toBeUndefined()
})
})
@@ -0,0 +1,89 @@
import { usedTriggerKinds } from '$lib/stores'
import { formatCron } from '$lib/utils'
import { saveScheduleFromCfg } from '$lib/components/flows/scheduleUtils'
import { saveEmailTriggerFromCfg } from '$lib/components/triggers/email/utils'
import { saveKafkaTriggerFromCfg } from '$lib/components/triggers/kafka/utils'
import { saveMqttTriggerFromCfg } from '$lib/components/triggers/mqtt/utils'
import { saveAmqpTriggerFromCfg } from '$lib/components/triggers/amqp/utils'
import { saveNatsTriggerFromCfg } from '$lib/components/triggers/nats/utils'
import { savePostgresTriggerFromCfg } from '$lib/components/triggers/postgres/utils'
import { saveSqsTriggerFromCfg } from '$lib/components/triggers/sqs/utils'
import { saveGcpTriggerFromCfg } from '$lib/components/triggers/gcp/utils'
import {
AmqpTriggerService,
EmailTriggerService,
GcpTriggerService,
KafkaTriggerService,
MqttTriggerService,
NatsTriggerService,
PostgresTriggerService,
ScheduleService,
SqsTriggerService
} from '$lib/gen'
import type { NativeTriggerKind, PipelineTriggerDraft } from './types'
/** Creates the trigger through the same helper its editor saves with; that helper
* reports the outcome as a toast and resolves to whether it succeeded. */
export function deployTriggerDraft(d: PipelineTriggerDraft, workspace: string): Promise<boolean> {
const cfg = d.config
switch (d.kind) {
case 'schedule':
return saveScheduleFromCfg({ ...cfg, schedule: formatCron(cfg.schedule) }, false, workspace)
case 'email':
return saveEmailTriggerFromCfg('', cfg, false, workspace, false, usedTriggerKinds)
case 'kafka':
return saveKafkaTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'mqtt':
return saveMqttTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'amqp':
return saveAmqpTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'nats':
return saveNatsTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'postgres':
return savePostgresTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'sqs':
return saveSqsTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
case 'gcp':
return saveGcpTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
}
}
/** Deletes a deployed trigger row. False for the kinds that have none (webhook,
* data upload). */
export async function deleteTriggerRow(
kind: NativeTriggerKind,
path: string,
workspace: string
): Promise<boolean> {
switch (kind) {
case 'schedule':
await ScheduleService.deleteSchedule({ workspace, path })
return true
case 'kafka':
await KafkaTriggerService.deleteKafkaTrigger({ workspace, path })
return true
case 'mqtt':
await MqttTriggerService.deleteMqttTrigger({ workspace, path })
return true
case 'amqp':
await AmqpTriggerService.deleteAmqpTrigger({ workspace, path })
return true
case 'nats':
await NatsTriggerService.deleteNatsTrigger({ workspace, path })
return true
case 'postgres':
await PostgresTriggerService.deletePostgresTrigger({ workspace, path })
return true
case 'sqs':
await SqsTriggerService.deleteSqsTrigger({ workspace, path })
return true
case 'gcp':
await GcpTriggerService.deleteGcpTrigger({ workspace, path })
return true
case 'email':
await EmailTriggerService.deleteEmailTrigger({ workspace, path })
return true
default:
return false
}
}
@@ -0,0 +1,26 @@
import type { PipelineTriggerDraftKind } from './types'
export const DRAFTABLE_TRIGGER_KINDS: readonly PipelineTriggerDraftKind[] = [
'schedule',
'email',
'kafka',
'mqtt',
'amqp',
'nats',
'postgres',
'sqs',
'gcp'
]
export function isDraftableTriggerKind(kind: string): kind is PipelineTriggerDraftKind {
return (DRAFTABLE_TRIGGER_KINDS as readonly string[]).includes(kind)
}
/** Drafts are keyed by kind too: two kinds may share a trigger path. */
export function triggerDraftKey(kind: string, path: string): string {
return `${kind}:${path}`
}
export function defaultTriggerPath(scriptPath: string, kind: PipelineTriggerDraftKind): string {
return `${scriptPath}_${kind}`
}
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest'
import {
resolveGraph,
computeMutedReadKeys,
pipelineNodeErrors,
dbtAssociations,
type ResolveGraphInput
} from './resolveGraph'
@@ -492,6 +493,49 @@ describe('resolveGraph', () => {
expect(r.edges.some((e) => e.asset_path === 'main/out' && e.access_type === 'w')).toBe(true)
})
it('drops a persisted asset the open script renamed away, unless something still uses it', () => {
const write = (runnable_path: string, asset_path: string) => ({
runnable_path,
runnable_kind: 'script' as const,
asset_kind: 's3object' as const,
asset_path,
access_type: 'w' as const
})
const base = baseGraph({
assets: [
{ kind: 's3object', path: '/old.json' },
{ kind: 's3object', path: '/shared.json' },
{ kind: 's3object', path: '/unreferenced.json' }
],
runnables: [
{ path: 'f/x/prod', usage_kind: 'script' },
{ path: 'f/x/other', usage_kind: 'script' }
],
edges: [write('f/x/prod', '/old.json'), write('f/x/prod', '/shared.json')],
triggers: [
{
trigger_kind: 'asset',
asset_kind: 's3object',
asset_path: '/shared.json',
runnable_kind: 'script',
runnable_path: 'f/x/other'
}
]
})
const r = resolveGraph(
input({
base,
liveAnnotations: { scriptPath: 'f/x/prod', annotations: ann() },
liveBodyAssets: { scriptPath: 'f/x/prod', assets: [s3('/new.json', 'w')] }
})
)
expect(r.assets.map((a) => a.path).sort()).toEqual([
'/new.json',
'/shared.json',
'/unreferenced.json'
])
})
it('editing a saved scd2 producer keeps both the base and _current persisted write edges', () => {
// Deploy persists a write to both `main/dim` and `main/dim_current`.
// Opening the producer for editing must not judge the companion `_current`
@@ -656,6 +700,31 @@ describe('resolveGraph', () => {
})
})
it('a trigger draft stands in for its draft script’s missing trigger of that kind', () => {
const drafts = new Map([['f/x/new', { script: { content: '// on schedule\n// on kafka' } }]])
const r = resolveGraph(
input({
drafts,
triggerDrafts: [
{ kind: 'schedule', config: { path: 'f/x/new_schedule', script_path: 'f/x/new' } }
]
})
)
expect(r.triggers.filter((t) => t.trigger_kind === 'kafka')).toEqual([
expect.objectContaining({ runnable_path: 'f/x/new', missing: true })
])
expect(r.triggers.filter((t) => t.trigger_kind === 'schedule')).toEqual([
{
trigger_kind: 'schedule',
path: 'f/x/new_schedule',
runnable_kind: 'script',
runnable_path: 'f/x/new',
unsaved: true,
draft: true
}
])
})
it('macro edges: base passes through; live `// use` adds an unsaved via_use edge', () => {
const base = baseGraph({
runnables: [
@@ -884,11 +953,11 @@ describe('live buffer overlays (open script)', () => {
access_type: 'w',
unsaved: true
})
// …the stale write edge is dropped, but the deployed dataset node stays.
// …the stale write edge is dropped, and with nothing else using it, so is the node.
expect(
r.edges.filter((e) => e.asset_path === 'main/orders' && e.runnable_path === 'f/x/prod')
).toEqual([])
expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/orders' })
expect(r.assets).not.toContainEqual({ kind: 'ducklake', path: 'main/orders' })
})
it('open saved script: unchanged materialize target dedups against the persisted write edge', () => {
@@ -1083,9 +1152,7 @@ describe('dbtAssociations', () => {
it('counts only what the project materializes as its transforms', () => {
const { writesByOwner } = dbtAssociations(runnables, edges)
// Hovering the project highlights the models it builds, not its inputs.
expect([...(writesByOwner.get('script:f/a/dbtproj') ?? [])]).toEqual([
'asset:dbt:wh/s/model_a'
])
expect([...(writesByOwner.get('script:f/a/dbtproj') ?? [])]).toEqual(['asset:dbt:wh/s/model_a'])
})
it('ignores runnables that are not dbt', () => {
@@ -1162,8 +1229,160 @@ describe('dbtAssociations with two writers of one relation', () => {
writeBy('f/a/downstream')
])
expect(writesByOwner.get('script:f/a/upstream')).toEqual(new Set(['asset:dbt:wh/s/shared']))
expect(writesByOwner.get('script:f/a/downstream')).toEqual(
new Set(['asset:dbt:wh/s/shared'])
expect(writesByOwner.get('script:f/a/downstream')).toEqual(new Set(['asset:dbt:wh/s/shared']))
})
})
describe('pipelineNodeErrors', () => {
const edge = (runnable_path: string, asset_path: string, access_type: 'r' | 'w') => ({
runnable_path,
runnable_kind: 'script' as const,
asset_kind: 'ducklake' as const,
asset_path,
access_type
})
const graph = baseGraph({
assets: [
{ kind: 'ducklake', path: 'main/orphan' },
{ kind: 'ducklake', path: 'main/produced' },
{ kind: 'ducklake', path: 'main/external' }
],
runnables: [
{ path: 'f/x/prod', usage_kind: 'script' },
{ path: 'f/x/cons', usage_kind: 'script' }
],
edges: [
edge('f/x/prod', 'main/produced', 'w'),
edge('f/x/cons', 'main/produced', 'r'),
edge('f/x/cons', 'main/external', 'r')
]
})
it('flags an unused asset, and a `// on` with no producer and its script, but not a plain read', () => {
const onRefs = (...paths: string[]) =>
paths.map((path) => ({ kind: 'ducklake' as const, path }))
const none = pipelineNodeErrors(graph, new Map())
expect([...none.assets.keys()]).toEqual(['ducklake:main/orphan'])
expect(none.scripts.size).toBe(0)
const errors = pipelineNodeErrors(
graph,
new Map([['f/x/cons', onRefs('main/produced', 'main/external')]])
)
// The subscriber is at fault, not the asset nothing writes.
expect([...errors.assets.keys()]).toEqual(['ducklake:main/orphan'])
expect([...errors.scripts.keys()]).toEqual(['f/x/cons'])
expect([...errors.unwritten.keys()]).toEqual(['ducklake:main/external'])
})
it('flags a pipeline script nothing starts, but not a macro library or a triggered one', () => {
const g = baseGraph({
runnables: [
{ path: 'f/x/stray', usage_kind: 'script', in_pipeline: true },
{ path: 'f/x/lib', usage_kind: 'script', in_pipeline: true, macros: [{ name: 'm' }] },
{ path: 'f/x/timed', usage_kind: 'script', in_pipeline: true }
] as any,
triggers: [
{
trigger_kind: 'schedule',
path: 'f/x/timed_schedule',
runnable_kind: 'script',
runnable_path: 'f/x/timed'
}
]
})
expect([...pipelineNodeErrors(g, new Map()).untriggered.keys()]).toEqual(['f/x/stray'])
})
it('flags a script only started by reads of assets nothing writes', () => {
const g = baseGraph({
assets: [{ kind: 'ducklake', path: 'main/raw' }],
runnables: [{ path: 'f/x/reader', usage_kind: 'script', in_pipeline: true }] as any,
edges: [
{
runnable_path: 'f/x/reader',
runnable_kind: 'script',
asset_kind: 'ducklake',
asset_path: 'main/raw',
access_type: 'r'
}
],
triggers: [
{
trigger_kind: 'asset',
asset_kind: 'ducklake',
asset_path: 'main/raw',
runnable_kind: 'script',
runnable_path: 'f/x/reader'
}
]
})
expect(pipelineNodeErrors(g, new Map()).untriggered.get('f/x/reader')).toBe('unwritten-inputs')
expect(
pipelineNodeErrors(g, new Map(), new Set(['ducklake:main/raw'])).untriggered.size
).toBe(0)
})
it('does not count a flow write as starting an asset-triggered script', () => {
const g = baseGraph({
assets: [{ kind: 'ducklake', path: 'main/raw' }],
runnables: [
{ path: 'f/x/reader', usage_kind: 'script', in_pipeline: true },
{ path: 'f/x/loader', usage_kind: 'flow' }
] as any,
edges: [
{
runnable_path: 'f/x/loader',
runnable_kind: 'flow',
asset_kind: 'ducklake',
asset_path: 'main/raw',
access_type: 'w'
}
],
triggers: [
{
trigger_kind: 'asset',
asset_kind: 'ducklake',
asset_path: 'main/raw',
runnable_kind: 'script',
runnable_path: 'f/x/reader'
}
]
} as any)
expect(pipelineNodeErrors(g, new Map()).untriggered.get('f/x/reader')).toBe('unwritten-inputs')
})
it('clears a subscription the workspace writes outside the pipeline', () => {
const errors = pipelineNodeErrors(
graph,
new Map([['f/x/cons', [{ kind: 'ducklake' as const, path: 'main/external' }]]]),
new Set(['ducklake:main/external'])
)
expect(errors.scripts.size).toBe(0)
expect(errors.unwritten.size).toBe(0)
})
it('treats dbt models as used and written by their (hidden) project, but not dbt sources', () => {
const dbt = (path: string, resource_type: string) => ({
kind: 'dbt' as const,
path,
dbt: { unique_id: `x.${path}`, resource_type } as any
})
const errors = pipelineNodeErrors(
baseGraph({
assets: [dbt('mart', 'model'), dbt('raw', 'source')],
runnables: [{ path: 'f/x/s', usage_kind: 'script' }]
}),
new Map([
[
'f/x/s',
[
{ kind: 'dbt', path: 'mart' },
{ kind: 'dbt', path: 'raw' }
]
]
])
)
expect([...errors.unwritten.keys()]).toEqual(['dbt:raw'])
expect([...errors.scripts.keys()]).toEqual(['f/x/s'])
})
})
@@ -1,4 +1,9 @@
import type { AssetGraphMacroEdge, AssetGraphResponse, NativeTriggerKind } from './types'
import type {
AssetGraphMacroEdge,
AssetGraphResponse,
NativeTriggerKind,
PipelineTriggerDraft
} from './types'
import {
mergeColumnLineage,
parsePipelineAnnotations,
@@ -46,6 +51,8 @@ export type ResolveGraphInput = {
* annotation has no matching trigger row in `base.triggers`.
*/
annotatedNativeKindsByPath: Map<string, Set<NativeTriggerKind>>
/** Undeployed triggers; each stands in for its script's missing trigger of that kind. */
triggerDrafts?: Iterable<PipelineTriggerDraft>
}
/** Mutable bag the sequential passes accumulate the resolved graph into. */
@@ -264,16 +271,200 @@ export function resolveGraph(input: ResolveGraphInput): AssetGraphResponse {
// there is nothing to show (also keeps the no-macros response shape
// byte-identical to before the feature).
const macroEdges = resolveMacroEdges(input)
const triggers = withTriggerDrafts(
[...baseTriggers, ...acc.extraTriggers],
input.triggerDrafts ?? []
)
return {
...base,
assets: acc.assets,
assets: dropOrphanedAssets(acc.assets, base, acc.edges, triggers),
runnables: acc.runnables,
edges: acc.edges,
triggers: [...baseTriggers, ...acc.extraTriggers],
triggers,
...(macroEdges.length > 0 || base.macro_edges ? { macro_edges: macroEdges } : {})
}
}
function withTriggerDrafts(
triggers: AssetGraphResponse['triggers'],
triggerDrafts: Iterable<PipelineTriggerDraft>
): AssetGraphResponse['triggers'] {
const drafts = [...triggerDrafts]
if (drafts.length === 0) return triggers
const drafted = new Set(drafts.map((d) => `${d.kind}:${d.config.script_path}`))
return [
...triggers.filter(
(t) =>
!(
t.trigger_kind !== 'asset' &&
t.missing &&
t.runnable_kind === 'script' &&
drafted.has(`${t.trigger_kind}:${t.runnable_path}`)
)
),
...drafts.map((d) => ({
trigger_kind: d.kind,
path: d.config.path,
runnable_kind: 'script' as const,
runnable_path: d.config.script_path,
unsaved: true,
draft: true,
...(d.kind === 'schedule'
? { schedule: d.config.schedule as string, timezone: d.config.timezone as string }
: {}),
...(d.config.summary ? { summary: d.config.summary as string } : {})
}))
]
}
/** Why a node is red, and what the Fix actions act on. */
export type PipelineNodeErrors = {
/** `kind:path` → reason. */
assets: Map<string, string>
/** Script path → reason. */
scripts: Map<string, string>
/** Subscribed assets nothing writes: `kind:path` → the asset and its subscribers. */
unwritten: Map<string, { asset: { kind: AssetKind; path: string }; subscribers: string[] }>
/** Script path → the unwritten asset it runs on (its first, when several). */
waitsOn: Map<string, { kind: AssetKind; path: string }>
/** Pipeline scripts nothing starts → why: no trigger at all, or only asset
* triggers on assets nothing writes (an auto-triggering read included). */
untriggered: Map<string, 'no-trigger' | 'unwritten-inputs'>
/** Assets some asset trigger waits on that nothing in the pipeline writes: the
* ones to look up across the workspace for `writtenElsewhere`. */
unwrittenTriggerAssets: Set<string>
}
/**
* Nodes the pipeline cannot work with, with the reason. Assets (keyed
* `kind:path`): one nothing uses at all. Scripts (keyed by path): one
* subscribed (`// on <asset>`) to an asset nothing writes, in the pipeline or,
* per `writtenElsewhere`, by a script anywhere else in the workspace, since the subscription can never fire; the asset itself is
* fine, and is reported in `unwritten`. A body read of an external asset is fine and is not passed here:
* only the explicit `// on` annotations, per script path, count as subscriptions.
*/
export function pipelineNodeErrors(
graph: AssetGraphResponse,
explicitOnByPath: Map<string, Array<{ kind: AssetKind; path: string }>>,
writtenElsewhere: ReadonlySet<string> = new Set()
): PipelineNodeErrors {
const referenced = referencedAssetKeys(graph)
// Only a script's write fires an asset trigger: a flow's never does.
const written = new Set(
graph.edges
.filter(
(e) => e.runnable_kind === 'script' && (e.access_type === 'w' || e.access_type === 'rw')
)
.map((e) => `${e.asset_kind}:${e.asset_path}`)
)
for (const k of writtenElsewhere) written.add(k)
// The dbt project's own edges are hidden from the pipeline graph (see
// `hideDbtRunnables`), so its relations are judged by their provenance: the
// project uses every one, and materializes all but its sources.
for (const a of graph.assets) {
if (!a.dbt) continue
referenced.add(`${a.kind}:${a.path}`)
if (a.dbt.resource_type !== 'source') written.add(`${a.kind}:${a.path}`)
}
const errors = new Map<string, string>()
const scriptErrors = new Map<string, string>()
const unwritten: PipelineNodeErrors['unwritten'] = new Map()
const waitsOn: PipelineNodeErrors['waitsOn'] = new Map()
for (const a of graph.assets) {
const key = `${a.kind}:${a.path}`
if (!referenced.has(key)) errors.set(key, 'nothing in the pipeline uses it')
}
const scripts = new Set(
graph.runnables.filter((r) => r.usage_kind === 'script').map((r) => r.path)
)
for (const [path, refs] of explicitOnByPath) {
if (!scripts.has(path)) continue
for (const ref of refs) {
const key = `${ref.kind}:${ref.path}`
if (written.has(key)) continue
const entry = unwritten.get(key) ?? { asset: ref, subscribers: [] }
if (!entry.subscribers.includes(path)) entry.subscribers.push(path)
unwritten.set(key, entry)
if (!scriptErrors.has(path)) {
scriptErrors.set(path, `runs on ${ref.path}, which nothing writes`)
waitsOn.set(path, ref)
}
}
}
// A pipeline script nothing starts runs only by hand. An asset trigger starts
// it only if something writes that asset, which covers the automatic trigger
// of a ducklake/s3 read too. dbt projects and macro libraries are never
// started on their own, so they are left out.
const assetKey = (t: { asset_kind: AssetKind; asset_path: string }) =>
`${t.asset_kind}:${t.asset_path}`
const unwrittenTriggerAssets = new Set<string>()
const hasTrigger = new Set<string>()
const started = new Set<string>()
for (const t of graph.triggers) {
hasTrigger.add(t.runnable_path)
if (t.trigger_kind !== 'asset') started.add(t.runnable_path)
else if (written.has(assetKey(t))) started.add(t.runnable_path)
else unwrittenTriggerAssets.add(assetKey(t))
}
for (const k of unwritten.keys()) unwrittenTriggerAssets.add(k)
const untriggered: PipelineNodeErrors['untriggered'] = new Map()
for (const r of graph.runnables) {
if (r.usage_kind !== 'script' || !r.in_pipeline || r.dbt || r.macros?.length) continue
if (started.has(r.path) || scriptErrors.has(r.path)) continue
const why = hasTrigger.has(r.path) ? 'unwritten-inputs' : 'no-trigger'
untriggered.set(r.path, why)
scriptErrors.set(
r.path,
why === 'no-trigger'
? 'nothing starts it: it has no trigger'
: 'nothing starts it: nothing writes the assets it runs after'
)
}
return {
assets: errors,
scripts: scriptErrors,
unwritten,
waitsOn,
untriggered,
unwrittenTriggerAssets
}
}
function referencedAssetKeys(
graph: Pick<AssetGraphResponse, 'edges' | 'triggers' | 'test_edges' | 'dbt_edges'>
): Set<string> {
const keys = new Set<string>()
for (const e of graph.edges) keys.add(`${e.asset_kind}:${e.asset_path}`)
for (const t of graph.triggers)
if (t.trigger_kind === 'asset') keys.add(`${t.asset_kind}:${t.asset_path}`)
for (const e of graph.test_edges ?? []) keys.add(`${e.asset_kind}:${e.asset_path}`)
for (const e of graph.dbt_edges ?? []) {
keys.add(`dbt:${e.from_asset_path}`)
keys.add(`dbt:${e.to_asset_path}`)
}
return keys
}
/**
* The backend only returns assets something references, so a deployed asset
* whose every edge and trigger the draft/live overlays stripped — an output the
* open script renamed — would linger as an edgeless node until the deploy
* lands. Assets that were already unreferenced in `base` are left alone.
*/
function dropOrphanedAssets(
assets: AssetGraphResponse['assets'],
base: AssetGraphResponse,
edges: AssetGraphResponse['edges'],
triggers: AssetGraphResponse['triggers']
): AssetGraphResponse['assets'] {
const before = referencedAssetKeys(base)
const after = referencedAssetKeys({ ...base, edges, triggers })
return assets.filter((a) => {
const key = `${a.kind}:${a.path}`
return !before.has(key) || after.has(key)
})
}
/**
* Macro-library → consumer edges: the deployed base edges, with `// use`
* declarations of overlaid scripts (drafts + the open buffer) taking over
@@ -848,7 +1039,6 @@ function overlayInferredLineage(acc: Accumulator, input: ResolveGraphInput) {
overlayLineage(inferredReadsByPath, 'r')
}
/** dbt project ↔ relation association, for a graph that does NOT draw it.
*
* A dbt script owns every relation of its project, so drawing one edge per
@@ -6,6 +6,12 @@ export type GraphUsageKind = 'script' | 'flow'
export interface AssetGraphAssetNode {
kind: AssetKind
path: string
// Filled in client-side: the node is misconfigured (see `pipelineNodeErrors`).
error?: string
// Filled in client-side: no script in the workspace writes it, so a step
// downstream of it would never run on its writes (a flow's write fires no
// asset trigger).
never_written?: boolean
// Fork workspaces only: 'fork' when this ducklake asset was materialized in
// the fork itself, 'deferred' when reads fall back to the parent workspace's
// current table via a defer view. Absent outside forks / for other kinds /
@@ -70,6 +76,12 @@ export interface DbtDataTest {
export interface AssetGraphRunnableNode {
path: string
usage_kind: GraphUsageKind
// Filled in client-side (not sent by the backend): the canvas titles the
// node with the summary over the path, and shows the language's icon.
summary?: string
language?: import('$lib/gen').ScriptLang
// Why the script is misconfigured, when it is (see `pipelineNodeErrors`).
error?: string
// Script has `// pipeline` annotation. Drives the pipeline-member
// visual state; unrelated to what the script actually writes (that's
// parsed separately into lineage edges).
@@ -170,6 +182,17 @@ export type NativeTriggerKind =
| 'gcp'
| 'data_upload'
export type PipelineTriggerDraftKind = Exclude<NativeTriggerKind, 'webhook' | 'data_upload'>
/** A trigger configured in the pipeline editor but not deployed yet. It lives only
* in the pipeline's draft bundle, never as a workspace trigger draft, and is
* created when the pipeline is saved. `config` is what the kind's editor saves
* (for a schedule, a `NewSchedule`). */
export type PipelineTriggerDraft = {
kind: PipelineTriggerDraftKind
config: Record<string, any> & { path: string; script_path: string }
}
export type AssetGraphTrigger =
| {
trigger_kind: 'asset'
@@ -192,6 +215,13 @@ export type AssetGraphTrigger =
// the canvas renders a red placeholder with a "Create trigger"
// affordance instead of a fully-wired source.
missing?: boolean
// A pipeline-local `PipelineTriggerDraft`: `path` is the trigger it
// will be deployed as, and no row exists yet.
draft?: boolean
// A schedule's cron, zone and summary, filled in client-side, for the node's labels.
schedule?: string
timezone?: string
summary?: string
}
// Macro-library → consumer edge: the consumer calls `macro_names` of
@@ -0,0 +1,32 @@
import type { Item } from '$lib/utils'
import type { ScriptLang } from '$lib/gen'
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
import { Code2 } from 'lucide-svelte'
export type UpstreamScript = { path: string; summary?: string; language?: ScriptLang }
/** A menu of the scripts that produce an asset, to pick the one to open. */
export function upstreamScriptItems(
scripts: UpstreamScript[],
onPick: (path: string) => void
): Item[] {
return scripts.map((s) => ({
displayName: s.summary || s.path,
description: s.summary ? s.path : undefined,
...(s.language
? { icon: LanguageIcon, iconProps: { lang: s.language, width: 14, height: 14 } }
: { icon: Code2 }),
action: () => onPick(s.path)
}))
}
/** An upstream trigger the asset pane can open, and how. */
export type UpstreamTriggerAction = { label: string; detail?: string; onOpen: () => void }
export function upstreamTriggerItems(triggers: UpstreamTriggerAction[]): Item[] {
return triggers.map((t) => ({
displayName: t.label,
description: t.detail,
action: () => t.onOpen()
}))
}
@@ -17,6 +17,8 @@
bgClass?: string | undefined
bgStyle?: string | undefined
iconClass?: string | undefined
/** Drop the type's icon; the text starts at the alert's edge. */
hideIcon?: boolean
iconStyle?: string | undefined
titleClass?: string | undefined
titleStyle?: string | undefined
@@ -41,6 +43,7 @@
bgClass = undefined,
bgStyle = undefined,
iconClass = undefined,
hideIcon = false,
iconStyle = undefined,
titleClass = undefined,
titleStyle = undefined,
@@ -78,14 +81,16 @@
<div class="flex flex-row items-center">
<!-- shrink-0: the badge's min-content width is the 16px icon, so without it a
narrow container squeezes the circle into an oval. -->
<div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-full">
<SvelteComponent
class={twMerge(classes[type].iconClass, iconClass)}
style={iconStyle}
size={16}
/>
</div>
<div class={twMerge('ml-1 w-full')}>
{#if !hideIcon}
<div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-full">
<SvelteComponent
class={twMerge(classes[type].iconClass, iconClass)}
style={iconStyle}
size={16}
/>
</div>
{/if}
<div class={twMerge(!hideIcon && 'ml-1', 'w-full')}>
{#if hasTitleRow}
<div class={twMerge('w-full flex flex-row items-center justify-between')}>
<span
@@ -0,0 +1,18 @@
import type { Item } from '$lib/utils'
import type { ContextMenuItem } from './ContextMenu.svelte'
/** A dropdown's items as the same entries of a right-click menu. */
export function contextMenuItemsFromMenu(items: Item[]): ContextMenuItem[] {
return items.flatMap((item) => [
...(item.separatorTop ? [{ id: `${item.displayName}-divider`, label: '', divider: true }] : []),
{
id: item.displayName,
label: item.displayName,
icon: item.icon,
disabled: item.disabled,
type: item.type,
shortcut: item.shortcut,
onClick: item.action as (() => void) | undefined
}
])
}
@@ -0,0 +1,155 @@
<script lang="ts">
import { base } from '$lib/base'
import Dropdown from '$lib/components/DropdownV2.svelte'
import ConfirmationModal from '../confirmationModal/ConfirmationModal.svelte'
import {
deletePipeline,
planPipelineDelete,
type PipelineDeletePlan
} from '$lib/components/assets/AssetGraph/pipelineDelete'
import { userStore, workspaceStore } from '$lib/stores'
import { sendUserToast } from '$lib/toast'
import { Loader2, Trash } from 'lucide-svelte'
import Row from './Row.svelte'
interface Props {
folder: string
depth?: number
/** Shown instead of `f/<folder>`, e.g. under the folder's own tree node. */
label?: string
onDeleted?: () => void
}
let { folder, depth = 0, label, onDeleted }: Props = $props()
let confirmOpen = $state(false)
let deleting = $state(false)
let plan = $state<PipelineDeletePlan | undefined>(undefined)
let planError = $state<string | undefined>(undefined)
// The backend deletes scripts for admins only; everyone else archives them.
let hardDelete = $derived(!!($userStore?.is_admin || $userStore?.is_super_admin))
async function openConfirm() {
const ws = $workspaceStore
if (!ws) return
plan = undefined
planError = undefined
confirmOpen = true
try {
plan = await planPipelineDelete(ws, folder)
} catch (e: any) {
planError = e?.body ?? e?.message ?? String(e)
}
}
async function confirmDelete() {
const ws = $workspaceStore
if (!ws || !plan) return
deleting = true
try {
const { removed, failures } = await deletePipeline(ws, folder, plan, hardDelete)
// Loaded on demand: the home list should not pull in the session runtime.
const { forgetDeletedPipeline } = await import(
'$lib/components/sessions/sessionRuntime.svelte'
)
forgetDeletedPipeline(ws, folder)
if (failures.length) {
sendUserToast(`Pipeline f/${folder} was only partly removed`, true, [], failures.join('\n'))
} else {
sendUserToast(`Removed pipeline f/${folder} (${removed} item${removed === 1 ? '' : 's'})`)
}
onDeleted?.()
} catch (e: any) {
sendUserToast(`Could not remove pipeline f/${folder}: ${e?.body ?? e?.message ?? e}`, true)
} finally {
deleting = false
confirmOpen = false
}
}
</script>
<Row
kind="data_pipeline"
marked={undefined}
href="{base}/pipeline/{encodeURIComponent(folder)}"
workspaceId={$workspaceStore ?? ''}
path={`f/${folder}`}
summary={label ?? `Pipeline · ${folder}`}
canFavorite={false}
{depth}
>
{#snippet actions()}
{#if !$userStore?.operator}
<Dropdown
items={[
{
displayName: 'Delete',
icon: Trash,
type: 'delete',
action: () => void openConfirm()
}
]}
/>
{/if}
{/snippet}
</Row>
<ConfirmationModal
open={confirmOpen}
title="Delete pipeline f/{folder}"
confirmationText={hardDelete ? 'Delete' : 'Archive'}
loading={deleting}
confirmDisabled={!plan}
trashbin={hardDelete}
on:canceled={() => (confirmOpen = false)}
on:confirmed={confirmDelete}
>
<div class="flex flex-col gap-3 text-xs">
{#if planError}
<span class="text-red-600 dark:text-red-400">Could not list the pipeline: {planError}</span>
{:else if !plan}
<span class="flex items-center gap-2 text-secondary">
<Loader2 size={14} class="animate-spin" /> Listing what this removes…
</span>
{:else}
{#if plan.scripts.some((s) => !s.draftOnly)}
{@render group(
hardDelete ? 'Scripts deleted' : 'Scripts archived',
plan.scripts.filter((s) => !s.draftOnly).map((s) => s.displayPath)
)}
{/if}
{#if plan.scripts.some((s) => s.draftOnly)}
{@render group(
'Draft scripts discarded',
plan.scripts.filter((s) => s.draftOnly).map((s) => s.displayPath)
)}
{/if}
{#if plan.triggers.length}
{@render group(
'Triggers deleted',
plan.triggers.map((t) => `${t.path} (${t.kind})`)
)}
{/if}
{#if plan.hasDraft}
{@render group('Unsaved pipeline changes discarded', ['Your pipeline draft'])}
{/if}
{#if !plan.scripts.length && !plan.triggers.length && !plan.hasDraft}
<span class="text-secondary">Nothing left to remove in this pipeline.</span>
{/if}
<span class="text-secondary">
The tables and files the pipeline wrote are kept, and so is the folder f/{folder}.
</span>
{/if}
</div>
</ConfirmationModal>
{#snippet group(title: string, entries: string[])}
<div class="flex flex-col gap-1">
<span class="font-semibold text-emphasis">{title} ({entries.length})</span>
<ul class="max-h-40 overflow-auto rounded-md border bg-surface-secondary px-3 py-1.5">
{#each entries as entry (entry)}
<li class="font-mono text-2xs text-primary truncate">{entry}</li>
{/each}
</ul>
</div>
{/snippet}
@@ -501,11 +501,17 @@ export class AIChatManager implements ChatViewHost {
skipResponsesApi = $state(false)
mode = $state<AIMode>(AIMode.NAVIGATOR)
pipelineAiChatHelpers = $state<PipelineAIChatHelpers | undefined>(undefined)
// One per mounted pipeline editor; a session can have several folders open.
#pipelineEditors = new Set<PipelineAIChatHelpers>()
// Resolved when a pipeline editor registers its tools. open_preview(pipeline)
// awaits this so the model's next build_pipeline_node call can't race ahead of
// the async canvas mount and hit "Unknown tool call".
#pipelineHelpersWaiters = new Set<() => void>()
#pipelineHelpersWaiters = new Set<(helpers: PipelineAIChatHelpers) => void>()
// A session's pipeline editor unmounts whenever another tab takes the preview
// panel. Its tools stay offered for every folder the session opened, and a call
// reopens the tab through this, so a page shown in between costs no failed call.
#pipelineReopener: ((folder: string) => void) | undefined = undefined
#openedPipelineFolders = new Set<string>()
readonly isOpen = $derived(chatState.size > 0)
savedSize = $state<number>(0)
instructions = $state<string>('')
@@ -900,6 +906,15 @@ export class AIChatManager implements ChatViewHost {
await this.#persistModifiedItems()
}
// Record the items deploying a chat-modified item produced — a pipeline's
// scripts — so a fork review selects exactly them. Persisted at once, like a
// discard: the deploy can run outside any turn.
async recordDeployedItems(itemKind: UserDraftItemKind, storagePaths: string[]) {
if (!this.modifiedItems || storagePaths.length === 0) return
for (const p of storagePaths) this.modifiedItems.add(maskKey(itemKind, p))
await this.#persistModifiedItems()
}
// Move a mask entry to the path a draft actually deployed to. A draft-only
// flow/app parks at a synthetic `draft_{uuid}` storage path and deploys to
// its chosen path — without the move, the existence check at the synthetic
@@ -2492,8 +2507,27 @@ export class AIChatManager implements ChatViewHost {
// Public because it is purely local, unlike `changeMode(GLOBAL)`, which also
// fires the three network refreshes.
setPipelineReopener = (reopen: (folder: string) => void) => {
this.#pipelineReopener = reopen
}
#pipelineFolders = (): string[] => [
...new Set([
...[...this.#pipelineEditors].map((p) => p.getFolder()),
...(this.#pipelineReopener ? this.#openedPipelineFolders : [])
])
]
#ensurePipeline = async (folder: string): Promise<PipelineAIChatHelpers | undefined> => {
const find = () => [...this.#pipelineEditors].find((p) => p.getFolder() === folder)
const live = find()
if (live || !this.#pipelineReopener) return live
this.#pipelineReopener(folder)
return (await this.waitForPipelineHelpers(folder)) ? find() : undefined
}
configureGlobalMode = () => {
const pipeline = this.pipelineAiChatHelpers
const hasPipelines = this.#pipelineFolders().length > 0
const opts = this.globalAssemblyOpts()
const baseHelpers: GlobalToolHelpers = {
// A session targets its own fixed (possibly forked) workspace, so capture it for
@@ -2525,7 +2559,14 @@ export class AIChatManager implements ChatViewHost {
}
}
this.#assembledTools = assembleGlobalTools(opts)
this.helpers = pipeline ? { ...baseHelpers, pipeline } : baseHelpers
this.helpers = hasPipelines
? {
...baseHelpers,
pipelines: () => [...this.#pipelineEditors],
pipelineFolders: this.#pipelineFolders,
ensurePipeline: this.#ensurePipeline
}
: baseHelpers
this.systemMessage = assembleGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), opts)
this.syncArtifactsSession()
}
@@ -2538,7 +2579,7 @@ export class AIChatManager implements ChatViewHost {
mcpServers: this.mcpServers,
access: this.sessionAccess,
sessionContext: this.sessionContextResolver?.(),
pipelineContext: this.pipelineAiChatHelpers?.getPipelineContext()
pipelineFolders: this.#pipelineFolders()
})
// Fetch the workspace's AI skills and, if GLOBAL mode is still active, rebuild
@@ -4870,19 +4911,18 @@ export class AIChatManager implements ChatViewHost {
// AI edits apply directly as drafts, so there is nothing to auto-accept.
// Returns a cleanup that tears the registration back down.
setPipelineHelpers = (pipelineHelpers: PipelineAIChatHelpers) => {
this.pipelineAiChatHelpers = pipelineHelpers
this.#pipelineEditors.add(pipelineHelpers)
this.#openedPipelineFolders.add(pipelineHelpers.getFolder())
untrack(() => {
if (this.mode === AIMode.GLOBAL) {
this.configureGlobalMode()
}
})
// The pipeline tools are now registered — release anything awaiting them.
const waiters = [...this.#pipelineHelpersWaiters]
this.#pipelineHelpersWaiters.clear()
waiters.forEach((resolve) => resolve())
;[...this.#pipelineHelpersWaiters].forEach((notify) => notify(pipelineHelpers))
return () => {
this.pipelineAiChatHelpers = undefined
this.#pipelineEditors.delete(pipelineHelpers)
untrack(() => {
if (this.mode === AIMode.GLOBAL) {
this.configureGlobalMode()
@@ -4892,15 +4932,16 @@ export class AIChatManager implements ChatViewHost {
}
/**
* Await the pipeline editor's tool registration. Resolves `true` immediately
* when a pipeline editor is already mounted, or when the next one registers;
* resolves `false` after `timeoutMs` if none registers (e.g. a backgrounded
* session whose preview tab is not mounted, or a closed tab). Callers must
* treat `false` as "tools NOT available" rather than silently reporting
* success — the open_preview handler surfaces that to the model.
* Await the tool registration of the pipeline editor for `folder` (any folder
* when unset). Resolves `true` immediately when one is already mounted, or when
* it registers; resolves `false` after `timeoutMs` if none registers (e.g. a
* session whose runtime is not mounted, or a closed tab). Callers must treat
* `false` as "tools NOT available" rather than silently reporting success —
* the open_preview handler surfaces that to the model.
*/
waitForPipelineHelpers = (timeoutMs = 8000): Promise<boolean> => {
if (this.pipelineAiChatHelpers) return Promise.resolve(true)
waitForPipelineHelpers = (folder?: string, timeoutMs = 8000): Promise<boolean> => {
const matches = (p: PipelineAIChatHelpers) => folder === undefined || p.getFolder() === folder
if ([...this.#pipelineEditors].some(matches)) return Promise.resolve(true)
return new Promise<boolean>((resolve) => {
let settled = false
const finish = (registered: boolean) => {
@@ -4909,7 +4950,9 @@ export class AIChatManager implements ChatViewHost {
this.#pipelineHelpersWaiters.delete(onRegister)
resolve(registered)
}
const onRegister = () => finish(true)
const onRegister = (p: PipelineAIChatHelpers) => {
if (matches(p)) finish(true)
}
this.#pipelineHelpersWaiters.add(onRegister)
setTimeout(() => finish(false), timeoutMs)
})
@@ -4329,11 +4329,15 @@ describe('DOM selector chips scoped by app path', () => {
describe('AIChatManager.waitForPipelineHelpers', () => {
function fakePipelineHelpers(): PipelineAIChatHelpers {
return {
getFolder: () => 'f',
hasDrafts: () => false,
getPipelineContext: () => ({ folder: 'f', mode: 'edit', nodes: [], assets: [] }),
getNodeBody: async () => undefined,
proposeNode: async () => ({ path: '', detectedReads: [], detectedWrites: [] }),
editNode: async () => ({ detectedReads: [], detectedWrites: [] }),
removeProposedNode: async () => {},
unconfiguredTriggers: async () => [],
setNodeTrigger: async () => ({ path: '', replaced: false }),
testNode: async () => undefined
}
}
@@ -4341,13 +4345,13 @@ describe('AIChatManager.waitForPipelineHelpers', () => {
it('resolves true immediately when a pipeline editor is already registered', async () => {
const manager = new AIChatManager()
manager.setPipelineHelpers(fakePipelineHelpers())
await expect(manager.waitForPipelineHelpers(1000)).resolves.toBe(true)
await expect(manager.waitForPipelineHelpers(undefined, 1000)).resolves.toBe(true)
})
it('resolves true once a pipeline editor registers', async () => {
const manager = new AIChatManager()
let outcome: boolean | undefined
const wait = manager.waitForPipelineHelpers(1000).then((v) => (outcome = v))
const wait = manager.waitForPipelineHelpers(undefined, 1000).then((v) => (outcome = v))
await Promise.resolve()
expect(outcome).toBeUndefined()
manager.setPipelineHelpers(fakePipelineHelpers())
@@ -4355,12 +4359,24 @@ describe('AIChatManager.waitForPipelineHelpers', () => {
expect(outcome).toBe(true)
})
it('waits for the editor of the requested folder, not any editor', async () => {
const manager = new AIChatManager()
manager.setPipelineHelpers(fakePipelineHelpers())
let outcome: boolean | undefined
const wait = manager.waitForPipelineHelpers('sales', 1000).then((v) => (outcome = v))
await Promise.resolve()
expect(outcome).toBeUndefined()
manager.setPipelineHelpers({ ...fakePipelineHelpers(), getFolder: () => 'sales' })
await wait
expect(outcome).toBe(true)
})
// The false result is the signal the open_preview handler needs: a backgrounded
// session's editor never mounts, so it must report "tools unavailable" rather
// than silently claim success.
it('resolves false after the timeout when no editor ever registers', async () => {
const manager = new AIChatManager()
await expect(manager.waitForPipelineHelpers(10)).resolves.toBe(false)
await expect(manager.waitForPipelineHelpers(undefined, 10)).resolves.toBe(false)
})
})
@@ -19,6 +19,7 @@ import { applyReasoningToConfig } from '../reasoningRegistry'
import {
appendPendingToolImages,
processToolCall,
type LiveToolSet,
queuedToolStatus,
type Tool,
type ToolCallbacks,
@@ -197,7 +198,11 @@ export async function parseAnthropicCompletion(
tools: Tool<any>[],
helpers: any,
abortController?: AbortController,
options?: { workspace?: string; onTokenUsage?: (usage: ChatTokenUsage) => void }
options?: {
workspace?: string
onTokenUsage?: (usage: ChatTokenUsage) => void
live?: LiveToolSet
}
): Promise<ParsedCompletionResult> {
let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = []
let error = null
@@ -447,7 +452,8 @@ export async function parseAnthropicCompletion(
helpers,
toolCallbacks: callbacks,
workspace: options?.workspace,
messages
messages,
live: options?.live
})
messages.push(messageToAdd)
addedMessages.push(messageToAdd)
@@ -453,7 +453,8 @@ export async function runChatLoop(config: ChatLoopConfig): Promise<ChatLoopResul
const parseOptions = {
workspace,
provider: modelProvider.provider,
onTokenUsage: reportUsage
onTokenUsage: reportUsage,
live: () => ({ tools: config.tools, helpers: config.helpers })
}
if (isOpenAI) {
@@ -1,6 +1,7 @@
import { randomUUID } from '$lib/utils/uuid'
import {
AppService,
AssetService,
AzureTriggerService,
EmailTriggerService,
FlowService,
@@ -67,6 +68,7 @@ import {
stripFileArgs
} from '$lib/components/job_args'
import { processSecretArgs } from '$lib/components/secretArgUtils'
import { parsePipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations'
import { PLAN_MODE_MESSAGES } from '../planModeMessages'
import {
DEFAULT_DATA as DEFAULT_RAW_APP_DATA,
@@ -423,6 +425,8 @@ const updateUserInstructionsSchema = z.object({
)
})
const listPipelinesSchema = z.object({})
const listWorkspaceItemsSchema = z.object({
types: z
.array(itemTypeSchema)
@@ -1424,7 +1428,7 @@ const buildGlobalSystemPrompt = (
const activePreviewRule = previewTools
? '\n- If the user message includes an ACTIVE PREVIEW section, that is the page the side panel is showing — resolve "this page", "here" and "it" against it, and against `open` (the item the user has open in its editor) when there is one. It already tells you what get_preview_status would, so do not call that tool to learn what is on screen; call it only to check the panel\'s *other* tabs.'
: ''
const pipelineBullet = `- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on <ref>\` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow.${pipelineAlphaNote}`
const pipelineBullet = `- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on <ref>\` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow. list_pipelines lists the existing ones by folder.${pipelineAlphaNote}`
// Hosting and edition come from the hostname and a store the app populates at init, so
// they are knowable only in the browser: a non-browser caller reads false for both and
// would be told "self-hosted Community Edition" whatever it targets. No base URL here
@@ -1524,7 +1528,7 @@ ${pipelineBullet}`
canWriteDraft,
`
- After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited.
- Building a data pipeline: call open_preview(kind="pipeline", path="<folder>") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty ${when(canCreateFolder, 'or not-yet-created ')}folder is fine${when(canCreateFolder, ' (create_folder first if needed, then open it)')}. Opening it registers build_pipeline_node / edit_pipeline_node — use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor.`
- Building a data pipeline: call open_preview(kind="pipeline", path="<folder>") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty ${when(canCreateFolder, 'or not-yet-created ')}folder is fine${when(canCreateFolder, ' (create_folder first if needed, then open it)')}. Opening it adds build_pipeline_node / edit_pipeline_node to your tools from the next step on — they are not in your tool list before open_preview returns, so call open_preview on its own, wait for its result, then build. They then stay available for the rest of the session even while another tab (a page, another preview) is shown: a call brings the pipeline editor back by itself, so do not re-open it first. Use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor.`
)}
- When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app").
- Writing an app file does not compile it: the open preview rebuilds it afterwards. After editing a raw app's frontend files with its preview open, call get_app_runtime_logs to check the build — when it failed, it returns the build errors (e.g. syntax or import errors) and bundler logs to fix.
@@ -3654,6 +3658,23 @@ export const globalTools: SessionTool<{}>[] = [
return JSON.stringify(results, null, 2)
}
},
{
requires: NONE,
def: createToolDef(
listPipelinesSchema,
'list_pipelines',
'List the data pipelines of the workspace: each folder holding at least one pipeline script, with its number of scripts. A pipeline is named by its folder, not by an item path.'
),
planModeSafe: true,
fn: async ({ workspace, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, { content: 'Listing data pipelines...' })
const pipelines = await AssetService.listPipelineFolders({ workspace })
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${pipelines.length} data pipeline(s)`
})
return JSON.stringify(pipelines)
}
},
{
requires: NONE,
def: createToolDef(
@@ -4291,7 +4312,7 @@ export const globalTools: SessionTool<{}>[] = [
toolCallbacks.setToolStatus(toolId, {
content: `Found ${results.length} resource type(s) for "${parsed.query}"`
})
return JSON.stringify(
const listing = JSON.stringify(
results.map((rt) => ({
name: rt.name,
schema: rt.schema
@@ -4299,6 +4320,15 @@ export const globalTools: SessionTool<{}>[] = [
null,
2
)
// The search is semantic and always returns its closest types, so a miss reads
// like a hit unless the absence is spelled out.
const words = (parsed.query.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter(
(w) => w.length >= 3
)
const named = results.some((rt) => words.some((w) => rt.name.toLowerCase().includes(w)))
return named || words.length === 0
? listing
: `${listing}\n\nNo resource type name contains "${parsed.query}": these are only the nearest matches, so no dedicated type exists. Searching again with synonyms will not find one — use a fitting generic type above, or create a custom c_<name> type.`
}
},
createDbSchemaTool<{}>({
@@ -5471,6 +5501,17 @@ function writeScriptDraft(
ctx: WriteDraftCtx,
codeDiff?: ToolCodeDiff
): Promise<string> {
// A session builds pipeline nodes into the folder's pipeline draft; a script draft
// written here never reaches the pipeline graph, and saving the pipeline drops it.
if (
(ctx.sessionId ?? sessionIdFromCtx(ctx)) &&
parsePipelineAnnotations(args.content).inPipeline
) {
const folder = args.path.match(/^f\/([^/]+)\//)?.[1] ?? '<folder>'
throw new Error(
`"${args.path}" is a data pipeline node (its source starts with the pipeline annotation). In a session, pipeline nodes are not script drafts: call open_preview(kind="pipeline", path="${folder}") if that folder's pipeline editor is not open yet, then use build_pipeline_node (new node) or edit_pipeline_node (existing node). Nothing was written.`
)
}
return writeDraft(SCRIPT_SPEC, 'script', args.path, args, ctx, {
override: args.override,
codeDiff:
@@ -15,22 +15,17 @@ vi.mock('@codingame/monaco-vscode-languages-service-override', () => ({ default:
vi.mock('$lib/components/vscode', () => ({}))
import { getSessionContextPromptSection, type SessionPromptContext } from './core'
import { getPipelinePromptSection, type PipelineContext } from '../pipeline/core'
import { getPipelinePromptSection } from '../pipeline/core'
import { assembleGlobalSystemMessage, assembleGlobalTools } from './globalAssembly'
const sessionContext: SessionPromptContext = { workspaceId: 'ws', parentWorkspaceId: 'parent' }
const pipelineContext: PipelineContext = {
folder: 'myfolder',
mode: 'edit',
nodes: [],
assets: []
}
const pipelineFolders = ['myfolder']
describe('assembleGlobalTools', () => {
it('includes the pipeline tools only when a pipeline editor is active', () => {
const names = (opts: Parameters<typeof assembleGlobalTools>[0]) =>
assembleGlobalTools(opts).map((t) => t.def.function.name)
expect(names({ pipelineContext })).toContain('get_pipeline_graph')
expect(names({ pipelineFolders })).toContain('get_pipeline_graph')
expect(names({})).not.toContain('get_pipeline_graph')
})
})
@@ -41,13 +36,13 @@ describe('assembleGlobalSystemMessage', () => {
it('carries both optional sections when both contexts are given', () => {
const content = assembleGlobalSystemMessage(undefined, {
sessionContext,
pipelineContext
pipelineFolders
}).content as string
expect(content).toContain(getSessionContextPromptSection(sessionContext))
expect(content).toContain(getPipelinePromptSection(pipelineContext))
expect(content).toContain(getPipelinePromptSection(pipelineFolders))
// Order is part of the assembled output, so it is pinned too.
expect(content.indexOf(getSessionContextPromptSection(sessionContext))).toBeLessThan(
content.indexOf(getPipelinePromptSection(pipelineContext))
content.indexOf(getPipelinePromptSection(pipelineFolders))
)
})
@@ -58,6 +53,6 @@ describe('assembleGlobalSystemMessage', () => {
// shares, so a stray section built from any other context still trips them.
const heading = (s: string) => s.trim().split('\n')[0]
expect(content).not.toContain(heading(getSessionContextPromptSection(sessionContext)))
expect(content).not.toContain(heading(getPipelinePromptSection(pipelineContext)))
expect(content).not.toContain(heading(getPipelinePromptSection(pipelineFolders)))
})
})
@@ -12,7 +12,7 @@ import {
type SessionPromptContext
} from './core'
import { createMcpTools } from './mcpTools'
import { getPipelinePromptSection, pipelineTools, type PipelineContext } from '../pipeline/core'
import { getPipelinePromptSection, pipelineTools } from '../pipeline/core'
import { getFolderInstructionsPromptSection, type FolderInstruction } from '../folderInstructions'
// Derived, not retyped: an option added to prepareGlobalSystemMessage is reachable
@@ -21,7 +21,8 @@ import { getFolderInstructionsPromptSection, type FolderInstruction } from '../f
// directly — would get it.
export type GlobalAssemblyOpts = NonNullable<Parameters<typeof prepareGlobalSystemMessage>[1]> & {
sessionContext?: SessionPromptContext
pipelineContext?: PipelineContext
/** Folders of the open pipeline editors; the pipeline tools and prompt come with any. */
pipelineFolders?: readonly string[]
folderInstructions?: readonly FolderInstruction[]
}
@@ -34,8 +35,8 @@ export function assembleGlobalSystemMessage(
if (opts.sessionContext) {
systemMessage.content += getSessionContextPromptSection(opts.sessionContext, opts.access)
}
if (opts.pipelineContext) {
systemMessage.content += getPipelinePromptSection(opts.pipelineContext, opts.access)
if (opts.pipelineFolders?.length) {
systemMessage.content += getPipelinePromptSection(opts.pipelineFolders, opts.access)
}
return systemMessage
}
@@ -43,7 +44,7 @@ export function assembleGlobalSystemMessage(
export function assembleGlobalTools(opts: GlobalAssemblyOpts): SessionTool<any>[] {
return [
...globalToolsFor({ sessionPreview: opts.previewTools ?? false }),
...(opts.pipelineContext ? pipelineTools : []),
...(opts.pipelineFolders?.length ? pipelineTools : []),
...createMcpTools(opts.mcpServers ?? [])
]
}
@@ -32,7 +32,7 @@ import type { SessionAccess, SessionCapability, SessionTool } from '../sessionCa
const ASSEMBLY_OPTS = {
previewTools: true,
pipelineContext: { folder: 'my_pipeline', mode: 'edit', nodes: [], assets: [] },
pipelineFolders: ['my_pipeline'],
mcpServers: [{ path: 'f/test/server' } as any]
}
@@ -168,7 +168,7 @@ describe('session tool policies', () => {
user: { username: 'alex', folders: ['shared'], folders_read: ['shared'] },
access,
sessionContext: ctx,
pipelineContext: { folder: 'my_pipeline', mode: 'edit', nodes: [], assets: [] }
pipelineFolders: ['my_pipeline']
})
// Both decoration variants: the escalation one adds its own tool mentions.
for (const blocks of [0, 9]) {
@@ -18,6 +18,7 @@ import { applyReasoningToConfig } from '../reasoningRegistry'
import {
appendPendingToolImages,
processToolCall,
type LiveToolSet,
queuedToolStatus,
type Tool,
type ToolCallbacks,
@@ -413,7 +414,11 @@ export async function parseOpenAIResponsesCompletion(
addedMessages: ChatCompletionMessageParam[],
tools: Tool<any>[],
helpers: any,
options?: { workspace?: string; onTokenUsage?: (usage: ChatTokenUsage) => void }
options?: {
workspace?: string
onTokenUsage?: (usage: ChatTokenUsage) => void
live?: LiveToolSet
}
): Promise<ParsedCompletionResult> {
let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = []
let error: OpenAIError | ResponseErrorEvent | null = null
@@ -620,7 +625,8 @@ export async function parseOpenAIResponsesCompletion(
helpers,
toolCallbacks: callbacks,
workspace: options?.workspace,
messages
messages,
live: options?.live
})
messages.push(messageToAdd)
addedMessages.push(messageToAdd)
@@ -39,7 +39,7 @@ const sampleContext: PipelineContext = {
}
function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
helpers: { pipeline: PipelineAIChatHelpers }
helpers: { pipelines: () => PipelineAIChatHelpers[] }
calls: Record<string, any[]>
} {
const calls: Record<string, any[]> = {}
@@ -50,6 +50,8 @@ function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
return ret
}
const pipeline: PipelineAIChatHelpers = {
getFolder: () => sampleContext.folder,
hasDrafts: () => false,
getPipelineContext: () => sampleContext,
getNodeBody: async (path: string) => {
calls.getNodeBody = [...(calls.getNodeBody ?? []), [path]]
@@ -59,15 +61,20 @@ function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
calls.proposeNode = [...(calls.proposeNode ?? []), [input]]
return { path: input.path, detectedReads: [], detectedWrites: [] }
},
editNode: async (path, content) => {
calls.editNode = [...(calls.editNode ?? []), [path, content]]
return { detectedReads: [], detectedWrites: [] }
editNode: async (path, content, summary) => {
calls.editNode = [...(calls.editNode ?? []), [path, content, summary]]
return { summary: summary ?? 'Existing summary', detectedReads: [], detectedWrites: [] }
},
removeProposedNode: record('removeProposedNode'),
unconfiguredTriggers: async () => [],
setNodeTrigger: async (path, kind, config) => {
calls.setNodeTrigger = [...(calls.setNodeTrigger ?? []), [path, kind, config]]
return { path: `${path}_${kind}`, replaced: false }
},
testNode: async () => 'job-123',
...overrides
}
return { helpers: { pipeline }, calls }
return { helpers: { pipelines: () => [pipeline] }, calls }
}
describe('pipeline tools', () => {
@@ -78,6 +85,7 @@ describe('pipeline tools', () => {
'get_pipeline_graph',
'read_pipeline_node',
'remove_pipeline_node',
'set_pipeline_trigger',
'test_pipeline_node'
])
})
@@ -96,18 +104,25 @@ describe('pipeline tools', () => {
it('build_pipeline_node forwards to proposeNode and does not deploy', async () => {
const { helpers, calls } = makeHelpers()
const modified: string[] = []
const out = await toolByName('build_pipeline_node').fn({
args: {
path: 'f/analytics/clean',
language: 'bun',
summary: 'Test node',
content: '// pipeline\nexport async function main() {}',
output_kind: 'ducklake'
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
toolCallbacks: {
...noopCallbacks(),
onItemModified: (kind, path) => modified.push(`${kind}:${path}`)
},
toolId: 't'
})
// The session tracks the folder's draft bundle, the unit it deploys.
expect(modified).toEqual(['data_pipeline:f/analytics/data_pipeline'])
expect(calls.proposeNode?.[0]?.[0]).toMatchObject({
path: 'f/analytics/clean',
language: 'bun',
@@ -116,6 +131,99 @@ describe('pipeline tools', () => {
expect(out).toContain('not deployed')
})
it('build_pipeline_node attaches declared triggers, and flags one left unset', async () => {
const { helpers, calls } = makeHelpers({ unconfiguredTriggers: async () => ['kafka'] })
const out = await toolByName('build_pipeline_node').fn({
args: {
path: 'f/analytics/ingest',
language: 'python3',
summary: 'Test node',
content: '# pipeline\n# on schedule\n# on kafka\ndef main():\n return 1',
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
toolId: 't'
})
expect(calls.setNodeTrigger?.[0]).toEqual([
'f/analytics/ingest',
'schedule',
{ args: {}, enabled: true, schedule: '0 0 6 * * *', timezone: 'UTC' }
])
expect(out).toContain("schedule 'f/analytics/ingest_schedule'")
expect(out).toContain('ACTION REQUIRED')
expect(out).toContain('set_pipeline_trigger')
})
it('build_pipeline_node rejects an invalid trigger config before staging the node', async () => {
const { helpers, calls } = makeHelpers()
await expect(
toolByName('build_pipeline_node').fn({
args: {
path: 'f/analytics/ingest',
language: 'python3',
summary: 'Test node',
content: '# pipeline\n# on schedule\ndef main():\n return 1',
triggers: [{ kind: 'schedule', config: { timezone: 'UTC' } }]
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
toolId: 't'
})
).rejects.toThrow(/schedule/)
expect(calls.proposeNode).toBeUndefined()
})
it('build_pipeline_node keeps and records the node when a trigger fails to attach', async () => {
const { helpers, calls } = makeHelpers({
setNodeTrigger: async () => {
throw new Error('path taken')
}
})
const modified: string[] = []
const out = await toolByName('build_pipeline_node').fn({
args: {
path: 'f/analytics/ingest',
language: 'python3',
content: '# pipeline\n# on schedule\ndef main():\n return 1',
summary: 'Ingest',
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
},
workspace: 'w',
helpers,
toolCallbacks: {
...noopCallbacks(),
onItemModified: (kind, path) => modified.push(`${kind}:${path}`)
},
toolId: 't'
})
expect(calls.proposeNode).toHaveLength(1)
expect(modified).toEqual(['data_pipeline:f/analytics/data_pipeline'])
expect(out).toContain('path taken')
})
it('build_pipeline_node refuses a trigger the content does not declare, before staging', async () => {
const { helpers, calls } = makeHelpers()
await expect(
toolByName('build_pipeline_node').fn({
args: {
path: 'f/analytics/ingest',
language: 'python3',
content: '# pipeline\ndef main():\n return 1',
summary: 'Ingest',
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
toolId: 't'
})
).rejects.toThrow(/on schedule/)
expect(calls.proposeNode).toBeUndefined()
})
it('build_pipeline_node reports the inferred asset lineage', async () => {
const { helpers } = makeHelpers({
proposeNode: async (input) => ({
@@ -125,7 +233,12 @@ describe('pipeline tools', () => {
})
})
const out = await toolByName('build_pipeline_node').fn({
args: { path: 'f/analytics/clean', language: 'duckdb', content: '-- pipeline' },
args: {
path: 'f/analytics/clean',
language: 'duckdb',
content: '-- pipeline',
summary: 'Test node'
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
@@ -140,7 +253,12 @@ describe('pipeline tools', () => {
proposeNode: async (input) => ({ path: input.path, detectedReads: [], detectedWrites: [] })
})
const out = await toolByName('build_pipeline_node').fn({
args: { path: 'f/analytics/clean', language: 'duckdb', content: '-- pipeline' },
args: {
path: 'f/analytics/clean',
language: 'duckdb',
content: '-- pipeline',
summary: 'Test node'
},
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
@@ -164,6 +282,34 @@ describe('pipeline tools', () => {
expect(calls.editNode?.[0]?.[1]).toContain('const x = 42')
})
it('edit_pipeline_node renames a node by summary alone, and asks for one when missing', async () => {
const body = { language: 'bun' as const, content: 'const x = 1\n' }
const { helpers, calls } = makeHelpers({ getNodeBody: async () => body })
const run = (args: Record<string, unknown>) =>
toolByName('edit_pipeline_node').fn({
args: { path: 'f/analytics/orders', ...args },
workspace: 'w',
helpers,
toolCallbacks: noopCallbacks(),
toolId: 't'
})
await run({ summary: 'Clean orders' })
expect(calls.editNode?.[0]).toEqual(['f/analytics/orders', body.content, 'Clean orders'])
const bare = makeHelpers({
getNodeBody: async () => body,
editNode: async () => ({ summary: '', detectedReads: [], detectedWrites: [] })
})
const out = await toolByName('edit_pipeline_node').fn({
args: { path: 'f/analytics/orders', old_string: 'x = 1', new_string: 'x = 2' },
workspace: 'w',
helpers: bare.helpers,
toolCallbacks: noopCallbacks(),
toolId: 't'
})
expect(out).toContain('has no summary')
})
it('edit_pipeline_node surfaces a clear error when old_string is absent', async () => {
const { helpers } = makeHelpers({
getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\n' })
@@ -182,7 +328,7 @@ describe('pipeline tools', () => {
it('mutation tools fail clearly when no pipeline editor is registered', async () => {
await expect(
toolByName('build_pipeline_node').fn({
args: { path: 'f/a/b', language: 'bun', content: 'x' },
args: { path: 'f/a/b', language: 'bun', content: 'x', summary: 'Test node' },
workspace: 'w',
helpers: {},
toolCallbacks: noopCallbacks(),
@@ -191,14 +337,114 @@ describe('pipeline tools', () => {
).rejects.toThrow(/No pipeline editor is open/)
})
it('remove_pipeline_node drops the pipeline from the changes once its last draft goes', async () => {
const run = async (hasDrafts: boolean) => {
const { helpers } = makeHelpers({ hasDrafts: () => hasDrafts })
const log: string[] = []
await toolByName('remove_pipeline_node').fn({
args: { path: 'f/analytics/clean' },
workspace: 'w',
helpers,
toolCallbacks: {
...noopCallbacks(),
onItemModified: (kind, path) => log.push(`modified ${kind}:${path}`),
onItemDiscarded: (kind, path) => log.push(`discarded ${kind}:${path}`)
},
toolId: 't'
})
return log
}
expect(await run(false)).toEqual(['discarded data_pipeline:f/analytics/data_pipeline'])
expect(await run(true)).toEqual(['modified data_pipeline:f/analytics/data_pipeline'])
})
it('test_pipeline_node requires confirmation', () => {
expect(toolByName('test_pipeline_node').requiresConfirmation).toBe(true)
})
})
describe('pipeline tools with several editors open', () => {
function editorFor(folder: string, built: string[]): PipelineAIChatHelpers {
return {
...makeHelpers().helpers.pipelines()[0],
getFolder: () => folder,
getPipelineContext: () => ({ ...sampleContext, folder }),
proposeNode: async (input) => {
built.push(`${folder}:${input.path}`)
return { path: input.path, detectedReads: [], detectedWrites: [] }
}
}
}
const call = (name: string, args: any, editors: PipelineAIChatHelpers[]) =>
toolByName(name).fn({
args,
workspace: 'w',
helpers: { pipelines: () => editors },
toolCallbacks: noopCallbacks(),
toolId: 't'
})
it('remounts an opened folder whose editor another tab replaced', async () => {
const built: string[] = []
const mounted: PipelineAIChatHelpers[] = []
const ensured: string[] = []
await toolByName('build_pipeline_node').fn({
args: {
path: 'f/sales/clean',
language: 'bun',
content: '// pipeline',
summary: 'Test node'
},
workspace: 'w',
helpers: {
pipelines: () => mounted,
pipelineFolders: () => ['sales'],
ensurePipeline: async (folder: string) => {
ensured.push(folder)
mounted.push(editorFor(folder, built))
return mounted[0]
}
},
toolCallbacks: noopCallbacks(),
toolId: 't'
})
expect(ensured).toEqual(['sales'])
expect(built).toEqual(['sales:f/sales/clean'])
})
it('routes a node to the editor of its folder', async () => {
const built: string[] = []
const editors = [editorFor('crm', built), editorFor('sales', built)]
await call(
'build_pipeline_node',
{ path: 'f/sales/clean', language: 'bun', content: '// pipeline', summary: 'Test node' },
editors
)
expect(built).toEqual(['sales:f/sales/clean'])
await expect(
call(
'build_pipeline_node',
{ path: 'f/other/clean', language: 'bun', content: '// pipeline', summary: 'Test node' },
editors
)
).rejects.toThrow(/open_preview\(kind="pipeline", path="other"\)/)
})
it('get_pipeline_graph needs the folder only when it is ambiguous', async () => {
const editors = [editorFor('crm', []), editorFor('sales', [])]
await expect(call('get_pipeline_graph', {}, editors)).rejects.toThrow(/pass the folder/)
expect(
JSON.parse(await call('get_pipeline_graph', { folder: 'f/sales' }, editors))
).toMatchObject({ folder: 'sales' })
expect(JSON.parse(await call('get_pipeline_graph', {}, [editors[0]]))).toMatchObject({
folder: 'crm'
})
})
})
describe('getPipelinePromptSection', () => {
it('names the active folder and the direct-draft workflow', () => {
const section = getPipelinePromptSection(sampleContext)
const section = getPipelinePromptSection([sampleContext.folder])
expect(section).toContain('/pipeline/analytics')
expect(section).toContain('build_pipeline_node')
expect(section).toContain('directly as unsaved drafts')
@@ -1,7 +1,8 @@
import { z } from 'zod'
import { $ScriptLang } from '$lib/gen/schemas.gen'
import type { ScriptLang } from '$lib/gen'
import { createToolDef, executeTestRun, findAndReplace } from '../shared'
import { createToolDef, executeTestRun, findAndReplace, type ToolCallbacks } from '../shared'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
import {
NONE,
RUN_PREVIEW,
@@ -10,6 +11,11 @@ import {
type SessionTool
} from '../sessionCapabilities'
import type { PipelineOutputKind } from '$lib/components/assets/AssetGraph/pipelineTemplates'
import type { PipelineTriggerDraftKind } from '$lib/components/assets/AssetGraph/types'
import { DRAFTABLE_TRIGGER_KINDS } from '$lib/components/assets/AssetGraph/pipelineTriggerDrafts'
import { scheduleRequestSchema, triggerRequestSchemas } from '../workspaceToolsZod.gen'
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
import { parsePipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations'
// ============================================================================
// Pipeline AI chat tools.
@@ -57,6 +63,10 @@ export type PipelineContext = {
* small — the page owns the draft Map and canvas rendering.
*/
export interface PipelineAIChatHelpers {
/** Folder name (no `f/` prefix) of the pipeline this editor shows. */
getFolder: () => string
/** Whether the folder has any script or trigger draft left. */
hasDrafts: () => boolean
getPipelineContext: () => PipelineContext
/** Read a node's source (the in-flight draft body if one exists, else deployed). */
getNodeBody: (path: string) => Promise<{ language: ScriptLang; content: string } | undefined>
@@ -68,29 +78,108 @@ export interface PipelineAIChatHelpers {
language: ScriptLang
content: string
outputKind?: PipelineOutputKind
summary?: string
}) => Promise<{ path: string; detectedReads: string[]; detectedWrites: string[] }>
/** Replace an existing node's body, applied as an unsaved draft. Returns the
* re-inferred asset lineage (URIs) so the caller sees the effect of the edit. */
editNode: (
path: string,
content: string
) => Promise<{ detectedReads: string[]; detectedWrites: string[] }>
content: string,
summary?: string
) => Promise<{ summary: string; detectedReads: string[]; detectedWrites: string[] }>
/** Discard the unsaved draft at a path (undo a build_pipeline_node). */
removeProposedNode: (path: string) => Promise<void>
/** Trigger kinds the node declares (`on <kind>`) that no trigger row or draft fills. */
unconfiguredTriggers: (path: string) => Promise<PipelineTriggerDraftKind[]>
/** Attach (or replace) the node's trigger draft of that kind, deployed with the
* pipeline. `config` is the kind's trigger config; path/script_path are filled in. */
setNodeTrigger: (
path: string,
kind: PipelineTriggerDraftKind,
config: Record<string, any>
) => Promise<{ path: string; replaced: boolean }>
/** Preview-run a node (draft body preferred). Returns the started job id. */
testNode: (path: string, args?: Record<string, any>) => Promise<string | undefined>
}
/** Helper bag the pipeline tools receive from the manager in global mode. */
export type PipelineToolHelpers = { pipeline?: PipelineAIChatHelpers }
/** Helper bag the pipeline tools receive from the manager in global mode: every
* pipeline editor currently open, one per folder. */
export type PipelineToolHelpers = {
pipelines?: () => readonly PipelineAIChatHelpers[]
/** Folders the chat may work on, including ones whose editor is not mounted now. */
pipelineFolders?: () => readonly string[]
/** Remounts the folder's editor (a session reopens its preview tab) and resolves
* to it, or `undefined` when it cannot be brought back. */
ensurePipeline?: (folder: string) => Promise<PipelineAIChatHelpers | undefined>
}
function requirePipeline(helpers: PipelineToolHelpers): PipelineAIChatHelpers {
if (!helpers?.pipeline) {
/** The folder's editor, remounted when the chat opened it before and something else
* has taken its tab since. */
async function remount(
helpers: PipelineToolHelpers,
folder: string | undefined
): Promise<PipelineAIChatHelpers | undefined> {
if (!folder || !helpers?.ensurePipeline) return undefined
if (!(helpers.pipelineFolders?.() ?? []).includes(folder)) return undefined
return helpers.ensurePipeline(folder)
}
function openPipelines(helpers: PipelineToolHelpers): readonly PipelineAIChatHelpers[] {
const open = helpers?.pipelines?.() ?? []
if (open.length === 0) {
throw new Error(
'No pipeline editor is open. Pipeline tools only work on a /pipeline/<folder> page in edit mode.'
)
}
return helpers.pipeline
return open
}
function notOpenError(folder: string, open: readonly PipelineAIChatHelpers[]): Error {
const names = open.map((p) => `'${p.getFolder()}'`).join(', ')
return new Error(
`No pipeline editor is open for folder '${folder}' (open: ${names}). Open it with open_preview(kind="pipeline", path="${folder}") first.`
)
}
/** The editor of the pipeline a node path belongs to (`f/<folder>/<node>`). With a
* single editor open, a path outside it still goes to that editor, whose own check
* names the corrected path. */
async function pipelineForPath(
helpers: PipelineToolHelpers,
path: string
): Promise<PipelineAIChatHelpers> {
const folder = path.match(/^f\/([^/]+)\//)?.[1]
const live = folder ? helpers?.pipelines?.().find((p) => p.getFolder() === folder) : undefined
const match = live ?? (await remount(helpers, folder))
if (match) return match
const open = openPipelines(helpers)
if (open.length === 1) return open[0]
throw folder
? notOpenError(folder, open)
: new Error(
`'${path}' is not a pipeline node path — nodes live at f/<folder>/<node_name>, with <folder> one of: ${open.map((p) => `'${p.getFolder()}'`).join(', ')}.`
)
}
async function pipelineForFolder(
helpers: PipelineToolHelpers,
folder: string | undefined
): Promise<PipelineAIChatHelpers> {
const known = helpers?.pipelineFolders?.() ?? []
const name = folder ? normalizePipelineFolder(folder) : known.length === 1 ? known[0] : undefined
const live = name ? helpers?.pipelines?.().find((p) => p.getFolder() === name) : undefined
const remounted = live ?? (await remount(helpers, name))
if (remounted) return remounted
const open = openPipelines(helpers)
if (folder) {
const match = open.find((p) => p.getFolder() === name)
if (!match) throw notOpenError(normalizePipelineFolder(folder), open)
return match
}
if (open.length === 1) return open[0]
throw new Error(
`Several pipeline editors are open (${open.map((p) => `'${p.getFolder()}'`).join(', ')}) — pass the folder to read.`
)
}
const scriptLangSchema = z.enum($ScriptLang.enum)
@@ -105,12 +194,20 @@ const outputKindSchema = z
// Read tools
// ----------------------------------------------------------------------------
const getPipelineGraphSchema = z.object({})
const getPipelineGraphSchema = z.object({
folder: z
.string()
.optional()
.describe(
'Folder of the pipeline to read (e.g. `analytics`). Required when several pipeline editors are open; may be omitted when only one is.'
)
})
const getPipelineGraphToolDef = createToolDef(
getPipelineGraphSchema,
'get_pipeline_graph',
"Read the live pipeline graph for the open /pipeline/<folder> editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG."
"Read the live pipeline graph of an open /pipeline/<folder> editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG.",
{ strict: false }
)
const readPipelineNodeSchema = z.object({
@@ -127,11 +224,75 @@ const readPipelineNodeToolDef = createToolDef(
// Mutation tools (apply directly as unsaved drafts; never deploy)
// ----------------------------------------------------------------------------
const triggerKindEnum = z.enum(DRAFTABLE_TRIGGER_KINDS as [PipelineTriggerDraftKind, ...PipelineTriggerDraftKind[]])
const nodeTriggerSchema = z.object({
kind: triggerKindEnum.describe('Trigger kind; the node must declare `on <kind>`.'),
config: z
.record(z.string(), z.any())
.describe(
'The trigger configuration. schedule: { schedule (6-field cron, e.g. "0 0 6 * * *"), timezone (IANA, e.g. "UTC"), args? (the node\'s inputs), summary? }. Any other kind: the fields get_trigger_schema returns for it. path, script_path and is_flow are filled in; pass path only to name the trigger yourself.'
)
})
const setPipelineTriggerSchema = nodeTriggerSchema.extend({
path: z.string().describe('Workspace path of the node the trigger runs.')
})
const setPipelineTriggerToolDef = createToolDef(
setPipelineTriggerSchema,
'set_pipeline_trigger',
'Attach the trigger a node declares with `on <schedule|email|kafka|mqtt|amqp|nats|postgres|sqs|gcp>` (its cron, topic, queue…) as an unsaved trigger draft, deployed together with the pipeline. Calling it again for the same node and kind replaces that draft. Webhook and data_upload need no trigger.',
{ strict: false }
)
/** Validates a node trigger's config against the kind's request schema, with the
* fields the pipeline fills in present so they are not reported missing. */
function validateTriggerConfig(
kind: PipelineTriggerDraftKind,
config: Record<string, any>,
nodePath: string
): Record<string, any> {
const filled = { path: `${nodePath}_${kind}`, script_path: nodePath, is_flow: false, ...config }
if (kind === 'schedule') {
const parsed = scheduleRequestSchema.safeParse({ args: {}, enabled: true, ...filled })
if (!parsed.success) {
throw new Error(
`Invalid schedule config: ${parsed.error.issues.map((i) => `${i.path.join('.') || '<root>'}: ${i.message}`).join('; ')}`
)
}
return { args: {}, enabled: true, ...config }
}
const parsed = triggerRequestSchemas[kind].safeParse(filled)
if (!parsed.success) {
throw new Error(
`Invalid config for a "${kind}" trigger. Call get_trigger_schema with kind "${kind}" for its exact fields. Issues: ${parsed.error.issues.map((i) => `${i.path.join('.') || '<root>'}: ${i.message}`).join('; ')}`
)
}
return config
}
/** The note every node write ends with while a declared trigger is still unset: the
* node would deploy with nothing to start it. */
function unconfiguredTriggersNote(path: string, kinds: readonly string[]): string {
if (kinds.length === 0) return ''
return ` ACTION REQUIRED: '${path}' declares \`on ${kinds.join('`, `on ')}\` but has no ${kinds.join('/')} trigger yet, so nothing would start it once deployed. Call set_pipeline_trigger for it now (a schedule needs its cron and timezone; ask the user only if the cadence cannot be inferred).`
}
const summaryField = z
.string()
.trim()
.min(1)
.max(80)
.describe(
'Succinct summary of what the node does, a few words in sentence case with no trailing period (e.g. "Ingest Stripe charges", "Daily revenue by region"). It is the title the node shows in the workspace.'
)
const buildPipelineNodeSchema = z.object({
path: z
.string()
.describe(
"Workspace path for the new node, e.g. f/<folder>/<name>. Use the open pipeline's folder. Must not collide with an existing node."
'Workspace path for the new node, e.g. f/<folder>/<name>, in the folder of an open pipeline editor. Must not collide with an existing node.'
),
language: scriptLangSchema.describe(
'Script language. SQL-shaped data work uses duckdb (DuckLake/S3) or postgresql (data tables); bun/python3 for general transforms.'
@@ -141,7 +302,14 @@ const buildPipelineNodeSchema = z.object({
.describe(
"Full script source. Start it with the `pipeline` annotation as a top-of-file comment in the LANGUAGE'S comment syntax — `-- pipeline` for SQL (duckdb/postgresql), `# pipeline` for python3/bash, `// pipeline` for bun/TS — to mark it a pipeline member; declare inputs the same way (e.g. `-- on <asset-uri|schedule|webhook|...>`), and write outputs via the wmill SDK / SQL so the lineage edges are inferred. A `// pipeline` line in a SQL node is a syntax error. Read existing node bodies first to match conventions."
),
output_kind: outputKindSchema.optional()
summary: summaryField,
output_kind: outputKindSchema.optional(),
triggers: z
.array(nodeTriggerSchema)
.optional()
.describe(
'Triggers for the `on <kind>` lines the content declares (schedule, email, kafka, …), attached as trigger drafts in the same step. Every such declaration needs one; webhook and data_upload need none.'
)
})
const buildPipelineNodeToolDef = createToolDef(
@@ -153,8 +321,17 @@ const buildPipelineNodeToolDef = createToolDef(
const editPipelineNodeSchema = z.object({
path: z.string().describe('Workspace path of the node to edit.'),
old_string: z.string().min(1).describe("Exact text to find in the node's current source."),
new_string: z.string().describe('Replacement text.'),
old_string: z
.string()
.min(1)
.optional()
.describe("Exact text to find in the node's current source. Omit to change only the summary."),
new_string: z.string().optional().describe('Replacement text.'),
summary: summaryField
.optional()
.describe(
'New succinct summary (a few words, no trailing period). Pass it whenever the node has none or the edit changes what the node does.'
),
replace_all: z
.boolean()
.optional()
@@ -197,6 +374,12 @@ const testPipelineNodeToolDef = createToolDef(
{ strict: false }
)
// A pipeline's drafts are one bundle for the whole folder, so that bundle is the
// item a node edit marks as modified — it is what the session deploys as a unit.
function recordPipelineModified(toolCallbacks: ToolCallbacks, pipeline: PipelineAIChatHelpers) {
toolCallbacks.onItemModified?.(PIPELINE_DRAFT_KIND, pipelineBundlePath(pipeline.getFolder()))
}
// Summarize the asset lineage the parser inferred from a just-applied node so the
// model gets same-turn feedback on whether its intended edges formed. An empty
// result is the useful signal: a write/read expressed via a variable or dynamic
@@ -217,8 +400,9 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
requires: NONE,
def: getPipelineGraphToolDef,
planModeSafe: true,
fn: async ({ helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const { folder } = getPipelineGraphSchema.parse(args ?? {})
const pipeline = await pipelineForFolder(helpers, folder)
toolCallbacks.setToolStatus(toolId, { content: 'Reading pipeline graph...' })
const ctx = pipeline.getPipelineContext()
toolCallbacks.setToolStatus(toolId, {
@@ -233,8 +417,8 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
def: readPipelineNodeToolDef,
planModeSafe: true,
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
const { path } = readPipelineNodeSchema.parse(args)
const pipeline = await pipelineForPath(helpers, path)
toolCallbacks.setToolStatus(toolId, { content: `Reading node '${path}'...` })
const node = await pipeline.getNodeBody(path)
if (!node) {
@@ -251,20 +435,51 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
showDetails: true,
showFade: true,
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
const { path, language, content, output_kind } = buildPipelineNodeSchema.parse(args)
const { path, language, content, summary, output_kind, triggers } =
buildPipelineNodeSchema.parse(args)
const pipeline = await pipelineForPath(helpers, path)
// Checked before the node is staged, so a bad trigger leaves nothing half-built.
const declared = new Set(parsePipelineAnnotations(content).nativeTriggers.map((n) => n.kind))
const configs = (triggers ?? []).map((t) => {
if (!declared.has(t.kind)) {
throw new Error(
`The content declares no \`on ${t.kind}\`, so a ${t.kind} trigger has nothing to bind to. Add the annotation line or drop that trigger. Nothing was built.`
)
}
return { kind: t.kind, config: validateTriggerConfig(t.kind, t.config, path) }
})
toolCallbacks.setToolStatus(toolId, { content: `Building node '${path}'...` })
const { detectedReads, detectedWrites } = await pipeline.proposeNode({
path,
language: language as ScriptLang,
content,
summary,
outputKind: output_kind as PipelineOutputKind | undefined
})
// Recorded as soon as it is staged: a trigger that fails below leaves the node.
recordPipelineModified(toolCallbacks, pipeline)
const attached: string[] = []
const failed: string[] = []
for (const t of configs) {
try {
const { path: triggerPath } = await pipeline.setNodeTrigger(path, t.kind, t.config)
attached.push(`${t.kind} '${triggerPath}'`)
} catch (e) {
failed.push(`${t.kind}: ${e instanceof Error ? e.message : String(e)}`)
}
}
toolCallbacks.setToolStatus(toolId, {
content: `Added draft node '${path}'`,
result: 'Success'
})
return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.${inferredLineageNote(detectedReads, detectedWrites)}`
const attachedNote =
(attached.length
? ` Attached trigger draft${attached.length > 1 ? 's' : ''}: ${attached.join(', ')}.`
: '') +
(failed.length
? ` The node was built, but these triggers were not attached — fix them with set_pipeline_trigger: ${failed.join('; ')}.`
: '')
return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.${attachedNote}${inferredLineageNote(detectedReads, detectedWrites)}${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
}
},
{
@@ -274,36 +489,74 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
showDetails: true,
showFade: true,
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
const { path, old_string, new_string, replace_all } = editPipelineNodeSchema.parse(args)
const { path, old_string, new_string, replace_all, summary } =
editPipelineNodeSchema.parse(args)
if (old_string === undefined && summary === undefined) {
throw new Error('Pass old_string/new_string to change the source, summary to rename it, or both.')
}
const pipeline = await pipelineForPath(helpers, path)
const node = await pipeline.getNodeBody(path)
if (!node) {
return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.`
}
toolCallbacks.setToolStatus(toolId, { content: `Editing node '${path}'...` })
const updated = findAndReplace(
node.content,
old_string,
new_string,
replace_all ?? false,
'node source'
)
const { detectedReads, detectedWrites } = await pipeline.editNode(path, updated)
const updated =
old_string === undefined
? node.content
: findAndReplace(
node.content,
old_string,
new_string ?? '',
replace_all ?? false,
'node source'
)
const edited = await pipeline.editNode(path, updated, summary)
const { detectedReads, detectedWrites } = edited
const summaryNote = edited.summary.trim()
? ''
: ` '${path}' has no summary: call edit_pipeline_node with just \`summary\` (a few words saying what it does).`
recordPipelineModified(toolCallbacks, pipeline)
toolCallbacks.setToolStatus(toolId, {
content: `Edited draft '${path}'`,
result: 'Success'
})
return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).${inferredLineageNote(detectedReads, detectedWrites)}`
return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).${inferredLineageNote(detectedReads, detectedWrites)}${summaryNote}${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
}
},
{
requires: WRITE_DRAFT,
def: setPipelineTriggerToolDef,
showDetails: true,
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const { path, kind, config } = setPipelineTriggerSchema.parse(args)
const pipeline = await pipelineForPath(helpers, path)
toolCallbacks.setToolStatus(toolId, { content: `Setting the ${kind} trigger of '${path}'...` })
const valid = validateTriggerConfig(kind, config, path)
const { path: triggerPath, replaced } = await pipeline.setNodeTrigger(path, kind, valid)
recordPipelineModified(toolCallbacks, pipeline)
toolCallbacks.setToolStatus(toolId, {
content: `${replaced ? 'Updated' : 'Attached'} ${kind} trigger '${triggerPath}'`,
result: 'Success'
})
return `${replaced ? 'Replaced' : 'Attached'} the ${kind} trigger draft '${triggerPath}' on '${path}'. It deploys with the pipeline.${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
}
},
{
requires: WRITE_DRAFT,
def: removePipelineNodeToolDef,
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
const { path } = removePipelineNodeSchema.parse(args)
const pipeline = await pipelineForPath(helpers, path)
toolCallbacks.setToolStatus(toolId, { content: `Discarding draft '${path}'...` })
await pipeline.removeProposedNode(path)
// With the last draft gone there is nothing left to deploy: the pipeline
// is no longer this chat's change, and listing it would show it deployed.
if (pipeline.hasDrafts()) recordPipelineModified(toolCallbacks, pipeline)
else
toolCallbacks.onItemDiscarded?.(
PIPELINE_DRAFT_KIND,
pipelineBundlePath(pipeline.getFolder())
)
toolCallbacks.setToolStatus(toolId, {
content: `Discarded draft '${path}'`,
result: 'Success'
@@ -319,8 +572,8 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
showDetails: true,
autoCollapseDetails: false,
fn: async ({ args, workspace, helpers, toolId, toolCallbacks }) => {
const pipeline = requirePipeline(helpers)
const { path, args: runArgs } = testPipelineNodeSchema.parse(args)
const pipeline = await pipelineForPath(helpers, path)
return executeTestRun({
jobStarter: async () => {
const jobId = await pipeline.testNode(path, runArgs ?? undefined)
@@ -350,22 +603,35 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
* since the tools it names are withheld from that session; the annotation model stays,
* as it is what lets the model read and explain an existing pipeline.
*/
export function getPipelinePromptSection(ctx: PipelineContext, access?: SessionAccess): string {
export function getPipelinePromptSection(
folders: readonly string[],
access?: SessionAccess
): string {
const canWriteDraft = !access || access.has('write_draft')
const [first] = folders
const openLine =
folders.length === 1
? `The user has the /pipeline/${first} editor open.`
: `The user has ${folders.length} pipeline editors open: ${folders.map((f) => `/pipeline/${f}`).join(', ')}. Each tool acts on the pipeline its node path's folder names; pass \`folder\` to get_pipeline_graph.`
const pathLine =
folders.length === 1
? `Every node of this pipeline lives at \`f/${first}/<node_name>\` — \`${first}\` is the folder name and \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`).`
: `Every node of a pipeline lives at \`f/<folder>/<node_name>\` (e.g. \`f/${first}/<node_name>\`) — \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`). A node can only be built in the folder of an open pipeline editor.`
return `
Data Pipeline editor (ACTIVE):
- The user has the /pipeline/${ctx.folder} editor open. A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers.
- ${openLine} A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers.
- Annotations are top-of-file comments in the NODE'S OWN comment syntax: \`--\` for SQL (duckdb/postgresql), \`#\` for python3/bash, \`//\` for bun/TS. The \`//\` shown below is the TS form — translate it (a \`// pipeline\` line in a SQL node is a syntax error that won't deploy).
- A script becomes a pipeline node when its source starts with the \`// pipeline\` annotation. Declare execution-DAG inputs with \`// on <asset-uri | schedule | webhook | email | kafka | mqtt | amqp | nats | postgres | sqs | gcp | data_upload>\` (e.g. \`// on ducklake://main/orders\`). Outputs are inferred from what the body writes (wmill SDK calls / SQL CREATE TABLE / writeS3File); declare a managed output with \`// materialize <asset-uri>\`. Optional badges: \`// partitioned <daily|hourly|weekly|monthly|dynamic>\`, \`// freshness <duration>\`, \`// tag <name>\`, \`// retry <count> [delay]\`, \`// data_test <kind> ...\` (managed DuckLake targets only — deploy rejects it beside a \`dbt://\` target), \`// measure <name> = <agg> [where <pred>]\`, \`// dimension <name> = <expr>\`.
- \`materialize\` (the managed output): a managed \`// materialize ducklake://<name>/<table>\` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. The \`dbt://\` target below is the opposite: the node writes its own DDL and none of the write strategies apply to it. IMPORTANT: a MANAGED \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake://<name>/<table>\`) — deploy rejects a \`ducklake://\` target on any other language. For a \`python3\`/\`bun\`/\`postgresql\` node writing the lake, do NOT use \`// materialize\`; write the output via the SDK instead (e.g. \`wmill.writeS3File(...)\`, a \`CREATE TABLE\` in postgresql, or \`wmill.databaseUrlFromResource\`/ducklake helpers) and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. The one target any language BUT DBT'S OWN may declare (a dbt project's writes come from its manifest, so \`// materialize\` on a dbt script is rejected at deploy) is a WAREHOUSE RELATION: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, with \`<warehouse>\` a warehouse the workspace configures under Settings → dbt. \`manual\` is its only mode — nothing generates warehouse DDL, so the node issues its own write and the annotation records the outcome. Use it on an ingestion node a dbt project reads as a \`source\`: the declared relation and the dbt model become ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when that node completes. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize <uri> append\` INSERT-appends rows (incremental); \`// materialize <uri> key=<col>\` merges/upserts on \`<col>\`. \`// materialize manual <uri>\` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned <daily|hourly|weekly|monthly|dynamic>\` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token — usable in any asset URI AND in the body SQL — is substituted with the current partition's IDENTITY string at run time. To filter the source to the active slice on a time grain, use the runtime-injected macro: \`WHERE wm_partition(<ts_col>) = {partition}\`. \`wm_partition(ts)\` buckets a timestamp with the exact identity format the runtime used (daily/hourly/weekly/monthly), so it always matches and you never hand-write a \`strftime\` format. Do NOT write \`= TIMESTAMP {partition}\`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at runtime. For \`dynamic\` partitioning the identity is your caller-supplied key (not a timestamp, no macro), so filter on it directly: \`WHERE <your_key_col> = {partition}\`. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run".
- \`materialize\` (the managed output): a managed \`// materialize ducklake://<name>/<table>\` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. The \`dbt://\` target below is the opposite: the node writes its own DDL and none of the write strategies apply to it. IMPORTANT: a MANAGED \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake://<name>/<table>\`) — deploy rejects a \`ducklake://\` target on any other language. For a \`python3\`/\`bun\`/\`postgresql\` node writing the lake, do NOT use \`// materialize\`; write the table straight into DuckLake with \`wmill.ducklake()\` (or a data table with \`wmill.datatable()\`), or a \`CREATE TABLE\` in postgresql and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. The one target any language BUT DBT'S OWN may declare (a dbt project's writes come from its manifest, so \`// materialize\` on a dbt script is rejected at deploy) is a WAREHOUSE RELATION: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, with \`<warehouse>\` a warehouse the workspace configures under Settings → dbt. \`manual\` is its only mode — nothing generates warehouse DDL, so the node issues its own write and the annotation records the outcome. Use it on an ingestion node a dbt project reads as a \`source\`: the declared relation and the dbt model become ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when that node completes. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize <uri> append\` INSERT-appends rows (incremental); \`// materialize <uri> key=<col>\` merges/upserts on \`<col>\`. \`// materialize manual <uri>\` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned <daily|hourly|weekly|monthly|dynamic>\` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token — usable in any asset URI AND in the body SQL — is substituted with the current partition's IDENTITY string at run time. To filter the source to the active slice on a time grain, use the runtime-injected macro: \`WHERE wm_partition(<ts_col>) = {partition}\`. \`wm_partition(ts)\` buckets a timestamp with the exact identity format the runtime used (daily/hourly/weekly/monthly), so it always matches and you never hand-write a \`strftime\` format. Do NOT write \`= TIMESTAMP {partition}\`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at runtime. For \`dynamic\` partitioning the identity is your caller-supplied key (not a timestamp, no macro), so filter on it directly: \`WHERE <your_key_col> = {partition}\`. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run".
- \`measure\` / \`dimension\` (declared metrics): on a node that materializes a DuckLake table, \`// measure <name> = <aggregate> [where <predicate>]\` names the canonical way to aggregate that table (e.g. \`// measure revenue = sum(amount) where not is_refund\`), and \`// dimension <name> = <expr>\` names a way to slice it (e.g. \`// dimension region = region\`, \`// dimension month = date_trunc('month', ordered_at)\`). They execute nothing: they are catalogued at deploy so the editor and other agents can reuse the definition instead of re-deriving it and silently disagreeing. Keep the predicate in the \`where\` clause rather than folding it into the aggregate: it is rendered as \`<agg> FILTER (WHERE <pred>)\`, which is what lets two measures with different predicates sit under one GROUP BY. DuckLake-only, and only meaningful next to \`// materialize\`. Declare one when a number carries a judgement call someone else would get wrong (refunds excluded, test rows dropped, which column is the amount); do NOT blanket every table with measures, an obvious \`count(*)\` earns nothing. To USE a metric another node declares, read that node with read_pipeline_node and reuse its exact expression rather than guessing it.
- Use get_pipeline_graph to see the current nodes/assets/triggers, and read_pipeline_node before editing one.
- Every node of this pipeline lives at \`f/${ctx.folder}/<node_name>\` — \`${ctx.folder}\` is the folder name and \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`).${
- ${pathLine}${
canWriteDraft
? `
- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open.
- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. Give every node a succinct \`summary\` (a few words saying what it does, e.g. "Ingest Stripe charges"), and update it when an edit changes what the node does. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open.
- Reuse existing asset paths from the graph when wiring a downstream node to an upstream one (read the upstream's write asset, then \`// on\` that same URI).
- TRIGGERS: an ingestion node (one with no upstream asset to react to) must declare how it starts, e.g. \`// on schedule\`, and every \`on <schedule|email|kafka|mqtt|amqp|nats|postgres|sqs|gcp>\` line needs its trigger: pass it in build_pipeline_node's \`triggers\` (or call set_pipeline_trigger) in the same step you build the node. A schedule needs a 6-field cron and an IANA timezone — pick a sensible cadence from the request (e.g. daily at 06:00 UTC for a daily sync) rather than leaving it empty; for the other kinds call get_trigger_schema(kind) for the config fields and ask the user for any connection/resource you cannot find. Never leave a declared trigger without one: the node would deploy with nothing to start it. \`webhook\` and \`data_upload\` need no trigger.
- Only deploy when the user explicitly asks; the user deploys drafts from the canvas.`
: ''
}`
@@ -395,6 +395,30 @@ describe('processToolCall', () => {
expect(result.content).toBe(`Error while calling tool: ${expectedError}`)
})
it('resolves a tool registered earlier in the same batch from the live set', async () => {
const { createToolDef, processToolCall } = await import('./shared')
const fn = vi.fn().mockResolvedValue('built')
const registered = {
def: createToolDef(z.object({}), 'build_pipeline_node', 'Build node'),
fn
}
const result = await processToolCall({
tools: [],
live: () => ({ tools: [registered], helpers: { pipelines: true } }),
toolCall: {
id: 'call_live',
type: 'function',
function: { name: 'build_pipeline_node', arguments: '{}' }
},
helpers: {},
workspace: 'test-workspace',
toolCallbacks: { setToolStatus: vi.fn(), removeToolStatus: vi.fn() }
})
expect(result.content).toBe('built')
expect(fn).toHaveBeenCalledWith(expect.objectContaining({ helpers: { pipelines: true } }))
})
it('continues to confirmation when pre-confirmation validation passes', async () => {
const { createToolDef, processToolCall } = await import('./shared')
const fn = vi.fn().mockResolvedValue('ok')
@@ -834,6 +834,13 @@ async function callTool<T>({
}): Promise<string> {
const tool = tools.find((t) => t.def.function.name === functionName)
if (!tool) {
// The pipeline tools exist only while a pipeline editor is mounted, and change_mode
// cannot mount one, so the generic hint would send the model probing.
if (/_pipeline_node$|^get_pipeline_graph$/.test(functionName)) {
throw new Error(
`${functionName} is not available yet: no pipeline editor is open for this chat. Call open_preview(kind="pipeline", path="<folder>") first and wait for its result; once it reports the editor open, the pipeline tools are callable. Do not fall back to write_script.`
)
}
throw new Error(
`Unknown tool call: ${functionName}. Probably not in the correct mode, use the change_mode tool to switch to the correct mode.`
)
@@ -924,22 +931,33 @@ type ToolCallStatus =
| 'blocked_plan_mode'
| 'held_for_instructions'
/** Re-reads the chat loop's current tools and helpers. A call earlier in the same
* response can register tools (open_preview mounting the pipeline editor, change_mode),
* so a later call in that batch must resolve against the live set, not the snapshot the
* response was requested with. */
export type LiveToolSet = () => { tools: Tool<any>[]; helpers: any }
export async function processToolCall<T>({
tools,
tools: snapshotTools,
toolCall,
helpers,
helpers: snapshotHelpers,
toolCallbacks,
workspace,
messages = []
messages = [],
live
}: {
tools: Tool<T>[]
toolCall: ChatCompletionMessageFunctionToolCall
helpers: T
live?: LiveToolSet
toolCallbacks: ToolCallbacks
workspace?: string
/** The conversation so far, for which folder instructions it already carries. */
messages?: readonly ChatCompletionMessageParam[]
}): Promise<ChatCompletionMessageParam> {
const current = live?.()
const tools: Tool<T>[] = current?.tools ?? snapshotTools
const helpers: T = current ? current.helpers : snapshotHelpers
const tool = tools.find((t) => t.def.function.name === toolCall.function.name)
const workspaceId = workspace ?? get(workspaceStore) ?? ''
+4 -1
View File
@@ -32,6 +32,7 @@ import { formatResourceTypes } from './utils'
import {
appendPendingToolImages,
processToolCall,
type LiveToolSet,
queuedToolStatus,
type Tool,
type ToolCallbacks
@@ -1372,6 +1373,7 @@ export async function parseOpenAICompletion(
workspace?: string
provider?: string
onTokenUsage?: (usage: ChatTokenUsage) => void
live?: LiveToolSet
}
): Promise<{ shouldContinue: boolean; tokenUsage: ChatTokenUsage }> {
const finalToolCalls: Record<number, ChatCompletionChunk.Choice.Delta.ToolCall> = {}
@@ -1583,7 +1585,8 @@ export async function parseOpenAICompletion(
helpers,
toolCallbacks: callbacks,
workspace: options?.workspace,
messages
messages,
live: options?.live
})
messages.push(messageToAdd)
addedMessages.push(messageToAdd)
@@ -3,6 +3,7 @@
import { Handle, Position } from '@xyflow/svelte'
import { twMerge } from 'tailwind-merge'
import ContextMenu, { type ContextMenuItem } from '../../../common/contextmenu/ContextMenu.svelte'
import { contextMenuItemsFromMenu } from '../../../common/contextmenu/fromMenuItems'
import { getGraphContext } from '../../graphContext'
import type { Item } from '$lib/utils'
@@ -28,21 +29,7 @@
}: Props = $props()
let resolvedContextMenuItems: ContextMenuItem[] | undefined = $derived(
contextMenuItems ??
menuItems?.flatMap((item) => [
...(item.separatorTop
? [{ id: `${item.displayName}-divider`, label: '', divider: true }]
: []),
{
id: item.displayName,
label: item.displayName,
icon: item.icon,
disabled: item.disabled,
type: item.type,
shortcut: item.shortcut,
onClick: item.action as (() => void) | undefined
}
])
contextMenuItems ?? (menuItems && contextMenuItemsFromMenu(menuItems))
)
// NodeWrapper is reused outside the flow graph (e.g. the app decision-tree
@@ -20,6 +20,7 @@
PanelLeftClose
} from 'lucide-svelte'
import BarsStaggered from '$lib/components/icons/BarsStaggered.svelte'
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
import { PythonIcon, TypeScriptIcon } from '$lib/components/common/languageIcons'
import { HOME_SHOW_CREATE_FLOW, HOME_SHOW_CREATE_APP } from '$lib/consts'
import { importFlowStore } from '$lib/components/flows/flowStore.svelte'
@@ -55,6 +56,23 @@
label: string
icon: typeof PythonIcon
onSelect: () => void
badge?: { label: string; class: string }
/** opened in a new tab from a link beside the variant */
docHref?: string
/** shown in the doc panel while the variant is hovered, instead of its option's */
doc?: Doc
}
/** what the doc panel shows */
type Doc = {
label: string
/** a lucide icon or a brand mark; rendered at 26px through both size APIs */
icon: any
accent: string
tagline: string
description: string
bullets: string[]
badge?: { label: string; class: string }
}
/** an importable artifact kind handled by the shared YAML/JSON import drawer */
@@ -87,6 +105,9 @@
const badgeAdvanced = 'bg-gray-100 text-gray-600 dark:bg-gray-700 dark:text-gray-300'
const badgeLegacy = 'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300'
const badgeAlpha = 'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300'
const badgeRecommended = 'bg-blue-100 text-blue-700 dark:bg-blue-900/40 dark:text-blue-300'
const DBT_DOCS = 'https://www.windmill.dev/docs/getting_started/scripts_quickstart/dbt'
const allOptions: Option[] = [
{
@@ -200,7 +221,34 @@
'Asset-aware lineage'
],
onSelect: () => goto(`${base}/pipeline`),
badge: { label: 'Alpha', class: badgeAlpha }
badge: { label: 'Alpha', class: badgeAlpha },
variants: [
{
label: 'Windmill pipelines',
icon: Workflow,
badge: { label: 'Recommended', class: badgeRecommended },
onSelect: () => goto(`${base}/pipeline`)
},
{
label: 'dbt',
icon: DbtIcon,
docHref: DBT_DOCS,
onSelect: () => goto(`${base}/scripts/add?lang=dbt`),
doc: {
label: 'dbt',
icon: DbtIcon,
accent: 'orange',
tagline: 'Run an existing dbt project',
description:
'A pipeline is Windmill’s own abstraction; an existing dbt project stays a dbt project, and runs here unchanged as its own kind of script, with dbt still owning its models, refs and tests. All it needs is a warehouse configured under Settings → dbt. Its models show up in the asset graph as dbt:// nodes, next to whatever pipelines you do build, so you can adopt pipelines later, or never.',
bullets: [
'Your dbt project, unchanged',
'Needs a warehouse under Settings → dbt',
'Models appear as dbt:// assets'
]
}
}
]
}
] as Option[])
: []),
@@ -301,18 +349,25 @@
// per-row fan-out submenus: open to the right when there's room. The popover hugs
// the right viewport edge on most screens, so the default flip (right → left) would
// cover the doc panel — fall back below the trigger row instead.
const variantSubmenu = () =>
builders.createSubmenu({
positioning: {
placement: 'right-start',
gutter: 4,
flip: { fallbackPlacements: ['bottom-end', 'bottom-start', 'top-end', 'top-start'] },
fitViewport: true,
overflowPadding: 8
}
})
const {
elements: { subTrigger: wacSubTrigger, subMenu: wacSubMenu },
states: { subOpen: wacSubOpen }
} = builders.createSubmenu({
positioning: {
placement: 'right-start',
gutter: 4,
flip: { fallbackPlacements: ['bottom-end', 'bottom-start', 'top-end', 'top-start'] },
fitViewport: true,
overflowPadding: 8
}
})
} = variantSubmenu()
const {
elements: { subTrigger: pipelineSubTrigger, subMenu: pipelineSubMenu },
states: { subOpen: pipelineSubOpen }
} = variantSubmenu()
let variantSubOpen = $derived($wacSubOpen || $pipelineSubOpen)
const {
elements: { subTrigger: importSubTrigger, subMenu: importSubMenu },
states: { subOpen: importSubOpen }
@@ -378,7 +433,10 @@
if (isOpen && !wasOpen) {
logFeatureUsage('home', 'new_menu_open', { key: source })
}
if (!isOpen) activeKey = undefined
if (!isOpen) {
activeKey = undefined
activeVariantDoc = undefined
}
wasOpen = isOpen
})
@@ -389,17 +447,25 @@
// only persist the non-default (hidden) state, so a cleared key means "shown"
storeLocalSetting(SHOW_DOC_SETTING, value ? undefined : 'false')
}
// The pointer crosses the gutter outside the menu on its way into the Workflow-as-Code
// submenu, so leaving the menu keeps the panel while that submenu is open; it clears
// once the submenu closes with neither pointer nor focus left on the menu.
// The pointer crosses the gutter outside the menu on its way into a variant submenu,
// so leaving the menu keeps the panel while that submenu is open; it clears once the
// submenu closes with neither pointer nor focus left on the menu.
let menuEl: HTMLDivElement | undefined = $state(undefined)
$effect(() => {
if ($wacSubOpen || !menuEl) return
if (variantSubOpen || !menuEl) return
if (!menuEl.matches(':hover') && !menuEl.contains(document.activeElement)) {
activeKey = undefined
}
})
let active = $derived(options.find((o) => o.key === activeKey))
// A hovered variant with its own doc (dbt under Data pipelines) takes over the panel.
let activeVariantDoc: Doc | undefined = $state(undefined)
function highlightOption(key: string | undefined) {
activeKey = key
activeVariantDoc = undefined
}
let active: Doc | undefined = $derived(
activeVariantDoc ?? options.find((o) => o.key === activeKey)
)
let activeAc = $derived(active ? accentClasses[active.accent] : undefined)
// shared YAML/JSON import drawer, reused by every "Import …" extra
@@ -476,7 +542,7 @@
style={showDoc ? 'width: 780px;' : ''}
bind:this={menuEl}
onpointerleave={() => {
if (!$wacSubOpen) activeKey = undefined
if (!variantSubOpen) highlightOption(undefined)
}}
>
<div
@@ -492,7 +558,7 @@
<div
class="w-12 h-12 rounded-xl flex items-center justify-center shrink-0 {activeAc.tile}"
>
<active.icon size={26} class={activeAc.iconText} />
<active.icon size={26} width={26} height={26} class={activeAc.iconText} />
</div>
<div class="min-w-0">
<div class="flex flex-row items-center gap-2">
@@ -551,48 +617,92 @@
</span>
{/if}
{/snippet}
{#snippet variantRow(
option: Option,
ac: (typeof accentClasses)[string],
rowClass: string,
subTrigger: any,
subMenu: any,
subOpen: boolean
)}
<button
use:melt={subTrigger}
class={rowClass}
onfocusin={() => highlightOption(option.key)}
onpointerenter={() => highlightOption(option.key)}
>
{@render rowBody(option, ac)}
<ChevronRight size={14} class="shrink-0 text-tertiary" />
</button>
{#if subOpen}
<!-- The invisible `before:` margin bridges the gutter to the trigger row: when
the submenu falls below its row, that gutter lies over the next row, and
crossing it highlights that item, which closes the submenu. -->
<div
use:melt={subMenu}
use:hugViewportRight
class="pointer-events-auto z-[6001] flex flex-col gap-0.5 p-1 min-w-52 w-max rounded-lg border border-gray-200 dark:border-gray-700 bg-surface shadow-xl focus:outline-none before:absolute before:-inset-2 before:-z-10 before:content-['']"
>
{#each option.variants ?? [] as variant (variant.label)}
{@const VariantIcon = variant.icon}
<div class="flex flex-row items-center gap-1">
<button
use:melt={$item}
class="flex flex-row flex-1 items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
onclick={() => variant.onSelect()}
onfocusin={() => (activeVariantDoc = variant.doc)}
onpointerenter={() => (activeVariantDoc = variant.doc)}
>
<VariantIcon width={14} height={14} size={14} />
<span class="text-xs font-medium text-primary whitespace-nowrap">
{variant.label}
</span>
{#if variant.badge}
<span
class="shrink-0 rounded px-1.5 py-0.5 text-[10px] font-semibold uppercase tracking-wide {variant
.badge.class}"
>
{variant.badge.label}
</span>
{/if}
</button>
{#if variant.docHref}
<a
href={variant.docHref}
target="_blank"
rel="noreferrer"
class="px-2 text-[10px] text-tertiary hover:text-secondary hover:underline whitespace-nowrap"
onpointerenter={() => (activeVariantDoc = variant.doc)}
>
See documentation
</a>
{/if}
</div>
{/each}
</div>
{/if}
{/snippet}
{#each options as option (option.key)}
{@const ac = accentClasses[option.accent]}
{@const rowClass =
'w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover'}
{#if option.variants}
<button
use:melt={$wacSubTrigger}
class={rowClass}
onfocusin={() => (activeKey = option.key)}
onpointerenter={() => (activeKey = option.key)}
>
{@render rowBody(option, ac)}
<ChevronRight size={14} class="shrink-0 text-tertiary" />
</button>
{#if $wacSubOpen}
<!-- The invisible `before:` margin bridges the gutter to the trigger row: when
the submenu falls below its row, that gutter lies over the next row, and
crossing it highlights that item, which closes the submenu. -->
<div
use:melt={$wacSubMenu}
use:hugViewportRight
class="pointer-events-auto z-[6001] flex flex-col gap-0.5 p-1 w-52 rounded-lg border border-gray-200 dark:border-gray-700 bg-surface shadow-xl focus:outline-none before:absolute before:-inset-2 before:-z-10 before:content-['']"
>
{#each option.variants ?? [] as variant (variant.label)}
{@const VariantIcon = variant.icon}
<button
use:melt={$item}
class="flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
onclick={() => variant.onSelect()}
>
<VariantIcon width={14} height={14} />
<span class="text-xs font-medium text-primary">{variant.label}</span>
</button>
{/each}
</div>
{/if}
{#if option.variants && option.key === 'pipeline'}
{@render variantRow(
option,
ac,
rowClass,
$pipelineSubTrigger,
$pipelineSubMenu,
$pipelineSubOpen
)}
{:else if option.variants}
{@render variantRow(option, ac, rowClass, $wacSubTrigger, $wacSubMenu, $wacSubOpen)}
{:else}
<button
use:melt={$item}
class={rowClass}
onfocusin={() => (activeKey = option.key)}
onpointerenter={() => (activeKey = option.key)}
onfocusin={() => highlightOption(option.key)}
onpointerenter={() => highlightOption(option.key)}
onclick={() => option.onSelect()}
>
{@render rowBody(option, ac)}
@@ -605,8 +715,8 @@
<button
use:melt={$importSubTrigger}
class="w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
onfocusin={() => (activeKey = undefined)}
onpointerenter={() => (activeKey = undefined)}
onfocusin={() => highlightOption(undefined)}
onpointerenter={() => highlightOption(undefined)}
>
<div
class="w-6 h-6 rounded-md flex items-center justify-center shrink-0 bg-gray-100 dark:bg-gray-700"
@@ -60,8 +60,7 @@
import { effectivePath, type ItemType } from './treeViewUtils'
import { tick, untrack } from 'svelte'
import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte'
import { NetworkIcon } from 'lucide-svelte'
import { base } from '$lib/base'
import PipelineRow from '../common/table/PipelineRow.svelte'
import BulkActionsBar from './BulkActionsBar.svelte'
import { HomeSelection, setHomeSelection, toBulkItem } from './homeSelection.svelte'
@@ -835,6 +834,8 @@
// keeping a dead path. `tick` lets the reloaded rows re-register first.
const renderedBefore = homeSelection.renderedKeys
void ownerCountsRes.refetch()
// Deleting a pipeline removes it from this list, which no item reload covers.
void pipelineFoldersRes.refetch()
await reloadItems()
await tick()
homeSelection.dropVanished(renderedBefore)
@@ -2168,13 +2169,7 @@
<div class="border rounded-md bg-surface-tertiary" class:wm-imported={justImported}>
{#if filter === ''}
{#each [...visiblePipelineFolders].sort() as folder (folder)}
<a
href="{base}/pipeline/{encodeURIComponent(folder)}"
class="w-full inline-flex items-center gap-4 px-4 py-3 border-b last:border-b-0 hover:bg-surface-hover transition-colors text-sm first-of-type:rounded-t-md"
>
<NetworkIcon size={16} class="text-emerald-600 dark:text-emerald-400" />
<span class="text-xs font-medium text-emphasis truncate">Pipeline · f/{folder}</span>
</a>
<PipelineRow {folder} onDeleted={reloadItemsAndCounts} />
{/each}
{/if}
{#each displayedItems as item, i (item.type + '/' + item.path + (item.hash ? '/' + item.hash : ''))}
@@ -1,14 +1,14 @@
<script lang="ts">
import TreeView from './TreeView.svelte'
import { onDestroy, untrack, type Snippet } from 'svelte'
import { createEventDispatcher, onDestroy, untrack, type Snippet } from 'svelte'
import ResizeTransitionWrapper from '$lib/components/common/ResizeTransitionWrapper.svelte'
import { ChevronDown, ChevronUp, Folder, FolderTree, NetworkIcon, User } from 'lucide-svelte'
import { ChevronDown, ChevronUp, Folder, FolderTree, User } from 'lucide-svelte'
import PipelineRow from '../common/table/PipelineRow.svelte'
import Item from './Item.svelte'
import { countLeaves, type FolderItem, type ItemType, type UserItem } from './treeViewUtils'
import { twMerge } from 'tailwind-merge'
import { pluralize } from '$lib/utils'
import { base } from '$lib/base'
import { Button } from '$lib/components/common'
interface Props {
@@ -74,6 +74,8 @@
let visualDepth = $derived(depth + indent)
const dispatch = createEventDispatcher<{ reload: void }>()
// Bounds the request burst from "expand all": however many root owners the tree
// renders (its slice grows as you scroll), it fetches at most this many. Lazy owners
// past the cap stay collapsed and load on a single click (see the effect).
@@ -316,17 +318,12 @@
{#if opened || isSearching}
<div>
{#if hasPipeline && isFolder(item)}
<!-- py-3 matches common/table/Row.svelte so this row sits at
the same height as the script/flow/app rows that follow
it under the same folder; py-2 was visibly shorter. -->
<a
href="{base}/pipeline/{encodeURIComponent(item.folderName)}"
class="flex items-center gap-4 px-4 py-3 border-b text-sm hover:bg-surface-hover transition-colors"
style="padding-left: {(visualDepth + 1) * 16}px;"
>
<NetworkIcon size={16} class="text-emerald-600 dark:text-emerald-400" />
<span class="text-xs font-medium text-emphasis">Pipeline</span>
</a>
<PipelineRow
folder={item.folderName}
depth={visualDepth + 1}
label="Pipeline"
onDeleted={() => dispatch('reload')}
/>
{/if}
{#each item.items.slice(0, effectiveMax) as subItem, index ((subItem['path'] ? subItem['type'] + '__' + subItem['path'] + '__' + index : undefined) ?? 'folder__' + subItem['folderName'] + '__' + index)}
<TreeView
@@ -1,9 +1,10 @@
<script lang="ts">
import { resource } from 'runed'
import { tick, untrack } from 'svelte'
import { Loader2, Workflow } from 'lucide-svelte'
import { onDestroy, tick, untrack } from 'svelte'
import { Loader2, Save, Workflow } from 'lucide-svelte'
import PipelineGraphEditor from '$lib/components/assets/AssetGraph/PipelineGraphEditor.svelte'
import PipelineTriggerEditors from '$lib/components/assets/AssetGraph/PipelineTriggerEditors.svelte'
import { usePipelineDataUploads } from '$lib/components/assets/AssetGraph/pipelineDataUploads.svelte'
import { resolveGraph } from '$lib/components/assets/AssetGraph/resolveGraph'
import { useActiveRunnableIds } from '$lib/components/assets/AssetGraph/activeRunnables.svelte'
import type {
@@ -11,10 +12,26 @@
AssetGraphSelection,
NativeTriggerKind
} from '$lib/components/assets/AssetGraph/types'
import { AssetService, JobService, type AssetKind } from '$lib/gen'
import { AssetService, JobService, type ScriptLang } from '$lib/gen'
import { DATA_ASSET_KINDS } from '$lib/components/assets/AssetGraph/cascadeRun'
import { usePipelineAssetPrefetch } from '$lib/components/assets/AssetGraph/pipelineAssetPrefetch.svelte'
import { sendUserToast } from '$lib/utils'
import { createPipelineAiHelpers } from '$lib/components/assets/AssetGraph/pipelineAiHelpers'
import { deployPipelineDrafts } from '$lib/components/assets/AssetGraph/pipelineDeploy.svelte'
import {
diffDeployedGraph,
extractCascadeFacts,
formatDrift
} from '$lib/components/assets/AssetGraph/deployGraphDiff'
import PipelineDeployErrors from '$lib/components/assets/AssetGraph/PipelineDeployErrors.svelte'
import PipelineDeployTriggersModal from '$lib/components/assets/AssetGraph/PipelineDeployTriggersModal.svelte'
import AutosaveIndicator from '$lib/components/AutosaveIndicator.svelte'
import { Button } from '$lib/components/common'
import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
import type { DeployResult } from '$lib/utils_workspace_deploy'
import type { SessionRuntime } from './sessionRuntime.svelte'
import { maskKey } from './modifiedItemsMask'
let {
runtime,
@@ -29,7 +46,7 @@
/** Folder name the pipeline graph is scoped to (not a workspace item path). */
path: string
workspaceId: string
/** Only the visible session registers the pipeline tools on its manager. */
/** Only the visible session polls the live run badges. */
isActiveSession?: boolean
/** Whether this is the foreground preview tab. */
active?: boolean
@@ -43,50 +60,36 @@
// wrong chat — leaving this session's chat unable to build canvas nodes.
const aiChatManager = runtime.manager
// Only the foreground tab of the foreground session owns the chat's pipeline
// tools and runs the live-badge poll — a background tab (another preview tab is
// showing, or another session is active) must not shadow that context or poll.
// Only the foreground tab of the foreground session runs the live-badge poll.
const engaged = $derived(isActiveSession && active)
// Externalized editor state — lives on the runtime so the drafts persist across
// hide/show of the preview pane (the pane unmounts on hide).
const pe = runtime.pipelineEditorState
// The reused `pe` is scoped to a folder. A same-folder remount (hide→show)
// keeps the drafts; a retarget to a different folder resets so stale drafts
// don't bleed across folders. untrack the writes so this can't self-loop.
$effect(() => {
const folder = path
untrack(() => {
if (pe.folder !== folder) {
if (pe.folder !== undefined) {
pe.reset()
// A retarget without remount re-scopes the poll, so the release
// effect could never match the old folder's job — drop the hint.
activeRunnable = undefined
activeRunnableJobId = undefined
// Re-scope the Global pipeline prompt to the new folder (the helper
// methods already read the reactive path, but the system message
// string was built for the old one). Only when this tab is engaged —
// its helpers are the registered set; a background tab reconfigures
// when it next becomes the foreground one.
if (engaged) aiChatManager.rebuildGlobalSystemMessage()
}
pe.folder = folder
}
})
})
// This folder's editor state lives on the runtime, so its drafts survive a
// remount. The host remounts this view when the tab moves to another folder, so
// `path` is fixed for its lifetime.
const pe = untrack(() => runtime.pipelineEditor(path))
const EMPTY_GRAPH: AssetGraphResponse = { assets: [], runnables: [], edges: [], triggers: [] }
const EMPTY_PATH_MAP = new Map<string, Array<{ kind: AssetKind; path: string }>>()
const EMPTY_NATIVE_MAP = new Map<string, Set<any>>()
// Data assets only, as on the pipeline page: variables and resources are
// config most scripts reference, and would swamp the layout as hub nodes.
const graphRes = resource(
() => ({ workspace: workspaceId, folder: path }),
async ({ workspace, folder }) =>
workspace && folder ? await AssetService.getAssetsGraph({ workspace, folder }) : EMPTY_GRAPH
workspace && folder
? await AssetService.getAssetsGraph({
workspace,
folder,
assetKinds: DATA_ASSET_KINDS.join(',')
})
: EMPTY_GRAPH
)
const assetPrefetch = usePipelineAssetPrefetch({
getWorkspace: () => workspaceId,
getGraph: () => graphRes.current as AssetGraphResponse | undefined,
editor: pe
})
// Folder whose graph is actually rendered — `graphRes.current` is stale-
// while-revalidate on a folder retarget, so the canvas's one-shot initial
// fit is keyed on the folder captured when a graph lands, not on `path`
@@ -97,9 +100,7 @@
})
// Deployed graph + the in-flight draft overlay (AI-built nodes render as plain
// dashed unsaved drafts, same as manual drafts). The session
// skips the route page's folder-wide asset prefetch (empty inferred maps); the
// open script's live overlays still feed the graph. (resolveGraph's base is the
// dashed unsaved drafts, same as manual drafts). (resolveGraph's base is the
// pipeline runnables subset; the 'job' usage_kind of the wire type never appears.)
let resolvedGraph = $derived.by<AssetGraphResponse>(() =>
resolveGraph({
@@ -107,9 +108,10 @@
drafts: pe.drafts,
liveBodyAssets: pe.liveBodyAssets,
liveAnnotations: pe.liveAnnotations,
inferredWritesByPath: EMPTY_PATH_MAP,
inferredReadsByPath: EMPTY_PATH_MAP,
annotatedNativeKindsByPath: EMPTY_NATIVE_MAP
inferredWritesByPath: assetPrefetch.inferredWritesByPath,
inferredReadsByPath: assetPrefetch.inferredReadsByPath,
annotatedNativeKindsByPath: assetPrefetch.annotatedNativeKindsByPath,
triggerDrafts: pe.triggerDrafts.values()
})
)
@@ -120,7 +122,13 @@
getDrafts: () => pe.drafts,
setDrafts: (next) => (pe.drafts = next),
newDraftLocalId: pe.newDraftLocalId,
onForgetPath: pe.forgetPath,
hasTriggerDrafts: () => pe.triggerDrafts.size > 0,
getTriggerDrafts: () => pe.triggerDrafts,
setTriggerDraft: pe.setTriggerDraft,
onForgetPath: (p) => {
pe.forgetPath(p)
pe.discardTriggerDraftsFor(p)
},
// Open the staged node in the details pane so its code is visible.
onProposeNode: (p) => {
pe.activeDraftPath = p
@@ -135,6 +143,7 @@
})
function handleCanvasSelect(s: AssetGraphSelection | undefined) {
uploadInputsPath = undefined
if (s && s.kind === 'runnable' && s.runnable_kind === 'script' && pe.drafts.has(s.path)) {
pe.activeDraftPath = s.path
pe.selection = undefined
@@ -145,6 +154,7 @@
}
async function afterSaved(savedPath: string) {
assetPrefetch.forget(savedPath)
const next = new Map(pe.drafts)
next.delete(savedPath)
pe.drafts = next
@@ -194,7 +204,11 @@
// Data upload has no trigger row — open the target in the details pane and pulse
// the signal so its auto-generated run form focuses the S3 input.
// A data-upload click opens the node on its inputs alone; any other selection
// (or "Open script") goes back to the editor.
let uploadInputsPath = $state<string | undefined>(undefined)
function openDataUploadRun(scriptPath: string) {
uploadInputsPath = scriptPath
if (pe.drafts.has(scriptPath)) {
pe.activeDraftPath = scriptPath
pe.selection = undefined
@@ -236,29 +250,40 @@
}
})
const dataUploads = usePipelineDataUploads(() => resolvedGraph)
function runProducer(producer: { kind: 'script' | 'flow'; path: string; cascade?: boolean }) {
// Pipeline nodes are scripts; a flow producer can't be preview/by-path run.
if (producer.kind !== 'script') return Promise.resolve(undefined)
return runNode(producer.path, {}, producer.cascade ?? false)
// A data-upload entry runs on the file staged in its run form.
return runNode(producer.path, dataUploads.argsFor(producer.path) ?? {}, producer.cascade ?? false)
}
// Tell the open node's runs list a job started, so it shows it right away.
let runsRefreshKey = $state(0)
let runsPendingJobId = $state<string | undefined>(undefined)
// `live` is an open editor's body, newer than what is stored: it is previewed,
// a deployed node included.
async function runNode(
nodePath: string,
args: Record<string, any> = {},
cascade = false
cascade = false,
live?: { content: string; language: ScriptLang }
): Promise<string | undefined> {
const draft = pe.drafts.get(nodePath)
const body = live ?? (draft && { content: draft.script.content, language: draft.script.language })
activeRunnables.arm(`script:${nodePath}`)
try {
let jobId: string
if (draft) {
// Preview-run the draft content; a preview never dispatches downstream.
if (body) {
// Preview-run the unsaved body; a preview never dispatches downstream.
jobId = await JobService.runScriptPreview({
workspace: workspaceId,
requestBody: {
path: nodePath,
content: draft.script.content,
language: draft.script.language,
content: body.content,
language: body.language,
args
}
})
@@ -275,6 +300,8 @@
}
activeRunnable = { kind: 'script', path: nodePath }
activeRunnableJobId = jobId
runsPendingJobId = jobId
runsRefreshKey++
return jobId
} catch (e: any) {
sendUserToast(`Run failed: ${e?.body ?? e?.message ?? e}`, true)
@@ -282,14 +309,126 @@
}
}
// Register the pipeline tools on this session's manager while this tab is the
// engaged (foreground) one. setPipelineHelpers rebuilds the global tool set to
// include the pipeline tools and tears them down on cleanup — so switching to
// another preview tab or session releases them.
$effect(() => {
if (!engaged) return
return aiChatManager.setPipelineHelpers(helpers)
})
// Register this folder's pipeline tools on the session's own manager for as long
// as the view is mounted — background tabs included, since each folder's tools
// act on its own state — and release them on unmount.
$effect(() => aiChatManager.setPipelineHelpers(helpers))
// ── Deploy all ───────────────────────────────────────────────────────────
// The same deploy as the pipeline page's "Save all", reached from this tab's
// button and from the session's changes list (through the runtime).
const pendingCount = $derived(
pe.drafts.size + pe.triggerDrafts.size + (pe.liveEditPath != undefined ? 1 : 0)
)
let deploying = $state(false)
let deployErrors = $state<Map<string, string>>(new Map())
let deployErrorsOpen = $state(false)
let triggerConfirmOpen = $state(false)
let resolveTriggerConfirm: ((ok: boolean) => void) | undefined
function confirmTriggers(): Promise<boolean> {
triggerConfirmOpen = true
return new Promise((resolve) => (resolveTriggerConfirm = resolve))
}
function answerTriggerConfirm(ok: boolean) {
triggerConfirmOpen = false
resolveTriggerConfirm?.(ok)
resolveTriggerConfirm = undefined
}
// A deploy asked for from the changes list can arrive while the tab is still
// loading this folder's drafts.
async function hydrated(timeoutMs = 10000): Promise<boolean> {
for (let waited = 0; !pe.hydratedFromDb && waited < timeoutMs; waited += 100) {
await new Promise((resolve) => setTimeout(resolve, 100))
}
return pe.hydratedFromDb
}
async function deployAll(): Promise<DeployResult> {
// Taken before any await: a second caller (the tab's button and the changes
// list both reach here) would otherwise replace the pending confirmation and
// leave the first one waiting forever.
if (deploying) return { success: false, error: 'This pipeline is already deploying.' }
deploying = true
try {
if (!(await hydrated())) return { success: false, error: 'The pipeline drafts did not load.' }
if (pendingCount === 0) {
// Nothing left to deploy, but an earlier deploy may have failed to save
// that: the server would still hold the deployed drafts.
const unsaved = await saveRemaining()
return unsaved ? { success: false, error: unsaved } : { success: true }
}
if (pe.triggerDrafts.size > 0 && !(await confirmTriggers())) {
return { success: false, error: 'Deploy cancelled.' }
}
// What the canvas promises for each draft, checked against what the
// backend derived once deployed.
const predicted = new Map(
[...pe.drafts.keys()].map((p) => [p, extractCascadeFacts(resolvedGraph, p)])
)
const { savedPaths, savedTriggers, errors } = await deployPipelineDrafts(pe, workspaceId)
assetPrefetch.forget(...savedPaths)
const bundleKey = maskKey(PIPELINE_DRAFT_KIND, pipelineBundlePath(path))
if (aiChatManager.modifiedItems?.has(bundleKey)) {
await aiChatManager.recordDeployedItems('script', savedPaths)
}
if (savedPaths.length > 0 || savedTriggers.length > 0) {
await graphRes.refetch()
const deployed = new Map([...predicted].filter(([p]) => savedPaths.includes(p)))
const drift = formatDrift(
diffDeployedGraph(deployed, (graphRes.current ?? EMPTY_GRAPH) as AssetGraphResponse)
)
if (drift) sendUserToast(drift, true)
}
// The session's changes list re-reads the draft once this resolves, so
// what is left of it has to be saved by then.
const unsaved = await saveRemaining()
deployErrors = errors
if (unsaved) return { success: false, error: `Deployed, but ${unsaved}` }
if (errors.size > 0) {
deployErrorsOpen = true
return {
success: false,
error: `${errors.size} of the pipeline's drafts failed to deploy, see its tab.`
}
}
return { success: true }
} finally {
deploying = false
}
}
// Saves what is left of the folder's draft and says why if that failed. A
// flush resolves even when the save failed, and the server would then keep
// drafts already deployed — deploying them again would recreate their
// triggers. A failed save stays queued until one succeeds, so the flush
// retries exactly what failed, an open script's unsaved edits included.
async function saveRemaining(): Promise<string | undefined> {
await tick()
const draft = {
workspace: workspaceId,
itemKind: PIPELINE_DRAFT_KIND,
path: pipelineBundlePath(path)
}
await UserDraftDbSyncer.flush(draft)
if (UserDraftDbSyncer.getConflict(draft).conflict)
return 'the remaining drafts conflict with a newer version.'
const sync = UserDraftDbSyncer.getState(draft)
if (sync.state === 'failed')
return `the remaining drafts could not be saved: ${sync.failureMessage ?? 'unknown error'}.`
return undefined
}
async function deployAllFromButton() {
const count = pendingCount
const res = await deployAll()
if (res.success) sendUserToast(`Deployed ${count} draft${count === 1 ? '' : 's'}`)
}
$effect(() => runtime.registerPipelineView(path, { deployAll }))
// A deploy waiting on the trigger confirmation must still settle when the tab goes.
onDestroy(() => answerTriggerConfirm(false))
</script>
<div class="flex flex-col h-full w-full bg-surface">
@@ -299,6 +438,31 @@
<Workflow size={14} />
<span class="font-mono text-emphasis truncate">f/{path}</span>
<span class="text-tertiary">· data pipeline</span>
{#if workspaceId}
<AutosaveIndicator
workspace={workspaceId}
itemKind={PIPELINE_DRAFT_KIND}
path={pipelineBundlePath(path)}
draftOnly
loadedFromDraft={pe.loadedFromDbDraft}
/>
{/if}
<div class="flex-1"></div>
{#if deployErrors.size > 0}
<PipelineDeployErrors errors={deployErrors} bind:open={deployErrorsOpen} />
{/if}
{#if pendingCount > 0}
<Button
variant="accent"
unifiedSize="xs"
startIcon={{ icon: deploying ? Loader2 : Save }}
onclick={deployAllFromButton}
disabled={deploying}
title={deploying ? 'Deploying drafts…' : `Deploy all ${pendingCount} drafts`}
>
{deploying ? 'Deploying…' : `Deploy all (${pendingCount})`}
</Button>
{/if}
</div>
<div class="flex-1 min-h-0">
<!-- Only block on the deployed-graph fetch when there's nothing to show yet.
@@ -328,6 +492,8 @@
folder={path}
viewportFitKey={viewportFitFolder}
persistDrafts={true}
modalPanel
prefetchingAssets={assetPrefetch.prefetching}
displayGraph={resolvedGraph}
mode="edit"
workspace={workspaceId}
@@ -338,12 +504,21 @@
onOpenWebhook={openWebhookDrawer}
onOpenDataUpload={openDataUploadRun}
focusUploadSignal={focusDataUploadSignal}
readyDataUploadPaths={dataUploads.readyPaths}
runFormInitialArgs={pe.openScriptPath ? dataUploads.argsFor(pe.openScriptPath) : undefined}
onRunFormArgsChange={dataUploads.stage}
{activeRunnable}
{runsRefreshKey}
{runsPendingJobId}
activeRunnableIds={activeRunnables.ids}
runStates={activeRunnables.states}
eventLogEvents={activeRunnables.events}
onRunProducer={runProducer}
onRunByPath={(path, args) => runNode(path, args)}
inputsOnly={uploadInputsPath !== undefined &&
(pe.openScriptPath === undefined || uploadInputsPath === pe.openScriptPath)}
onRunInputs={(path, args, live) => runNode(path, args, false, live)}
onShowScript={() => (uploadInputsPath = undefined)}
canRunByPath
onTestStateChange={(running) => {
const openPath = pe.openScriptPath
@@ -370,6 +545,7 @@
await graphRes.refetch()
}}
onScriptRenamed={async (oldPath, newPath) => {
assetPrefetch.forget(oldPath, newPath)
// Repoint the selection so the canvas follows the renamed node instead
// of staying on the now-gone old path until an unrelated refetch.
if (pe.selection?.kind === 'runnable' && pe.selection.path === oldPath) {
@@ -402,3 +578,10 @@
workspace={workspaceId}
onUpdate={() => graphRes.refetch()}
/>
<PipelineDeployTriggersModal
open={triggerConfirmOpen}
triggerDrafts={pe.triggerDrafts}
onConfirmed={() => answerTriggerConfirm(true)}
onCanceled={() => answerTriggerConfirm(false)}
/>
@@ -16,8 +16,9 @@ section or the flat layout.
<script lang="ts">
import { untrack } from 'svelte'
import { Compass, FileText, Pen } from 'lucide-svelte'
import { Compass, FileText, NetworkIcon, Pen } from 'lucide-svelte'
import { resource } from 'runed'
import { AssetService } from '$lib/gen'
import { workspaceStore } from '$lib/stores'
import RowIcon from '$lib/components/common/table/RowIcon.svelte'
import {
@@ -228,9 +229,38 @@ section or the flat layout.
: undefined
)
// Data pipelines are folders, not items, so they get a branch of their own
// rather than a leaf in the workspace tree.
const pipelinesResource = resource(
() => effectiveWorkspace,
async (ws) => (ws ? await AssetService.listPipelineFolders({ workspace: ws }) : [])
)
const pipelinesBranch = $derived<DrillBranch<PreviewTarget> | undefined>(
pipelinesResource.current && pipelinesResource.current.length > 0
? {
type: 'branch',
key: 'pipelines',
label: 'Data pipelines',
icon: NetworkIcon,
searchGroup: true,
children: pipelinesResource.current.map(({ folder }) => {
const path = `/pipeline/${encodeURIComponent(folder)}`
return {
type: 'leaf' as const,
key: pageKey(path),
label: `f/${folder}`,
icon: NetworkIcon,
data: { type: 'page', href: pageHref(path), label: folder }
}
})
}
: undefined
)
const tree = $derived<DrillNode<PreviewTarget>[]>([
pagesBranch,
...(artifactsBranch ? [artifactsBranch] : []),
...(pipelinesBranch ? [pipelinesBranch] : []),
...tagItems(
buildWorkspaceTree({
loaded: loader.loaded,
@@ -354,7 +354,10 @@
{#await import('./PipelineEditorView.svelte')}
{@render editorLoading()}
{:then Module}
<Module.default {runtime} path={itemPath} {workspaceId} {isActiveSession} {active} />
<!-- Keyed: the view binds one folder's state for its lifetime. -->
{#key itemPath}
<Module.default {runtime} path={itemPath} {workspaceId} {isActiveSession} {active} />
{/key}
{/await}
{:else}
{#await import('./RawAppEditorView.svelte')}
@@ -22,7 +22,7 @@
import { badgeCounts, badgeOf, buildDeployItems, type DeployItem } from './sessionDeployModel'
import { useExistingMaskKeys } from './sessionDeployModel.svelte'
import { previewTargetForDeployKind } from './sessionPreviewTabs.svelte'
import { pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
import { PIPELINE_DRAFT_KIND, pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
import type { PreviewTarget } from './previewRouter'
import JobsSegment from '$lib/components/copilot/chat/JobsSegment.svelte'
import RowIcon from '$lib/components/common/table/RowIcon.svelte'
@@ -378,6 +378,11 @@
onDataChanged={refreshDock}
onItemDeployed={(item) =>
void runtime?.manager.renameModifiedItem(item.draftKind, item.path, item.displayPath)}
onItemDiscarded={(item) => void runtime?.manager.removeModifiedItem(item.draftKind, item.path)}
onItemDiscarded={(item) => {
void runtime?.manager.removeModifiedItem(item.draftKind, item.path)
const folder = pipelineFolderFromBundlePath(item.path)
if (item.draftKind === PIPELINE_DRAFT_KIND && folder) void runtime?.forgetPipelineDrafts(folder)
}}
deployPipeline={runtime?.deployPipeline}
/>
{/if}
@@ -5,6 +5,7 @@
import DevWorkspaceBadge from '$lib/components/DevWorkspaceBadge.svelte'
import { useSessionDeployModel } from './sessionDeployModel.svelte'
import type { DeployItem } from './sessionDeployModel'
import type { DeployResult } from '$lib/utils_workspace_deploy'
// Session "Edits" drawer. Builds the deploy model over the session
// workspace's drafts and hands it to WorkspaceDiffDrawer, which renders the
@@ -18,7 +19,8 @@
keys,
onDataChanged,
onItemDeployed,
onItemDiscarded
onItemDiscarded,
deployPipeline
}: {
workspaceId: string
parentWorkspaceId?: string
@@ -35,6 +37,8 @@
* drops it. */
onItemDeployed?: (item: DeployItem) => void
onItemDiscarded?: (item: DeployItem) => void
/** Forwarded to the deploy model: deploys a pipeline folder's draft bundle. */
deployPipeline?: (folder: string) => Promise<DeployResult>
} = $props()
const isFork = $derived(!!parentWorkspaceId)
@@ -50,7 +54,8 @@
mask: keys,
onDataChanged,
onItemDeployed,
onItemDiscarded
onItemDiscarded,
deployPipeline
}))
// Editor URL for a row (every item lives in the session workspace).
@@ -1338,8 +1338,8 @@
bind:open={seedOfferOpen}
kind="X"
enterConfirms={false}
title="Keep this {seedOffer?.route.kind ?? 'item'} in the new session?"
description="You came here from {seedOffer?.route.itemPath ?? ''}."
title="Keep this {seedOffer?.kind ?? 'item'} in the new session?"
description="You came here from {seedOffer?.path ?? ''}."
>
<p class="text-sm text-secondary">
Keeping it opens it in the preview, so the chat starts with it as context.
@@ -62,6 +62,12 @@ describe('diffInMask', () => {
expect(diffInMask(diff('script', 'u/me/s'), mask)).toBe(true)
})
it('selects only the scripts a masked pipeline deployed, not the rest of its folder', () => {
const mask = new Set(['data_pipeline:f/crm/data_pipeline', 'script:f/crm/clean'])
expect(diffInMask(diff('script', 'f/crm/clean'), mask)).toBe(true)
expect(diffInMask(diff('script', 'f/crm/unrelated'), mask)).toBe(false)
})
it('matches a legacy app diff under both its identity and bridged mask keys', () => {
expect(diffInMask(diff('app', 'u/me/legacy'), new Set(['app:u/me/legacy']))).toBe(true)
expect(diffInMask(diff('app', 'u/me/legacy'), new Set(['raw_app:u/me/legacy']))).toBe(true)
@@ -1,4 +1,5 @@
import type { UserDraftItemKind, WorkspaceItemDiff } from '$lib/gen'
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
// The "modified items mask" tracks which workspace items an AI chat touched via
// tool calls. Each key is `${UserDraftItemKind}:${storagePath}`. UserDraftItemKind
@@ -45,6 +46,21 @@ export function forkDiffKindToUserDraftKind(kind: ForkDiffKind): UserDraftItemKi
return FORK_DIFF_KIND_TO_USER_DRAFT_KIND[kind]
}
// A pipeline the chat edited is masked as its folder's draft bundle, and each
// script that bundle deploys is masked too, so a fork review selects exactly
// those. The session's changes list shows deployed scripts of a masked
// pipeline's folder as that pipeline's one row — any of them, since a script's
// mask entry does not say whether a pipeline deploy or the chat put it there.
export function foldedIntoPipelineRow(
mask: ReadonlySet<string>,
kind: UserDraftItemKind,
path: string
): boolean {
if (kind !== 'script') return false
const folder = path.match(/^f\/([^/]+)\//)?.[1]
return !!folder && mask.has(maskKey(PIPELINE_DRAFT_KIND, pipelineBundlePath(folder)))
}
// True when a fork-comparison diff names an item present in the chat-modified mask.
export function diffInMask(diff: WorkspaceItemDiff, mask: Set<string>): boolean {
const kind = forkDiffKindToUserDraftKind(diff.kind)
@@ -1,6 +1,6 @@
import { getContext, onDestroy, setContext } from 'svelte'
import type { OpenInSessionSource } from './OpenInSessionButton.svelte'
import type { PreviewItemRoute } from './previewPaths'
import type { SessionTarget } from './sessionState.svelte'
// The "Open in AI session" hand-off, published by the component that owns the
// item being edited (FlowBuilder, RawAppEditor) for AI entry points too deep in
@@ -38,20 +38,17 @@ export function getOpenInSessionHandoff(): OpenInSessionHandoff | undefined {
return getContext<OpenInSessionHandoff | undefined>(KEY)
}
/** The mounted editor's hand-off for the item `route` names, or undefined when
* no editor on screen publishes one for it (a legacy app, a detail page).
* Matched on the item rather than taken as "the latest registered": a script
* editor mounted in a flow's drawer registers too, and the rail wants the
* page's own item. */
/** The mounted editor's hand-off for `target`, or undefined when no editor on
* screen publishes one for it (a detail page). Matched on the item rather than
* taken as "the latest registered": a script editor mounted in a flow's drawer
* registers too, and the rail wants the page's own item. */
export function findMountedOpenInSessionSource(
route: PreviewItemRoute
target: SessionTarget
): OpenInSessionSource | undefined {
const kind = route.kind === 'app' ? (route.raw_app ? 'raw_app' : undefined) : route.kind
if (!kind) return undefined
for (const handoff of mounted) {
const source = handoff.source()
const target = source?.target
if (target && target.kind === kind && target.path === route.itemPath) return source
const t = source?.target
if (t && t.kind === target.kind && t.path === target.path) return source
}
return undefined
}
@@ -1,5 +1,6 @@
import { base } from '$lib/base'
import type { WorkspaceItemKind } from '$lib/components/workspacePicker'
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
// The paths a preview location can point at, and the base handling around them. Kept apart
// from `previewRouter`, which reads a location's *view* from each page's filter schema and
@@ -186,3 +187,11 @@ export function parseHistoricalScriptEdit(url: string): { path: string; hash: st
const hash = new URL(url, 'http://x').searchParams.get('hash')
return hash ? { path: route.itemPath, hash } : null
}
// A `/pipeline/<folder>` route is the data-pipeline graph editor for that folder
// (the folder is a single path segment, not a workspace item path). The bare
// `/pipeline` list page is not an editor. Returns the folder name, or null.
export function parsePipelineRoute(fullPath: string): string | null {
const m = stripBase(fullPath).match(/^\/pipeline\/([^/?#]+)/)
return m ? normalizePipelineFolder(decodeURIComponent(m[1])) : null
}
@@ -11,6 +11,7 @@ import {
pageKey,
pageHref,
parsePageItemRoute,
parsePipelineRoute,
parseHistoricalScriptEdit,
parsePreviewItemRoute,
RESOURCES_PATH,
@@ -35,6 +36,7 @@ export {
pageHref,
parseHistoricalScriptEdit,
parsePageItemRoute,
parsePipelineRoute,
parsePreviewItemRoute,
stripBase,
TRIGGER_PAGES,
@@ -63,7 +65,6 @@ import { buildVariablesFilterSchema } from '$lib/components/variables/variablesF
import { buildResourcesFilterSchema } from '$lib/components/resources/resourcesFilter'
import { buildAssetsFilterSchema } from '$lib/components/assets/assetsFilter'
import { COMPARE_ITEMS_PARAM } from './modifiedItemsMask'
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
import type { WorkspaceItem } from '$lib/components/workspacePicker'
import type { SessionTargetKind } from './sessionRuntime.svelte'
@@ -502,13 +503,6 @@ export function parsePreviewSelectedId(url: string): string | undefined {
}
}
// A `/pipeline/<folder>` route is the data-pipeline graph editor for that folder
// (the folder is a single path segment, not a workspace item path). The bare
// `/pipeline` list page is not an editor. Returns the folder name, or null.
export function parsePipelineRoute(fullPath: string): string | null {
const m = stripBase(fullPath).match(/^\/pipeline\/([^/?#]+)/)
return m ? normalizePipelineFolder(decodeURIComponent(m[1])) : null
}
// The id (before the query) is the artifact's stable routing identity; the name rides in the
// hash so the tab strip labels it without a store lookup, and the version in the query so the
@@ -562,9 +556,9 @@ export const isArtifactKey = (key: string) => key.startsWith('artifact:')
// How a preview tab should render: as an in-process live editor, the deployed
// item's view page, or an iframe fallback. An item of a wrappable kind (script,
// flow, raw app) mounts its per-(kind,path) cell editor on `/edit/` and its viewer
// on `/get/`; a `/pipeline/<folder>` route mounts the data-pipeline graph editor
// (single, shared runtime.pipelineEditorState — `path` is the folder); the list page
// of a page item kind mounts that list; everything
// on `/get/`; a `/pipeline/<folder>` route mounts the data-pipeline graph editor of
// that folder (`path` is the folder); the list page of a page item kind mounts that
// list; everything
// else (static pages, regular drag-and-drop apps, any other route) stays an iframe.
export type PreviewSlot =
| { kind: 'editor'; editorKind: SessionTargetKind | 'pipeline'; path: string }
@@ -27,6 +27,8 @@ import {
import { maskKey } from './modifiedItemsMask'
import { sessionState } from './sessionState.svelte'
import { logFeatureUsage } from '$lib/utils/featureUsage'
import { AssetService } from '$lib/gen'
import { PIPELINE_DRAFT_KIND, pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
export type DeploymentStatus = { status: 'loading' | 'failed'; error?: string }
@@ -52,6 +54,9 @@ export interface SessionDeployModelArgs {
onItemDiscarded?: (item: DeployItem) => void
/** Chat-modified-items mask; undefined shows every draft. */
mask?: Set<string>
/** Deploys a pipeline folder's draft bundle, which no per-item deploy can: it
* expands into the folder's scripts and triggers. */
deployPipeline?: (folder: string) => Promise<DeployResult>
}
export interface DiffValues {
@@ -59,6 +64,15 @@ export interface DiffValues {
after: unknown
}
// A pipeline has no row of its own: it exists while its folder holds pipeline scripts.
async function itemExists(kind: Kind, path: string, workspace: string): Promise<boolean> {
if (kind !== PIPELINE_DRAFT_KIND) return checkItemExists(kind, path, workspace)
const folder = pipelineFolderFromBundlePath(path)
if (!folder) return false
const pipelines = await AssetService.listPipelineFolders({ workspace })
return pipelines.some((p) => p.folder === folder)
}
/** Resolves which mask-only candidates still exist in the session workspace
* (deployed) vs are gone (discarded drafts / deleted items) — the
* `existingKeys` input of `buildDeployItems`. Undefined while resolving, so
@@ -100,7 +114,7 @@ export function useExistingMaskKeys(
// workspace; a discarded or deleted one doesn't.
void Promise.all(
cands.map((c) =>
checkItemExists(c.deployKind, c.path, workspaceId)
itemExists(c.deployKind, c.path, workspaceId)
.then((exists) => (exists ? c.key : null))
.catch(() => null)
)
@@ -246,11 +260,18 @@ export function useSessionDeployModel(getArgs: () => SessionDeployModelArgs) {
async function runPlan(entry: DeployPlanEntry): Promise<DeployResult> {
const cur = getArgs().workspaceId
switch (entry.op) {
case 'deploy_draft':
case 'deploy_draft': {
const folder =
entry.draftKind === PIPELINE_DRAFT_KIND
? pipelineFolderFromBundlePath(entry.path)
: undefined
const deployPipeline = getArgs().deployPipeline
if (folder && deployPipeline) return deployPipeline(folder)
return deployDraft(entry.draftKind, entry.path, cur, {
draftOnly: entry.draftOnly,
rawApp: entry.rawApp
})
}
case 'discard':
return discardDraft(entry.draftKind, entry.path, cur, entry.draftOnly, entry.legacy)
}
@@ -314,6 +335,16 @@ export function useSessionDeployModel(getArgs: () => SessionDeployModelArgs) {
// ── Diff values (one resolver for tree + column) ─────────────────────────
async function loadDiffValues(item: DeployItem): Promise<DiffValues> {
const pipelineFolder =
item.draftKind === PIPELINE_DRAFT_KIND ? pipelineFolderFromBundlePath(item.path) : undefined
if (pipelineFolder) {
// Dynamic: the pipeline deploy module reaches every trigger kind's save utils.
const { loadPipelineDraftDiff } = await import(
'$lib/components/assets/AssetGraph/pipelineDeploy.svelte'
)
if (item.hasDraft) return loadPipelineDraftDiff(getArgs().workspaceId, pipelineFolder)
return { before: undefined, after: undefined }
}
const base = diffBaseFor(item, getArgs().workspaceId)
if (base.kind === 'draft') {
const { deployed, draft } = await getDraftDiffValues(
@@ -101,6 +101,15 @@ describe('buildDeployItems', () => {
expect(items[0].hasDraft).toBe(true)
})
it("a pipeline's deployed scripts fold into its one row", () => {
const bundle = maskKey('data_pipeline', 'f/crm/data_pipeline')
const node = maskKey('script', 'f/crm/clean')
const mask = new Set([bundle, node])
const items = buildDeployItems({ draftItems: [], mask, existingKeys: new Set([bundle, node]) })
expect(items.map((i) => i.key)).toEqual([bundle])
expect(maskOnlyCandidates({ draftItems: [], mask }).map((c) => c.key)).toEqual([bundle])
})
it('maskOnlyCandidates returns keys not covered by a draft', () => {
const cands = maskOnlyCandidates({
draftItems: [draft('script', 'u/a/draft')],

Some files were not shown because too many files have changed in this diff Show More