mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-10-13 00:03:00 +00:00
feat: pipeline editor: assets-only view, Fix actions and draft triggers (#11440)
* wip: one sidebar for operators and developers, with a detach mode Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: snap-detach the operator sidebar, and join the handle to the picker Dragging the sidebar into the left snap zone detaches it for an operator, who has no icon-only rail to collapse into; the width the drag started from is what gets persisted, so re-attaching restores the sidebar's size. The handle's colour pastille slides onto the card's workspace pastille as the card opens, alongside the workspace name, and both buttons that attach or detach the sidebar carry a tooltip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: attach and detach from the sidebar header, open the card under its handle The button that swaps the sidebar between docked and detached moves to the header, next to the workspace name, so it stays in one place across the swap; the icon-only rail has no room for it and keeps the footer copy. The handle itself becomes a component of its own, carries the workspace icon its picker uses, and opens the card rather than docking — docking is the header button's job. The card is placed under the handle measured at open time instead of at a fixed offset. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one page header across routes, with the workspace picker in it Every route now wears the same bar above its content: the sidebar toggle, the workspace picker, and whatever the route registers — an item's breadcrumb and summary, or a section name — with the route's own buttons at the far end. The bar's row is level with the sidebar's own top row, which the workspace ⇄ sessions switch now owns, so the two read as one band. A route fills the bar by mounting PageHeaderContent, which registers a getter rather than a snapshot: the layout calls it while rendering, so a path or summary the page edits reaches the bar without an effect keeping two copies in step. Nothing registers yet; the routes follow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: detail pages fill the page header, which now spans above the sidebar A script or flow detail page no longer draws a bar of its own: its path becomes the header's breadcrumb, its summary keeps the rename-and-labels popover next to it, and its badges, menu and buttons ride along as the header's actions. The band runs the full width with the sidebar hanging below it. A snippet rendered in the header initialises its components under the header, not under the page that wrote it, so anything the page's tree provides is lost — the detail page's buttons read TriggerContext and threw. ContextBridge carries the entries a registration names back around the render. The floating handle is gone with it: the header's own toggle is the one control, hovering it peeks the sidebar in and clicking pins it. The workspace picker in the header is a denser variant of the sidebar's, and the card's outside-click catcher starts at the card rather than the viewport top, so it no longer swallows clicks meant for the toggle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one breadcrumb in the page header, and the banners that used to sit under it The header band now carries a single breadcrumb — workspace, fork, then the item's path — built from one component, so every segment has the same size, weight and icon size, and slashes separate them. The item's kind icon rides with its summary, after a dot, rather than with the path. The fork and drafts banners move into the band as badges plus a subtle Review button: ahead/behind counts instead of a bare change total, and the per-kind breakdown stays on the compare page. The home hero's CLI, MCP and Hub links move into an ellipsis menu, alongside the toggle that pins the session chat to the page. The flow editor and the app viewer hand their bar to the band too, and a route can drop the band's bottom border when its own content draws one — the border stays in the box with a transparent colour, so contents never shift a pixel across a navigation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the runs page's own header row moves into the page header The title, the queue counters, the kind and status filters, the schedule and future-job switches, the search field and the timeframe picker all ride the band now, and the page starts at the chart — one row of vertical space back. The two toggle groups appear from 1500px: below that the bar cannot hold them next to the search field, and the same filtering is reachable from the search field's own kind and status chips. The search field is what gives width back, and the breadcrumb yields last, so a page's own name stays whole while the workspace and fork names truncate. The queue counters keep their icon form at every width, since the bar has room for two counters and never for the two labelled blocks. Their counts hang outside their icons, so the group carries the margins that keep a badge off its neighbour, and the suspended counter answers a hover at zero rather than sitting there as a dead icon. The band is 44px rather than 40, and a page can hand the breadcrumb a hint — the documentation tooltip that belongs with the page's name rather than with its buttons. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: no border under the page header, and the peek card closes on the way out The band draws no bottom edge and no page asks it to, so `barBorder` goes with it — a page that needs the separation draws its own line, as the detail pages now do above the run form and the tab strip. With nothing toggling a border, the band is a flat 44px on every route. The detached sidebar's card is a hover affordance, so it leaves when the pointer does: off the handle and off the card both, on the same debounce a hover menu uses, because the two do not touch and the leave of one fires before the enter of the other. The mobile drawer is opened deliberately and still stays until it is dismissed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one band, one workspace part, and the pickers that hang off it The band is a single element whose placement is styled rather than branched: a page registering a side-panel inset used to swap it between two `{#if}` arms, destroying the header and rebuilding it a frame later — a blink of no header with the content sliding up into the gap. The workspace and its fork become one breadcrumb part: the name leads home (inert in a session, where the workspace is the session's scope rather than a place to go) and one chevron opens the picker, which already expands a family to reach its forks. The picker's panel hangs from the part's left edge, opens with the active family expanded, and closes before the switch runs — the switch renames the workspace, which moves the trigger the panel hangs from, and floating-ui was chasing it across the viewport. The item's path is a plain label rather than a row of pickers: same separators and weight, and a click copies it with its own confirmation. The kind icon goes, the page below says what kind it is. Around them: a "prod" badge for a root workspace, the session's acting fork beside the session's name rather than in the breadcrumb, the edit-in-dev affordance as a button in the header, the drafts count as a sentence, and a fade when the workspace changes under a page that stays mounted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep a page's own controls when the workspace navigation is hidden `menuHidden` means "this is an embed, don't show the workspace's navigation" — but Runs, the detail pages and the flow editor now keep their controls in the band and nowhere else, so hiding it took the filters, the timeframe and the page's buttons with it. A session's preview iframe opens exactly that way. The band now renders for an embed too, without its breadcrumb, so the page keeps its controls and the embed still gains no way to navigate the host's workspace. The flow editor's autosave indicator moves into the header's actions with the rest of its bar: saving state, the autosave toggle and other people's drafts were left behind in the branch the full-page editor no longer takes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the header carries what it promised, and nothing it does not The registry's item described pickers the breadcrumb no longer has: nine fields — the editability flags, the change callbacks, the navigation hook — that nothing read, while the flow editor passed eight of them and quietly lost its inline rename. The type now says what the bar does: a path, and either a summary or the page's own control for one. The flow editor renames through that control, as the detail page already did. `barPlacement` went the same way: no page registered 'inline', so the layout's guard never fired. The sessions page registered an inset of 0 whenever its preview pane never mounted — the "no session selected" and opted-out arms — and 0 reads as "a page owns the right edge", floating the band over the page's first line. It registers an inset only once the pane has a width. The band's sidebar handle docks through `setDetached` like the card's own toggle, so docking still suppresses the rail's entry animation and closes the card; a summary without its breadcrumb no longer leads with a stray separator in an embed; a page's hint sits with its name whichever way the page named itself; and `Menu.closeInstantly` and `EditorHeader`'s `inline`/`hideSummary` go, their callers having been replaced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: a deployed app gets the viewport, and the header waits in the corner An app is the page, not a page inside Windmill's chrome, so the band leaves the flow and floats above the viewport until the handle in the app's corner calls it down. The handle is faded until the pointer nears it, hovering peeks the band over the app, and a click pins it — one flag for every deployed app, in this browser, because someone who wants the breadcrumb over a dashboard wants it over the next one too. Peeking overlays rather than pushes: the app is an iframe, and taking 44px off its box on every pass of the pointer would re-lay-out whatever it draws. Pinned, the band is in flow like any other route's and the app starts under it. The sidebar is untouched — docked stays docked, detached stays a peek — since the pin is about the header alone. A page says it wants the viewport with one registry field, so the layout keeps asking the registry rather than knowing about apps. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one corner control for the chrome an app page hides Hovering the button overlaid on a deployed app brings back whatever that page is hiding: the band always, and the sidebar card too when the sidebar is detached. The card hangs from this button rather than from the header's own handle, which is off-screen with the band. The handle in the band no longer attaches the sidebar — it reveals it and nothing more. A control that hides the thing it sits on cannot also be the way to bring it back, so attaching stays with the toggle in the sidebar's own footer, where detaching happens. The band's two buttons go in the order the things nest: the sidebar's handle, then the header's pin. The overlaid button stands where the pin will be once pinned — same 28px box, and a slot further along when the sidebar's handle takes the first — so pinning moves nothing. It wears the panel pair the sidebar's toggle uses: solid while the band is away, dashed while it is shown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the access point sits in the corner, and the header covers it The button is in the page's top-left corner — 8px in, or just past the rail when the sidebar is docked — rather than shifted along to where the header's pin would land. Hovering brings the header down over it: an access point that the regular view covers once it arrives, so there is nothing to line up and nothing to move. Because the header lands on the button, the pointer leaves it the instant it arrives. The band holds both itself and the sidebar card open while the pointer is anywhere on it, or the card would retract under a header that stayed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: on an app page the floating button owns both, and they move together The band drops its sidebar handle there: the floating control in the corner reveals the sidebar along with the band, and two handles for one gesture is one too many. The two are then linked — the pointer resting on either holds both, and leaving either collapses both — so they behave as the single thing the one button revealed. The card meets the header in a line: it hangs from the header's bottom edge rather than from whichever control opened it, which had it starting a couple of pixels high. With no header to sit under — an embed — it still hangs from the control that reached for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the revealed sidebar is a panel under the header, not a card over the page No rounded corners, no inset from the window: it starts at the header's bottom edge and runs to the left and bottom edges, with one border down its right side and a shadow to lift it off the page. A card floating in from nowhere read as a separate thing; this reads as the sidebar the header sits on top of. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: only an app page gets the flush panel; elsewhere the card still floats Over an app the sidebar reads as the panel the header sits on — flush left and bottom, square, one border down its right side. On every other page it is what it was: a card inset from the window with rounded corners and a border all round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: an app opens with the header already away Opening an app showed the header and then slid it out: the page claimed the viewport on mount, and the band animated from where it had been rendering. The slide belongs to the hover, not to arriving, so it is armed a frame after the claim — the first hide is a cut. The claim moves to the route, which mounts before the viewer waiting on the workspace to resolve. A cold load now paints the band away from the first frame instead of showing a header that leaves again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: an app's Edit sits with its name At the far end of the bar it was a journey across the window from the thing it edits — on a page whose header is only there while the pointer holds it, a journey with a time limit. It goes beside the app's path instead. The slot it uses is the one Runs' documentation tooltip already had, renamed from `hint` to `afterName` and documented for both: what belongs to the page's name, whether that is a mark or a control that acts on what the name points at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: developers collapse, operators detach One control each, rather than three for everyone. A developer switches the sidebar between icon-only and full width and never detaches it; an operator, who has no icon-only rail to collapse to, hides it instead — by the button in its footer or by the same drag that collapses it for a developer. The way back survives either way: a detached sidebar keeps its "Show sidebar" button for anyone, so a developer who detached before this was so is not left with a sidebar they cannot return. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: configure pipeline triggers and outputs as drafts in the editor Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: one card for pipeline nodes, and the pipeline pages in the page header Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: drop a removed node's trigger drafts, and confirm triggers when leaving Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: show pipeline assets red when unused or awaited but never produced Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: route long pipeline edges around nodes in the target column, and flag scripts waiting on unproduced assets Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a page is as tall as it is given, not as tall as the screen The band takes 44px off the top of every route, so a page sized to the viewport overflows by exactly that much: h-screen inside a column that already lost the band's height. These pages measure the box they are in instead. CenteredModal gains `inPage` for the five pages that render it inside the layout rather than over it — the same modal, centred in what is left of the page rather than in the window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the band narrows, wraps, and draws its own edge The band publishes its width, so the pages filling it can answer it. Two tiers: under 1000px the breadcrumb keeps the workspace disc and its env badge and drops the name, and under PHONE_BAR (800) the trail stops yielding at 10rem — that floor is what makes the bar wrap, so a page's buttons take the line below rather than squeezing the name to an ellipsis. The band also draws the line between itself and the page, and only while something is scrolled under it. One capture listener on the window sees every scroller, since most pages scroll inside their own box and never move the layout's. An inset shadow, not a border: a border would move every page down a pixel the moment the line appeared. DetailPageLayout and InWorkspaceAppViewer drew that first line themselves; two lines would stack, so theirs go. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the script, app, runs and audit pages fill the band Each of these kept a header of its own above the page. They register with the band instead: the breadcrumb names what the page is on, and the page's buttons sit at the far end of the one row. They answer the band's width rather than the window's, since the buttons share that row with the breadcrumb. The script editor folds Diff and Settings into the menu it already shows when compact, the app editor drops the panel buttons, Awareness and Export and goes icon-only on Deploy and the mode switch, the flow editor menus its diff, and runs hides its counters. Runs also declares `actionsFlexible`: its search is the one control here that can give width back, and a box that refused to shrink would make it take its maximum instead. The run detail page loses its ActionRow, and the audit log its big title — the breadcrumb says where you are, and the page keeps its tooltip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: one sidebar column, shown as a rail or as a panel The rail and the drawer held two copies of the same column — the same sections, in the same order, in two pieces of markup, so a change to one silently skipped the other. Both now render `sidebarColumn`, and the shells keep only what differs: the rail its width, resize handle and collapse button, the panel its outside-click catcher and its transform. A phone gets that panel rather than a modal. There was a width test deciding between two presentations; the card under the band's handle works at any width, so it goes. What is left of the width test is the choice between a rail and a panel, plus a `(hover: hover)` test — the old `innerWidth >= 768` was standing in for "has a pointer", which is what decides whether the panel closes when the pointer leaves it. With the card everywhere, the backdrop, the `aria-modal` (nothing dims and nothing traps focus) and the drawer's close cross all go: the band's handle opens the panel and a tap outside closes it. The burger row goes with them. Entering panel width now collapses the rail transiently, so coming back to a wide window does not land on a rail the page was resized away from. The band also stops easing its right edge. That transition belongs to the rail, which moves the band's left; its right follows a splitter the page owns, and easing it left the band lying over a session's side panel — a blank strip across that panel's own header for 200ms after every drag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the home composer flies to the menu that restores it Removing the composer puts it away rather than throws it out, and the way back is one item in the band's menu — easy to miss on a page the composer has just vacated. A mark flies from the control that removed it to that menu, and the menu answers with one ring as the mark dissolves, so where it went is witnessed rather than explained. The cross and the menu item now read "Remove session chat", one name for one thing whichever a reader reaches for. Reduced motion skips both. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: give pipeline asset nodes the trigger nodes' background again Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: taller pipeline nodes, with the asset + back on the node's edge Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: judge dbt relations by provenance, keep dynamic_skip on reopen, refresh saved scripts' titles Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: label the pipeline + buttons: add data source, add downstream step on hover Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: name a schedule node's cadence, like "Every day at 12:00" Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: keep the downstream-step pill open while its menu is, without flicker Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: room between pipeline node chips and the node's edge Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: refuse renaming a pipeline draft out of the pipeline's folder Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: title pipeline schedule nodes with the schedule's summary Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * revert: keep operators out of AI sessions That change is its own PR; this one is about the navigation. `prefersSessionHandoff` takes the role again and every entry point passes it, the sessions page refuses an operator arriving by URL, and the layout leaves them on the docked chat. Two clauses are new rather than restored, because the nav rework is what gives an operator a sidebar at all: the Workspace/AI Sessions switch and session mode are both hidden from them, since the page behind them declines. `askAiShown` gains the operator case for the same reason — they keep the docked pane whatever the beta gate says, so the sidebar has to offer a way into it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: assets-only view of a pipeline, with producer and trigger chips Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: default schedule handlers, unique trigger draft paths, zoned cadence, valid script names Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: setup banner without icon, button actions, plainer activity rows Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: assets-only trigger chip under the title, editable and selectable Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: await schedule defaults before confirming, refuse colliding trigger renames Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: size assets-only cards to their content Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: pipeline draft sync status beside the folder name in edit mode Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: autosave edits to a deployed pipeline script from the first keystroke Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: show schedule handlers once their defaults load, draft suffix on static chips Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: keep the pipeline folder while the page tears down on leaving Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: discard the open pane's autosaved edits with Discard all Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: pipeline node context menus, add-at-cursor, and assets-only delete Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: borderless pipeline cards on the main surface, like flow steps Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: remember which pipelines are in the assets-only view Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: edit an asset's producing script from its pane and multi-script chip Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: more room between pipeline layers for the bottom + button Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: background on the trigger chip only when it opens an editor Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: assets-only delete in both modes, why it is blocked, and a Show scripts toggle Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: remove the script before its trigger, archive it for non-admins Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: scope assets-only memory by workspace, no delete for operators, flow reason Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: fly in the pipeline node menus Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: size the Show drafts toggle like the other header buttons Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: a No asset node for scripts that build nothing, in the assets-only view Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: describe a pipeline edge's relationship on hover Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: name assets-only edges by what is built from what Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: open an asset's upstream trigger from its pane header Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: reopen the edit-mode pane when coming back from View Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * style: no red border on errored pipeline nodes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: Fix actions on misconfigured pipeline nodes, and trigger toggles on edges Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: folded errors and edge toggles in assets-only view, preset writer wizard Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: flag pipeline scripts nothing starts, with schedule/webhook/remove fixes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: an unwritten subscription is fixed on its script only, by removing it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: hide subscribed-only unwritten assets, review fixes for the edit pane and edge actions Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: an unwritten subscription is the subscriber's error, not the asset's Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a script only started by reads of unwritten assets is flagged too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: no add-downstream + on assets nothing in the workspace writes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: Save all treats a draft identical to the deployed script as saved Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: remove-from-View opens its dialog, error chip fits, no link to an undrawn asset Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: drag a script's dot onto an asset to subscribe it; unbuilt asset opens its script Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: toggle to hide reads that don't trigger runs in the pipeline graph Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: drag an asset's + onto a script or the canvas; tell reactive assets-only edges apart Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a programmatic on/mute line joins the existing ones in the header Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: address review — live-version save check, bounded writer lookups, script-only trigger writes, dbt sources, hidden pane Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: review — compare every deployed field, passive-read filter only with its toggle, drag cancel, asset escape hatch Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: keep every control the band took over reachable Four controls the page-header conversion could put out of reach, each found by a review pass and reproduced in the browser: - The flow editor's preview buttons. `topbarWidth` is bound only in the branch a full-page editor does not render, so `compactTopbar` was permanently false there; below the 800px phone bar `phoneTopbar` dropped the buttons inline and the graph overlay never took them, leaving FlowPreviewButtons mounted nowhere and step testing, whole-flow testing and "Test flow & record" no-ops through optional calls. It now measures the header the way ScriptBuilder and AppEditorHeader do, and the inline and overlay conditions are exact complements. - The pin on a deployed app. The band lands over the corner handle that called it down, so the pointer that arrives there hits the band. The band carries the pin itself now, on the same pixels, in place of the sidebar handle it hides. - The detached sidebar's own menus. They portal to `body`, so moving into one left the card and closed it 200ms later, sliding it out from under the menu being read. The card holds while the pointer is in a menu and takes the close from the leave of that. - A deployed app's name for a read-only viewer. The breadcrumb was registered behind write access, so without it the trail fell back to the section name "Apps"; and the Edit button rode `afterName`, which the band drops for an embed along with the trail. The name is registered for whoever opened the app, and what sits after it survives `navHidden` as the page's own control. navPageFor gets a table test: it answers both the sidebar links and the breadcrumb label, and its longest-prefix and first-segment rules are what made the "Apps" fallback look like a correct answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: drop what the operator revert and this refactor left behind The operator tour's last step named AI sessions among the pages the sidebar opens, which operators have no way to reach. The nav handle's comment named the AI chat layout's menu button as a second way into the card; that button and its prop are gone, and the corner control on a page that hides the band is what stands in for the handle now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: prettier the three files this branch left unformatted `prettier --check` fails on all three at the previous head. Whitespace only, apart from one Button tag that fits on a line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: assets-only nodes name their script in a header tab instead of a side chip Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: no loading spinner in the pane while a clicked script loads Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: no subscribe drag dot in view mode Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: opening a draft no longer autosaves; header tab styling, selected edge, taller cards Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * revert: clicking an unbuilt asset opens the asset again, not its script Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: header tab sets card width, edit icons on script header and trigger chip, no pane header shift on open Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: no trash in the pane header; card width ignores the run chip so a click never re-lays the graph Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: taller trigger chip; node menu and guard badge on the card corner, below the header Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: give a page without the band's chrome what the band was carrying Second review round, three reachable defects plus one decision: - A public raw app was 150px tall. The bundle's iframe is `h-full`, and dropping its `min-h-screen` left nothing to resolve against on the share, embed and custom-path routes — none of which has a layout above them. Measured: 150px in a 900px viewport. The floor goes back for those routes and stays off inside the workspace, where the layout hands a height down and `min-h-screen` would overflow it by the band. - The band retracted out from under its own open menu on a deployed app. Last round taught the sidebar card to hold while the pointer is in a portaled menu and left its sibling alone, but the band opens menus too — the workspace disc, the fork family, every path level. One helper now, used by both closers. - `bandPeeked` outlived the page it belonged to. Clicking a control in the band navigates away, and the next page may not be one the band floats over, so the hover handlers that would clear it are gone — leaving the band already down when a full-bleed page came back. `beforeNavigate` clears it. - An embed has no band to carry a control, so the app's Edit floats over the canvas there, where it sat before the band existed. Registering it as the band's `afterName` under `navHidden` instead would have cost the app 44px of the iframe for a row holding one button. Reading "is the workspace navigation hidden" now has one definition (`isMenuHidden`), since a page has to answer it too: the flag is sticky inside an iframe, so the raw query param is the wrong thing to read. Also inset the band by the rail only when there is a rail — an embed reserved its width and crowded its own controls into a compact layout early. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: a path that names nothing yet has no breadcrumb leaf `splitItemPath` documents that it gives up on a path still being typed, but `f/demo/` is three parts and returned a leaf with an empty name — a blank segment in the trail. The editor binds the path live, so the breadcrumb sees every keystroke. Tested here rather than `navPageFor`'s sort: no two `NAV_PAGES` paths nest, so the longest-prefix case that test claimed to pin has only one candidate. Renamed it to what it actually checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: prettier the four remaining files this branch left unformatted All four fail `prettier --check` on the branch and pass at the merge base. `raw_apps/utils.ts` also fails but already did at the base, so it is left alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: pipelines open in the assets-only view by default Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: a No asset node per script, banner-only when writes start it; no On asset change chip Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: delete a node's script, trigger, or both from its menu; red header for a script with errors Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: an errored script header stays red when selected Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: coloured run badge border; per-script delete when no combined delete; run button centred on the card Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: Add data source pill sits one small gap above the graph Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: Multiple triggers chip picks the trigger to edit; each folded trigger deletable from the node menu Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: add-step menu labels its path field Path Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: iterate NodeLists with Array.from so svelte-check passes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: no trigger delete for flow-built assets; Multiple triggers chip shows the one being edited Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: Archive button for archive-only deletes, single delete doc, inert Multiple triggers label in View Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: the raw app editor fills the band like the other three It was the one editor route still rendering its own bar under the band, so it showed two header rows and the band named the app "Apps raw" — `navPageFor`'s first-segment fallback, because nothing on the route registered an item. Its top bar now becomes the page header the way the script, flow and low-code app editors' do: the path and summary are the breadcrumb's, the file-sidebar toggle, autosave cloud, presence mark, menu, Diff, Jobs, export, AI and Deploy are the header's actions, and the path is a plain label since this editor renames from its deploy drawer. The actions render under the band, outside the subtree `RawAppEditor` declares the operating workspace in, and `OpenInSessionButton` asks that context who the acting user is — so the workspace travels with them. `operatingWorkspaceContextEntry()` hands it over without exporting the key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: keep the folders a half-typed path already names The trailing-separator guard gave up on `f/demo/` entirely, which dropped the whole trail to the kind chip and brought `f/demo` back on the first character of the name. The levels the path does name come back now; only the leaf waits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: pin holds the first slot, and the nav table says what it shares Pinning brings the sidebar's handle back, and from behind it the unpin button sat one slot over from where the pin had just been clicked — so a second click in the same place opened the sidebar instead. An operator is detached by default, which makes that the default state of the feature rather than an edge case. `NAV_PAGES` claimed one table stops a page being called one thing in the nav and another in the header, which holds for the seven entries the sidebar's main group is built from and not for the four grouped separately — `/routes` was "Triggers" in the header and "HTTP" in the nav. The comment now says which is which, and that entry is named for the page rather than its group. `sidebarPageAllowed` gets a table test: it decides what an operator sees, and its `=== true` is what a future `Option`-style default would silently change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the band names the path Deploy will create, and keeps the summary readable A brand-new app is parked at a `draft_<uuid>` placeholder, and the band named that rather than the path its Deploy will create — `newEditedPath` refuses the placeholder for exactly this reason, and the editor's own bar read it. Both app editors register it now. Verified: the trail reads `u/admin/gentle_app` while the URL is still the draft. The band's summary gets the `title` the editor's own bar had, so one the bar truncates can still be read on hover. The wrapper that hid AppExportButton on a phone never hid anything: the component draws no trigger — it is the export drawer, which portals to `body` — so the class toggle was inert in both states and the comment read as if something moved. Dropped from both app editors. Two comments corrected: `NAV_PAGES`' `icon` does reach the breadcrumb for all eleven entries (`NavBreadcrumb` renders `routePage.icon`), so only `operatorKey` goes unread for the four the sidebar groups separately; and the raw editor's phone note described folding that does not happen — the file-sidebar toggle stays in the band, Jobs and the export are already in the menu, and AI is what stands down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: an item the picker files under no kind keeps its name in the band The merge brought main's agents detail page, which reaches DetailPageHeader with no `errorHandlerKind` — and the band's item required one. `WorkspaceItemKindIcon` falls through to the script glyph for anything it does not know, so defaulting would have labelled an agent a script; the kind is optional now and the trail names the item without claiming it is one of the three. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the file sidebar's toggle leaves the band The band is the workspace's header, not the raw editor's layout controls. ⌘B still folds the file sidebar, and the editor's own bar keeps the button where it draws one — a session pane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the scroll edge stops measuring, and the comments describe the code The band's edge listener captures every scroller in the app — Monaco, the audit table, a run's logs — about once a frame while any of them moves, and it took two `getBoundingClientRect()` reads before looking at `scrollTop`. It now answers from `scrollTop` first and only measures when that answer differs from what the band draws, which is the transition rather than the steady state. The rect check still authorizes every change, so a scroller that does not start at the content's top still cannot claim the edge. The two agent routes main just added size themselves to the viewport, which is 44px more than the box the band leaves them. They measure 856 either way — the layout's `min-h-0` column clamps them — so this is the sweep the rest of the branch already had, making the height follow the box rather than depend on a parent's clamp. Six comments described the conversion rather than the code, which AGENTS.md calls out by name. Each now states the constraint a reader could break: why the summary keeps its popover here, why the audit hint is `afterName` and not an action, why a run registers a `section` rather than an item, why the Runs filters are `actionsFlexible`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Revert "fix: the file sidebar's toggle leaves the band" This reverts commit1da8d16740. A folded file sidebar renders nothing of its own, so the toggle on the bar is the only way back short of knowing ⌘B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the editors name their item in the band, and a pen renames it The band showed each editor's summary as an inline field, which put editing where the page names what it is working on. The summary is text there now, with its `title` for one the band truncates, and the pen that opens the path for editing sits after it — the same popover the editors' own bar carries, at the end of the name it renames. Every editor already has a field for the summary itself: the script's settings, the flow's Settings tab, and both app editors' deploy drawer. The popover moves into `PathEditPopover` rather than being written out four more times. It owns the open state and hands back the path as it was when it opened: the band's trail reads that while the popover is open, so a path growing under the pointer does not drag the pen — and floating-ui the popover — along with it. `EditorHeader` renders the same component, so the session pane and the band cannot drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the pen renames the item and its summary The pen opened the path alone, which left the summary with nowhere to be edited from the band. It carries a Summary field above the path now — the item's name first, the address it lives at below — and the band's text follows it as the user types. `TextInput` with a label rather than the bar's inline field: `EditableInput`'s own docs rule itself out inside a popover, and it wraps this same input in its edit mode, so the field, the "Add a summary..." placeholder and the trim on commit are the behaviour that was there, in the shape the deploy drawer's Summary field already uses. The keydown stops at the field because every editor binds keys of its own and a summary is prose. `summaryEditable` carries the hosts' `customUi.topBar.editableSummary` through, and the editors' own bar passes it false: that bar keeps its inline field, and `hidePath` would take the pen with it, leaving a condensed session pane no way to name the item at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the agent editor fills the band too It was the last editor route drawing its own header row under the band, and the band named it "Agents" — `navPageFor`'s first-segment fallback, because nothing on the route registered an item. The agent's path is the breadcrumb now, its version badge reads after that name, the sentence that was the row's second line became the hint beside it, and the Chat/Form switch, Settings, Evals, version history and Deploy are the band's actions. No trail levels in the registration: this layout opens settings in a drawer and evals over the page, so unlike the dialog it never stands on a level — `trail` is its root alone here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the pipeline page fills the band Found by sweeping every route under the band: the pipeline editor drew its own row beneath it, 48px of a second header, while the band named only the section. Its name is the band's now, the folder it is open on sits with that name — the switcher is how you change pipelines, so it belongs to the name rather than to the actions — and the mode switch, deploy-all, activity, refresh and overflow menu are the bar's actions. The arrow back to the list stays: nothing else on the page points there, and the sidebar does not list pipelines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the app canvas controls stay reachable, and an unnamed item says so Codex found the app editor's canvas controls unreachable on an ordinary laptop: the owning route collapsed at 1350px, and a 1440 window with the rail out leaves 1232, so the width, theme and breakpoint toggles were hidden with nothing in the menu to replace them — and the breakpoint is the only way onto the mobile layout. Measured on the band, the toggles are 240px beside 334 of other actions and a ~430 trail, so they fit to about 1050: the threshold is 1100 now, and below it each of the three keeps a menu entry rather than disappearing. An item with no summary read as a bare `·` with an invisible pen after it, since the plain text replaced a field that had a placeholder. It now reads "Add a summary..." in the italic SummaryPathDisplay uses for the same state, and the pen stops hiding when there is nothing to hover. Three more from the review: - `actionsFlexible` says "these actions can give width back", which is true of the Runs filter row and not of the pipeline's row of fixed buttons. - `pathSnapshot ?? (newEditedPath || …)` keeps the `||` chain the five rename sites read and the declaration documents; `??` had made the rest unreachable, since `newEditedPath` is a string that is never undefined. - The band bridged `contexts` onto the actions alone, while the pen that decides whether a path is the caller's to change renders in `summaryContent`. The breadcrumb's `afterName` and the summary are bridged the same way now, so that permission check reads the tree that registered it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the widest action rows scroll, and the canvas toggles show only where they fit Measuring the band instead of estimating it: with the canvas toggles shown the app editor's actions are 826-911px and the trail with the summary another ~500, so the row overflowed the bar at every width up to about 1470 — the 1100 I set from an estimate was wrong, and the 1350 before it was too. The toggles come out at 1500 now, and the menu entries are what keeps all three reachable below that rather than merely out of the way. The two widest rows — this editor's and the pipeline's — scroll inside their own box, which is the escape the rows they replaced had (`overflow-x-auto whitespace-nowrap`). That answers pi's finding and clears the last 23-40px of band overflow at middling widths, which predated the conversion. Measured after: zero horizontal overflow from 852px of bar to 1392. From the CI round's other findings: - The pen is offered when either the summary or the path is editable, not the path alone: a host that allowed one and refused the other could no longer edit the summary from the band. `PathEditPopover` takes `pathEditable` to match, and its label names whichever it is given. - The four editors' comments called the summary read-only, which stopped being true when the pen beside it started editing it. - The pipeline's "centre group" comment described a layout it no longer has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: actionsFlexible covers both ways a row gives width back It read as if a search field were the only way to earn it, which stopped being true when the app and pipeline editors' rows started scrolling inside their own box. Both are the same bargain — the box may shrink because its contents can — and the flag is what the bar reads to allow it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: data pipelines in AI sessions (alpha) (#11453) * feat: pipelines in AI sessions with several folders open at once Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: deploy and discard a session's pipeline edits as one changes-list row Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: pipelines in the chat's listing and the preview picker; session tab shares the page's asset prefetch Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a masked pipeline preselects only the scripts and triggers of its folder Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: review round — exact pipeline scripts in the fork mask, refuse a failed hand-off flush, deploy guard before awaits Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: review round — failed bundle save fails the deploy, removing a draft records no change Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: a session's pipeline tab opens node details in a modal, as the flow editor does Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: route pipeline graph edges orthogonally with spread ports, separate tracks and lanes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: pipeline graph edges merge by shared end, enter nodes by the side, keep fewest crossings Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: review round — incremental crossing search and cached routes, retry the leftover draft save, drop the pipeline row when its last draft goes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: retry the leftover draft save with what failed, never an empty draft Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * revert: pipeline graph edges back to bezier curves, detours curved too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: hovering a pipeline graph node fades everything outside its lineage Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: hover fade eases edges too, through a stylesheet instead of new node objects Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: gentler hover fade without flicker on click; implied trigger edges recede in the assets-only view Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * docs: pipeline nodes ingest straight into DuckLake or data tables from Python/TS * fix: pipeline tools callable right after open_preview; write_script refuses pipeline nodes in sessions * feat: delete a data pipeline from the home list; pipeline rows match item rows * fix: upward detour edges keep their gutter lane * fix: pipeline delete takes only its own triggers and discards draft-only nodes by storage key Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: AI attaches trigger drafts to the pipeline nodes that declare them; materialize nodes keep no stale output Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: AI gives every pipeline node a succinct summary Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: session pipeline tools survive another tab taking the panel, reopening the editor on call Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: pipeline trigger lookups skip same-path flows; a failed trigger never hides its built node Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: deleting a pipeline clears it from open sessions so autosave cannot restore it Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: data upload opens on its inputs alone, works in sessions, with a prominent Upload data button Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: closing the pipeline pane closes its modal instead of leaving it blank Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat: a DuckLake asset opens on a Data tab with the database manager Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: inputs-only view infers draft inputs, keeps stored lineage, runs the live body Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: lineage follows the editor that inferred it; upload runs preview unsaved edits of deployed nodes Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: readyDataUploadPaths is a ReadonlySet in the pipeline editor Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: selected pipeline nodes get their blue edge in dark mode too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: no red flash on load while writer lookups and script bodies are in flight Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: deploy an open deployed script's edits, and don't resurrect them after a discard Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * refactor: deploy and discard close the open pane first instead of patching its edits Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: keep the pane closed while deploying, and clear its buffer once its edits are a draft Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a draft edited while it deploys keeps the newer edits, chained on the new version Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: a draft whose summary or labels changed mid-deploy is kept too Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: the app viewer names the page it owns, and stops claiming the viewport A deployed app opened in an AI session's preview panel renamed the band from the session to the app, and put the app's Edit beside the session's name. The viewer registered wherever it was mounted; it takes `ownsPageHeader` now, the way the detail pages already did, and keeps Edit over its own canvas otherwise. Both viewers also claimed the whole viewport under a band that takes 44px of it, so a deployed app scrolled by exactly the height of its own header (the low-code one measured 900px in an 856px box). The height comes from the host now: the three standalone routes floor themselves at the viewport and everything inside them is `h-full`. A raw app reached through `/app_embed` had no height above it at all and collapsed to nothing; it fills the frame now, though that route still fetches no bundle secret, so what it shows is blank — as it is on main. The agent editor's pen was a second editor of the same path that reported nothing: an invalid path never reached the deploy gate the settings drawer's field reports to. It binds to the same slot now. Two things that surfaced with it: a rename never reached the trail, which read the deployed path; and typing the saved path back raised "path already used", since the popover's `initialPath` moves with the rename — `allowedExistingPath` is what says the item already occupies it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: the band's controls come from the design system, and the row has one copy `NavBreadcrumbTrigger` was `MeltButton` written again — same `use:melt` on a plain button, same spread — so the workspace chevron uses the component that already exists, and the sidebar handle is a `Button`. `DetailPageHeader` carried its action row twice, once for the band and once for the row it draws inside a session's preview panel, and the two had already drifted: different dropdown size, no `unifiedSize` on the buttons, and the deprecated `variant="contained" color="light" size="xs"` on the schedule button. Both placements render the same two snippets now, at the size each needs. `OperatorPageKey` admitted every `operator_settings` field, including the three write rights, which `sidebarPageAllowed`'s `=== true` rule answers wrongly: a write right is granted unless withdrawn, so its absence means "never configured". It is the ten visibility flags now. A test over the registry's merge: later registration wins field by field, actions sort by `actionsOrder` then registration, release gives its fields back. That rule is why a component registering where it is not the page renames someone else's band. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the band owns its bottom edge, and one pen renames everywhere The page header decides whether a separator is drawn, so a page cannot draw a second line under it. A registration asks for `onScroll` (the default, drawn only while something passes under the bar), `always` for a page whose content starts right beneath it, or `none` where the page genuinely draws its own first line. Detail pages, the agent editor, workspace settings and the pipeline list take `always`; the raw app editor takes `none`, since its splitpanes already draw that line. The pen that opens the rename popover becomes PathEditPen, shared by the editors and the deployed pages so the two cannot drift, and the deployed pages gain it. SummaryPathDisplay makes the pen the only trigger rather than the whole name block, so the name is selectable text again. Renaming an agent goes through ResourceService with only the fields the popover edits: the backend sets a column per field it is given, so writing the config back would clobber a concurrent edit. The popover also stops clearing a real collision verdict by telling Path which path is its own. Two Button fixes, both reaching call sites beyond this work: - `dropdownItems` given as a function was asked for `.length`, which is its arity and therefore 0, so a split button built that way rendered no caret. Both app editors' Deploy dropdowns were unreachable. - the anchor branch dropped `title`, so a link button had no tooltip. The audit log filters move into the header actions at the design system's sizes and trade absolutely positioned labels for the form layout in brand-guidelines.md: a label above its input, 4px apart, fields 8px apart. The deployed app viewer floors itself at `min-h-full` instead of claiming the viewport, which would overhang the page box by the height of the band, and carries Edit and a public-url link that fold into a menu on a narrow bar. The root layout drops the legacy menu block, which rendered a second hidden sidebar and 54 invisible tab stops, and marks the closed detached sidebar `inert`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: service logs and the audit filters move into the band Service logs: the page's name is the breadcrumb's, so the `<h1>` is gone and the search field, the timeframe and the two switches sit in the band beside it, with the documentation tooltip after the name. ServiceLogsInner registers the band itself, the way RunsPage does, since that route is its only caller, and the `title` snippet the page used to hand it goes with the heading. The search is the widest control there, because searching is what the page is for, and it is the one that gives width back under pressure. The two switches fold into a popover below a 1200px bar rather than at a phone's width, so the field keeps its full width down to a much narrower window — folded rather than dropped, because the page no longer carries a toolbar they could fall back to. The timeframe control took no size of its own, so it defaulted to `md` beside a `sm` field, and sat in a `w-48` box 20px narrower than its content, whose overflow is visible and so spilled into its neighbours. No `separator`: the two panes below start under the bar with borders of their own, so at rest there is nothing for the band to divide. Audit logs: the filters lose their labels. Each select says what it filters while unset — "all usernames", "all resources" — and once set the value names the field by itself. From and To become the TimeframeSelect the runs page uses, which absorbs the standalone Refresh button, since that control's left half is the refresh. `auditTimeframes` joins the runs and service-logs presets, with "All time" first. The bounds stay the filters' own `after`/`before` rather than a remembered preset label: the timeline's zoom writes them too, and a label could then contradict the chart. That takes the row from 1326px to 898px, so it stays inline down to a ~1500px viewport instead of ~2500. The fold measures the row against the bar less what the breadcrumb takes, plus a gap the two must never close — the breadcrumb yields width grudgingly and the row not at all, so sized to just meet they overlap rather than compress, and mid-resize the tooltip after the page's name lands on the timeframe button. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: an embedded page keeps its name in the band A page in a session's preview panel loads in a frame that sets `nomenubar`, so the layout renders the band with `navHidden`. That flag hid the whole breadcrumb — which is what carries the page's name — while the actions beside it had no such guard, leaving a row of controls belonging to nothing. Hiding the trail is still right: it leads to the host's workspace and offers a picker that would take the reader out of the session. So the breadcrumb gains a `nameOnly` mode that drops the workspace part and its picker and keeps the page's own name, and the band asks for it instead of rendering nothing. The summary line loses the same guard, so an item in there shows its path and summary. Only the pages whose own title the band took over are affected. Variables, Resources, Schedules and Triggers still draw their own `PageHeader` and render no band when embedded, so nothing is doubled. The bar's left padding is there to sit a button against — the sidebar handle, or the workspace disc, both of which carry their own margin. Embedded there is neither, and the name started as bare text 8px from the edge, so that case takes the same 16px the bar already keeps on its right. Workspace settings drops the copy-id button from the band: the General tab names the workspace ID with a field of its own, which is where someone looks for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: a session in full screen keeps its tabs in the band Full screen used to drop the session's name from the breadcrumb and leave the tab strip inside the panel. The band now carries both: the session's name where "Sessions" stood, and from the breadcrumb to the right edge one filled surface holding the tabs and the preview's controls, the bar's full height. The switch between the two layouts is the panel expanding to the left. The chat column is resized to nothing rather than unmounted, so it can be seen taking its place -- and so a toggle no longer tears down every mounted chat and rebuilds it on the way back. Panes transition their width only while that switch is in flight, since a splitter drag sets the same inline width and would trail the pointer. The band floats over the content in full screen as it already did beside a panel, so only its right edge moves and the page below never shifts by the bar's height. Leaving full screen, the panel travels back across ground the band has already claimed, so it passes one layer above it for the length of the move: pinned to the width the panel will end at, the bar would otherwise show through as a white patch over the panel's left edge. The breadcrumb also gains the acting fork, the way every other page shows it, which replaces the "Acting on" control the session header carried. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the band's inset survives a resize made in full screen The panel was remembered as a width, sampled while it stood still. A window resized during full screen left that width describing a layout that no longer existed, and the way out pinned the band to it for the length of the move. It is now remembered as the share of the row the panel takes, which the new row width turns back into the right inset: leaving full screen at 1200px after entering it at 1600px, the band starts at 628 rather than at 909. The pane transition's duration is read from the property the timer is set from, so the class cannot outlive the movement it describes, and two comments that described earlier drafts are rewritten: the band's name in full screen comes from the session's own header once it has mounted, and the tab strip's wrapper carries a fade, not a measured move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: an agent's pen keeps the labels the agent already carries The pen on an agent's page was handed the summary and the path but not the labels, so the popover offered an empty list over a resource that had some. It saves every field it shows in one write, and the resource endpoint sets the column it is given, so adding a label there replaced the agent's own: an agent labelled `production` and `scheduled` came back from one edit labelled only with the label just added. A flow's page and a script's page both pass theirs; this one now does too. Verified against the running instance: an agent created with `["production","scheduled"]`, one label added through the pen, reads back `["production","scheduled","new"]` — and `["new"]` with the prop removed again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the sidebar remembers detaching per role, not per account Whether the sidebar starts detached depends on the role: an operator gets it out of the way, a developer gets it docked. That default held only until the control was pressed once, because the choice went to a single key read before the role. From then on both roles wore whichever was pressed last, and the one that had not asked for it could never get its own default back — switch a workspace where you are an operator for one where you are a developer, and the sidebar stayed detached. Each role now keeps its own choice, and falls back to its own default until it has one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: the handle's comment describes what the handle does It claimed a click holds the sidebar card open. Nothing distinguishes a click from a hover there — both call the same opener, and the pointer leaving schedules the close either way — so on a desktop the card follows the pointer and goes when it goes. Measured: hovering slides the rail from -213 to 5, the click moves nothing, and the pointer leaving puts it back at -213. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: a dev workspace's badge wears its fork's own colour Every surface that names a dev workspace drew the same blue badge, which said nothing about which workspace it belonged to. One component, DevWorkspaceBadge, now draws it: a fork with a colour fills the badge with that hue and writes the label in white, and a fork without one keeps whatever its row had before. The hue cannot carry white at a fixed lightness — HSL lightness is not perceived brightness, so yellow and purple at the same L differ by several stops — so forkAccentStyle solves the badge's lightness by bisection instead, taking the lightest shade of the hue that still holds white at WCAG AA. Measured across the wheel the contrast lands between 4.52 and 4.64. The session picker opens with every family collapsed, since the fork a session runs in is the session's own and not something that picker should offer to change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the name is what opens its rename editor, and the trail truncates from its start The pen beside a name is gone. The name itself is the trigger, with a hover fill behind it where the click opens an editor rather than going anywhere — the path segment that only copies keeps no fill, which is what tells the two apart. A bar with a summary field of its own (EditorHeader's condensed pane) still gets the pen, since it has no name to hang the trigger on. Clicking either half of a name opens the editor under that half with the cursor in the matching field. The band draws the path and the page draws the summary, so one popover cannot anchor under both: a page hands the band a `pathTrigger` snippet and the band renders its path segment inside it, passing the classes its own segment wears. Each editor supplies a second PathEditPopover for it, and `focusField` names the field to focus — `#path` explicitly, since melt otherwise lands on the row's first button. Those snippets render from the band's tree, which outlives the values they read by a flush, so each is gated on the value it reads. Without that an agent page whose draft had not loaded threw inside the band and left the breadcrumb empty. The trail now carries two weights: the segment that ends it is what the page is about and takes `font-medium text-emphasis`, everything before it is a step quieter. Which segment that is follows the route — the workspace on home (the fork alone inside a fork), a list page's own name, an item's summary. And it truncates instead of switching to icons at a breakpoint. The workspace part yields first, the family name before the fork, then the path's folders, and the item's own name last. The bar no longer wraps on a phone: a wrapping row never squeezes anything, it hands the trail a line of its own and leaves it whole. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: a page keeps its path segment when it cannot offer a rename Four things the local reviews caught in the rename work: A detail page handed the band a `pathTrigger` whether or not it could save one. Without `onSaved` — every script, flow or agent page the reader cannot write — `SummaryPathDisplay` fell through to its read-only branch, which ignores the `label` it was given and draws its own block: the trail lost its path and the summary appeared twice, two lines deep, in a 44px bar. The page now offers the trigger only where a rename can be saved, and the component renders what it was handed rather than its own row. The band gives up its own segment the moment a page offers a `pathTrigger`, so a trigger whose own gate is closed left the trail with no path at all — for the whole of an agent draft's load, not just a flush. Each gated snippet now draws the path itself in that case. `openFocus` named the path field as `#path`, which melt resolves with a document-wide query; `Path` hardcodes that id at every call site, and the popover is portalled to the body, so it lost document order to any other `Path` on the page. It is scoped to the popover's own field now. And `open` was an exported `$bindable(false)` on an optional prop, which the frontend's own rules ban. No caller bound it, so it is local state. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the badge solves its lightness against the colour the browser paints Three nits from the review round. The contrast bisection measured the hue at full float precision, but a browser paints 8-bit channels: rounding them moved the result up to 0.025 off the target, which took #993388 to 4.478 against white — under the AA floor the comment claims. Quantizing the channels inside `paintedLuminance` puts the worst case across 2880 hue and saturation combinations at 4.500, and your own fork's badge measures 4.53 as painted. An agent nobody can write still offered the rename behind its path segment: an editor with two read-only fields, in place of the click that copies the path. It is gated on `readOnly` now, the way a detail page is gated on `onSaved`. The script and flow editors take the same gate from the `customUi` flags that already decide whether their summary offers one — unreachable today, since the editors own the band only on routes that pass no `customUi`, but the four editors now answer the question the same way. And the registry's doc named an `onEditName` field that has never existed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: an operator is not offered a fork of a workspace A fork does not carry the role. The server writes the forker's membership row in the new workspace from the parent's `is_admin` and lets `operator` fall to its `false` default, so an operator who forks lands in a full-rights copy of the workspace they could only run things in. `canCreateFork` refuses an operator, which takes the entry out of the workspace picker, the session family picker and the session changes bar; `editInForkAllowed` refuses one too, so no editor offers "Edit in fork". The fork page refuses on its own, since a URL reaches it whatever the pickers show. And a workspace the user is an operator in is no longer a base the fork form will accept, so the base picker cannot route around the gate. This hides the affordance; it does not enforce the rule. `create_fork` asks only for membership — `require_admin` covers the dev-workspace case alone — so the API still answers an operator who calls it directly. The server-side fix is to carry `operator` into that membership row, the way the member-copy path already does, and it belongs in a backend change rather than here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * style: the fork refusal uses the design system's alert The refusal panel was hand-built from a div and two paragraphs with literal colors, copied from the Runs page's own operator refusal. The frontend's rules say to use the design-system component, and `Alert` already draws exactly this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: frontend type errors failing npm_check The dev workspace badge accepts the nullable label the API returns, the session strip transitions take the node Svelte passes them, and an unused import is dropped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix: the frontend type-checks again `check frontend build` has been red on every head for a day, through four review rounds that each came back clean — none of the reviewers type-check, and they say so in their own coverage sections. Ten errors: The dev badge's `label` and `color` were typed `string | undefined`, but the workspace rows carry `string | null`, so all seven of its call sites failed. `forkAccentStyle` took the same widening. A Svelte transition is called with the node it animates, and `stripArrive` and `stripLeave` were declared with no parameters at all. And `DevWorkspaceSetting` still imported `Badge` after its badge moved into the shared component. `svelte-check` now reports 0 errors. Note that it only agrees with CI once `src/lib/gen` has been regenerated — the gitignored client goes stale against main and invents errors CI never sees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the home bar carries the CLI and hub buttons while it has the room A menu is where a thing goes when the bar cannot hold it, not where it belongs. Above 1024px of bar, "CLI / MCP" and "Hub" stand as their own buttons; below it they fold back into the ellipsis, which is still where they live on a phone. The threshold reads `pageHeader.barWidth`, not the viewport, so it tracks what the sidebar actually leaves behind. Home's trail is ~250px and its other registered actions ~400, so the two labelled buttons fit from about 850 up; 1024 leaves margin before the trail would start truncating, and it does not truncate at any width between 760 and 1600. The ellipsis hides itself when it would open an empty menu, which happens only once both side trips are in the bar and the composer is already on the page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: a fork's band counts its drafts beside its drift, and every count opens what it names A fork that had drifted stopped saying anything about its drafts: the banner printed the draft count only while the fork was in sync, and the drift badges replaced it the moment there was anything to deploy. A fork can be 86 ahead, 39 behind and holding 128 drafts; it now says so. All four counts open the page that holds them — ahead the deploy direction, behind `?dir=update`, conflicts the side they are resolved from before deploying, drafts the drafts half. The button beside them still chooses a direction on the reader's behalf; a badge means the one thing it names. The colours and the icons are the compare page's, so the counts in the bar and the rows behind them say the same thing the same way: ahead green, behind blue, conflicts orange. Drafts share blue with behind and are told apart by a pencil and by naming what they count, rather than by an arrow that would read as a third direction. Both arrows are straight, here and on the compare page. "Review changes" replaces "Review & Deploy Changes", and it renders in every state. What the comparison has to say — still counting, or failed — is a mark beside the button rather than a branch that replaces it: a fork whose count is unfinished is exactly when a reader wants the page that counts properly. The drafts banner outside a fork drops "This workspace has", so the count reads the same wherever it appears. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the home band keeps its hero's own cross, and the side trips stay quiet The CLI and hub buttons take the shape the hero gave them before the bar did — `xs`, hint-coloured, the hub keeping its outbound mark — so a side trip does not read as loudly as the thing the page is for. The ellipsis is gone while the hero is on the page: the hero's cross already puts it away, so a menu whose only item repeats that is a button that does nothing new. It comes back when it has something of its own to carry, which is the side trips on a narrow bar and the way back to a hero that has been put away. That entry is now "Pin session hero to homepage", and "Remove session chat" survives only as the cross's own words, where one action keeps one name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: a fork's counts and the way to review them are one control Four badges beside a button read as four things that happen to sit together. They are one thing — every segment opens the same page on the part it names — so they now share a box: one outline, square corners inside it, and the rounding on the box alone, which means no segment has to know where in the row it sits. The only rule in the strip is the one before the end cap: everything left of it is a reading, that is an action. With room every count says what it counts; below 1280px of bar they keep their icon and their number, which is the part a reader scans for, and the tooltip names them either way. The end cap keeps its words at every width — the counts have icons to fall back on and it has none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the compare page asks once, says it is working, and knows its own name The page re-requested the same summaries and on-behalf-of emails forever. Two faults compounding: `fetchOnBehalfOfInfo` skipped a key it had already answered with `onBehalfOfInfo[k] !== undefined`, but stored `undefined` on failure — so "asked, and there was nothing" could not be told from "never asked". An item that is only ahead does not exist in the parent, so that side always fails: a permanently un-cacheable set, re-requested on every pass. And the effect that drives both fetchers depended on what they write. Each reads its cache synchronously before its first `await`, which registers that cache as a dependency; the write that follows re-runs the effect. Write, re-run, write. Both now record what they asked for in a plain Set, marked before the request rather than after the answer, and the effect is untracked so it depends on the comparison alone. The caches are cleared when the compared pair changes, since a summary's key carries no workspace and those Sets would otherwise never ask again. While the comparison was in flight the page printed "No changes between this workspace and X" — an empty list and an unanswered one look alike from the layout, and that is a claim with no answer behind it. The page tracks the request now, and the layout spins instead. The breadcrumb called the page "Forks", which is what `navPageFor` falls back to when a route names itself nowhere: the first segment, capitalised. It registers "Compare" instead — in the page rather than in NAV_PAGES, which is what the sidebar renders from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Revert "feat: a fork's counts and the way to review them are one control" This reverts commite4504d2843. The counts go back to separate badges and "Review changes" to a button of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the compare page follows the fork picked in the trail Switching a fork from the breadcrumb left the page comparing the pair it opened with. Two reasons, neither reactive: the page seeds `currentWorkspaceId` from `?workspace_id` once at mount and never reads it again, and the switch rewrites `?workspace`, a different param that this page does not read. It follows `$workspaceStore` now — picking a fork in the trail means comparing that fork. The param is rewritten so a reload keeps the pair on screen, and an arbitrary `?target=` is dropped, since it named a destination for the workspace just left. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: the band is a step taller, and the runs row sizes with it The bar goes from h-11 to h-12, taken from one constant beside the top padding a floating band owes the page under it, so the two cannot drift. The runs page's controls now share the sm height the rest of a header's actions use: FilterSearchbar learns the size, dropping the top margin that only centres text at md and taking the shared leading instead, and sm buttons get px-3 so a label reads as a button. The actions sit in their own row with a wider gap, and the chart below meets the header with no gap of its own. The search field holds one width through typing. As flex-1 it was whatever the row had left over, so committing a chip grew it from 226 to 288 and took those pixels off the breadcrumb. A fork's count badges keep their labels while the bar is at least 1280 wide and fall back to icon and number below that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the trail holds a path's place instead of naming its kind A page registers its kind when it mounts and its path once it has loaded one. In between the trail printed "script" or "flow", which reads as the answer and then jumps when the real path lands. A skeleton of a plausible path's width holds the place, with aria-busy and a label naming what is loading. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the compare page reads the workspace it was linked with The effect that follows the trail's fork also ran on mount, so a `?workspace_id` that differs from the navigation workspace was thrown away. That difference is the contract: a session's Review button passes the fork it committed to and leaves the navigation workspace alone, as does the prod -> dev link in UpdateDevWorkspaceModal. Both opened on the sidebar's workspace instead, comparing the wrong pair or none at all. Only a change in the store now retargets the page, measured against what it held when the page opened. The URL rewriting goes with it. The picker's items are links that carry every other param forward, so their navigation lands after the effect and put back the `workspace_id` it had just rewritten: switching fork and reloading left the trail naming one workspace and the page comparing another. The page reads `?workspace=` ahead of `?workspace_id=` instead — the param the picker writes and the root layout already applies to the store — and `?target=` lapses while the page is on another workspace rather than being deleted from a URL someone else is about to replace. Two 44px literals left over from the h-11 band: the detached sidebar card's measured-top fallback and the rail morph's end state, both now from BAR_HEIGHT_PX beside the class the band wears. The runs search field keeps its 18rem without growing, but as `shrink-0` nothing in that row could give width back, so a short bar had no answer but the trail's truncation and an overlong row would have run past the edge. `min-w-0` restores the yield without restoring the growth: a flex item's floor is its min-content width, which the chips inside the field raise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the compare page's target picker works, and a switch survives reload `?target=` was gated on the page still being on the workspace the URL named. Opened from "Merge into another workspace" in the workspace settings the URL names no workspace at all, so that comparison was `<id> === undefined` and the param was ignored for good: picking a target left the page on the "no parent workspace" prompt, which is the only place the migration can be started from. Read the param as it was read before. The follow effect writes `?workspace=` again. A plain click in the picker preventDefaults its own link and switches the store directly, so nothing navigates afterwards and nothing overwrites the write — while `fixupUrlAfterWorkspaceSwitch` rewrites the param only once it is already there, which on a page opened with `workspace_id` alone it never is. Without it, switching fork and reloading left the trail naming one workspace and the page comparing another. Exercised on a fork pair: the settings entry's picker loads the comparison it names, a plain-click switch survives a reload, and a `workspace_id` deep link still outranks the navigation workspace. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: the compare page reads its pair instead of following it Net effect count: three removed, none added. The page derives the workspace it compares from the URL and the store rather than keeping it in state and running an effect to follow a switch. The workspace-switch link is what moves it: the picker's items carry every param forward, so a page-side rewrite raced that navigation and lost — the trace showed the fixup's replaceState landing first and the link's pushState putting the old params straight back. The href now swaps `workspace_id` alongside `workspace` and drops `target`, and `fixupUrlAfterWorkspaceSwitch` does the same for the paths that do not navigate, so both writers agree and the order stops mattering. Dropping `target` on a switch also closes the case where switching onto the workspace that was the target left the page comparing it with itself. CompareWorkspaces remounts on either side of the pair changing, not just the target: two forks of one parent shared a mount, since the target is the parent they have in common, and with it the selection, the deploy direction, the summaries, the pinned rows, the deploy permissions and the CI results. The cache-clearing effect that stood in for that is gone. Its summary loops outlive their mount, so they read the pair once at the start and stop writing when the instance is destroyed. The app viewer asked for a public secret on every app view to decide whether to draw a button; it reads the mode off the app it has already loaded and fetches the secret when the button is used. The summary popover seeds its fields on the open transition rather than from an effect that also read the item: a save landing while it was open reseeded the fields over what was being typed. Exercised against a running instance, 18 cases: idle issues nothing, a switch moves page, trail and URL together and survives a reload, a deep `workspace_id` link still outranks the navigation workspace, four fast switches settle with no foreign row, and the migration target picker loads the pair it names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the public url opens the tab the click itself opened The secret is fetched when the button is used, and the tab was opened once it arrived — after an await, where Safari no longer counts the click as the gesture that opened it and blocks the popup without saying so. Open the tab synchronously and point it at the url once the secret is known, closing it and saying why when there is none. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the public url button survives a sandboxed app A sandboxed low-code app loads inside the opaque /app_embed iframe, so `loadApp` never runs on this page and `app` stays undefined — taking the button and its compact-menu entry with it, for the owner too. The mode now comes from the lite read that already runs for the Edit button, which is there for exactly this reason: it is the one description of the app this page has whether or not the app itself loads here. Exercised on four apps: anonymous sandboxed low-code, anonymous unsandboxed low-code and an anonymous raw app all show the button and fetch the secret only on click; a publisher app shows neither and asks nothing. At 640px the button folds into the menu, which carries it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: the removed hero leaves a blue flash on the ellipsis The mark that flew from the cross to the menu is gone, with its two element refs, its flight state and timer, and the `data-home-ai-menu` hook that existed only as its destination. What is left is the thing it was there to say: the ellipsis holds the way back, so it flashes blue as it appears. Written as a keyframe animation, not a transition. A transition needs the element painted at its start value before the end value is set, and this mounts in the same flush that collapses the hero, so the frame that would have shown it at rest is the frame it is already told to leave — one `requestAnimationFrame` is not reliably past that paint, and the flash was never drawn at all. `animate-fade-out` already exists for this (AssetsDropdownButton flashes a background the same way); mounting it only while pulsing gets a fresh run per removal without that component's reflow juggling. The flash rides the ellipsis rather than the row: with the CLI and hub buttons beside it the row is three controls wide, and marking all three names none of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the flash rests invisible, and its comment describes what is there `animate-fade-out` carries no fill mode, so at the end of the run the element returns to its own opacity. At the default 1 that is the flash coming back at full strength for as long as the unmount timer lags the animation — which the hero's collapse, happening right there, is well placed to cause. `opacity-0` at rest makes the revert land on invisible. The comment above it still described the mark that used to fly to the menu, and made its point three times over. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the compare page asks only about the pair it is showing Switching workspace on this page fetched the previous pair's item paths from the new workspace, where none of them exist: 97 of 174 requests 404ing on one switch. The comparison was plain state, and the card is remounted when the pair changes — in the same render that changes it, before the effect that cleared the comparison could run, so the new mount started from the old pair's diffs. It now carries the pair it describes and is read only while that matches, which leaves nothing to clear. `computeFullScan` had the same shape: it read the target again after its await, so a switch mid-scan invalidated another pair's cache, named the wrong workspace in its toast, and left the scan button spinning on a pair that was never scanned. It captures the pair up front, re-compares only while the page is still on it, and reports it as its own. Each diff says which side holds the item, so the summary and on-behalf-of loops stop asking the side that does not: an item that is only ahead cannot answer in the parent, and asking was a guaranteed 404 and a console error per one-sided row. Those two loops also read the same object — a summary and an identity are fields of one payload — so every row was fetched twice per workspace. One read per item per workspace now serves both. A failed read is not kept: held, it would read as "this item has no identity", which is what decides whether a deploy offers to preserve one, and an item deployed without it silently becomes the deployer's. The header names the workspace being compared when that is not the one the app is pointed at, which is how a session's Review button and the prod -> dev link open this page. The trail said the parent while the page compared the fork. Measured on a 90-item pair, before -> after: a cold load 247 -> 170 requests and 167 -> 90 item reads; a workspace switch 174 requests with 97 404s -> 71 with none. Sixteen cases over cold load, idle, switching in both directions, four switches with no time to settle, a cross-family switch, reload, deep link, the migration picker and leaving an arbitrary target: no request in any of them names a pair the page has left. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: the scan toast reports the pair it scanned Off the scanned pair the re-compare is skipped, so there is nothing to report as compared — but the success branch said "Compared N items" anyway, and the error it checked first belonged to whichever pair the page had moved to. Each outcome now says what happened: the seed alone when the page has moved on, and the comparison's result only where one was asked for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Guilhem Lemouel <guilhemlemouel@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
This commit is contained in:
co-authored by
Claude Opus 5
Guilhem Lemouel
Ruben Fiszel
parent
a85d0b76cc
commit
3dae553576
Generated
+40
-2
@@ -7915,10 +7915,48 @@ A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates step
|
||||
A pipeline node that produces a table should almost always be a **\`duckdb\`** node that materializes its output into a **DuckLake** table with \`-- materialize ducklake://<name>/<table>\` (in a DuckDB node the annotation uses SQL \`--\` comment syntax; write the body as a bare \`SELECT\` and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else:
|
||||
|
||||
- \`postgresql\` / data tables — only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live).
|
||||
- \`bun\` / \`python3\` — only for non-tabular work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node still produces tabular data for downstream steps, land it in DuckLake (write it with the wmill SDK / ducklake helpers) rather than inventing a parallel store.
|
||||
- \`bun\` / \`python3\` — only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it **straight into DuckLake** (or a data table) from the same script — see "Ingesting from Python / TypeScript" below — rather than inventing a parallel store.
|
||||
|
||||
Do not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal.
|
||||
|
||||
## Ingesting from Python / TypeScript
|
||||
|
||||
A node that fetches data in code (an API, a SaaS export, a scraper) writes the rows **directly** into their DuckLake table with \`wmill.ducklake()\`, or into a data table with \`wmill.datatable()\`. DuckLake already stores every table as parquet files in the workspace object storage, so **never stage the raw payload first** — no intermediate JSON / CSV / parquet file in S3 and no second "load the raw file" node. One ingestion node, one table.
|
||||
|
||||
Pass the rows as a query argument (a list of dicts / array of objects is sent as JSON) and let SQL unnest them; declare the column types so the table gets a real schema:
|
||||
|
||||
\`\`\`python
|
||||
# pipeline
|
||||
# on schedule
|
||||
import wmill
|
||||
|
||||
def main():
|
||||
rows = fetch_charges() # list[dict]
|
||||
lake = wmill.ducklake()
|
||||
lake.query("""
|
||||
CREATE OR REPLACE TABLE stripe_charges AS
|
||||
SELECT unnest(from_json($rows, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true)
|
||||
""", rows=rows).fetch()
|
||||
\`\`\`
|
||||
|
||||
\`\`\`ts
|
||||
// pipeline
|
||||
// on schedule
|
||||
import * as wmill from "windmill-client"
|
||||
|
||||
export async function main() {
|
||||
const rows = await fetchCharges()
|
||||
const lake = wmill.ducklake()
|
||||
await lake\`CREATE OR REPLACE TABLE stripe_charges AS
|
||||
SELECT unnest(from_json(\${rows}, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true)\`.fetch()
|
||||
}
|
||||
\`\`\`
|
||||
|
||||
- Use \`INSERT INTO <table> SELECT ...\` instead of \`CREATE OR REPLACE\` to append (an incremental pull), with a one-time \`CREATE TABLE IF NOT EXISTS\` first.
|
||||
- Assign the client to a variable (\`lake = wmill.ducklake()\`) and keep the SQL a **string literal** (Python) or a template without \`sql.raw\` (TS): that is what lets the graph record the node's write to \`ducklake://main/<table>\`. A chained \`wmill.ducklake().query(...)\` or SQL built in a variable records no output, so the downstream edge never forms.
|
||||
- \`wmill.ducklake("<name>")\` targets a non-default catalog; \`wmill.datatable("<name>")\` writes a Postgres data table the same way (Python \`db.query(sql, *args)\` with \`$1\` placeholders, TS the same template form) — use it only when the data belongs in Postgres (see above).
|
||||
- The downstream node triggers on the table itself: \`-- on ducklake://main/stripe_charges\`.
|
||||
|
||||
## Storage prerequisites
|
||||
|
||||
A DuckLake pipeline only runs once the workspace has **object storage** (S3 / Azure Blob / GCS) **and a DuckLake catalog** configured — DuckLake tables and \`s3://\` assets can't be materialized or read without it. Check which DuckLake catalogs the workspace has before you build.
|
||||
@@ -7962,7 +8000,7 @@ The key must be a **string literal** — the graph parser is static and cannot f
|
||||
|
||||
## Materialize (the managed output)
|
||||
|
||||
> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake://<name>/<table>\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the output via the SDK (\`wmill.writeS3File(...)\`, ducklake helpers, …) and let it be inferred. Use \`duckdb\` when a node should materialize a DuckLake table.
|
||||
> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake://<name>/<table>\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the table itself with \`wmill.ducklake()\` (see "Ingesting from Python / TypeScript") and let the output be inferred. Use \`duckdb\` when a node should materialize a DuckLake table.
|
||||
>
|
||||
> The one target ANY language may declare (except a dbt script, whose writes come from its manifest) is a **warehouse relation**: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, where \`<warehouse>\` is a warehouse the workspace configures under Settings → dbt. \`manual\` is the only mode it has — nothing generates warehouse DDL, so the node issues its own write (a postgresql \`CREATE TABLE\` / \`INSERT\`, an SDK load, …) and the annotation records the outcome. Use it on an ingestion node whose output a dbt project reads as a \`source\`: the declared relation and the dbt model land on ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when the ingestion node completes.
|
||||
|
||||
|
||||
@@ -142,7 +142,13 @@
|
||||
let hintTimer: ReturnType<typeof setTimeout> | undefined
|
||||
|
||||
const kindLabel = $derived(
|
||||
itemKind === 'flow' ? 'flow' : itemKind === 'app' || itemKind === 'raw_app' ? 'app' : 'script'
|
||||
itemKind === 'flow'
|
||||
? 'flow'
|
||||
: itemKind === 'app' || itemKind === 'raw_app'
|
||||
? 'app'
|
||||
: itemKind === 'data_pipeline'
|
||||
? 'pipeline'
|
||||
: 'script'
|
||||
)
|
||||
|
||||
// Each false → true transition fires a hint (others > loaded). Seeded to
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
contentClasses="p-4"
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<Button variant="accent" size="xs" nonCaptureEvent={true} startIcon={{ icon: Clock }}>
|
||||
<Button variant="default" unifiedSize="sm" nonCaptureEvent={true} startIcon={{ icon: Clock }}>
|
||||
Cron builder
|
||||
</Button>
|
||||
{/snippet}
|
||||
|
||||
@@ -17,6 +17,9 @@
|
||||
selectInputClass?: string
|
||||
class?: string
|
||||
onClear?: () => void
|
||||
/** Size the input to its content, like Path's folder picker. */
|
||||
useContentEditable?: boolean
|
||||
RightIcon?: any
|
||||
}
|
||||
|
||||
let {
|
||||
@@ -27,7 +30,9 @@
|
||||
placeholder = undefined,
|
||||
selectInputClass = '',
|
||||
class: className = '',
|
||||
onClear = undefined
|
||||
onClear = undefined,
|
||||
useContentEditable = false,
|
||||
RightIcon = undefined
|
||||
}: Props = $props()
|
||||
|
||||
let datatables = usePromise(() =>
|
||||
@@ -47,6 +52,8 @@
|
||||
{placeholder}
|
||||
inputClass={selectInputClass}
|
||||
{onClear}
|
||||
{useContentEditable}
|
||||
{RightIcon}
|
||||
/>
|
||||
{#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'datatable', path: value })}
|
||||
<ExploreAssetButton class="mt-1 w-fit" asset={{ kind: 'datatable', path: value }} />
|
||||
|
||||
@@ -17,6 +17,9 @@
|
||||
selectInputClass?: string
|
||||
class?: string
|
||||
onClear?: () => void
|
||||
/** Size the input to its content, like Path's folder picker. */
|
||||
useContentEditable?: boolean
|
||||
RightIcon?: any
|
||||
}
|
||||
|
||||
let {
|
||||
@@ -27,7 +30,9 @@
|
||||
placeholder = undefined,
|
||||
selectInputClass = '',
|
||||
class: className = '',
|
||||
onClear = undefined
|
||||
onClear = undefined,
|
||||
useContentEditable = false,
|
||||
RightIcon = undefined
|
||||
}: Props = $props()
|
||||
|
||||
let ducklakes = usePromise(() =>
|
||||
@@ -45,6 +50,8 @@
|
||||
{placeholder}
|
||||
inputClass={selectInputClass}
|
||||
{onClear}
|
||||
{useContentEditable}
|
||||
{RightIcon}
|
||||
/>
|
||||
{#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'ducklake', path: value })}
|
||||
<ExploreAssetButton class="mt-1 w-fit" asset={{ kind: 'ducklake', path: value }} />
|
||||
|
||||
@@ -1523,6 +1523,9 @@
|
||||
} catch (e) {
|
||||
console.log('error initializing services', e)
|
||||
}
|
||||
// Unmounted while the services loaded: the container is detached, and
|
||||
// creating the editor on it throws.
|
||||
if (destroyed || !divEl?.isConnected) return
|
||||
|
||||
// vscode.languages.registerDefinitionProvider('*', {
|
||||
// provideDefinition(document, position, token) {
|
||||
|
||||
@@ -2101,43 +2101,15 @@
|
||||
</div>
|
||||
{/if}
|
||||
{#if customUi?.previewPanel?.hideArgs}
|
||||
<!-- Compact preview layout used by the pipeline editor:
|
||||
no args column (the script is known to take no
|
||||
inputs), LogPanel takes the full width, with a
|
||||
small Test/Cancel button at the top-left of the
|
||||
preview band. The earlier `-translate-y-1/2`
|
||||
"float onto the editor" version was clipped in
|
||||
Firefox (transform + overflow: visible interaction
|
||||
on the parent splitpane Pane), so the button is
|
||||
now positioned inside the panel — `top-1` keeps
|
||||
it visually pinned to the top edge without
|
||||
relying on cross-browser overflow behaviour. -->
|
||||
<div class="relative h-full pt-9 flex flex-col">
|
||||
{#if testJob?.id && testJob.type === 'CompletedJob' && opWs}
|
||||
<!-- Right-side affordances when we're displaying a *completed*
|
||||
job (either the user just ran a test, or the on-mount
|
||||
last-run loader populated the panel). The job-id link
|
||||
opens the full run page in a new tab; the dispatch
|
||||
button mounts a popover that shows what downstream
|
||||
jobs this run triggered (rendered only when the run
|
||||
actually dispatched anything). -->
|
||||
<div class="absolute top-1 right-2 z-10 flex items-center gap-2">
|
||||
<a
|
||||
class="text-3xs text-blue-600 hover:underline font-mono"
|
||||
href={`${base}/run/${testJob.id}?workspace=${opWs}`}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
title="Open this run"
|
||||
>
|
||||
{testJob.id.slice(0, 8)}… ↗
|
||||
</a>
|
||||
<DispatchEventsButton
|
||||
workspace={testJob.workspace_id ?? opWs}
|
||||
jobId={testJob.id}
|
||||
/>
|
||||
</div>
|
||||
{/if}
|
||||
<div class="absolute top-1 left-2 z-10">
|
||||
<!-- Compact preview layout used by the pipeline editor: no args
|
||||
column (most pipeline scripts take no inputs) and a full-width
|
||||
LogPanel. The controls sit in normal flow rather than floating
|
||||
onto the editor band, which Firefox clipped (transform +
|
||||
overflow: visible on the parent splitpane Pane). -->
|
||||
<div class="h-full flex flex-col">
|
||||
<!-- Test sits at the top right, the last run's link under it. -->
|
||||
<div class="shrink-0 flex flex-col items-end gap-1 px-2 pt-2 pb-1">
|
||||
<div>
|
||||
{#if testIsLoading}
|
||||
{@render cancelTestButton('sm', 'shadow-md')}
|
||||
{:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0 || customUi?.previewPanel?.onBoundedRun}
|
||||
@@ -2271,6 +2243,31 @@
|
||||
{@render runTestButton('sm', 'shadow-md')}
|
||||
{/if}
|
||||
</div>
|
||||
{#if testJob?.id && testJob.type === 'CompletedJob' && opWs}
|
||||
<!-- Right-side affordances when we're displaying a *completed*
|
||||
job (either the user just ran a test, or the on-mount
|
||||
last-run loader populated the panel). The job-id link
|
||||
opens the full run page in a new tab; the dispatch
|
||||
button mounts a popover that shows what downstream
|
||||
jobs this run triggered (rendered only when the run
|
||||
actually dispatched anything). -->
|
||||
<div class="flex items-center gap-2">
|
||||
<a
|
||||
class="text-3xs text-blue-600 hover:underline font-mono"
|
||||
href={`${base}/run/${testJob.id}?workspace=${opWs}`}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
title="Open this run"
|
||||
>
|
||||
{testJob.id.slice(0, 8)}… ↗
|
||||
</a>
|
||||
<DispatchEventsButton
|
||||
workspace={testJob.workspace_id ?? opWs}
|
||||
jobId={testJob.id}
|
||||
/>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
{#if customUi?.previewPanel?.argsAboveLogs && schema?.properties && Object.keys(schema.properties).length > 0}
|
||||
<div
|
||||
class="shrink-0 overflow-auto max-h-[45%] border-b border-gray-200 dark:border-gray-700 px-4 py-3"
|
||||
|
||||
@@ -31,6 +31,13 @@
|
||||
/** What is being renamed. An agent is a resource, so the path field is scoped to one and
|
||||
* the write goes through the agent branch of the rename manager. */
|
||||
kind?: 'flow' | 'script' | 'agent'
|
||||
/** Saves instead of the API, for an item that exists only locally (a draft).
|
||||
* Returns an error message to keep the popover open. */
|
||||
saveOverride?: (next: {
|
||||
path: string
|
||||
summary: string
|
||||
labels: string[] | undefined
|
||||
}) => string | undefined
|
||||
/** Header variant: no path line (the host shows it) and a lighter summary. */
|
||||
compact?: boolean
|
||||
/** Rendered inside the trigger in place of the summary, for a host that hangs a second copy
|
||||
@@ -51,6 +58,7 @@
|
||||
editable = false,
|
||||
onSaved,
|
||||
kind = 'flow',
|
||||
saveOverride = undefined,
|
||||
compact = false,
|
||||
label,
|
||||
triggerClass = 'block min-w-0 max-w-full px-1 py-0.5 rounded text-left cursor-pointer hover:bg-surface-hover transition-colors',
|
||||
@@ -67,7 +75,9 @@
|
||||
let popoverOpen = $state(false)
|
||||
// Resolved before the popover opens, not on opening it: the path field only exists for an owner,
|
||||
// and `openFocus` looks for it in the same flush the popover renders in.
|
||||
const own = $derived(isOwner(path ?? '', actingUser, $operatingWorkspace))
|
||||
const own = $derived(
|
||||
saveOverride !== undefined || isOwner(path ?? '', actingUser, $operatingWorkspace)
|
||||
)
|
||||
let onBehalfOfEmail = $state<string | undefined>(undefined)
|
||||
let summaryInput: ReturnType<typeof TextInput> | undefined = $state()
|
||||
let labelsDirty = $state(false)
|
||||
@@ -101,6 +111,17 @@
|
||||
async function save(close: () => void) {
|
||||
const initialPath = path ?? ''
|
||||
const newPath = own ? editPath : initialPath
|
||||
if (saveOverride) {
|
||||
const error = saveOverride({ path: newPath, summary: editSummary, labels })
|
||||
if (error) {
|
||||
sendUserToast(error, true)
|
||||
return
|
||||
}
|
||||
labelsDirty = false
|
||||
close()
|
||||
onSaved?.(newPath)
|
||||
return
|
||||
}
|
||||
|
||||
try {
|
||||
await updateItemPathAndSummary({
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
<!--
|
||||
@component
|
||||
The menu that adds a pipeline script started by a data source (a schedule, a
|
||||
webhook, a queue…): the add node's, and the canvas's right-click one.
|
||||
-->
|
||||
<script lang="ts">
|
||||
import type { Snippet } from 'svelte'
|
||||
import PipelineInsertMenu, {
|
||||
type PipelineInsertOptions,
|
||||
type PipelineInsertPick
|
||||
} from './PipelineInsertMenu.svelte'
|
||||
import {
|
||||
Clock,
|
||||
Webhook,
|
||||
Mail,
|
||||
Zap,
|
||||
Radio,
|
||||
MessageSquare,
|
||||
Database,
|
||||
Send,
|
||||
CloudCog,
|
||||
Upload
|
||||
} from 'lucide-svelte'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import type { NativeTriggerKind } from './types'
|
||||
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
|
||||
import type { PipelineOutputKind } from './pipelineTemplates'
|
||||
|
||||
interface Props {
|
||||
onAddPipelineScript: (
|
||||
language: ScriptLang,
|
||||
path: string,
|
||||
source: { kind: NativeTriggerKind; path: string | undefined },
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string,
|
||||
options?: PipelineInsertOptions
|
||||
) => void
|
||||
pathPrefix: string
|
||||
defaultPathSuffix: string
|
||||
trigger: Snippet<[{ open: boolean }]>
|
||||
/** Opens the menu each time it changes. */
|
||||
openSignal?: number
|
||||
}
|
||||
let {
|
||||
onAddPipelineScript,
|
||||
pathPrefix,
|
||||
defaultPathSuffix,
|
||||
trigger,
|
||||
openSignal
|
||||
}: Props = $props()
|
||||
|
||||
function handlePick(pick: PipelineInsertPick) {
|
||||
if (!pick.language || !pick.path) return
|
||||
// Native trigger annotation is marker-only — the binding lives on the
|
||||
// trigger row's own `script_path`, which is created separately.
|
||||
onAddPipelineScript(
|
||||
pick.language as ScriptLang,
|
||||
pick.path,
|
||||
{ kind: pick.kindId as NativeTriggerKind, path: undefined },
|
||||
(pick.outputKind ?? 'none') as PipelineOutputKind,
|
||||
pick.aiPrompt,
|
||||
{ schedule: pick.schedule, outputAsset: pick.outputAsset }
|
||||
)
|
||||
}
|
||||
</script>
|
||||
|
||||
<PipelineInsertMenu
|
||||
kinds={[
|
||||
{
|
||||
id: 'data_upload',
|
||||
label: 'On data upload',
|
||||
description: 'UI-first: run by uploading a file via the S3 picker',
|
||||
icon: Upload
|
||||
},
|
||||
{
|
||||
id: 'schedule',
|
||||
label: 'On schedule',
|
||||
description: 'Triggered by a schedule you create',
|
||||
icon: Clock
|
||||
},
|
||||
{
|
||||
id: 'webhook',
|
||||
label: 'On webhook',
|
||||
description: 'Triggered by an HTTP webhook',
|
||||
icon: Webhook
|
||||
},
|
||||
{
|
||||
id: 'email',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On email',
|
||||
description: 'Triggered by incoming email',
|
||||
icon: Mail
|
||||
},
|
||||
{
|
||||
id: 'kafka',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On Kafka',
|
||||
description: 'Triggered by a Kafka message',
|
||||
icon: Zap
|
||||
},
|
||||
{
|
||||
id: 'mqtt',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On MQTT',
|
||||
description: 'Triggered by an MQTT message',
|
||||
icon: Radio
|
||||
},
|
||||
{
|
||||
id: 'amqp',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On AMQP',
|
||||
description: 'Triggered by an AMQP (RabbitMQ) message',
|
||||
icon: Radio
|
||||
},
|
||||
{
|
||||
id: 'nats',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On NATS',
|
||||
description: 'Triggered by a NATS message',
|
||||
icon: MessageSquare
|
||||
},
|
||||
{
|
||||
id: 'postgres',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On Postgres',
|
||||
description: 'Triggered by a Postgres event',
|
||||
icon: Database
|
||||
},
|
||||
{
|
||||
id: 'sqs',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On SQS',
|
||||
description: 'Triggered by an SQS message',
|
||||
icon: Send
|
||||
},
|
||||
{
|
||||
id: 'gcp',
|
||||
configuredAfterCreate: true,
|
||||
label: 'On GCP Pub/Sub',
|
||||
description: 'Triggered by a Pub/Sub message',
|
||||
icon: CloudCog
|
||||
}
|
||||
]}
|
||||
languages={PIPELINE_LANGUAGES as any}
|
||||
{pathPrefix}
|
||||
{defaultPathSuffix}
|
||||
onPick={handlePick}
|
||||
{openSignal}
|
||||
{trigger}
|
||||
/>
|
||||
@@ -0,0 +1,73 @@
|
||||
<!--
|
||||
@component
|
||||
The menu adding a pipeline script downstream of an asset: opened from the asset's
|
||||
"+", or where that "+" was dropped on the blank canvas.
|
||||
-->
|
||||
<script lang="ts" module>
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import type { AssetKind } from '$lib/components/assets/lib'
|
||||
import type { PipelineInsertOptions } from './PipelineInsertMenu.svelte'
|
||||
import type { PipelineOutputKind } from './pipelineTemplates'
|
||||
|
||||
export type AddScriptForAsset = (
|
||||
asset: { kind: AssetKind; path: string },
|
||||
language: ScriptLang,
|
||||
scriptPath: string,
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string,
|
||||
options?: PipelineInsertOptions
|
||||
) => void
|
||||
</script>
|
||||
|
||||
<script lang="ts">
|
||||
import { Code2 } from 'lucide-svelte'
|
||||
import type { Snippet } from 'svelte'
|
||||
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
|
||||
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
|
||||
|
||||
let {
|
||||
asset,
|
||||
onAddScript,
|
||||
pathPrefix = '',
|
||||
defaultPathSuffix = '',
|
||||
openSignal,
|
||||
trigger
|
||||
}: {
|
||||
asset: { kind: AssetKind; path: string }
|
||||
onAddScript: AddScriptForAsset
|
||||
pathPrefix?: string
|
||||
defaultPathSuffix?: string
|
||||
openSignal?: number
|
||||
trigger: Snippet<[{ open: boolean }]>
|
||||
} = $props()
|
||||
|
||||
function handlePick(pick: PipelineInsertPick) {
|
||||
if (pick.kindId === 'pipeline_script' && pick.language && pick.path) {
|
||||
onAddScript(
|
||||
asset,
|
||||
pick.language as ScriptLang,
|
||||
pick.path,
|
||||
(pick.outputKind ?? 'none') as PipelineOutputKind,
|
||||
pick.aiPrompt,
|
||||
{ outputAsset: pick.outputAsset }
|
||||
)
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<PipelineInsertMenu
|
||||
kinds={[
|
||||
{
|
||||
id: 'pipeline_script',
|
||||
label: 'Add downstream pipeline script',
|
||||
description: 'Triggered when this asset changes',
|
||||
icon: Code2
|
||||
}
|
||||
]}
|
||||
languages={PIPELINE_LANGUAGES as any}
|
||||
{pathPrefix}
|
||||
{defaultPathSuffix}
|
||||
{openSignal}
|
||||
onPick={handlePick}
|
||||
{trigger}
|
||||
/>
|
||||
@@ -0,0 +1,23 @@
|
||||
<!--
|
||||
@component
|
||||
The "+" on an asset's bottom edge: a circle at rest, widened into a labelled pill
|
||||
while its menu is open or while it is being dragged. The label animates its
|
||||
max-width (width can't transition to auto) and never wraps.
|
||||
-->
|
||||
<script lang="ts">
|
||||
import { Plus } from 'lucide-svelte'
|
||||
|
||||
let { expanded }: { expanded: boolean } = $props()
|
||||
</script>
|
||||
|
||||
<span
|
||||
class="group/add h-6 min-w-6 rounded-full flex items-center justify-center bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:text-primary hover:border-gray-500 shadow-sm leading-none overflow-hidden transition-[padding] duration-200 ease-out data-[open=true]:pl-1"
|
||||
data-open={expanded}
|
||||
>
|
||||
<Plus size={16} strokeWidth={2.5} class="shrink-0" />
|
||||
<span
|
||||
class="max-w-0 opacity-0 whitespace-nowrap text-3xs font-normal transition-[max-width,opacity,margin] duration-200 ease-out group-data-[open=true]/add:max-w-32 group-data-[open=true]/add:opacity-100 group-data-[open=true]/add:ml-1 group-data-[open=true]/add:mr-2"
|
||||
>
|
||||
Add downstream step
|
||||
</span>
|
||||
</span>
|
||||
@@ -1,145 +1,36 @@
|
||||
<script lang="ts">
|
||||
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
|
||||
import {
|
||||
Plus,
|
||||
Clock,
|
||||
Webhook,
|
||||
Mail,
|
||||
Zap,
|
||||
Radio,
|
||||
MessageSquare,
|
||||
Database,
|
||||
Send,
|
||||
CloudCog,
|
||||
Upload
|
||||
} from 'lucide-svelte'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import type { NativeTriggerKind } from './types'
|
||||
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
|
||||
import type { PipelineOutputKind } from './pipelineTemplates'
|
||||
|
||||
// Each left-column kind is just "pipeline script triggered by <trigger
|
||||
// source>". id === the SCRIPT_TRIGGER_KIND value, so the handler can
|
||||
// dispatch on it uniformly. Asset-triggered scripts are not in this
|
||||
// menu; those live under the per-asset + inside the graph.
|
||||
type KindId = NativeTriggerKind
|
||||
import AddDataSourceMenu from './AddDataSourceMenu.svelte'
|
||||
import { Plus } from 'lucide-svelte'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
|
||||
interface Props {
|
||||
data: {
|
||||
onAddPipelineScript: (
|
||||
language: ScriptLang,
|
||||
path: string,
|
||||
source: { kind: NativeTriggerKind; path: string | undefined },
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string
|
||||
) => void
|
||||
pathPrefix: string
|
||||
defaultPathSuffix: string
|
||||
}
|
||||
data: Pick<
|
||||
import('svelte').ComponentProps<typeof AddDataSourceMenu>,
|
||||
'onAddPipelineScript' | 'pathPrefix' | 'defaultPathSuffix'
|
||||
>
|
||||
}
|
||||
let { data }: Props = $props()
|
||||
|
||||
function handlePick(pick: PipelineInsertPick) {
|
||||
if (!pick.language || !pick.path) return
|
||||
const kindId = pick.kindId as KindId
|
||||
const outputKind = (pick.outputKind ?? 'none') as PipelineOutputKind
|
||||
// Native trigger annotation is marker-only — the binding lives on
|
||||
// the trigger row's own `script_path`, which the user creates
|
||||
// separately. Seed with `path: undefined`.
|
||||
data.onAddPipelineScript(
|
||||
pick.language as ScriptLang,
|
||||
pick.path,
|
||||
{ kind: kindId, path: undefined },
|
||||
outputKind,
|
||||
pick.aiPrompt
|
||||
)
|
||||
}
|
||||
</script>
|
||||
|
||||
<PipelineInsertMenu
|
||||
kinds={[
|
||||
{
|
||||
id: 'data_upload',
|
||||
label: 'On data upload',
|
||||
description: 'UI-first: run by uploading a file via the S3 picker',
|
||||
icon: Upload
|
||||
},
|
||||
{
|
||||
id: 'schedule',
|
||||
label: 'On schedule',
|
||||
description: 'Triggered by a schedule you create',
|
||||
icon: Clock
|
||||
},
|
||||
{
|
||||
id: 'webhook',
|
||||
label: 'On webhook',
|
||||
description: 'Triggered by an HTTP webhook',
|
||||
icon: Webhook
|
||||
},
|
||||
{
|
||||
id: 'email',
|
||||
label: 'On email',
|
||||
description: 'Triggered by incoming email',
|
||||
icon: Mail
|
||||
},
|
||||
{
|
||||
id: 'kafka',
|
||||
label: 'On Kafka',
|
||||
description: 'Triggered by a Kafka message',
|
||||
icon: Zap
|
||||
},
|
||||
{
|
||||
id: 'mqtt',
|
||||
label: 'On MQTT',
|
||||
description: 'Triggered by an MQTT message',
|
||||
icon: Radio
|
||||
},
|
||||
{
|
||||
id: 'amqp',
|
||||
label: 'On AMQP',
|
||||
description: 'Triggered by an AMQP (RabbitMQ) message',
|
||||
icon: Radio
|
||||
},
|
||||
{
|
||||
id: 'nats',
|
||||
label: 'On NATS',
|
||||
description: 'Triggered by a NATS message',
|
||||
icon: MessageSquare
|
||||
},
|
||||
{
|
||||
id: 'postgres',
|
||||
label: 'On Postgres',
|
||||
description: 'Triggered by a Postgres event',
|
||||
icon: Database
|
||||
},
|
||||
{
|
||||
id: 'sqs',
|
||||
label: 'On SQS',
|
||||
description: 'Triggered by an SQS message',
|
||||
icon: Send
|
||||
},
|
||||
{
|
||||
id: 'gcp',
|
||||
label: 'On GCP Pub/Sub',
|
||||
description: 'Triggered by a Pub/Sub message',
|
||||
icon: CloudCog
|
||||
}
|
||||
]}
|
||||
languages={PIPELINE_LANGUAGES as any}
|
||||
pathPrefix={data.pathPrefix}
|
||||
defaultPathSuffix={data.defaultPathSuffix}
|
||||
onPick={handlePick}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<!-- Quiet insert affordance, mirroring the flow editor's inline +
|
||||
buttons (bg-surface + gray border + secondary text) — a filled
|
||||
accent circle outweighed every real node on the canvas. -->
|
||||
<button
|
||||
type="button"
|
||||
class="w-8 h-8 rounded-full flex items-center justify-center bg-surface border border-gray-400 dark:border-gray-600 text-secondary shadow-sm hover:bg-surface-hover transition-colors cursor-pointer"
|
||||
title="Add to pipeline"
|
||||
>
|
||||
<Plus size={16} />
|
||||
</button>
|
||||
{/snippet}
|
||||
</PipelineInsertMenu>
|
||||
<!-- The layout gives this node a full node-width slot; centering the pill in
|
||||
it lines it up with the nodes below whatever its label's width. -->
|
||||
<div class="flex justify-center" style="width: {NODE.width}px;">
|
||||
<AddDataSourceMenu
|
||||
onAddPipelineScript={data.onAddPipelineScript}
|
||||
pathPrefix={data.pathPrefix}
|
||||
defaultPathSuffix={data.defaultPathSuffix}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<!-- Quiet insert affordance, mirroring the flow editor's inline +
|
||||
buttons (bg-surface + gray border + secondary text) — a filled
|
||||
accent pill would outweigh every real node on the canvas. -->
|
||||
<button
|
||||
type="button"
|
||||
class="h-8 px-3 rounded-full flex items-center gap-1.5 whitespace-nowrap bg-surface border border-gray-400 dark:border-gray-600 text-xs font-normal text-secondary shadow-sm hover:bg-surface-hover transition-colors cursor-pointer"
|
||||
>
|
||||
<Plus size={16} class="shrink-0" />
|
||||
Add data source
|
||||
</button>
|
||||
{/snippet}
|
||||
</AddDataSourceMenu>
|
||||
</div>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -20,10 +20,17 @@
|
||||
SquareFunction,
|
||||
Trash2,
|
||||
X,
|
||||
Pencil
|
||||
Pencil,
|
||||
ChevronDown,
|
||||
Zap
|
||||
} from 'lucide-svelte'
|
||||
import Popover from '$lib/components/meltComponents/Popover.svelte'
|
||||
import { tick } from 'svelte'
|
||||
import DropdownV2 from '$lib/components/DropdownV2.svelte'
|
||||
import {
|
||||
upstreamScriptItems,
|
||||
upstreamTriggerItems,
|
||||
type UpstreamScript,
|
||||
type UpstreamTriggerAction
|
||||
} from './upstreamScriptItems'
|
||||
import { inferArgs } from '$lib/infer'
|
||||
import { emptySchema, sendUserToast } from '$lib/utils'
|
||||
import type { Schema } from '$lib/common'
|
||||
@@ -41,6 +48,7 @@
|
||||
import { extractDraftMacros } from './resolveGraph'
|
||||
import { assetColumnNodes, type ColumnLineageGraph } from './columnLineageGraph'
|
||||
import SummaryPathDisplay from '$lib/components/SummaryPathDisplay.svelte'
|
||||
import Badge from '$lib/components/common/badge/Badge.svelte'
|
||||
import S3FilePreview from '$lib/components/S3FilePreview.svelte'
|
||||
import DataTablePreview from './DataTablePreview.svelte'
|
||||
import DucklakeAssetPanel from './DucklakeAssetPanel.svelte'
|
||||
@@ -117,7 +125,9 @@
|
||||
// (`onDraftPersist`). `onDraftPersist` stays the authoritative commit on
|
||||
// navigate-away; this is the continuous feed that keeps the draft's
|
||||
// server copy current while the user is still typing.
|
||||
onContentChange?: (scriptPath: string | undefined, content: string) => void
|
||||
// `base`: the deployed copy being edited, so the parent can autosave unsaved
|
||||
// edits to it before they are promoted to a draft. Unset for a draft.
|
||||
onContentChange?: (scriptPath: string | undefined, content: string, base?: Script) => void
|
||||
// Fires when the user navigates away from a draft (selects another
|
||||
// node, closes the pane, etc.) — the parent must persist the new
|
||||
// content + write outputs into its drafts Map. Without this,
|
||||
@@ -133,7 +143,8 @@
|
||||
writes: { kind: AssetWithAltAccessType['kind']; path: string }[]
|
||||
// Body-inferred reads, so the parent's draft keeps its input
|
||||
// lineage while inactive (no live inference runs for it).
|
||||
reads: { kind: AssetWithAltAccessType['kind']; path: string }[]
|
||||
// Undefined when nothing was inferred: keep what the draft has.
|
||||
reads?: { kind: AssetWithAltAccessType['kind']; path: string }[]
|
||||
// Full edited script — set when the source is a persisted
|
||||
// script (needed to seed a brand-new draft entry).
|
||||
script?: Script
|
||||
@@ -156,6 +167,12 @@
|
||||
path: string
|
||||
unsaved?: boolean
|
||||
}>
|
||||
/** The selected asset's producing scripts, for its "Edit script" action,
|
||||
* which calls `onEditScript` with the one picked. */
|
||||
producerScripts?: UpstreamScript[]
|
||||
onEditScript?: (path: string) => void
|
||||
/** The selected asset's upstream triggers that can be opened from here. */
|
||||
producerTriggers?: UpstreamTriggerAction[]
|
||||
// Pipeline-wide column-lineage graph (built by the parent page from the
|
||||
// resolved graph). Drives the transitive column-lineage trace shown for a
|
||||
// selected materialized asset.
|
||||
@@ -215,15 +232,18 @@
|
||||
// the edges. Counter rather than boolean so back-to-back runs
|
||||
// re-fire the effect even if no other prop changed.
|
||||
requestRunSignal?: number
|
||||
// Folder-scoped non-editable prefix shown next to the suffix
|
||||
// editor when the user renames a draft (e.g. `f/<folder>/`). The
|
||||
// new path = pathPrefix + suffix.
|
||||
pathPrefix?: string
|
||||
// Called when the user renames a draft — the parent reseats the
|
||||
// path key in its drafts map and updates activeDraftPath. Returns
|
||||
// true on success; false on collision/validation failure so the
|
||||
// popover can keep itself open and surface the error inline.
|
||||
onDraftPathChange?: (oldPath: string, newPath: string) => boolean | string
|
||||
onDraftMetaChange?: (
|
||||
path: string,
|
||||
meta: { summary: string; labels: string[] | undefined }
|
||||
) => void
|
||||
// The pipeline's folder prefix (`f/<folder>/`). A draft renamed outside it
|
||||
// would be deployed out of the pipeline, so the rename is refused.
|
||||
pathPrefix?: string
|
||||
// Bumped by the parent (e.g. from the runnable-node action menu) to
|
||||
// auto-open the archive/delete confirmation modal for the currently
|
||||
// loaded persisted script. No-ops while the pane is showing a draft
|
||||
@@ -280,6 +300,19 @@
|
||||
// can persist a data-upload entry's staged input (drives node readiness).
|
||||
// `isValid` is the full-schema validity, not just "file present".
|
||||
onRunFormArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
|
||||
// Show only the open script's inputs and runs (a data-upload click), not
|
||||
// its editor. `onRunInputs` runs it with them, drafts included, and
|
||||
// `onShowScript` leaves this view for the script.
|
||||
inputsOnly?: boolean
|
||||
/** `live` is the pane's buffer when it differs from what is stored: always
|
||||
* for a draft, and for a deployed script edited before switching to the
|
||||
* inputs view. A run with it previews that body. */
|
||||
onRunInputs?: (
|
||||
path: string,
|
||||
args: Record<string, any>,
|
||||
live?: { content: string; language: ScriptLang }
|
||||
) => Promise<string | undefined>
|
||||
onShowScript?: () => void
|
||||
}
|
||||
let {
|
||||
selection,
|
||||
@@ -301,6 +334,9 @@
|
||||
onScriptRenamed,
|
||||
onScriptRemoved,
|
||||
selectionProducers = [],
|
||||
producerScripts = [],
|
||||
onEditScript,
|
||||
producerTriggers = [],
|
||||
selectionColumnGraph,
|
||||
selectionColumnLoading = false,
|
||||
selectionColumnTruncated = false,
|
||||
@@ -315,8 +351,9 @@
|
||||
activeRunnable,
|
||||
onTestStateChange,
|
||||
requestRunSignal,
|
||||
pathPrefix = '',
|
||||
onDraftPathChange,
|
||||
onDraftMetaChange,
|
||||
pathPrefix = '',
|
||||
requestRemoveSignal,
|
||||
downstreamSubscribers = 0,
|
||||
onStartBoundedRun,
|
||||
@@ -328,10 +365,15 @@
|
||||
onRunByPath,
|
||||
onRunCascadeByPath,
|
||||
runFormInitialArgs,
|
||||
onRunFormArgsChange
|
||||
onRunFormArgsChange,
|
||||
inputsOnly = false,
|
||||
onRunInputs,
|
||||
onShowScript
|
||||
}: Props = $props()
|
||||
|
||||
let readOnly = $derived(mode !== 'edit')
|
||||
// Someone who cannot switch the pipeline to edit mode only gets to read it.
|
||||
let editScriptLabel = $derived(readOnly && !onRequestEdit ? 'Open script' : 'Edit script')
|
||||
|
||||
// Root element of the pane — used to scope the S3-input lookup for the
|
||||
// data_upload focus effect.
|
||||
@@ -408,7 +450,7 @@
|
||||
// PipelineScriptView renders instead of the test panel.
|
||||
let viewReadyForTarget = $derived.by(
|
||||
() =>
|
||||
readOnly &&
|
||||
(readOnly || inputsOnly) &&
|
||||
runTargetPath !== undefined &&
|
||||
script?.path === runTargetPath &&
|
||||
(isDraft || (!scriptRes.loading && scriptRes.current?.path === runTargetPath))
|
||||
@@ -484,6 +526,12 @@
|
||||
// alongside `liveBodyAssets` and forwarded so the live graph can show
|
||||
// inferred column lineage on the edited script before it deploys.
|
||||
let liveColumnLineage = $state<ColumnLineage[] | undefined>(undefined)
|
||||
// The script whose editor last bound `liveBodyAssets` (undefined there means it
|
||||
// inferred nothing). They stay bound after that editor unmounts, so a script
|
||||
// that never had one (the inputs-only view) must not report or persist them as
|
||||
// its lineage. Set by an effect declared after the save-on-switch one, whose
|
||||
// cleanup still needs the previous owner when the selection moves.
|
||||
let liveAssetsOwner: string | undefined = undefined
|
||||
|
||||
// Bumped when the runs panel reports a watched job has reached a
|
||||
// terminal state. Drives S3FilePreview's refreshKey so the preview
|
||||
@@ -549,7 +597,9 @@
|
||||
let args = $state<Record<string, any>>({})
|
||||
let argsSeedPath: string | undefined = undefined
|
||||
$effect.pre(() => {
|
||||
const p = script?.path
|
||||
// Re-seeded on leaving the inputs-only view too, so the file picked there
|
||||
// shows in the editor's test form.
|
||||
const p = script?.path === undefined ? undefined : `${script.path}|${inputsOnly}`
|
||||
if (p === argsSeedPath) return
|
||||
argsSeedPath = p
|
||||
args = runFormInitialArgs ? structuredClone($state.snapshot(runFormInitialArgs)) : {}
|
||||
@@ -559,7 +609,8 @@
|
||||
// required field (not just the S3 file) is satisfied.
|
||||
let runFormIsValid = $state(true)
|
||||
$effect(() => {
|
||||
if (readOnly || !script) return
|
||||
// Inputs-only mounts no ScriptEditor: PipelineScriptView reports its own args.
|
||||
if (readOnly || inputsOnly || !script) return
|
||||
onRunFormArgsChange?.(script.path, $state.snapshot(args), runFormIsValid)
|
||||
})
|
||||
|
||||
@@ -596,18 +647,25 @@
|
||||
b: Array<{ kind: string; path: string }>
|
||||
) => a.length === b.length && a.every((x, i) => x.kind === b[i]?.kind && x.path === b[i]?.path)
|
||||
return () => {
|
||||
const writes = (liveBodyAssets ?? [])
|
||||
// No editor inferred this script (the inputs-only view): keep its stored
|
||||
// lineage rather than persisting it as empty.
|
||||
const inferred = liveAssetsOwner === captured.path
|
||||
const writes = !inferred
|
||||
? (writesAtRegister ?? [])
|
||||
: (liveBodyAssets ?? [])
|
||||
.filter((a) => {
|
||||
const t = a.access_type ?? a.alt_access_type
|
||||
return t === 'w' || t === 'rw'
|
||||
})
|
||||
.map((a) => ({ kind: a.kind, path: a.path }))
|
||||
const reads = (liveBodyAssets ?? [])
|
||||
.filter((a) => {
|
||||
const t = a.access_type ?? a.alt_access_type
|
||||
return t === 'r' || t === 'rw'
|
||||
})
|
||||
.map((a) => ({ kind: a.kind, path: a.path }))
|
||||
const reads = !inferred
|
||||
? readsAtRegister
|
||||
: (liveBodyAssets ?? [])
|
||||
.filter((a) => {
|
||||
const t = a.access_type ?? a.alt_access_type
|
||||
return t === 'r' || t === 'rw'
|
||||
})
|
||||
.map((a) => ({ kind: a.kind, path: a.path }))
|
||||
// Draft runs: emit only when content or lineage changed in THIS clone.
|
||||
// An unconditional emit ping-pongs forever when the entry is rewritten
|
||||
// externally while the pane stays mounted (rename rekey, AI edit):
|
||||
@@ -617,6 +675,7 @@
|
||||
(captured.content ?? '') === contentAtRegister &&
|
||||
refsEq(writesAtRegister ?? [], writes) &&
|
||||
readsAtRegister != undefined &&
|
||||
reads != undefined &&
|
||||
refsEq(readsAtRegister, reads)
|
||||
)
|
||||
return
|
||||
@@ -651,12 +710,42 @@
|
||||
}
|
||||
})
|
||||
|
||||
$effect(() => {
|
||||
if (script && !readOnly && !inputsOnly) liveAssetsOwner = script.path
|
||||
})
|
||||
|
||||
let saving = $state(false)
|
||||
// What this pane last deployed, so the persist-back cleanup can tell "the
|
||||
// buffer equals the new deployed head" from real unsaved edits (plain
|
||||
// variable: only read inside the untracked cleanup).
|
||||
let deployedFromPane: { path: string; content: string } | undefined = undefined
|
||||
let isDraft = $derived(draftScript != undefined)
|
||||
function liveBody(): { content: string; language: ScriptLang } | undefined {
|
||||
if (!script) return undefined
|
||||
const live = { content: script.content ?? '', language: script.language as ScriptLang }
|
||||
if (isDraft) return live
|
||||
const orig = scriptRes.current
|
||||
return orig?.path === script.path && (orig.content ?? '') !== live.content ? live : undefined
|
||||
}
|
||||
// A draft's stored schema is empty until an editor infers it, and the inputs
|
||||
// view mounts none: infer from the body here so its fields (the file picker)
|
||||
// show up.
|
||||
const inputsSchema = resource(
|
||||
[() => (inputsOnly && isDraft ? script?.content : undefined), () => script?.language],
|
||||
async ([content, language]) => {
|
||||
if (content === undefined || !language) return undefined
|
||||
const schema = emptySchema()
|
||||
try {
|
||||
await inferArgs(language, content, schema as Schema)
|
||||
} catch {
|
||||
return undefined
|
||||
}
|
||||
return schema
|
||||
}
|
||||
)
|
||||
let inputsScript = $derived(
|
||||
script && inputsSchema.current ? { ...script, schema: inputsSchema.current } : undefined
|
||||
)
|
||||
|
||||
// Single trash-bin button opens one modal that exposes both Archive
|
||||
// (always available) and Delete permanently (admin-only). Archive is
|
||||
@@ -763,12 +852,17 @@
|
||||
onAnnotationsChange?.(script?.path, liveAnnotations)
|
||||
})
|
||||
$effect(() => {
|
||||
if (readOnly) return
|
||||
if (readOnly || (inputsOnly && liveAssetsOwner !== script?.path)) return
|
||||
onAssetsChange?.(script?.path, liveBodyAssets ?? [], liveColumnLineage)
|
||||
})
|
||||
$effect(() => {
|
||||
if (readOnly) return
|
||||
onContentChange?.(script?.path, script?.content ?? '')
|
||||
const base = draftScript ? undefined : scriptRes.current
|
||||
onContentChange?.(
|
||||
script?.path,
|
||||
script?.content ?? '',
|
||||
base && base.path === script?.path ? base : undefined
|
||||
)
|
||||
})
|
||||
|
||||
async function save() {
|
||||
@@ -910,49 +1004,6 @@
|
||||
JSON.stringify(script.schema ?? null) === JSON.stringify(orig.schema ?? null)
|
||||
)
|
||||
})
|
||||
|
||||
// Suffix editor for the draft-path popover. Seeded from the current
|
||||
// path each time the popover opens so the user starts with what they
|
||||
// see, not stale state from an earlier rename.
|
||||
let draftPathSuffix = $state('')
|
||||
let draftPathError = $state<string | undefined>(undefined)
|
||||
let draftPathInput: HTMLInputElement | undefined = $state(undefined)
|
||||
|
||||
function suffixOf(fullPath: string): string {
|
||||
return fullPath.startsWith(pathPrefix) ? fullPath.slice(pathPrefix.length) : fullPath
|
||||
}
|
||||
|
||||
async function openDraftPathEditor() {
|
||||
draftPathSuffix = script ? suffixOf(script.path) : ''
|
||||
draftPathError = undefined
|
||||
await tick()
|
||||
draftPathInput?.focus()
|
||||
draftPathInput?.select()
|
||||
}
|
||||
|
||||
function confirmDraftPath(close: () => void) {
|
||||
if (!script) return
|
||||
const suffix = draftPathSuffix.trim()
|
||||
if (!suffix) {
|
||||
draftPathError = 'Path cannot be empty'
|
||||
return
|
||||
}
|
||||
const newPath = pathPrefix + suffix
|
||||
if (newPath === script.path) {
|
||||
close()
|
||||
return
|
||||
}
|
||||
const result = onDraftPathChange?.(script.path, newPath)
|
||||
if (result === true || result === undefined) {
|
||||
script.path = newPath
|
||||
draftPathError = undefined
|
||||
close()
|
||||
} else if (typeof result === 'string') {
|
||||
draftPathError = result
|
||||
} else {
|
||||
draftPathError = 'Path already in use'
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col h-full bg-surface" bind:this={paneEl}>
|
||||
@@ -964,80 +1015,34 @@
|
||||
{@const draftScriptPath = script.path}
|
||||
<Code2 size={16} class="shrink-0 text-tertiary" />
|
||||
{#if onDraftPathChange && !readOnly}
|
||||
<!-- Inline rename popover for drafts. The persisted-script
|
||||
branch uses SummaryPathDisplay which round-trips through
|
||||
updateItemPathAndSummary; drafts have no server row yet,
|
||||
so we just rekey the parent's drafts map locally. -->
|
||||
<Popover
|
||||
placement="bottom-start"
|
||||
contentClasses="p-3"
|
||||
usePointerDownOutside
|
||||
on:openChange={(e) => {
|
||||
if (e.detail) openDraftPathEditor()
|
||||
<!-- Same summary/path popover as a deployed script, saved into the
|
||||
drafts map instead of through the API: a draft has no row yet. -->
|
||||
<SummaryPathDisplay
|
||||
summary={script.summary ?? ''}
|
||||
path={draftScriptPath}
|
||||
labels={script.labels ?? []}
|
||||
kind="script"
|
||||
onSaved={() => {}}
|
||||
saveOverride={({ path, summary, labels }) => {
|
||||
if (pathPrefix && (!path.startsWith(pathPrefix) || path === pathPrefix)) {
|
||||
return `A pipeline script stays in ${pathPrefix}: pick a name inside that folder`
|
||||
}
|
||||
// Before the rename, which carries the draft entry over as it is.
|
||||
onDraftMetaChange?.(draftScriptPath, { summary, labels })
|
||||
if (script) {
|
||||
script.summary = summary
|
||||
script.labels = labels
|
||||
}
|
||||
if (path === draftScriptPath) return undefined
|
||||
const result = onDraftPathChange(draftScriptPath, path)
|
||||
if (result !== true && result !== undefined) {
|
||||
return typeof result === 'string' ? result : 'Path already in use'
|
||||
}
|
||||
if (script) script.path = path
|
||||
return undefined
|
||||
}}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<button
|
||||
type="button"
|
||||
class="flex flex-col min-w-0 text-left px-2 py-1 rounded-md hover:bg-surface-hover transition-colors group"
|
||||
title="Edit draft path"
|
||||
>
|
||||
<span
|
||||
class="text-3xs uppercase tracking-wide text-tertiary flex items-center gap-1"
|
||||
>
|
||||
Draft pipeline script
|
||||
<Pencil size={9} class="opacity-0 group-hover:opacity-60 transition-opacity" />
|
||||
</span>
|
||||
<span class="text-xs font-mono truncate" title={draftScriptPath}
|
||||
>{draftScriptPath}</span
|
||||
>
|
||||
</button>
|
||||
{/snippet}
|
||||
{#snippet content({ close })}
|
||||
<div class="flex flex-col gap-2 w-[420px]">
|
||||
<span class="text-2xs font-normal text-secondary">Path</span>
|
||||
<div
|
||||
class="flex items-stretch border rounded-md bg-surface overflow-hidden focus-within:ring-2 focus-within:ring-blue-400"
|
||||
>
|
||||
{#if pathPrefix}
|
||||
<span
|
||||
class="flex items-center px-2 bg-surface-secondary text-tertiary text-sm font-mono border-r select-none"
|
||||
>
|
||||
{pathPrefix}
|
||||
</span>
|
||||
{/if}
|
||||
<input
|
||||
bind:this={draftPathInput}
|
||||
bind:value={draftPathSuffix}
|
||||
onkeydown={(e) => {
|
||||
if (e.key === 'Enter') {
|
||||
e.preventDefault()
|
||||
confirmDraftPath(close)
|
||||
} else if (e.key === 'Escape') {
|
||||
e.preventDefault()
|
||||
close()
|
||||
}
|
||||
}}
|
||||
class="flex-1 min-w-0 px-2 py-1.5 text-sm font-mono bg-transparent focus:outline-none"
|
||||
placeholder="my_script"
|
||||
/>
|
||||
</div>
|
||||
{#if draftPathError}
|
||||
<span class="text-2xs text-red-500">{draftPathError}</span>
|
||||
{/if}
|
||||
<div class="flex justify-end">
|
||||
<Button
|
||||
variant="accent"
|
||||
unifiedSize="sm"
|
||||
disabled={!draftPathSuffix.trim()}
|
||||
onClick={() => confirmDraftPath(close)}
|
||||
>
|
||||
Rename
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
{/snippet}
|
||||
</Popover>
|
||||
/>
|
||||
<Badge color="gray">Draft</Badge>
|
||||
{:else}
|
||||
<div class="flex flex-col min-w-0">
|
||||
<span class="text-3xs uppercase tracking-wide text-tertiary">
|
||||
@@ -1097,6 +1102,66 @@
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex items-center gap-1 shrink-0">
|
||||
{#if selection?.kind === 'asset' && onEditScript && producerScripts.length === 1}
|
||||
<Button
|
||||
variant="default"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Pencil }}
|
||||
onclick={() => onEditScript(producerScripts[0].path)}
|
||||
title={`Open ${producerScripts[0].path}, which builds this asset`}
|
||||
>
|
||||
{editScriptLabel}
|
||||
</Button>
|
||||
{:else if selection?.kind === 'asset' && onEditScript && producerScripts.length > 1}
|
||||
<DropdownV2
|
||||
items={upstreamScriptItems(producerScripts, onEditScript)}
|
||||
placement="bottom-end"
|
||||
enableFlyTransition
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<Button
|
||||
nonCaptureEvent
|
||||
variant="default"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Pencil }}
|
||||
endIcon={{ icon: ChevronDown }}
|
||||
title={`${producerScripts.length} scripts build this asset`}
|
||||
>
|
||||
{editScriptLabel}
|
||||
</Button>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
{/if}
|
||||
{#if selection?.kind === 'asset' && producerTriggers.length === 1}
|
||||
<Button
|
||||
variant="default"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Zap }}
|
||||
onclick={() => producerTriggers[0].onOpen()}
|
||||
title={producerTriggers[0].detail}
|
||||
>
|
||||
{producerTriggers[0].label}
|
||||
</Button>
|
||||
{:else if selection?.kind === 'asset' && producerTriggers.length > 1}
|
||||
<DropdownV2
|
||||
items={upstreamTriggerItems(producerTriggers)}
|
||||
placement="bottom-end"
|
||||
enableFlyTransition
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<Button
|
||||
nonCaptureEvent
|
||||
variant="default"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Zap }}
|
||||
endIcon={{ icon: ChevronDown }}
|
||||
title={`${producerTriggers.length} triggers start the scripts that build this asset`}
|
||||
>
|
||||
Triggers
|
||||
</Button>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
{/if}
|
||||
{#if readOnly && onRequestEdit}
|
||||
<Button
|
||||
variant="accent"
|
||||
@@ -1118,21 +1183,10 @@
|
||||
title="Discard draft"
|
||||
/>
|
||||
{/if}
|
||||
<!-- Action order, left → right: trash, external link, save, close.
|
||||
Save sits closest to Close so the primary commit action is
|
||||
anchored at the right edge of the bar; trash lives at the
|
||||
far left so destructive ops are visually separated from
|
||||
navigation/commit. Mirrors the draft Discard placement. -->
|
||||
{#if !readOnly && !isDraft && isScriptView && script?.hash}
|
||||
<Button
|
||||
variant="subtle"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Trash2 }}
|
||||
onclick={() => (removeOpen = true)}
|
||||
iconOnly
|
||||
title="Archive or delete"
|
||||
/>
|
||||
{/if}
|
||||
<!-- Action order, left → right: external link, save, close. Save sits
|
||||
closest to Close so the primary commit action is anchored at the
|
||||
right edge of the bar. Removing a script goes through its node's
|
||||
menu. -->
|
||||
{#if !readOnly && !isDraft && selection?.kind === 'runnable'}
|
||||
<Button
|
||||
variant="subtle"
|
||||
@@ -1338,14 +1392,31 @@
|
||||
{/if}
|
||||
</div>
|
||||
{:else if !isDraft && scriptRes.loading && !script}
|
||||
<div class="absolute inset-0 flex items-center justify-center gap-2 text-tertiary">
|
||||
<Loader2 size={16} class="animate-spin" />
|
||||
<span class="text-xs">Loading script…</span>
|
||||
</div>
|
||||
<!-- Blank while the script loads: the fetch is short, and a spinner only flashes. -->
|
||||
{:else if !isDraft && scriptRes.error}
|
||||
<div class="p-3 text-xs text-red-500">
|
||||
Failed to load: {scriptRes.error.message}
|
||||
</div>
|
||||
{:else if script && inputsOnly}
|
||||
{#key script.path}
|
||||
<PipelineScriptView
|
||||
{isDraft}
|
||||
canRun
|
||||
runsDrafts
|
||||
inputsOnly
|
||||
{onShowScript}
|
||||
onRun={onRunInputs && ((path, args) => onRunInputs(path, args, liveBody()))}
|
||||
script={inputsScript ?? script}
|
||||
{runsRefreshKey}
|
||||
{runsPendingJobId}
|
||||
initialArgs={runFormInitialArgs}
|
||||
onArgsChange={onRunFormArgsChange}
|
||||
onRunCompleted={() => {
|
||||
previewRefreshKey += 1
|
||||
onRunCompleted?.()
|
||||
}}
|
||||
/>
|
||||
{/key}
|
||||
{:else if script && readOnly}
|
||||
<!-- Read-only modes: no Monaco/ScriptEditor (operators are
|
||||
backend-blocked from previews anyway) — highlighted source,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
<script lang="ts">
|
||||
import { BaseEdge, getBezierPath, type EdgeProps } from '@xyflow/svelte'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import { PIPELINE_NODE_HEIGHT } from './assetGraphLayout'
|
||||
import { FlaskConical, Columns3, SquareFunction, BellOff } from 'lucide-svelte'
|
||||
import type { ColumnLineage, DataTest } from './parsePipelineAnnotations'
|
||||
|
||||
@@ -103,33 +104,42 @@
|
||||
// straight under the nodes in between — the child-and-grandchild-of-the-
|
||||
// same-node case, where the tidy-tree layout puts all three on one
|
||||
// column. Detour those through the gutter beside the column.
|
||||
const SKIP_DY = NODE.gap.vertical * 2 + NODE.height
|
||||
const SKIP_DY = NODE.gap.vertical * 2 + PIPELINE_NODE_HEIGHT
|
||||
const NEAR_VERTICAL_DX = NODE.width / 4
|
||||
// Lane offset: half a node plus a margin that stays inside the
|
||||
// inter-column gutter (gap.horizontal / 2).
|
||||
const LANE = NODE.width / 2 + NODE.gap.horizontal / 2
|
||||
const CORNER = 14
|
||||
|
||||
// Rounded-orthogonal detour: drop out of the source, swing into the
|
||||
// gutter lane at `midX`, run it down past the intermediate rows, swing
|
||||
// back into the target.
|
||||
// Detour through the gutter lane at `midX`, drawn with the same curves as
|
||||
// every other edge: curve out of the source into the lane within the gap
|
||||
// below it, run down the lane past the rows in between, curve into the
|
||||
// target within the gap above it.
|
||||
function gutterPath(midX: number): string {
|
||||
const yTop = sourceY + 24
|
||||
const yBot = targetY - 24
|
||||
const outDir = midX > sourceX ? 1 : -1
|
||||
const inDir = targetX > midX ? 1 : -1
|
||||
return [
|
||||
`M ${sourceX} ${sourceY}`,
|
||||
`L ${sourceX} ${yTop - CORNER}`,
|
||||
`Q ${sourceX} ${yTop} ${sourceX + outDir * CORNER} ${yTop}`,
|
||||
`L ${midX - outDir * CORNER} ${yTop}`,
|
||||
`Q ${midX} ${yTop} ${midX} ${yTop + CORNER}`,
|
||||
`L ${midX} ${yBot - CORNER}`,
|
||||
`Q ${midX} ${yBot} ${midX + inDir * CORNER} ${yBot}`,
|
||||
`L ${targetX - inDir * CORNER} ${yBot}`,
|
||||
`Q ${targetX} ${yBot} ${targetX} ${yBot + CORNER}`,
|
||||
`L ${targetX} ${targetY}`
|
||||
].join(' ')
|
||||
const bend = NODE.gap.vertical
|
||||
const yTop = sourceY + bend
|
||||
// Clamp only downward edges: an upward (feedback) edge must still run up the
|
||||
// lane, or its last curve cuts across the node the detour was clearing.
|
||||
const yBot = targetY > sourceY ? Math.max(yTop, targetY - bend) : targetY - bend
|
||||
const [out] = getBezierPath({
|
||||
sourceX,
|
||||
sourceY,
|
||||
sourcePosition,
|
||||
targetX: midX,
|
||||
targetY: yTop,
|
||||
targetPosition,
|
||||
curvature: 0.25
|
||||
})
|
||||
const [into] = getBezierPath({
|
||||
sourceX: midX,
|
||||
sourceY: yBot,
|
||||
sourcePosition,
|
||||
targetX,
|
||||
targetY,
|
||||
targetPosition,
|
||||
curvature: 0.25
|
||||
})
|
||||
// Continue from the lane rather than lifting the pen.
|
||||
return `${out} L${midX},${yBot} ${into.replace(/^M/, 'L')}`
|
||||
}
|
||||
|
||||
// Long edges bend out of the source within the layer gap and descend in
|
||||
@@ -178,7 +188,7 @@
|
||||
path={edgePath}
|
||||
{markerEnd}
|
||||
{style}
|
||||
interactionWidth={0}
|
||||
interactionWidth={12}
|
||||
label={undefined}
|
||||
labelStyle={undefined}
|
||||
/>
|
||||
|
||||
@@ -2,9 +2,14 @@
|
||||
import { Handle, Position } from '@xyflow/svelte'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import AssetGenericIcon from '$lib/components/icons/AssetGenericIcon.svelte'
|
||||
import { formatShortAssetPath, type AssetKind } from '$lib/components/assets/lib'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import PipelineInsertMenu, { type PipelineInsertPick } from './PipelineInsertMenu.svelte'
|
||||
import { formatAssetKind, formatShortAssetPath, type AssetKind } from '$lib/components/assets/lib'
|
||||
import PipelineNodeCard from './PipelineNodeCard.svelte'
|
||||
import NodeActionsMenu from './NodeActionsMenu.svelte'
|
||||
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
|
||||
import NodeOnDot from './NodeOnDot.svelte'
|
||||
import { PIPELINE_NODE_HEADER } from './assetGraphLayout'
|
||||
import AddDownstreamMenu from './AddDownstreamMenu.svelte'
|
||||
import AddDownstreamPill from './AddDownstreamPill.svelte'
|
||||
import {
|
||||
ArrowUpRight,
|
||||
Code2,
|
||||
@@ -12,19 +17,28 @@
|
||||
History,
|
||||
Play,
|
||||
Loader2,
|
||||
Plus,
|
||||
ShieldCheck,
|
||||
ShieldAlert,
|
||||
CheckCircle2,
|
||||
ChevronDown,
|
||||
Pencil,
|
||||
CircleSlash,
|
||||
Trash2,
|
||||
Upload,
|
||||
XCircle
|
||||
} from 'lucide-svelte'
|
||||
import { Button } from '$lib/components/common'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import { enterpriseLicense } from '$lib/stores'
|
||||
import { sendUserToast } from '$lib/utils'
|
||||
import { PIPELINE_LANGUAGES } from './pipelineLanguages'
|
||||
import { sendUserToast, type Item } from '$lib/utils'
|
||||
import type { PipelineOutputKind } from './pipelineTemplates'
|
||||
import type { DbtAssetProvenance } from './types'
|
||||
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
|
||||
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
|
||||
import RunStateChip from './RunStateChip.svelte'
|
||||
import DropdownV2 from '$lib/components/DropdownV2.svelte'
|
||||
import { upstreamScriptItems } from './upstreamScriptItems'
|
||||
import type { RunnableRunState } from './activeRunnables.svelte'
|
||||
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
|
||||
|
||||
const operatingWorkspace = useOperatingWorkspace()
|
||||
@@ -33,6 +47,49 @@
|
||||
// `content` / `language` so the page-level run handler can dispatch to
|
||||
// `runScriptPreview` instead of `runScriptByPath` (which 404s for
|
||||
// non-deployed scripts).
|
||||
/** What starts a producing script, as its chip shows it. */
|
||||
export type TriggerChipData = {
|
||||
label: string
|
||||
/** The native trigger kind; a data upload renders as a call to action. */
|
||||
kind?: string
|
||||
/** No chip: the node's incoming edges already show what starts it. */
|
||||
hidden?: boolean
|
||||
missing?: boolean
|
||||
draft?: boolean
|
||||
/** Opens the trigger's editor; unset for what has none (manual, asset change). */
|
||||
onOpen?: () => void
|
||||
selected?: boolean
|
||||
}
|
||||
|
||||
export type AssetUpstreamChips =
|
||||
| { none: true }
|
||||
| {
|
||||
multiple: true
|
||||
/** The producing scripts; one when only its triggers are ambiguous. */
|
||||
scripts: Array<{
|
||||
runnableId: string
|
||||
path: string
|
||||
summary?: string
|
||||
language?: ScriptLang
|
||||
unsaved?: boolean
|
||||
onOpen: () => void
|
||||
}>
|
||||
/** The scripts' own triggers, each named with its script. */
|
||||
triggers: Array<TriggerChipData & { script: string }>
|
||||
selected?: boolean
|
||||
}
|
||||
| {
|
||||
runnableId: string
|
||||
path: string
|
||||
summary?: string
|
||||
language?: ScriptLang
|
||||
unsaved?: boolean
|
||||
runState?: RunnableRunState
|
||||
trigger: TriggerChipData
|
||||
onOpen: () => void
|
||||
selected?: boolean
|
||||
}
|
||||
|
||||
export type AssetProducer = {
|
||||
kind: 'script' | 'flow'
|
||||
path: string
|
||||
@@ -51,6 +108,8 @@
|
||||
// "current view of <dim>" marker so it reads as a derived node, not an
|
||||
// unrelated table.
|
||||
derived_from?: string
|
||||
// Why the node is misconfigured, when it is: renders it red.
|
||||
error?: string
|
||||
// dbt provenance when this warehouse table is a dbt node: which model
|
||||
// it is, how dbt materializes it, its tags and its generic tests.
|
||||
dbt?: DbtAssetProvenance
|
||||
@@ -65,10 +124,14 @@
|
||||
language: ScriptLang,
|
||||
scriptPath: string,
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string
|
||||
aiPrompt?: string,
|
||||
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
|
||||
) => void
|
||||
pathPrefix?: string
|
||||
defaultPathSuffix?: string
|
||||
/** Starts dragging the "+" onto a script (subscribing it to this asset)
|
||||
* or onto the blank canvas (opening the add menu there). */
|
||||
onStartFeedDrag?: (e: PointerEvent) => void
|
||||
// Producer scripts/flows that write to this asset, supplied by
|
||||
// the canvas from the graph's write/rw edges. Drives the on-hover
|
||||
// "Run" button. Includes unsaved/draft producers so the button
|
||||
@@ -99,6 +162,41 @@
|
||||
// instead of only settling once the job ends.
|
||||
runStatus?: 'running' | 'materialized' | 'failed'
|
||||
runRowCount?: number | null
|
||||
/** Set in the assets-only view, where the producing script and its
|
||||
* trigger are folded into the asset: `none` for an asset nothing in the
|
||||
* pipeline writes, `multiple` when the producer or its trigger is
|
||||
* ambiguous. */
|
||||
upstream?: AssetUpstreamChips
|
||||
/** What is wrong with the node, and how to fix it: replaces the "+". */
|
||||
fix?: NodeFix
|
||||
/** Assets-only: errors of the scripts and triggers folded into this node. */
|
||||
foldedFixes?: NodeFix[]
|
||||
/** Assets-only: starts dragging the top dot onto an asset, subscribing the
|
||||
* scripts that build this one to it. */
|
||||
onStartOnDrag?: (e: PointerEvent) => void
|
||||
/** The assets-only node for the scripts that build no asset. */
|
||||
noAsset?: boolean
|
||||
/** The No-asset node's script, by its file name. */
|
||||
noAssetName?: string
|
||||
/** Assets-only: the node is only its script's banner. */
|
||||
bannerOnly?: boolean
|
||||
/** Assets-only: a script folded into the node has an error. */
|
||||
scriptError?: boolean
|
||||
/** No script in the workspace writes it; a flow's write fires no asset trigger. */
|
||||
neverWritten?: boolean
|
||||
/** Card width in the assets-only view, sized to its text. */
|
||||
width?: number
|
||||
/** Assets-only view: the producing script and trigger no other asset
|
||||
* shares, which the menu offers to delete one by one or together. */
|
||||
upstreamDelete?: import('./assetsOnlyView').AssetUpstreamDelete
|
||||
/** When no combined delete is possible: the folded scripts, each deletable
|
||||
* on its own. */
|
||||
scriptDeletes?: Array<{ path: string; unsaved: boolean }>
|
||||
triggerDeletes?: Array<NonNullable<import('./assetsOnlyView').AssetUpstreamDelete['trigger']>>
|
||||
onDeleteUpstream?: (target: import('./assetsOnlyView').AssetUpstreamDelete) => void
|
||||
deleteVerb?: 'Delete' | 'Archive'
|
||||
/** Why nothing can be deleted, when the producer is shared. */
|
||||
deleteBlocked?: string
|
||||
}
|
||||
// SvelteFlow injects this on the node component when the user clicks
|
||||
// the node. Combined with our own `hovered` state to drive the
|
||||
@@ -140,19 +238,94 @@
|
||||
}
|
||||
}
|
||||
|
||||
function handlePick(pick: PipelineInsertPick) {
|
||||
if (pick.kindId === 'pipeline_script' && pick.language && pick.path) {
|
||||
data.onAddScript?.(
|
||||
{ kind: data.asset_kind, path: data.path },
|
||||
pick.language as ScriptLang,
|
||||
pick.path,
|
||||
(pick.outputKind ?? 'none') as PipelineOutputKind,
|
||||
pick.aiPrompt
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
let showAdd = $derived(data.onAddScript != undefined)
|
||||
|
||||
// A red asset needs fixing where its error comes from, not a downstream step;
|
||||
// and a step downstream of what nothing writes would never run on its writes.
|
||||
let showAdd = $derived(data.onAddScript != undefined && !data.error && !data.neverWritten)
|
||||
let menuItems: Item[] = $derived(
|
||||
data.onDeleteUpstream && data.upstreamDelete
|
||||
? deleteItems(data.upstreamDelete, data.onDeleteUpstream)
|
||||
: data.onDeleteUpstream && (data.scriptDeletes?.length || data.triggerDeletes?.length)
|
||||
? [
|
||||
...(data.scriptDeletes ?? []).map((script) => ({ script })),
|
||||
...(data.triggerDeletes ?? []).map((trigger) => ({ trigger }))
|
||||
].flatMap((t) => deleteItems(t, data.onDeleteUpstream!))
|
||||
: data.deleteBlocked
|
||||
? [
|
||||
{
|
||||
displayName: `Can't delete: ${data.deleteBlocked}`,
|
||||
icon: Trash2,
|
||||
disabled: true
|
||||
}
|
||||
]
|
||||
: []
|
||||
)
|
||||
const lastSegment = (p: string) => p.split('/').pop() ?? p
|
||||
// A script is archived rather than deleted when the user may not delete it;
|
||||
// a trigger is always deleted.
|
||||
function deleteItems(
|
||||
t: import('./assetsOnlyView').AssetUpstreamDelete,
|
||||
run: (target: import('./assetsOnlyView').AssetUpstreamDelete) => void
|
||||
): Item[] {
|
||||
const items: Item[] = []
|
||||
if (t.script) {
|
||||
const script = t.script
|
||||
items.push({
|
||||
displayName: `${data.deleteVerb ?? 'Delete'} script ${lastSegment(script.path)}`,
|
||||
icon: Trash2,
|
||||
type: 'delete' as const,
|
||||
action: () => run({ script })
|
||||
})
|
||||
}
|
||||
if (t.trigger) {
|
||||
const trigger = t.trigger
|
||||
items.push({
|
||||
displayName: `Delete ${trigger.kind} trigger ${lastSegment(trigger.path)}`,
|
||||
icon: Trash2,
|
||||
type: 'delete' as const,
|
||||
action: () => run({ trigger })
|
||||
})
|
||||
}
|
||||
if (t.script && t.trigger) {
|
||||
items.push({
|
||||
displayName: `${data.deleteVerb === 'Archive' ? 'Remove' : 'Delete'} all 2 items`,
|
||||
icon: Trash2,
|
||||
type: 'delete' as const,
|
||||
action: () => run(t)
|
||||
})
|
||||
}
|
||||
return items
|
||||
}
|
||||
let upstream = $derived(data.upstream)
|
||||
let HEADER_SELECTED = $derived(
|
||||
data.scriptError
|
||||
? 'bg-red-200 dark:bg-red-600 hover:bg-red-200 dark:hover:bg-red-600'
|
||||
: 'bg-surface-accent-selected hover:bg-surface-accent-selected text-accent'
|
||||
)
|
||||
const CHIP_SELECTED =
|
||||
'bg-surface-accent-selected border-border-selected hover:bg-surface-accent-selected text-accent'
|
||||
let producer = $derived(upstream && 'path' in upstream ? upstream : undefined)
|
||||
let coProducers = $derived(upstream && 'scripts' in upstream ? upstream.scripts : [])
|
||||
let hasHeader = $derived(!!producer || coProducers.length > 0)
|
||||
// The corner controls sit on the card's corner, below the header tab:
|
||||
// -top-2 plus PIPELINE_NODE_HEADER (22px).
|
||||
const CARD_CORNER_TOP = 'top-[14px]'
|
||||
let coSelected = $derived(!!(upstream && 'scripts' in upstream && upstream.selected))
|
||||
let coProducerItems = $derived(
|
||||
upstreamScriptItems(coProducers, (path) => coProducers.find((s) => s.path === path)?.onOpen())
|
||||
)
|
||||
let scriptMenuOpen = $state(false)
|
||||
let coTriggers = $derived(upstream && 'triggers' in upstream ? upstream.triggers : [])
|
||||
let coTriggerItems: Item[] = $derived(
|
||||
coTriggers.map((t) => ({
|
||||
displayName: `${t.label}${t.draft ? ' · draft' : ''}`,
|
||||
description: coProducers.length > 1 ? t.script : undefined,
|
||||
disabled: !t.onOpen,
|
||||
action: () => t.onOpen?.()
|
||||
}))
|
||||
)
|
||||
let triggerMenuOpen = $state(false)
|
||||
|
||||
// dbt badge. `materialized` is dbt's own word rather than the Windmill
|
||||
// strategy because `view` and `ephemeral` have no strategy, and showing the
|
||||
@@ -220,6 +393,185 @@
|
||||
)
|
||||
</script>
|
||||
|
||||
{#snippet scriptHeader()}
|
||||
<!-- The script building this asset, folded into it: a strip across the top
|
||||
naming it, which opens it. Several scripts open a menu to pick one. -->
|
||||
{@const one = producer ?? (coProducers.length === 1 ? coProducers[0] : undefined)}
|
||||
{@const headerClass = twMerge(
|
||||
'flex items-center gap-1.5 w-full min-w-0 px-2 text-3xs font-normal leading-none text-secondary hover:bg-surface-hover hover:text-primary',
|
||||
data.scriptError &&
|
||||
'bg-red-100 dark:bg-red-700 text-red-700 dark:text-red-200 hover:bg-red-200 dark:hover:bg-red-600 hover:text-red-800'
|
||||
)}
|
||||
{#if one}
|
||||
{@const isSelected = producer ? !!producer.selected : coSelected}
|
||||
<button
|
||||
type="button"
|
||||
class={twMerge(headerClass, isSelected && HEADER_SELECTED)}
|
||||
title={`Open ${one.summary ? `${one.summary} (${one.path})` : one.path}${one.unsaved ? ' · draft' : ''}`}
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
onclick={(e) => {
|
||||
e.stopPropagation()
|
||||
one.onOpen()
|
||||
}}
|
||||
>
|
||||
{#if one.language}
|
||||
<LanguageIcon lang={one.language} width={11} height={11} />
|
||||
{:else}
|
||||
<Code2 size={11} class="shrink-0" />
|
||||
{/if}
|
||||
<span class="truncate">{one.summary || one.path}</span>
|
||||
{#if one.unsaved}<span class="shrink-0 text-tertiary">draft</span>{/if}
|
||||
{#if producer?.runState}
|
||||
<RunStateChip runState={producer.runState} class="ml-auto h-4 px-1 rounded" />
|
||||
{/if}
|
||||
<Pencil size={10} class={twMerge('shrink-0 opacity-70', !producer?.runState && 'ml-auto')} />
|
||||
</button>
|
||||
{:else if coProducers.length > 1}
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div
|
||||
class="w-full min-w-0 flex"
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
>
|
||||
<DropdownV2
|
||||
items={coProducerItems}
|
||||
placement="bottom-start"
|
||||
bind:open={scriptMenuOpen}
|
||||
fixedHeight={false}
|
||||
usePointerDownOutside
|
||||
enableFlyTransition
|
||||
class="w-full"
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<span
|
||||
class={twMerge(headerClass, 'h-full', (coSelected || scriptMenuOpen) && HEADER_SELECTED)}
|
||||
title={`Built by ${coProducers.length} scripts: pick one to open`}
|
||||
>
|
||||
{#each coProducers.slice(0, 2) as sc (sc.path)}
|
||||
{#if sc.language}
|
||||
<LanguageIcon lang={sc.language} width={11} height={11} />
|
||||
{:else}
|
||||
<Code2 size={11} class="shrink-0" />
|
||||
{/if}
|
||||
{/each}
|
||||
<span class="truncate">{coProducers.length} scripts</span>
|
||||
<ChevronDown size={11} class="shrink-0 ml-auto" />
|
||||
</span>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
</div>
|
||||
{/if}
|
||||
{/snippet}
|
||||
|
||||
{#snippet upstreamRow()}
|
||||
{#if data.foldedFixes?.length}
|
||||
<!-- A folded script or trigger is misconfigured: its Fix, one at a time. -->
|
||||
<NodeFixButton
|
||||
chip
|
||||
fix={data.foldedFixes[0]}
|
||||
label={`${data.foldedFixes.length} error${data.foldedFixes.length === 1 ? '' : 's'} · Fix`}
|
||||
/>
|
||||
{:else if upstream && coTriggers.length > 1 && !coTriggers.some((t) => t.onOpen)}
|
||||
<!-- Several triggers, none editable here (View mode): a plain label. -->
|
||||
<span
|
||||
class="truncate rounded-md border px-1.5 py-1 text-3xs leading-none font-normal border-gray-300 dark:border-gray-600 text-secondary"
|
||||
>Multiple triggers</span
|
||||
>
|
||||
{:else if upstream && coTriggers.length > 1}
|
||||
<!-- Several triggers start what builds this asset: one chip, a menu to pick
|
||||
the one to edit. -->
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div
|
||||
class="min-w-0 flex"
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
>
|
||||
<DropdownV2
|
||||
items={coTriggerItems}
|
||||
placement="bottom-start"
|
||||
bind:open={triggerMenuOpen}
|
||||
fixedHeight={false}
|
||||
usePointerDownOutside
|
||||
enableFlyTransition
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<span
|
||||
class={twMerge(
|
||||
'flex items-center gap-1 min-w-0 rounded-md border px-1.5 py-1 text-3xs leading-none font-normal border-gray-300 dark:border-gray-600 text-secondary bg-surface hover:bg-surface-hover cursor-pointer',
|
||||
(triggerMenuOpen || coTriggers.some((t) => t.selected)) && CHIP_SELECTED
|
||||
)}
|
||||
title="Started by several triggers: pick one to edit"
|
||||
>
|
||||
<span class="truncate">Multiple triggers</span>
|
||||
<ChevronDown size={9} class="shrink-0 opacity-70" />
|
||||
</span>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
</div>
|
||||
{:else if upstream}
|
||||
{@const trigger = producer?.trigger ?? coTriggers[0]}
|
||||
{@const triggerScript = producer?.path ?? coTriggers[0]?.script}
|
||||
{@const chipClass = twMerge(
|
||||
'truncate rounded-md border px-1.5 py-1 text-3xs leading-none font-normal',
|
||||
trigger?.missing
|
||||
? 'border-red-300 dark:border-red-600 text-red-700 dark:text-red-300'
|
||||
: 'border-gray-300 dark:border-gray-600 text-secondary',
|
||||
trigger?.draft && 'border-dashed',
|
||||
trigger?.onOpen && 'bg-surface hover:bg-surface-hover cursor-pointer',
|
||||
trigger?.selected && CHIP_SELECTED
|
||||
)}
|
||||
{#if trigger?.kind === 'data_upload' && trigger.onOpen}
|
||||
<!-- The one trigger that needs the user: nothing runs until a file is
|
||||
provided, so it reads as the node's main action rather than a chip. -->
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<span
|
||||
class="flex min-w-0"
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Button
|
||||
variant="accent"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Upload }}
|
||||
title={`Upload the data ${triggerScript} runs on`}
|
||||
onClick={(e) => {
|
||||
e?.stopPropagation()
|
||||
trigger.onOpen?.()
|
||||
}}
|
||||
>
|
||||
Upload data
|
||||
</Button>
|
||||
</span>
|
||||
{:else if trigger?.onOpen}
|
||||
<button
|
||||
type="button"
|
||||
class={twMerge(chipClass, 'flex items-center gap-1 min-w-0')}
|
||||
title={trigger.missing
|
||||
? `No ${trigger.label} trigger targets ${triggerScript} yet: click to create one`
|
||||
: `Edit the ${trigger.label} trigger`}
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
onclick={(e) => {
|
||||
e.stopPropagation()
|
||||
trigger.onOpen?.()
|
||||
}}
|
||||
>
|
||||
<span class="truncate">{trigger.label}{trigger.draft ? ' · draft' : ''}</span>
|
||||
{#if !trigger.missing}<Pencil size={9} class="shrink-0 opacity-70" />{/if}
|
||||
</button>
|
||||
{:else}
|
||||
<span class={chipClass}>
|
||||
{#if 'none' in upstream}
|
||||
External source
|
||||
{:else if trigger}
|
||||
{trigger.label}{trigger.draft ? ' · draft' : ''}
|
||||
{/if}
|
||||
</span>
|
||||
{/if}
|
||||
{/if}
|
||||
{/snippet}
|
||||
|
||||
<!-- onmouseenter/leave on the wrapper (not the inner card) so the run
|
||||
button — which floats outside the card — keeps the hover state alive
|
||||
when the cursor moves between the card and the button. -->
|
||||
@@ -236,123 +588,169 @@
|
||||
<div class="wm-recompute-flash pointer-events-none absolute inset-0 z-10 rounded-md"></div>
|
||||
{/key}
|
||||
{/if}
|
||||
<!-- Mirrors the flow editor's asset pill: quiet surface + gray border at
|
||||
rest, accent reserved for the selected state. -->
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden border',
|
||||
'bg-surface-secondary border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500 transition-colors',
|
||||
selected && 'bg-surface-accent-selected border-border-selected'
|
||||
<NodeActionsMenu
|
||||
items={menuItems}
|
||||
hover={hovered}
|
||||
kebabClass={twMerge(
|
||||
showGuardBadge && '-right-8',
|
||||
hasHeader && !data.bannerOnly && CARD_CORNER_TOP
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={data.path}
|
||||
>
|
||||
<!-- Data identity carries the accent (luminance blue), pairing with
|
||||
the blue write edges that produce these assets — scripts stay
|
||||
neutral, so script vs data reads at a glance. -->
|
||||
<AssetGenericIcon
|
||||
assetKind={data.asset_kind}
|
||||
class={`shrink-0 ml-2 mr-2 ${selected ? 'text-accent' : 'text-blue-600 dark:text-blue-400'}`}
|
||||
size="14px"
|
||||
/>
|
||||
{#if data.runStatus}
|
||||
<!-- The run in view, per relation: a spinner while its producer is
|
||||
building it, then its outcome. Left of the name so the eye finds
|
||||
the moving nodes first on a wide graph. -->
|
||||
<span
|
||||
class="shrink-0 mr-1 {data.runStatus === 'failed'
|
||||
? 'text-red-600 dark:text-red-400'
|
||||
: data.runStatus === 'materialized'
|
||||
? 'text-green-600 dark:text-green-400'
|
||||
: 'text-blue-600 dark:text-blue-400'}"
|
||||
title={data.runStatus}
|
||||
{#if data.bannerOnly}
|
||||
<!-- A script that builds nothing and runs only on writes to what it reads:
|
||||
the incoming edges say how it starts, so its banner is the whole node. -->
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex items-stretch min-w-0 rounded-md overflow-hidden bg-surface border border-gray-200 dark:border-gray-700 drop-shadow-sm',
|
||||
(producer ? producer.selected : coSelected) &&
|
||||
'border-border-selected dark:border-border-selected'
|
||||
)}
|
||||
style="width: {data.width ?? 160}px; height: {PIPELINE_NODE_HEADER}px;"
|
||||
>
|
||||
{#if data.runStatus === 'running'}
|
||||
<Loader2 size={11} class="animate-spin" />
|
||||
{:else if data.runStatus === 'failed'}
|
||||
<XCircle size={11} />
|
||||
{@render scriptHeader()}
|
||||
</div>
|
||||
{:else}
|
||||
<PipelineNodeCard
|
||||
kindLabel={data.noAsset ? 'No asset' : formatAssetKind(asset)}
|
||||
title={data.noAsset ? (data.noAssetName ?? '') : formatShortAssetPath(asset)}
|
||||
tooltip={data.noAsset
|
||||
? 'This script builds no asset'
|
||||
: data.error
|
||||
? `${data.path}: ${data.error}`
|
||||
: data.path}
|
||||
{selected}
|
||||
tone={data.error ? 'error' : undefined}
|
||||
subtitle={data.foldedFixes?.length ||
|
||||
(upstream &&
|
||||
!producer?.trigger?.hidden &&
|
||||
!('triggers' in upstream && upstream.triggers.length === 0))
|
||||
? upstreamRow
|
||||
: undefined}
|
||||
header={hasHeader ? scriptHeader : undefined}
|
||||
headerTone={data.scriptError ? 'error' : undefined}
|
||||
headerSelected={producer ? !!producer.selected : coSelected}
|
||||
width={data.width}
|
||||
surface={upstream ? 'primary' : 'secondary'}
|
||||
>
|
||||
{#snippet icon()}
|
||||
<!-- Data identity carries the accent (luminance blue), pairing with
|
||||
the blue write edges that produce these assets — scripts stay
|
||||
neutral, so script vs data reads at a glance. -->
|
||||
{#if data.noAsset}
|
||||
<CircleSlash size={14} class="text-tertiary" />
|
||||
{:else}
|
||||
<CheckCircle2 size={11} />
|
||||
<AssetGenericIcon
|
||||
assetKind={data.asset_kind}
|
||||
class={data.error
|
||||
? 'text-red-600 dark:text-red-200'
|
||||
: selected
|
||||
? 'text-accent'
|
||||
: 'text-blue-600 dark:text-blue-400'}
|
||||
size="14px"
|
||||
/>
|
||||
{/if}
|
||||
</span>
|
||||
<!-- Rows the run wrote. Recorded per relation already, and the cheapest
|
||||
answer to "did this model actually produce anything" — a model that
|
||||
built green but emitted 0 rows is the failure that looks like a
|
||||
success. Only once settled: mid-build the number is not yet real. -->
|
||||
{#if data.runStatus === 'materialized' && data.runRowCount != undefined}
|
||||
<span
|
||||
class="shrink-0 mr-1 text-3xs tabular-nums text-tertiary"
|
||||
title="{data.runRowCount} rows written by this run"
|
||||
>
|
||||
{Intl.NumberFormat().format(data.runRowCount)}
|
||||
</span>
|
||||
{/if}
|
||||
{/if}
|
||||
<span class="flex-1 min-w-0 pr-1 py-0.5 text-2xs font-mono text-emphasis truncate">
|
||||
{formatShortAssetPath(asset)}
|
||||
</span>
|
||||
<!-- Fork data-environment chip: in a fork every asset shares its parent's
|
||||
name, so the env it resolves to must read at a glance. Labeled + tinted
|
||||
(amber "parent" = deferred read of the parent's current table via a
|
||||
view; emerald "fork" = the fork's own materialized copy) rather than a
|
||||
bare icon, which was too easy to miss. The title carries the detail. -->
|
||||
{#if data.fork_materialization === 'deferred'}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300 border border-amber-300 dark:border-amber-700"
|
||||
title="Deferred to parent workspace: reads the parent's current data. Materialize it in this fork to iterate on it."
|
||||
>
|
||||
<ArrowUpRight size={10} />
|
||||
parent
|
||||
</span>
|
||||
{:else if data.fork_materialization === 'fork'}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border border-emerald-300 dark:border-emerald-700"
|
||||
title="Materialized in this fork: reads and writes use the fork's isolated copy."
|
||||
>
|
||||
<GitFork size={10} />
|
||||
fork
|
||||
</span>
|
||||
{/if}
|
||||
<!-- dbt chip: names the materialization dbt declares, plus a test count
|
||||
when the model carries generic tests. Orange keeps it visually
|
||||
separate from the fork/SCD2 chips, which describe Windmill state.
|
||||
Where the project node is on the graph it also stands in for the edge
|
||||
to it: hovering lights that node up, clicking selects it. Where it is
|
||||
not — the run page, and a pipeline holding a dbt project — there is
|
||||
nothing to point at, so the chip renders inert. -->
|
||||
{#if data.dbt}
|
||||
<button
|
||||
type="button"
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold tracking-wide bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300 border border-orange-300 dark:border-orange-700 {data.onDbtSelect
|
||||
? 'hover:brightness-95 cursor-pointer'
|
||||
: 'cursor-default'}"
|
||||
title={dbtTitle}
|
||||
onmouseenter={() => data.onDbtHover?.(true)}
|
||||
onmouseleave={() => data.onDbtHover?.(false)}
|
||||
onclick={(e) => {
|
||||
e.stopPropagation()
|
||||
data.onDbtSelect?.()
|
||||
}}
|
||||
>
|
||||
<DbtIcon width={9} height={9} />
|
||||
{dbtLabel}
|
||||
{#if data.dbt.data_tests?.length}
|
||||
<span class="opacity-70">· {data.dbt.data_tests.length}T</span>
|
||||
{/snippet}
|
||||
{#snippet leading()}
|
||||
{#if data.runStatus}
|
||||
<!-- The run in view, per relation: a spinner while its producer is
|
||||
building it, then its outcome. Left of the name so the eye finds
|
||||
the moving nodes first on a wide graph. -->
|
||||
<span
|
||||
class="shrink-0 mr-1 {data.runStatus === 'failed'
|
||||
? 'text-red-600 dark:text-red-400'
|
||||
: data.runStatus === 'materialized'
|
||||
? 'text-green-600 dark:text-green-400'
|
||||
: 'text-blue-600 dark:text-blue-400'}"
|
||||
title={data.runStatus}
|
||||
>
|
||||
{#if data.runStatus === 'running'}
|
||||
<Loader2 size={11} class="animate-spin" />
|
||||
{:else if data.runStatus === 'failed'}
|
||||
<XCircle size={11} />
|
||||
{:else}
|
||||
<CheckCircle2 size={11} />
|
||||
{/if}
|
||||
</span>
|
||||
<!-- Rows the run wrote. Recorded per relation already, and the cheapest
|
||||
answer to "did this model actually produce anything" — a model that
|
||||
built green but emitted 0 rows is the failure that looks like a
|
||||
success. Only once settled: mid-build the number is not yet real. -->
|
||||
{#if data.runStatus === 'materialized' && data.runRowCount != undefined}
|
||||
<span
|
||||
class="shrink-0 mr-1 text-3xs tabular-nums text-tertiary"
|
||||
title="{data.runRowCount} rows written by this run"
|
||||
>
|
||||
{Intl.NumberFormat().format(data.runRowCount)}
|
||||
</span>
|
||||
{/if}
|
||||
{/if}
|
||||
</button>
|
||||
{/snippet}
|
||||
{#snippet trailing()}
|
||||
<!-- Fork data-environment chip: in a fork every asset shares its parent's
|
||||
name, so the env it resolves to must read at a glance. Labeled + tinted
|
||||
(amber "parent" = deferred read of the parent's current table via a
|
||||
view; emerald "fork" = the fork's own materialized copy) rather than a
|
||||
bare icon, which was too easy to miss. The title carries the detail. -->
|
||||
{#if data.fork_materialization === 'deferred'}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300 border border-amber-300 dark:border-amber-700"
|
||||
title="Deferred to parent workspace: reads the parent's current data. Materialize it in this fork to iterate on it."
|
||||
>
|
||||
<ArrowUpRight size={10} />
|
||||
parent
|
||||
</span>
|
||||
{:else if data.fork_materialization === 'fork'}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold uppercase tracking-wide bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border border-emerald-300 dark:border-emerald-700"
|
||||
title="Materialized in this fork: reads and writes use the fork's isolated copy."
|
||||
>
|
||||
<GitFork size={10} />
|
||||
fork
|
||||
</span>
|
||||
{/if}
|
||||
<!-- dbt chip: names the materialization dbt declares, plus a test count
|
||||
when the model carries generic tests. Orange keeps it visually
|
||||
separate from the fork/SCD2 chips, which describe Windmill state.
|
||||
Where the project node is on the graph it also stands in for the edge
|
||||
to it: hovering lights that node up, clicking selects it. Where it is
|
||||
not — the run page, and a pipeline holding a dbt project — there is
|
||||
nothing to point at, so the chip renders inert. -->
|
||||
{#if data.dbt}
|
||||
<button
|
||||
type="button"
|
||||
class="shrink-0 mr-1.5 flex items-center gap-0.5 rounded px-1 py-px text-3xs font-semibold tracking-wide bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300 border border-orange-300 dark:border-orange-700 {data.onDbtSelect
|
||||
? 'hover:brightness-95 cursor-pointer'
|
||||
: 'cursor-default'}"
|
||||
title={dbtTitle}
|
||||
onmouseenter={() => data.onDbtHover?.(true)}
|
||||
onmouseleave={() => data.onDbtHover?.(false)}
|
||||
onclick={(e) => {
|
||||
e.stopPropagation()
|
||||
data.onDbtSelect?.()
|
||||
}}
|
||||
>
|
||||
<DbtIcon width={9} height={9} />
|
||||
{dbtLabel}
|
||||
{#if data.dbt.data_tests?.length}
|
||||
<span class="opacity-70">· {data.dbt.data_tests.length}T</span>
|
||||
{/if}
|
||||
</button>
|
||||
{/if}
|
||||
<!-- SCD2 companion marker: this node is the `<dim>_current` "latest row
|
||||
per key" view its producer maintains alongside the base dimension.
|
||||
Icon-only (the pill already truncates); the title names the base. -->
|
||||
{#if data.derived_from}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 text-violet-600 dark:text-violet-400"
|
||||
title={`SCD2 current view of ${data.derived_from}: latest row per key, maintained by the same producer as the base dimension.`}
|
||||
>
|
||||
<History size={12} />
|
||||
</span>
|
||||
{/if}
|
||||
{/snippet}
|
||||
</PipelineNodeCard>
|
||||
{/if}
|
||||
<!-- SCD2 companion marker: this node is the `<dim>_current` "latest row
|
||||
per key" view its producer maintains alongside the base dimension.
|
||||
Icon-only (the pill already truncates); the title names the base. -->
|
||||
{#if data.derived_from}
|
||||
<span
|
||||
class="shrink-0 mr-1.5 text-violet-600 dark:text-violet-400"
|
||||
title={`SCD2 current view of ${data.derived_from}: latest row per key, maintained by the same producer as the base dimension.`}
|
||||
>
|
||||
<History size={12} />
|
||||
</span>
|
||||
{/if}
|
||||
</div>
|
||||
</NodeActionsMenu>
|
||||
{#if showGuardBadge}
|
||||
<!-- Data-test outcome badge. Floats off the TOP-RIGHT corner (opposite the
|
||||
left-edge run button and the bottom + inserter) so it never collides
|
||||
@@ -361,6 +759,7 @@
|
||||
<div
|
||||
class={twMerge(
|
||||
'absolute -top-2 -right-2 z-10 rounded-full w-5 h-5 grid place-items-center border shadow-sm',
|
||||
hasHeader && CARD_CORNER_TOP,
|
||||
guardClass
|
||||
)}
|
||||
title={guardTitle}
|
||||
@@ -376,7 +775,13 @@
|
||||
cards. Drafts are runnable too via runScriptPreview, so no
|
||||
greyed-out state — the page-supplied callback handles the
|
||||
dispatch. -->
|
||||
<div class="absolute -left-3 top-1/2 -translate-y-1/2 z-10">
|
||||
<!-- Centred on the card, below the header tab when there is one. -->
|
||||
<div
|
||||
class={twMerge(
|
||||
'absolute -left-3 top-1/2 -translate-y-1/2 z-10',
|
||||
hasHeader && !data.bannerOnly && 'top-[calc(50%+11px)]'
|
||||
)}
|
||||
>
|
||||
<button
|
||||
type="button"
|
||||
onclick={runProducers}
|
||||
@@ -398,53 +803,47 @@
|
||||
</button>
|
||||
</div>
|
||||
{/if}
|
||||
{#if showAdd}
|
||||
<!-- Always-visible + below the asset for downstream pipeline-script
|
||||
creation. Half-overlapping the bottom edge so it visually attaches
|
||||
to the node like the flow editor's between-step inserter. -->
|
||||
{#if data.onStartOnDrag}
|
||||
<NodeOnDot onStart={data.onStartOnDrag} visible={hovered} />
|
||||
{/if}
|
||||
{#if data.fix}
|
||||
<!-- A red node's way out replaces its "+": adding downstream of a broken
|
||||
node is not what it needs. -->
|
||||
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
|
||||
<PipelineInsertMenu
|
||||
kinds={[
|
||||
{
|
||||
id: 'pipeline_script',
|
||||
label: 'Add downstream pipeline script',
|
||||
description: 'Triggered when this asset changes',
|
||||
icon: Code2
|
||||
}
|
||||
]}
|
||||
languages={PIPELINE_LANGUAGES as any}
|
||||
pathPrefix={data.pathPrefix ?? ''}
|
||||
defaultPathSuffix={data.defaultPathSuffix ?? ''}
|
||||
onPick={handlePick}
|
||||
<NodeFixButton fix={data.fix} />
|
||||
</div>
|
||||
{:else if showAdd}
|
||||
<!-- Always-visible + on the asset's bottom edge for downstream pipeline-
|
||||
script creation, half-overlapping it like the flow editor's
|
||||
between-step inserter. -->
|
||||
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
|
||||
<AddDownstreamMenu
|
||||
asset={{ kind: data.asset_kind, path: data.path }}
|
||||
onAddScript={data.onAddScript!}
|
||||
pathPrefix={data.pathPrefix}
|
||||
defaultPathSuffix={data.defaultPathSuffix}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<!--
|
||||
Sizing notes for the round + button:
|
||||
- w-6/h-6 (24px) chosen so that with border-2 (2px each
|
||||
side) the inner area is exactly 20px — divisible by
|
||||
the 16px icon to leave a 2px gap on every side. At
|
||||
20px / 12px before, the gap was 4px-each but the
|
||||
odd rounding interacted badly with svelte-flow's
|
||||
fractional zoom transforms and the icon drifted
|
||||
half a pixel off-center on certain zoom levels.
|
||||
- `grid place-items-center` instead of flex centering:
|
||||
flex's baseline alignment introduces a sub-pixel
|
||||
nudge on small elements that grid avoids.
|
||||
- `leading-none` strips the default line-height
|
||||
contribution that the SVG inherits via the parent's
|
||||
text rendering, otherwise the icon is shifted
|
||||
downward by ~0.5px at fractional zooms.
|
||||
-->
|
||||
{#snippet trigger({ open })}
|
||||
<!-- A click opens the menu; a drag goes to the canvas, which
|
||||
tells a real drag from a click by how far the pointer moves. -->
|
||||
<button
|
||||
type="button"
|
||||
class="nopan nodrag block"
|
||||
onclick={(e) => e.stopPropagation()}
|
||||
class="bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:bg-surface-hover rounded-full w-6 h-6 grid place-items-center shadow-sm leading-none"
|
||||
title="Add downstream pipeline script"
|
||||
onpointerdown={(e) => {
|
||||
if (e.button !== 0) return
|
||||
e.preventDefault()
|
||||
data.onStartFeedDrag?.(e)
|
||||
}}
|
||||
aria-label="Add downstream step"
|
||||
title={data.onStartFeedDrag
|
||||
? 'Click to add a downstream step, or drag onto a script to run it after each write'
|
||||
: undefined}
|
||||
>
|
||||
<Plus size={16} strokeWidth={2.5} />
|
||||
<AddDownstreamPill expanded={open} />
|
||||
</button>
|
||||
{/snippet}
|
||||
</PipelineInsertMenu>
|
||||
</AddDownstreamMenu>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -11,7 +11,8 @@
|
||||
// can't be read), so a selectable version always exists in the table.
|
||||
import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte'
|
||||
import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte'
|
||||
import { Table2, History, Columns3 } from 'lucide-svelte'
|
||||
import { Table2, History, Columns3, Database } from 'lucide-svelte'
|
||||
import DBManagerContent from '$lib/components/DBManagerContent.svelte'
|
||||
import { Pane, Splitpanes } from 'svelte-splitpanes'
|
||||
import { resource } from 'runed'
|
||||
import { fetchDucklakeSnapshots } from '$lib/components/dbOps'
|
||||
@@ -31,7 +32,7 @@
|
||||
}
|
||||
let { path, workspace, schemaCanEvolve = true }: Props = $props()
|
||||
|
||||
let tab = $state<'partitions' | 'history' | 'schema'>('partitions')
|
||||
let tab = $state<'data' | 'partitions' | 'history' | 'schema'>('data')
|
||||
// The user's explicit snapshot pick (undefined until they click a row).
|
||||
let selectedVersion = $state<number | undefined>(undefined)
|
||||
|
||||
@@ -72,24 +73,28 @@
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col h-full">
|
||||
{#if !qualifiedTable}
|
||||
<!-- Catalog-level ducklake node (no table segment, e.g. `ducklake://main`):
|
||||
snapshot history / time-travel are per-table, so only the partition
|
||||
grid applies here. -->
|
||||
<PartitionStatusGrid {path} {workspace} />
|
||||
{:else}
|
||||
<div class="flex items-center gap-2 px-3 py-2 border-b shrink-0">
|
||||
<ToggleButtonGroup selected={tab} on:selected={(e) => (tab = e.detail)}>
|
||||
{#snippet children({ item })}
|
||||
<ToggleButton size="sm" value="partitions" label="Partitions" icon={Table2} {item} />
|
||||
<div class="flex items-center gap-2 px-3 py-2 border-b shrink-0">
|
||||
<ToggleButtonGroup selected={tab} on:selected={(e) => (tab = e.detail)}>
|
||||
{#snippet children({ item })}
|
||||
<ToggleButton size="sm" value="data" label="Data" icon={Database} {item} />
|
||||
<ToggleButton size="sm" value="partitions" label="Partitions" icon={Table2} {item} />
|
||||
<!-- Snapshot history and time-travel are per table: a catalog-level node
|
||||
(`ducklake://main`, no table segment) has only its data and partitions. -->
|
||||
{#if qualifiedTable}
|
||||
<ToggleButton size="sm" value="schema" label="Schema" icon={Columns3} {item} />
|
||||
<ToggleButton size="sm" value="history" label="History" icon={History} {item} />
|
||||
{/snippet}
|
||||
</ToggleButtonGroup>
|
||||
</div>
|
||||
{/if}
|
||||
{/snippet}
|
||||
</ToggleButtonGroup>
|
||||
</div>
|
||||
|
||||
<div class="flex-1 min-h-0">
|
||||
{#if tab === 'partitions'}
|
||||
<div class="flex-1 min-h-0">
|
||||
{#if tab === 'data' && parsed}
|
||||
<!-- Keyed on the asset: the manager opens its table once, on mount. -->
|
||||
{#key path}
|
||||
<DBManagerContent input={parsed} {workspace} />
|
||||
{/key}
|
||||
{:else if tab === 'partitions' || !qualifiedTable}
|
||||
<PartitionStatusGrid {path} {workspace} />
|
||||
{:else if tab === 'schema'}
|
||||
<SchemaHistoryPanel {path} {workspace} canEvolve={schemaCanEvolve} />
|
||||
@@ -117,7 +122,6 @@
|
||||
</Pane>
|
||||
</Splitpanes>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -25,6 +25,8 @@
|
||||
if (!nodesInitialized.current) return
|
||||
if (!nodes.some((n) => n.type !== 'add')) return
|
||||
fittedFor = fitKey
|
||||
void fitView({ padding: 0.1, maxZoom: 1 })
|
||||
// A re-fit over nodes that keep their ids finds them already measured, one
|
||||
// render before their new positions reach the flow.
|
||||
requestAnimationFrame(() => void fitView({ padding: 0.1, maxZoom: 1 }))
|
||||
})
|
||||
</script>
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
<!--
|
||||
@component
|
||||
A pipeline node's actions, reachable two ways: the kebab revealed on hover at the
|
||||
node's top-right corner, and a right-click menu over the node carrying the same
|
||||
items. Place it inside the node's `relative` wrapper, around the node's card.
|
||||
-->
|
||||
<script lang="ts">
|
||||
import type { Snippet } from 'svelte'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import { EllipsisVertical } from 'lucide-svelte'
|
||||
import { stopPropagation, preventDefault } from 'svelte/legacy'
|
||||
import DropdownV2 from '$lib/components/DropdownV2.svelte'
|
||||
import ContextMenu from '$lib/components/common/contextmenu/ContextMenu.svelte'
|
||||
import { contextMenuItemsFromMenu } from '$lib/components/common/contextmenu/fromMenuItems'
|
||||
import type { Item } from '$lib/utils'
|
||||
|
||||
interface Props {
|
||||
items: Item[]
|
||||
/** The pointer is over the node: shows the kebab. */
|
||||
hover: boolean
|
||||
/** Moves the kebab, for a node with its own badge on that corner. */
|
||||
kebabClass?: string
|
||||
children: Snippet
|
||||
}
|
||||
|
||||
let { items, hover, kebabClass, children }: Props = $props()
|
||||
|
||||
let menuOpen = $state(false)
|
||||
let contextItems = $derived(contextMenuItemsFromMenu(items))
|
||||
</script>
|
||||
|
||||
{#if items.length > 0}
|
||||
<ContextMenu items={contextItems}>
|
||||
{@render children()}
|
||||
</ContextMenu>
|
||||
<!-- Rendered only on hover or while open, so the canvas stays clean at rest.
|
||||
`pointerdown` is stopped so svelte-flow doesn't select the node when the
|
||||
user reaches for the menu. -->
|
||||
<div
|
||||
class={twMerge('absolute -top-2 -right-2 h-7 p-1 min-w-7', kebabClass)}
|
||||
style="will-change: transform;"
|
||||
>
|
||||
<DropdownV2
|
||||
{items}
|
||||
placement="bottom-end"
|
||||
bind:open={menuOpen}
|
||||
fixedHeight={false}
|
||||
usePointerDownOutside
|
||||
enableFlyTransition
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<button
|
||||
class={twMerge(
|
||||
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
|
||||
hover || menuOpen ? 'block' : '!hidden',
|
||||
'shadow-md rounded-md'
|
||||
)}
|
||||
onpointerdown={stopPropagation(preventDefault(() => {}))}
|
||||
title="Actions"
|
||||
>
|
||||
<EllipsisVertical size={12} />
|
||||
</button>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
</div>
|
||||
{:else}
|
||||
{@render children()}
|
||||
{/if}
|
||||
@@ -0,0 +1,124 @@
|
||||
<!--
|
||||
@component
|
||||
The "Fix" pill on the bottom edge of a misconfigured (red) pipeline node: what is
|
||||
wrong, and the actions that fix it. Styled like the node's "+" pill; the node's
|
||||
red fill already says something is wrong.
|
||||
-->
|
||||
<script lang="ts" module>
|
||||
export type NodeFixAction = {
|
||||
label: string
|
||||
/** The side effect, spelled out: what changes, where. */
|
||||
detail?: string
|
||||
/** `anchor`: where the Fix pill sits, for an action opening a menu there. */
|
||||
run: (anchor: { x: number; y: number }) => void
|
||||
}
|
||||
/** Prose, with the entities it names as links that select them. */
|
||||
export type FixExplainer = Array<string | { text: string; onClick: () => void }>
|
||||
export type NodeFix = { explainer: FixExplainer; actions: NodeFixAction[] }
|
||||
|
||||
/** A fix as its supplier declares it: an action, or opening the schedule
|
||||
* wizard (which the canvas owns). */
|
||||
export type NodeFixSpec = {
|
||||
explainer: FixExplainer
|
||||
actions: Array<{
|
||||
label: string
|
||||
detail?: string
|
||||
run?: () => void
|
||||
/** Opens the schedule wizard for this script; its confirm calls `onSchedule`. */
|
||||
scheduleFor?: {
|
||||
script: string
|
||||
onSchedule: (s: import('./PipelineInsertMenu.svelte').PipelineInsertSchedule) => void
|
||||
}
|
||||
}>
|
||||
}
|
||||
</script>
|
||||
|
||||
<script lang="ts">
|
||||
import Popover from '$lib/components/meltComponents/Popover.svelte'
|
||||
import { Button } from '$lib/components/common'
|
||||
import { Wrench } from 'lucide-svelte'
|
||||
|
||||
let {
|
||||
fix,
|
||||
label = 'Fix',
|
||||
chip = false
|
||||
}: {
|
||||
fix: NodeFix
|
||||
label?: string
|
||||
/** A small chip in a node's text, rather than the pill on its edge. */
|
||||
chip?: boolean
|
||||
} = $props()
|
||||
|
||||
let open = $state(false)
|
||||
let anchorEl: HTMLElement | undefined = $state()
|
||||
</script>
|
||||
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div
|
||||
bind:this={anchorEl}
|
||||
onpointerdown={(e) => e.stopPropagation()}
|
||||
onkeydown={(e) => e.stopPropagation()}
|
||||
onclick={(e) => e.stopPropagation()}
|
||||
>
|
||||
<Popover
|
||||
bind:isOpen={open}
|
||||
placement="bottom"
|
||||
enableFlyTransition
|
||||
usePointerDownOutside
|
||||
contentClasses="p-3 w-96"
|
||||
>
|
||||
{#snippet trigger()}
|
||||
{#if chip}
|
||||
<span
|
||||
class="flex items-center gap-1 rounded-md border px-1.5 py-1 text-3xs leading-none font-normal whitespace-nowrap bg-surface border-red-300 dark:border-red-600 text-red-700 dark:text-red-300 hover:bg-surface-hover"
|
||||
>
|
||||
<Wrench size={10} class="shrink-0" />
|
||||
{label}
|
||||
</span>
|
||||
{:else}
|
||||
<span
|
||||
class="h-6 px-2 rounded-full flex items-center gap-1 bg-surface border border-gray-400 dark:border-gray-600 text-secondary hover:text-primary hover:border-gray-500 shadow-sm text-2xs font-normal leading-none whitespace-nowrap"
|
||||
>
|
||||
<Wrench size={12} class="shrink-0" />
|
||||
{label}
|
||||
</span>
|
||||
{/if}
|
||||
{/snippet}
|
||||
{#snippet content()}
|
||||
<div class="flex flex-col gap-3">
|
||||
<p class="text-xs text-primary">
|
||||
{#each fix.explainer as part}
|
||||
{#if typeof part === 'string'}{part}{:else}<button
|
||||
type="button"
|
||||
class="text-accent hover:underline"
|
||||
onclick={() => {
|
||||
open = false
|
||||
part.onClick()
|
||||
}}>{part.text}</button
|
||||
>{/if}
|
||||
{/each}
|
||||
</p>
|
||||
<div class="flex flex-col gap-2">
|
||||
{#each fix.actions as action, i (action.label)}
|
||||
<div class="flex flex-col gap-0.5">
|
||||
<Button
|
||||
variant={i === 0 ? 'accent' : 'default'}
|
||||
unifiedSize="sm"
|
||||
onclick={() => {
|
||||
const r = anchorEl?.getBoundingClientRect()
|
||||
open = false
|
||||
action.run({ x: r ? r.left + r.width / 2 : 0, y: r ? r.bottom : 0 })
|
||||
}}
|
||||
>
|
||||
{action.label}
|
||||
</Button>
|
||||
{#if action.detail}
|
||||
<span class="text-2xs text-secondary">{action.detail}</span>
|
||||
{/if}
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
{/snippet}
|
||||
</Popover>
|
||||
</div>
|
||||
@@ -0,0 +1,23 @@
|
||||
<!--
|
||||
@component
|
||||
The dot on a pipeline node's top edge: dragged onto an asset, it makes the node's
|
||||
script run after each write to that asset.
|
||||
-->
|
||||
<script lang="ts">
|
||||
let { onStart, visible }: { onStart: (e: PointerEvent) => void; visible: boolean } = $props()
|
||||
</script>
|
||||
|
||||
<button
|
||||
type="button"
|
||||
class="absolute -top-1.5 left-1/2 -translate-x-1/2 z-10 w-3 h-3 rounded-full bg-surface border-2 border-blue-500 cursor-crosshair transition-opacity {visible
|
||||
? 'opacity-100'
|
||||
: 'opacity-0'}"
|
||||
title="Drag onto an asset to run after each write to it"
|
||||
aria-label="Drag onto an asset to run after each write to it"
|
||||
onpointerdown={(e) => {
|
||||
e.stopPropagation()
|
||||
e.preventDefault()
|
||||
onStart(e)
|
||||
}}
|
||||
onclick={(e) => e.stopPropagation()}
|
||||
></button>
|
||||
@@ -2,6 +2,7 @@
|
||||
import { useSvelteFlow, type Node } from '@xyflow/svelte'
|
||||
import { untrack } from 'svelte'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import { PIPELINE_NODE_HEIGHT } from './assetGraphLayout'
|
||||
|
||||
// Smoothly pans the viewport so a freshly-created node lands in the middle,
|
||||
// keeping the current zoom. Lives inside <SvelteFlow> so useSvelteFlow() has
|
||||
@@ -27,8 +28,8 @@
|
||||
untrack(() => {
|
||||
const node = ns.find((n) => n.id === id)
|
||||
if (!node) return // not laid out yet — re-runs when `nodes` updates
|
||||
const cx = node.position.x + NODE.width / 2
|
||||
const cy = node.position.y + NODE.height / 2
|
||||
const cx = node.position.x + (node.measured?.width ?? NODE.width) / 2
|
||||
const cy = node.position.y + PIPELINE_NODE_HEIGHT / 2
|
||||
const key = `${id}:${cx}:${cy}`
|
||||
if (key === lastKey) return
|
||||
lastKey = key
|
||||
|
||||
@@ -416,7 +416,7 @@
|
||||
onmouseenter={() => onHoverRun?.([e.path])}
|
||||
onmouseleave={() => onHoverRun?.(undefined)}
|
||||
class={twMerge(
|
||||
'w-full flex items-center gap-2 px-2 py-1 rounded-sm hover:bg-surface-hover text-xs text-left',
|
||||
'w-full flex items-center gap-2 px-2 py-1.5 rounded-sm font-normal hover:bg-surface-hover text-xs text-left',
|
||||
isMember && 'pl-7',
|
||||
expanded && 'bg-surface-selected'
|
||||
)}
|
||||
@@ -445,7 +445,7 @@
|
||||
{:else}
|
||||
<Code2 size={12} class="shrink-0 text-tertiary" />
|
||||
{/if}
|
||||
<span class="flex-1 min-w-0 truncate font-mono text-2xs" title={e.path}>
|
||||
<span class="flex-1 min-w-0 truncate text-xs" title={e.path}>
|
||||
{e.path}
|
||||
</span>
|
||||
{#if showSource}
|
||||
@@ -533,7 +533,7 @@
|
||||
onclick={() => toggleGroup(g.key)}
|
||||
onmouseenter={() => onHoverRun?.(g.members.map((m) => m.path))}
|
||||
onmouseleave={() => onHoverRun?.(undefined)}
|
||||
class="w-full flex items-center gap-2 px-2 py-1 rounded-sm hover:bg-surface-hover text-xs text-left"
|
||||
class="w-full flex items-center gap-2 px-2 py-1.5 rounded-sm font-normal hover:bg-surface-hover text-xs text-left"
|
||||
title={collapsed ? 'Expand cascade runs' : 'Collapse cascade runs'}
|
||||
>
|
||||
<span class="shrink-0 text-tertiary">
|
||||
@@ -549,7 +549,7 @@
|
||||
{/if}
|
||||
</span>
|
||||
<Workflow size={12} class="shrink-0 text-indigo-500" />
|
||||
<span class="flex-1 min-w-0 truncate font-mono text-2xs" title={groupTitle(g)}>
|
||||
<span class="flex-1 min-w-0 truncate text-xs" title={groupTitle(g)}>
|
||||
{groupTitle(g)}
|
||||
</span>
|
||||
<span
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
<script lang="ts">
|
||||
import { base } from '$lib/base'
|
||||
import Alert from '$lib/components/common/alert/Alert.svelte'
|
||||
import Button from '$lib/components/common/button/Button.svelte'
|
||||
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
|
||||
import { BookOpen } from 'lucide-svelte'
|
||||
|
||||
const DBT_DOCS = 'https://www.windmill.dev/docs/getting_started/scripts_quickstart/dbt'
|
||||
</script>
|
||||
|
||||
<Alert type="info" title="Prefer using dbt? Windmill also runs dbt" size="xs">
|
||||
<div class="flex flex-col gap-2">
|
||||
<p>
|
||||
A pipeline is Windmill's own abstraction; an existing dbt project stays a dbt project, and
|
||||
runs here unchanged as its own kind of script, with dbt still owning its models, refs and
|
||||
tests. All it needs is a warehouse configured under
|
||||
<a class="underline" href="{base}/workspace_settings?tab=dbt">Settings → dbt</a>. Its models
|
||||
show up in the asset graph as <span class="font-mono">dbt://</span> nodes, visible next to whatever
|
||||
pipelines you do build, so you can adopt pipelines later, or never.
|
||||
</p>
|
||||
<div class="flex flex-row flex-wrap items-center gap-2">
|
||||
<!-- The dbt mark goes in as a child rather than `startIcon`: that slot
|
||||
sizes its icon with lucide's `size`, which DbtIcon has no prop for. -->
|
||||
<Button variant="default" unifiedSize="sm" href="{base}/scripts/add?lang=dbt">
|
||||
<DbtIcon width={14} height={14} />
|
||||
New dbt script
|
||||
</Button>
|
||||
<Button
|
||||
variant="subtle"
|
||||
unifiedSize="sm"
|
||||
href={DBT_DOCS}
|
||||
target="_blank"
|
||||
rel="noreferrer"
|
||||
startIcon={{ icon: BookOpen }}
|
||||
>
|
||||
dbt documentation
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
</Alert>
|
||||
@@ -0,0 +1,49 @@
|
||||
<script lang="ts">
|
||||
import { AlertTriangle } from 'lucide-svelte'
|
||||
import Popover from '$lib/components/meltComponents/Popover.svelte'
|
||||
|
||||
// Which drafts failed to deploy and why, anchored next to the deploy button so
|
||||
// the reasons are one click away without leaving the editor. Deployed drafts
|
||||
// leave the map; what is listed is still unresolved.
|
||||
let {
|
||||
errors,
|
||||
open = $bindable()
|
||||
}: {
|
||||
errors: ReadonlyMap<string, string>
|
||||
open?: boolean
|
||||
} = $props()
|
||||
</script>
|
||||
|
||||
<Popover
|
||||
placement="bottom-end"
|
||||
contentClasses="p-3 max-w-[480px]"
|
||||
usePointerDownOutside
|
||||
enableFlyTransition
|
||||
bind:isOpen={open}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
<button
|
||||
type="button"
|
||||
class="flex items-center gap-1.5 px-2 py-1 rounded-md text-red-600 dark:text-red-400 bg-red-50 dark:bg-red-900/30 hover:bg-red-100 dark:hover:bg-red-900/50 transition-colors text-xs font-medium"
|
||||
title="View save errors"
|
||||
>
|
||||
<AlertTriangle size={14} />
|
||||
<span>{errors.size} failed</span>
|
||||
</button>
|
||||
{/snippet}
|
||||
{#snippet content()}
|
||||
<div class="flex flex-col gap-2">
|
||||
<span class="text-xs font-semibold text-emphasis">Save errors</span>
|
||||
<div class="flex flex-col gap-2 max-h-72 overflow-y-auto">
|
||||
{#each [...errors.entries()] as [path, message] (path)}
|
||||
<div class="flex flex-col gap-0.5 border-l-2 border-red-400 pl-2">
|
||||
<span class="text-2xs font-mono text-emphasis">{path}</span>
|
||||
<span class="text-2xs text-red-600 dark:text-red-400 break-words">
|
||||
{message}
|
||||
</span>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
{/snippet}
|
||||
</Popover>
|
||||
@@ -0,0 +1,47 @@
|
||||
<script lang="ts">
|
||||
import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte'
|
||||
import type { PipelineTriggerDraft } from './types'
|
||||
|
||||
// Triggers start running once deployed, so deploying a pipeline's trigger
|
||||
// drafts asks first, listing what will start.
|
||||
let {
|
||||
open,
|
||||
triggerDrafts,
|
||||
onConfirmed,
|
||||
onCanceled
|
||||
}: {
|
||||
open: boolean
|
||||
triggerDrafts: ReadonlyMap<string, PipelineTriggerDraft>
|
||||
onConfirmed: () => void
|
||||
onCanceled: () => void
|
||||
} = $props()
|
||||
</script>
|
||||
|
||||
<ConfirmationModal
|
||||
{open}
|
||||
title="Deploy triggers?"
|
||||
confirmationText="Deploy"
|
||||
type="info"
|
||||
{onConfirmed}
|
||||
{onCanceled}
|
||||
>
|
||||
<div class="flex flex-col gap-2 text-xs text-secondary">
|
||||
<p>
|
||||
{triggerDrafts.size === 1
|
||||
? 'Saving the pipeline also creates this trigger. It starts running its script once deployed.'
|
||||
: `Saving the pipeline also creates these ${triggerDrafts.size} triggers. They start running their scripts once deployed.`}
|
||||
</p>
|
||||
<ul class="flex flex-col gap-1">
|
||||
{#each [...triggerDrafts] as [key, d] (key)}
|
||||
<li class="flex flex-col">
|
||||
<span class="font-mono text-2xs text-emphasis truncate">{d.config.path}</span>
|
||||
<span class="text-2xs text-hint truncate">
|
||||
{d.kind}{#if d.kind === 'schedule'}
|
||||
· <span class="font-mono">{d.config.schedule}</span> ({d.config.timezone}){/if}
|
||||
→ {d.config.script_path}
|
||||
</span>
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
</div>
|
||||
</ConfirmationModal>
|
||||
@@ -18,11 +18,18 @@
|
||||
mode?: 'view' | 'edit'
|
||||
// Called right before navigating — lets a wrapping modal close itself.
|
||||
onOpen?: () => void
|
||||
// Only the folder picker, for hosts that don't list pipelines.
|
||||
hideExisting?: boolean
|
||||
}
|
||||
let { currentFolder = undefined, mode = 'view', onOpen = undefined }: Props = $props()
|
||||
let {
|
||||
currentFolder = undefined,
|
||||
mode = 'view',
|
||||
onOpen = undefined,
|
||||
hideExisting = false
|
||||
}: Props = $props()
|
||||
|
||||
let pipelines = resource(
|
||||
() => $workspaceStore,
|
||||
() => (hideExisting ? undefined : $workspaceStore),
|
||||
async (ws) => {
|
||||
if (!ws) return [] as ListPipelineFoldersResponse
|
||||
return await AssetService.listPipelineFolders({ workspace: ws })
|
||||
@@ -49,50 +56,52 @@
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col gap-6 w-full min-w-0">
|
||||
<section class="flex flex-col gap-2">
|
||||
<h3 class="text-xs font-semibold text-secondary uppercase tracking-wide">
|
||||
Existing pipelines
|
||||
</h3>
|
||||
{#if pipelines.loading && !pipelines.current}
|
||||
<div class="text-tertiary text-sm flex items-center gap-2">
|
||||
<Loader2 size={14} class="animate-spin" />
|
||||
Loading…
|
||||
</div>
|
||||
{:else if pipelines.error}
|
||||
<div class="text-red-500 text-sm">Failed: {pipelines.error.message}</div>
|
||||
{:else if visiblePipelines.length === 0}
|
||||
<div class="text-tertiary text-sm">
|
||||
{currentFolder
|
||||
? 'No other pipelines in this workspace.'
|
||||
: 'No pipelines yet. A pipeline is any folder whose scripts carry pipeline annotations.'}
|
||||
</div>
|
||||
{:else}
|
||||
<div class="flex flex-col border rounded-md overflow-hidden max-h-64 overflow-y-auto">
|
||||
{#each visiblePipelines as p (p.folder)}
|
||||
<button
|
||||
type="button"
|
||||
class="flex items-center justify-between px-3 py-2 border-b last:border-b-0 bg-surface hover:bg-surface-hover transition-colors text-left"
|
||||
onclick={() => openExistingPipeline(p.folder)}
|
||||
>
|
||||
<span class="font-mono text-sm">f/{p.folder}</span>
|
||||
<span class="text-2xs text-tertiary">
|
||||
{p.script_count}
|
||||
{p.script_count === 1 ? 'script' : 'scripts'}
|
||||
</span>
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</section>
|
||||
{#if !hideExisting}
|
||||
<section class="flex flex-col gap-2">
|
||||
<h3 class="text-sm font-semibold text-emphasis">Existing pipelines</h3>
|
||||
{#if pipelines.loading && !pipelines.current}
|
||||
<div class="text-tertiary text-sm flex items-center gap-2">
|
||||
<Loader2 size={14} class="animate-spin" />
|
||||
Loading…
|
||||
</div>
|
||||
{:else if pipelines.error}
|
||||
<div class="text-red-500 text-sm">Failed: {pipelines.error.message}</div>
|
||||
{:else if visiblePipelines.length === 0}
|
||||
<div class="text-tertiary text-sm">
|
||||
{currentFolder
|
||||
? 'No other pipelines in this workspace.'
|
||||
: 'No pipelines yet. A pipeline is any folder whose scripts carry pipeline annotations.'}
|
||||
</div>
|
||||
{:else}
|
||||
<div class="flex flex-col border rounded-md overflow-hidden max-h-64 overflow-y-auto">
|
||||
{#each visiblePipelines as p (p.folder)}
|
||||
<button
|
||||
type="button"
|
||||
class="flex items-center justify-between px-3 py-2 border-b last:border-b-0 bg-surface hover:bg-surface-hover transition-colors text-left"
|
||||
onclick={() => openExistingPipeline(p.folder)}
|
||||
>
|
||||
<span class="font-mono text-sm">f/{p.folder}</span>
|
||||
<span class="text-2xs text-tertiary">
|
||||
{p.script_count}
|
||||
{p.script_count === 1 ? 'script' : 'scripts'}
|
||||
</span>
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
</section>
|
||||
{/if}
|
||||
|
||||
{#if !$userStore?.operator}
|
||||
<!-- Operators only view existing pipelines — opening an arbitrary
|
||||
folder is a build-a-new-pipeline affordance. -->
|
||||
<section class="flex flex-col gap-2">
|
||||
<h3 class="text-xs font-semibold text-secondary uppercase tracking-wide">
|
||||
Pick or create a folder
|
||||
</h3>
|
||||
<div class="flex items-center gap-2">
|
||||
<section class="flex flex-col gap-1">
|
||||
<h3 class="text-sm font-semibold text-emphasis">Pick or create a folder</h3>
|
||||
<p class="text-xs text-secondary">
|
||||
Each folder holds one pipeline, made of its pipeline scripts. Pick a folder to open its
|
||||
pipeline, or create one to start a new pipeline.
|
||||
</p>
|
||||
<div class="flex items-center gap-2 mt-1">
|
||||
<div class="flex-1 min-w-0">
|
||||
<FolderPicker bind:folderName={pickedFolder} />
|
||||
</div>
|
||||
|
||||
@@ -1,6 +1,15 @@
|
||||
<script lang="ts">
|
||||
import { untrack, type Snippet } from 'svelte'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
|
||||
import { MousePointerClick } from 'lucide-svelte'
|
||||
import Disposable from '$lib/components/common/drawer/Disposable.svelte'
|
||||
import Portal from '$lib/components/Portal.svelte'
|
||||
import { getOverlayHost } from '$lib/components/common/overlayHost.svelte'
|
||||
import { useFlowPanelMode } from '$lib/components/flows/flowPanelMode.svelte'
|
||||
import {
|
||||
PIPELINE_DRAFT_KIND,
|
||||
pipelineBundlePath,
|
||||
pipelineLocalMirrorKey
|
||||
} from '$lib/pipelinePaths'
|
||||
import { Loader2 } from 'lucide-svelte'
|
||||
import { Pane, Splitpanes } from 'svelte-splitpanes'
|
||||
import { DraftService } from '$lib/gen'
|
||||
@@ -11,6 +20,11 @@
|
||||
import type { ColumnLineageGraph } from './columnLineageGraph'
|
||||
import HideButton from '$lib/components/apps/editor/settingsPanel/HideButton.svelte'
|
||||
import AssetGraphCanvas from './AssetGraphCanvas.svelte'
|
||||
import { TRIGGER_NODE_STYLE } from './TriggerNode.svelte'
|
||||
import { describeSchedule } from '$lib/utils/describeCron'
|
||||
import type { UpstreamTriggerAction } from './upstreamScriptItems'
|
||||
import type { AssetUpstreamDelete } from './assetsOnlyView'
|
||||
import type { NodeFixSpec } from './NodeFixButton.svelte'
|
||||
import AssetGraphDetailsPane from './AssetGraphDetailsPane.svelte'
|
||||
import PipelineEventLog from './PipelineEventLog.svelte'
|
||||
import type {
|
||||
@@ -18,8 +32,10 @@
|
||||
AssetGraphSelection,
|
||||
NativeTriggerKind,
|
||||
PipelineMode,
|
||||
PipelineTriggerDraft,
|
||||
DbtAssetProvenance
|
||||
} from './types'
|
||||
import { isDraftableTriggerKind, triggerDraftKey } from './pipelineTriggerDrafts'
|
||||
import type { AssetKind, Script, ScriptLang } from '$lib/gen'
|
||||
import type { RunnableRunState, PipelineEvent } from './activeRunnables.svelte'
|
||||
import type { PipelineOutputKind } from './pipelineTemplates'
|
||||
@@ -41,6 +57,7 @@
|
||||
defaultPathSuffix = 'new_pipeline_script',
|
||||
panelHidden = false,
|
||||
onTogglePanelHidden,
|
||||
modalPanel = false,
|
||||
prefetchingAssets = false,
|
||||
hoveredPaths = [],
|
||||
selectedRunPaths = [],
|
||||
@@ -65,6 +82,11 @@
|
||||
onAddScriptForAsset,
|
||||
onAddPipelineScript,
|
||||
onRunnableMenuRemove,
|
||||
onDeleteAssetUpstream,
|
||||
nodeFixes,
|
||||
assetDeleteVerb,
|
||||
onEdgeAction,
|
||||
onAddAssetTrigger,
|
||||
onRunProducer,
|
||||
idlePane,
|
||||
onRequestEdit,
|
||||
@@ -73,6 +95,9 @@
|
||||
onRunCascadeByPath,
|
||||
runFormInitialArgs,
|
||||
onRunFormArgsChange,
|
||||
inputsOnly = false,
|
||||
onRunInputs,
|
||||
onShowScript,
|
||||
readyDataUploadPaths,
|
||||
resolveLocalScript,
|
||||
localScriptsVersion,
|
||||
@@ -126,6 +151,9 @@
|
||||
pathPrefix: string
|
||||
defaultPathSuffix?: string
|
||||
panelHidden?: boolean
|
||||
/** Let the details pane open as a modal over a full-width graph when the editor
|
||||
* is too narrow to dock it — the flow editor's step panel rule. */
|
||||
modalPanel?: boolean
|
||||
onTogglePanelHidden?: () => void
|
||||
prefetchingAssets?: boolean
|
||||
hoveredPaths?: string[]
|
||||
@@ -153,16 +181,29 @@
|
||||
language: ScriptLang,
|
||||
scriptPath: string,
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string
|
||||
aiPrompt?: string,
|
||||
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
|
||||
) => void
|
||||
onAddPipelineScript?: (
|
||||
language: ScriptLang,
|
||||
path: string,
|
||||
source: { kind: NativeTriggerKind; path: string | undefined },
|
||||
outputKind: PipelineOutputKind,
|
||||
aiPrompt?: string
|
||||
aiPrompt?: string,
|
||||
options?: import('./PipelineInsertMenu.svelte').PipelineInsertOptions
|
||||
) => void
|
||||
onRunnableMenuRemove?: (...args: any[]) => void
|
||||
onDeleteAssetUpstream?: (target: AssetUpstreamDelete) => void
|
||||
nodeFixes?: ReadonlyMap<string, NodeFixSpec>
|
||||
assetDeleteVerb?: 'Delete' | 'Archive'
|
||||
onEdgeAction?: (a: {
|
||||
action: 'listen' | 'stop'
|
||||
targets: Array<{ script: string; asset: { kind: AssetKind; path: string }; reads: boolean }>
|
||||
}) => void
|
||||
onAddAssetTrigger?: (a: {
|
||||
scripts: Array<{ script: string; reads: boolean }>
|
||||
asset: { kind: AssetKind; path: string }
|
||||
}) => void
|
||||
onRunProducer?: (producer: RunProducer) => Promise<string | undefined>
|
||||
idlePane?: Snippet
|
||||
onRequestEdit?: () => void
|
||||
@@ -175,8 +216,16 @@
|
||||
// callback to persist them as they change — see the page's dataUploadArgs.
|
||||
runFormInitialArgs?: Record<string, any>
|
||||
onRunFormArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
|
||||
/** See AssetGraphDetailsPane's `inputsOnly`. */
|
||||
inputsOnly?: boolean
|
||||
onRunInputs?: (
|
||||
path: string,
|
||||
args: Record<string, any>,
|
||||
live?: { content: string; language: ScriptLang }
|
||||
) => Promise<string | undefined>
|
||||
onShowScript?: () => void
|
||||
// Data-upload entry scripts whose staged upload is ready (green node).
|
||||
readyDataUploadPaths?: Set<string>
|
||||
readyDataUploadPaths?: ReadonlySet<string>
|
||||
/** Local-dev (`/pipeline_dev`): resolve a node to its working-tree content
|
||||
* so the details pane skips the (nonexistent) deployed-script fetch. May
|
||||
* be async (to infer the args schema for the run form). */
|
||||
@@ -247,18 +296,81 @@
|
||||
? (editor.selection != undefined || editor.activeDraftPath != undefined) && !panelHidden
|
||||
: !panelHidden
|
||||
)
|
||||
// Docked beside the graph, or in a modal over it (modal mode). In modal mode the
|
||||
// details only show once asked for: a double-click on a node, a click on the
|
||||
// node already selected, or a script's open button — never on a selection the
|
||||
// editor makes itself (an AI-built node), which would pop a modal unasked.
|
||||
const panelController = useFlowPanelMode({ enabled: () => modalPanel })
|
||||
$effect(() => panelController.measure(containerWidth))
|
||||
const panelMode = $derived(panelController.mode)
|
||||
let panelModalOpen = $state(false)
|
||||
const dockedPaneOpen = $derived(detailsPaneOpen && panelMode === 'docked')
|
||||
$effect(() => {
|
||||
if (untrack(() => panelModalOpen) && (panelMode === 'docked' || !detailsPaneOpen)) {
|
||||
panelModalOpen = false
|
||||
}
|
||||
})
|
||||
let panelDisposable: Disposable | undefined = $state(undefined)
|
||||
// Disposable joins the overlay stack through its methods, not by watching `open`.
|
||||
$effect(() => {
|
||||
panelModalOpen
|
||||
untrack(() => {
|
||||
panelModalOpen ? panelDisposable?.openDrawer() : panelDisposable?.closeDrawer()
|
||||
})
|
||||
})
|
||||
const overlayHost = getOverlayHost()
|
||||
const modalHost = $derived(overlayHost?.el())
|
||||
|
||||
function selectableNodeAt(e: MouseEvent): HTMLElement | null {
|
||||
return (e.target as HTMLElement | null)?.closest('.svelte-flow__node.selectable') ?? null
|
||||
}
|
||||
// Read in the capture phase: once the click bubbles, the canvas has applied
|
||||
// its selection and a first click looks like a click on the selected node.
|
||||
let clickStartedOnSelected = false
|
||||
function noteSelectionBeforeClick(e: MouseEvent) {
|
||||
clickStartedOnSelected = !!selectableNodeAt(e)?.classList.contains('selected')
|
||||
}
|
||||
function openPanelModalIfReselected(e: MouseEvent) {
|
||||
if (clickStartedOnSelected && selectableNodeAt(e)) panelModalOpen = true
|
||||
}
|
||||
function openPanelModalFromGraph(e: MouseEvent) {
|
||||
if (selectableNodeAt(e)) panelModalOpen = true
|
||||
}
|
||||
// A button in the pane that ends it closes the modal in the same update: left to
|
||||
// the effect above, the modal would first render the pane without its selection.
|
||||
function closePanelModal() {
|
||||
if (panelMode === 'modal') panelModalOpen = false
|
||||
}
|
||||
function handleCanvasSelect(s: AssetGraphSelection | undefined, opts?: { open?: boolean }) {
|
||||
onSelect(s)
|
||||
if (opts?.open && s && panelMode === 'modal') panelModalOpen = true
|
||||
}
|
||||
// A data-upload node's click asks for its run form; in modal mode nothing shows
|
||||
// it unless the modal opens too.
|
||||
$effect(() => {
|
||||
if (focusUploadSignal > 0 && untrack(() => panelMode) === 'modal') {
|
||||
untrack(() => (panelModalOpen = true))
|
||||
}
|
||||
})
|
||||
const showNodeHint = $derived(panelMode === 'modal' && !panelModalOpen)
|
||||
const nodeHintText = $derived(
|
||||
effectiveSelection
|
||||
? 'Click the selected node to open it'
|
||||
: 'Double click a node to open it'
|
||||
)
|
||||
|
||||
let idleView = $derived(
|
||||
mode !== 'edit' && editor.selection == undefined && editor.activeDraftPath == undefined
|
||||
)
|
||||
|
||||
// Wrap the writes (and the rightPaneSize read in the else branch) in untrack so
|
||||
// the effect tracks only `detailsPaneOpen` / `storedRightPaneSize` — without it,
|
||||
// the effect tracks only `dockedPaneOpen` / `storedRightPaneSize` — without it,
|
||||
// the Pane `bind:size` feedback loops the effect and pegs the main thread.
|
||||
$effect(() => {
|
||||
// Stacked (vertical) splits give the details pane more room — the run
|
||||
// form + result need height more than the graph needs it.
|
||||
const fallback = stacked ? 55 : 40
|
||||
if (detailsPaneOpen) {
|
||||
if (dockedPaneOpen) {
|
||||
const restore = storedRightPaneSize
|
||||
untrack(() => {
|
||||
rightPaneSize = restore > 0 ? restore : fallback
|
||||
@@ -272,6 +384,67 @@
|
||||
}
|
||||
})
|
||||
|
||||
// The selected asset's producing scripts, named and iconed from the graph, for
|
||||
// the pane's "Edit script".
|
||||
let producerScripts = $derived(
|
||||
selectionProducers
|
||||
.filter((p) => p.kind === 'script')
|
||||
.map((p) => {
|
||||
const r = displayGraph.runnables.find((x) => x.usage_kind === 'script' && x.path === p.path)
|
||||
return { path: p.path, summary: r?.summary, language: r?.language }
|
||||
})
|
||||
)
|
||||
// The selected asset's upstream triggers that can be opened from here, each
|
||||
// with the same action its trigger node has: edit (edit mode), create a
|
||||
// missing one (edit mode), or the webhook / data-upload drawers.
|
||||
let producerTriggers = $derived.by((): UpstreamTriggerAction[] => {
|
||||
const scripts = new Set(producerScripts.map((p) => p.path))
|
||||
const seen = new Set<string>()
|
||||
const out: UpstreamTriggerAction[] = []
|
||||
for (const t of displayGraph.triggers ?? []) {
|
||||
if (t.trigger_kind === 'asset' || !scripts.has(t.runnable_path)) continue
|
||||
const kind = t.trigger_kind
|
||||
const rowless = kind === 'webhook' || kind === 'data_upload'
|
||||
const key = `${kind}:${rowless || t.missing ? t.runnable_path : t.path}`
|
||||
if (seen.has(key)) continue
|
||||
seen.add(key)
|
||||
const label = TRIGGER_NODE_STYLE[kind]?.label ?? kind
|
||||
const script = t.runnable_path
|
||||
let action: UpstreamTriggerAction | undefined
|
||||
if (kind === 'webhook') {
|
||||
if (onOpenWebhook)
|
||||
action = { label: 'Open webhook', detail: script, onOpen: () => onOpenWebhook(script) }
|
||||
} else if (kind === 'data_upload') {
|
||||
if (onOpenDataUpload)
|
||||
action = { label: 'Upload data', detail: script, onOpen: () => onOpenDataUpload(script) }
|
||||
} else if (t.missing && !rowless) {
|
||||
if (onCreateMissingTrigger)
|
||||
action = {
|
||||
label: `Create ${label} trigger`,
|
||||
detail: script,
|
||||
onOpen: () => onCreateMissingTrigger(kind, script)
|
||||
}
|
||||
} else if (t.path && onEditTrigger) {
|
||||
const path = t.path
|
||||
action = {
|
||||
label: kind === 'schedule' ? 'Edit schedule' : `Edit ${label} trigger`,
|
||||
detail:
|
||||
kind === 'schedule' && t.schedule
|
||||
? (t.summary ?? describeSchedule(t.schedule, t.timezone) ?? path)
|
||||
: path,
|
||||
onOpen: () => onEditTrigger(kind, path, script)
|
||||
}
|
||||
}
|
||||
if (action) out.push(action)
|
||||
}
|
||||
return out
|
||||
})
|
||||
|
||||
function editProducerScript(path: string) {
|
||||
if (mode !== 'edit') onRequestEdit?.()
|
||||
onSelect({ kind: 'runnable', runnable_kind: 'script', path })
|
||||
}
|
||||
|
||||
// ===================== Draft autosave (persistDrafts only) =====================
|
||||
// All of this folder's in-flight drafts live in ONE per-user DB draft (typ
|
||||
// `data_pipeline`) keyed at the folder, syncing across devices + surfacing in
|
||||
@@ -279,12 +452,21 @@
|
||||
// for the one-time migration below; the DB is the source of truth on load.
|
||||
// FlowBuilder's autosave analogue — gated by `persistDrafts`.
|
||||
let pipelineDraftPath = $derived(pipelineBundlePath(folder))
|
||||
let storageKey = $derived(`pipeline-${folder}`)
|
||||
type PipelineDraftBundle = { drafts: Array<[string, PipelineDraft]>; activeDraftPath?: string }
|
||||
let storageKey = $derived(pipelineLocalMirrorKey(folder))
|
||||
type PipelineDraftBundle = {
|
||||
drafts: Array<[string, PipelineDraft]>
|
||||
activeDraftPath?: string
|
||||
triggerDrafts?: PipelineTriggerDraft[]
|
||||
}
|
||||
// Hydration is tracked on the editor instance (`editor.hydratedFromDb`), not a
|
||||
// component flag: the in-session preview reuses one instance across editor
|
||||
// hide/show, and it must hydrate ONCE per instance, not on every remount.
|
||||
let lastPersistedBundle: string | undefined = undefined
|
||||
// What decides whether the bundle needs saving: its drafts, not which one is
|
||||
// open. Opening a draft is not an edit, so it rides along with the next real
|
||||
// change instead of saving (and spinning the sync indicator) on a click.
|
||||
const persistKey = (b: PipelineDraftBundle | undefined) =>
|
||||
b ? JSON.stringify({ drafts: b.drafts, triggerDrafts: b.triggerDrafts }) : undefined
|
||||
|
||||
function restoreBundle(bundle: PipelineDraftBundle) {
|
||||
if (Array.isArray(bundle.drafts)) {
|
||||
@@ -300,6 +482,18 @@
|
||||
if (loaded.size > 0) editor.drafts = loaded
|
||||
}
|
||||
if (typeof bundle.activeDraftPath === 'string') editor.activeDraftPath = bundle.activeDraftPath
|
||||
if (Array.isArray(bundle.triggerDrafts)) {
|
||||
editor.triggerDrafts = new Map(
|
||||
bundle.triggerDrafts
|
||||
.filter(
|
||||
(d) =>
|
||||
isDraftableTriggerKind(d?.kind) &&
|
||||
typeof d.config?.path === 'string' &&
|
||||
typeof d.config?.script_path === 'string'
|
||||
)
|
||||
.map((d) => [triggerDraftKey(d.kind, d.config.path), d])
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function readLocalBundle(): PipelineDraftBundle | undefined {
|
||||
@@ -311,7 +505,8 @@
|
||||
if (s && (Array.isArray(s.drafts) || typeof s.activeDraftPath === 'string')) {
|
||||
return {
|
||||
drafts: Array.isArray(s.drafts) ? s.drafts : [],
|
||||
activeDraftPath: typeof s.activeDraftPath === 'string' ? s.activeDraftPath : undefined
|
||||
activeDraftPath: typeof s.activeDraftPath === 'string' ? s.activeDraftPath : undefined,
|
||||
triggerDrafts: Array.isArray(s.triggerDrafts) ? s.triggerDrafts : undefined
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
@@ -339,8 +534,8 @@
|
||||
fromServer = true
|
||||
}
|
||||
}
|
||||
// A folder retarget during the await (the route-page header switcher, or a
|
||||
// session retarget) changed the target and reset hydratedFromDb. Don't
|
||||
// A folder switch during the await (the route page's header switcher)
|
||||
// changed the target and reset hydratedFromDb. Don't
|
||||
// apply this now-stale folder's bundle or mark the new folder hydrated —
|
||||
// otherwise the stale result blocks the new folder's hydrate and its
|
||||
// drafts bleed across folders.
|
||||
@@ -359,7 +554,7 @@
|
||||
{ workspace: ws ?? '', itemKind: PIPELINE_DRAFT_KIND, path },
|
||||
migratedFromLocal ? undefined : serverSavedAt
|
||||
)
|
||||
if (!migratedFromLocal) lastPersistedBundle = bundle ? JSON.stringify(bundle) : undefined
|
||||
if (!migratedFromLocal) lastPersistedBundle = persistKey(bundle)
|
||||
} catch (e) {
|
||||
console.warn('failed to load pipeline drafts', e)
|
||||
} finally {
|
||||
@@ -412,30 +607,52 @@
|
||||
return [p, d] as [string, PipelineDraft]
|
||||
return [p, { ...d, script, outputAssets, inputAssets }] as [string, PipelineDraft]
|
||||
})
|
||||
// A deployed script being edited is only promoted to a draft when its pane
|
||||
// closes; until then its buffer is saved as the draft it will become.
|
||||
const { scriptPath: livePath, content: liveBody, base } = editor.liveContent
|
||||
if (
|
||||
livePath != undefined &&
|
||||
base &&
|
||||
!editor.drafts.has(livePath) &&
|
||||
liveBody !== (base.content ?? '')
|
||||
) {
|
||||
const live = editor.liveBodyAssets.scriptPath === livePath
|
||||
const writes = live ? extractWrites(editor.liveBodyAssets.assets) : []
|
||||
serialized.push([
|
||||
livePath,
|
||||
{
|
||||
localId: editor.promotedDraftLocalId(livePath),
|
||||
script: { ...base, content: liveBody },
|
||||
outputAssets: writes.length > 0 ? writes : undefined,
|
||||
inputAssets: live ? extractReads(editor.liveBodyAssets.assets) : undefined
|
||||
}
|
||||
])
|
||||
}
|
||||
const activePath = editor.activeDraftPath
|
||||
const triggerDrafts = [...editor.triggerDrafts.values()]
|
||||
const key = storageKey
|
||||
const ws = workspace
|
||||
const path = pipelineDraftPath
|
||||
const hydrated = editor.hydratedFromDb
|
||||
untrack(() => {
|
||||
if (!hydrated) return
|
||||
const isEmpty = serialized.length === 0 && !activePath
|
||||
const isEmpty = serialized.length === 0 && !activePath && triggerDrafts.length === 0
|
||||
const bundle: PipelineDraftBundle | undefined = isEmpty
|
||||
? undefined
|
||||
: { drafts: serialized, activeDraftPath: activePath }
|
||||
: {
|
||||
drafts: serialized,
|
||||
activeDraftPath: activePath,
|
||||
...(triggerDrafts.length > 0 ? { triggerDrafts } : {})
|
||||
}
|
||||
try {
|
||||
if (typeof localStorage !== 'undefined') {
|
||||
if (isEmpty) localStorage.removeItem(key)
|
||||
else
|
||||
localStorage.setItem(
|
||||
key,
|
||||
encodeState({ drafts: serialized, activeDraftPath: activePath })
|
||||
)
|
||||
else localStorage.setItem(key, encodeState(bundle))
|
||||
}
|
||||
} catch (e) {
|
||||
console.warn('failed to mirror pipeline state', e)
|
||||
}
|
||||
const serializedBundle = bundle ? JSON.stringify(bundle) : undefined
|
||||
const serializedBundle = persistKey(bundle)
|
||||
if (serializedBundle === lastPersistedBundle) return
|
||||
lastPersistedBundle = serializedBundle
|
||||
if (!ws) return
|
||||
@@ -450,10 +667,90 @@
|
||||
})
|
||||
</script>
|
||||
|
||||
<div class="h-full w-full" bind:clientWidth={containerWidth}>
|
||||
{#snippet detailsBody(ws: string)}
|
||||
{#if idleView && idlePane}
|
||||
{@render idlePane()}
|
||||
{:else}
|
||||
<AssetGraphDetailsPane
|
||||
{mode}
|
||||
{onRequestEdit}
|
||||
{canRunByPath}
|
||||
{onRunByPath}
|
||||
{onRunCascadeByPath}
|
||||
{runFormInitialArgs}
|
||||
{onRunFormArgsChange}
|
||||
{inputsOnly}
|
||||
{onRunInputs}
|
||||
{onShowScript}
|
||||
{resolveLocalScript}
|
||||
{localScriptsVersion}
|
||||
selection={activeDraft ? undefined : editor.selection}
|
||||
selectionProducers={activeDraft ? [] : selectionProducers}
|
||||
{producerScripts}
|
||||
onEditScript={editProducerScript}
|
||||
{producerTriggers}
|
||||
{selectionColumnGraph}
|
||||
{selectionColumnLoading}
|
||||
{selectionColumnTruncated}
|
||||
{selectionColumnFailed}
|
||||
{selectionDbt}
|
||||
{schemaCanEvolve}
|
||||
{selectionForkMaterialization}
|
||||
{schemaContractContext}
|
||||
{runsRefreshKey}
|
||||
{runsPendingJobId}
|
||||
{activeRunnable}
|
||||
{downstreamSubscribers}
|
||||
onStartBoundedRun={canBoundedRunOpenScript &&
|
||||
editor.openScriptPath &&
|
||||
onStartBoundedRunForOpen
|
||||
? () => onStartBoundedRunForOpen?.(editor.openScriptPath!)
|
||||
: undefined}
|
||||
{onRunCompleted}
|
||||
{onTestStateChange}
|
||||
{requestRemoveSignal}
|
||||
{requestRunSignal}
|
||||
{requestRunCascadeSignal}
|
||||
{focusUploadSignal}
|
||||
draftScript={activeDraft?.script}
|
||||
draftOutputAssets={activeDraft?.outputAssets}
|
||||
draftInputAssets={activeDraft?.inputAssets}
|
||||
{onDraftPathChange}
|
||||
onDraftMetaChange={editor.setDraftMeta}
|
||||
{pathPrefix}
|
||||
workspace={ws}
|
||||
onAnnotationsChange={editor.handleAnnotationsChange}
|
||||
onAssetsChange={editor.handleAssetsChange}
|
||||
onContentChange={editor.handleContentChange}
|
||||
onDraftPersist={editor.handleDraftPersist}
|
||||
onclose={() => {
|
||||
closePanelModal()
|
||||
onClose()
|
||||
}}
|
||||
onHide={onTogglePanelHidden &&
|
||||
(() => {
|
||||
closePanelModal()
|
||||
onTogglePanelHidden()
|
||||
})}
|
||||
{onDiscard}
|
||||
{onDraftSaved}
|
||||
{onPersistedSaved}
|
||||
{onScriptRenamed}
|
||||
{onScriptRemoved}
|
||||
/>
|
||||
{/if}
|
||||
{/snippet}
|
||||
|
||||
<div class="relative h-full w-full" bind:clientWidth={containerWidth}>
|
||||
<Splitpanes class="!h-full" horizontal={stacked}>
|
||||
<Pane bind:size={leftPaneSize}>
|
||||
<div class="relative h-full">
|
||||
<!-- svelte-ignore a11y_click_events_have_key_events, a11y_no_static_element_interactions -->
|
||||
<div
|
||||
class="relative h-full"
|
||||
ondblclick={panelMode === 'modal' ? openPanelModalFromGraph : undefined}
|
||||
onpointerdowncapture={panelMode === 'modal' ? noteSelectionBeforeClick : undefined}
|
||||
onclick={panelMode === 'modal' ? openPanelModalIfReselected : undefined}
|
||||
>
|
||||
<AssetGraphCanvas
|
||||
graph={displayGraph}
|
||||
selection={effectiveSelection}
|
||||
@@ -470,10 +767,15 @@
|
||||
{onOpenWebhook}
|
||||
{onOpenDataUpload}
|
||||
{readyDataUploadPaths}
|
||||
onselect={onSelect}
|
||||
onselect={handleCanvasSelect}
|
||||
{onAddScriptForAsset}
|
||||
{onAddPipelineScript}
|
||||
{onRunnableMenuRemove}
|
||||
{onDeleteAssetUpstream}
|
||||
{nodeFixes}
|
||||
{assetDeleteVerb}
|
||||
{onEdgeAction}
|
||||
{onAddAssetTrigger}
|
||||
{onRunProducer}
|
||||
{validStartPaths}
|
||||
{onStartBoundedRun}
|
||||
@@ -482,6 +784,8 @@
|
||||
{panToNodeId}
|
||||
showMinimap={!stacked}
|
||||
viewportFitKey={viewportFitKey ?? folder}
|
||||
assetsOnlyToggle
|
||||
assetsOnlyFolder={workspace ? `${workspace}/${folder}` : undefined}
|
||||
/>
|
||||
{#if boundBar}{@render boundBar()}{/if}
|
||||
{#if mode === 'edit'}
|
||||
@@ -509,66 +813,47 @@
|
||||
{/if}
|
||||
</div></Pane
|
||||
>
|
||||
{#if detailsPaneOpen && workspace}
|
||||
{#if dockedPaneOpen && workspace}
|
||||
<Pane bind:size={rightPaneSize} minSize={25}>
|
||||
{#if idleView && idlePane}
|
||||
{@render idlePane()}
|
||||
{:else}
|
||||
<AssetGraphDetailsPane
|
||||
{mode}
|
||||
{onRequestEdit}
|
||||
{canRunByPath}
|
||||
{onRunByPath}
|
||||
{onRunCascadeByPath}
|
||||
{runFormInitialArgs}
|
||||
{onRunFormArgsChange}
|
||||
{resolveLocalScript}
|
||||
{localScriptsVersion}
|
||||
selection={activeDraft ? undefined : editor.selection}
|
||||
selectionProducers={activeDraft ? [] : selectionProducers}
|
||||
{selectionColumnGraph}
|
||||
{selectionColumnLoading}
|
||||
{selectionColumnTruncated}
|
||||
{selectionColumnFailed}
|
||||
{selectionDbt}
|
||||
{schemaCanEvolve}
|
||||
{selectionForkMaterialization}
|
||||
{schemaContractContext}
|
||||
{runsRefreshKey}
|
||||
{runsPendingJobId}
|
||||
{activeRunnable}
|
||||
{downstreamSubscribers}
|
||||
onStartBoundedRun={canBoundedRunOpenScript &&
|
||||
editor.openScriptPath &&
|
||||
onStartBoundedRunForOpen
|
||||
? () => onStartBoundedRunForOpen?.(editor.openScriptPath!)
|
||||
: undefined}
|
||||
{onRunCompleted}
|
||||
{onTestStateChange}
|
||||
{requestRemoveSignal}
|
||||
{requestRunSignal}
|
||||
{requestRunCascadeSignal}
|
||||
{focusUploadSignal}
|
||||
draftScript={activeDraft?.script}
|
||||
draftOutputAssets={activeDraft?.outputAssets}
|
||||
draftInputAssets={activeDraft?.inputAssets}
|
||||
{pathPrefix}
|
||||
{onDraftPathChange}
|
||||
{workspace}
|
||||
onAnnotationsChange={editor.handleAnnotationsChange}
|
||||
onAssetsChange={editor.handleAssetsChange}
|
||||
onContentChange={editor.handleContentChange}
|
||||
onDraftPersist={editor.handleDraftPersist}
|
||||
onclose={onClose}
|
||||
onHide={onTogglePanelHidden}
|
||||
{onDiscard}
|
||||
{onDraftSaved}
|
||||
{onPersistedSaved}
|
||||
{onScriptRenamed}
|
||||
{onScriptRemoved}
|
||||
/>
|
||||
{/if}
|
||||
{@render detailsBody(workspace)}
|
||||
</Pane>
|
||||
{/if}
|
||||
</Splitpanes>
|
||||
{#if showNodeHint}
|
||||
<!-- Above the event log's bar, which owns the canvas's bottom edge. -->
|
||||
<div
|
||||
class="pointer-events-none absolute bottom-10 left-3 z-30 flex items-center gap-1.5 text-xs text-hint"
|
||||
>
|
||||
<MousePointerClick size={13} />
|
||||
{nodeHintText}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<!-- Portalled to the host's overlay anchor (a session preview tab) so the modal covers
|
||||
that tab rather than the whole page; Disposable owns its place in the overlay stack
|
||||
and its Escape. -->
|
||||
<Disposable bind:open={panelModalOpen} bind:this={panelDisposable}>
|
||||
{#snippet children({ zIndex })}
|
||||
{#if panelMode === 'modal' && panelModalOpen && workspace}
|
||||
<Portal target={modalHost ?? 'body'} class="contents">
|
||||
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div
|
||||
class="{modalHost ? 'absolute' : 'fixed'} inset-0 flex justify-center px-2 py-6"
|
||||
style="z-index: {zIndex}"
|
||||
role="dialog"
|
||||
>
|
||||
<div class="absolute inset-0 bg-black/20" onclick={() => (panelModalOpen = false)}></div>
|
||||
<div
|
||||
class="relative flex w-full max-w-4xl flex-col overflow-hidden rounded-md border bg-surface shadow-xl"
|
||||
>
|
||||
<div class="min-h-0 flex-1 overflow-auto">
|
||||
{@render detailsBody(workspace)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</Portal>
|
||||
{/if}
|
||||
{/snippet}
|
||||
</Disposable>
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
<script lang="ts" module>
|
||||
import type { ComponentType } from 'svelte'
|
||||
import type { SupportedLanguage } from '$lib/common'
|
||||
import type { AssetKind, NewSchedule } from '$lib/gen'
|
||||
|
||||
export type PipelineInsertKind = {
|
||||
// Machine-readable id; drives which right-column panel renders.
|
||||
@@ -8,6 +9,9 @@
|
||||
label: string
|
||||
description?: string
|
||||
icon?: ComponentType
|
||||
// Its trigger is configured in its own editor right after the script is
|
||||
// created, so the wizard's last button reads "Configure".
|
||||
configuredAfterCreate?: boolean
|
||||
}
|
||||
|
||||
export type PipelineInsertPick = {
|
||||
@@ -22,7 +26,18 @@
|
||||
// from this prompt via AI (using language + outputKind + the
|
||||
// upstream input as context) instead of using the seeded template.
|
||||
aiPrompt?: string
|
||||
// Set by the schedule step when the picked kind is `schedule`.
|
||||
schedule?: PipelineInsertSchedule
|
||||
// Set by the asset step for the table output kinds, replacing the
|
||||
// generated name.
|
||||
outputAsset?: { kind: AssetKind; path: string }
|
||||
}
|
||||
|
||||
export type PipelineInsertOptions = Pick<PipelineInsertPick, 'schedule' | 'outputAsset'>
|
||||
|
||||
// Everything but the target, which is the script being created. An empty
|
||||
// `path` means the default `<script>_schedule`.
|
||||
export type PipelineInsertSchedule = Omit<NewSchedule, 'script_path'>
|
||||
</script>
|
||||
|
||||
<script lang="ts">
|
||||
@@ -36,10 +51,28 @@
|
||||
PIPELINE_OUTPUT_KINDS,
|
||||
type PipelineOutputKind
|
||||
} from './pipelineTemplates'
|
||||
import CronInput from '$lib/components/CronInput.svelte'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import Label from '$lib/components/Label.svelte'
|
||||
import TextInput from '$lib/components/text_input/TextInput.svelte'
|
||||
import { CornerDownLeft, Sparkles } from 'lucide-svelte'
|
||||
import TextInput, {
|
||||
inputBaseClass,
|
||||
inputBorderClass,
|
||||
inputSizeClasses
|
||||
} from '$lib/components/text_input/TextInput.svelte'
|
||||
import DucklakePicker from '$lib/components/DucklakePicker.svelte'
|
||||
import DatatablePicker from '$lib/components/DatatablePicker.svelte'
|
||||
import Path from '$lib/components/Path.svelte'
|
||||
import Section from '$lib/components/Section.svelte'
|
||||
import Toggle from '$lib/components/Toggle.svelte'
|
||||
import DateTimeInput from '$lib/components/DateTimeInput.svelte'
|
||||
import ScheduleAdvancedOptions, {
|
||||
emptyScheduleAdvanced,
|
||||
loadDefaultScheduleAdvanced,
|
||||
scheduleAdvancedCfg
|
||||
} from '$lib/components/triggers/schedules/ScheduleAdvancedOptions.svelte'
|
||||
import { workspaceStore } from '$lib/stores'
|
||||
import { ArrowLeft, ChevronDown, CornerDownLeft, Loader2, Sparkles } from 'lucide-svelte'
|
||||
import { tick, untrack } from 'svelte'
|
||||
import { arrowTabNav } from '$lib/attachments/arrowTabNav'
|
||||
import { selectAndAdvanceTo } from '$lib/attachments/selectAndAdvanceTo'
|
||||
|
||||
@@ -49,8 +82,15 @@
|
||||
pathPrefix?: string
|
||||
defaultPathSuffix?: string
|
||||
onPick: (pick: PipelineInsertPick) => void
|
||||
trigger: import('svelte').Snippet
|
||||
/** Receives whether the menu is open, e.g. to keep a hover state while it is. */
|
||||
trigger: import('svelte').Snippet<[{ open: boolean }]>
|
||||
placement?: 'bottom' | 'top' | 'left' | 'right'
|
||||
/** Opens the menu each time it changes, for an entry point other than the trigger. */
|
||||
openSignal?: number
|
||||
/** With `openSignal`: only the schedule step, for this existing script; its
|
||||
* confirm hands the schedule to `onSchedule` instead of creating a script. */
|
||||
scheduleFor?: string
|
||||
onSchedule?: (schedule: PipelineInsertSchedule) => void
|
||||
}
|
||||
|
||||
let {
|
||||
@@ -60,7 +100,10 @@
|
||||
trigger: triggerSnippet,
|
||||
placement = 'bottom',
|
||||
defaultPathSuffix,
|
||||
onPick
|
||||
onPick,
|
||||
openSignal,
|
||||
scheduleFor,
|
||||
onSchedule
|
||||
}: Props = $props()
|
||||
|
||||
// When there's only one trigger kind, hide the Trigger column entirely
|
||||
@@ -75,6 +118,107 @@
|
||||
aiPrompt: undefined as undefined | string
|
||||
})
|
||||
let selected = $state(buildEmptySelected())
|
||||
let menuOpen = $state(false)
|
||||
$effect(() => {
|
||||
if (!openSignal) return
|
||||
untrack(() => {
|
||||
if (scheduleFor) {
|
||||
resetWizard()
|
||||
selected.triggerId = 'schedule'
|
||||
step = 1
|
||||
config.schedulePath = `${scheduleFor}_schedule`
|
||||
loadAdvancedDefaults()
|
||||
void focusCurrentStep()
|
||||
}
|
||||
menuOpen = true
|
||||
})
|
||||
})
|
||||
let selectedKind = $derived(kinds.find((k) => k.id === selected.triggerId))
|
||||
|
||||
// Steps after the first, each configuring something the picks call for.
|
||||
type ConfigStep = 'schedule' | 'asset'
|
||||
const NAMED_OUTPUT_KINDS: PipelineOutputKind[] = ['materialize', 'datatable', 'ducklake']
|
||||
let configSteps = $derived.by<ConfigStep[]>(() => {
|
||||
const steps: ConfigStep[] = []
|
||||
if (selected.triggerId === 'schedule') steps.push('schedule')
|
||||
if (selected.outputId && NAMED_OUTPUT_KINDS.includes(selected.outputId)) steps.push('asset')
|
||||
return steps
|
||||
})
|
||||
// 0 is the trigger/language/output/path step; `n` is `configSteps[n - 1]`.
|
||||
let step = $state(0)
|
||||
let currentConfigStep = $derived(step > 0 ? configSteps[step - 1] : undefined)
|
||||
let isLastStep = $derived(step >= configSteps.length)
|
||||
|
||||
const buildEmptyConfig = () => ({
|
||||
schedule: '0 0 12 * *',
|
||||
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
|
||||
cronVersion: 'v2',
|
||||
validCron: true,
|
||||
// Follows the script path until the Path field is edited.
|
||||
schedulePath: '',
|
||||
schedulePathDirty: false,
|
||||
schedulePathError: '',
|
||||
summary: '',
|
||||
description: '',
|
||||
pauseUntil: false,
|
||||
pausedUntil: undefined as string | undefined,
|
||||
advanced: emptyScheduleAdvanced(),
|
||||
// The workspace's default handlers, loaded when the schedule step opens.
|
||||
advancedDefaults: 'unloaded' as 'unloaded' | 'loading' | 'loaded',
|
||||
// `store` is the data table or DuckLake catalog name. The table name
|
||||
// follows the script name until the user types in it.
|
||||
asset: undefined as
|
||||
| undefined
|
||||
| { kind: 'ducklake' | 'datatable'; store: string | undefined; table: string },
|
||||
tableEdited: false
|
||||
})
|
||||
let config = $state(buildEmptyConfig())
|
||||
|
||||
// A new schedule starts from the workspace's default handlers, as in the schedule
|
||||
// drawer. The step cannot be confirmed until they are in.
|
||||
function loadAdvancedDefaults() {
|
||||
const ws = $workspaceStore
|
||||
if (!ws || config.advancedDefaults !== 'unloaded') return
|
||||
const target = config
|
||||
target.advancedDefaults = 'loading'
|
||||
loadDefaultScheduleAdvanced(ws)
|
||||
.then((d) => (target.advanced = d))
|
||||
.catch(() => {})
|
||||
.finally(() => (target.advancedDefaults = 'loaded'))
|
||||
}
|
||||
|
||||
function resetWizard() {
|
||||
selected = buildEmptySelected()
|
||||
config = buildEmptyConfig()
|
||||
step = 0
|
||||
}
|
||||
|
||||
const TABLE_NAME_RE = /^[^/\s]+$/
|
||||
// The path grammar of `Path.svelte`: word segments, no empty or trailing ones.
|
||||
const SCRIPT_NAME_RE = /^[\w-]+(\/[\w-]+)*$/
|
||||
let scriptNameValid = $derived(SCRIPT_NAME_RE.test(selected.scriptPath.trim()))
|
||||
let assetValid = $derived(!!config.asset?.store && TABLE_NAME_RE.test(config.asset.table.trim()))
|
||||
let stepValid = $derived(
|
||||
currentConfigStep === 'schedule'
|
||||
? config.validCron &&
|
||||
!!config.schedule.trim() &&
|
||||
!config.schedulePathError &&
|
||||
config.advancedDefaults === 'loaded'
|
||||
: currentConfigStep === 'asset'
|
||||
? assetValid
|
||||
: scriptNameValid && !!selected.triggerId && !!selected.language && !!selected.outputId
|
||||
)
|
||||
|
||||
function enterAssetStep() {
|
||||
const kind = selected.outputId === 'datatable' ? 'datatable' : 'ducklake'
|
||||
if (config.asset?.kind !== kind) {
|
||||
config.asset = { kind, store: 'main', table: '' }
|
||||
config.tableEdited = false
|
||||
}
|
||||
if (!config.tableEdited && config.asset) {
|
||||
config.asset.table = selected.scriptPath.trim().split('/').pop() ?? ''
|
||||
}
|
||||
}
|
||||
|
||||
// Refs to the column containers so Enter inside one column can hand
|
||||
// focus off to the first tabbable in the next column.
|
||||
@@ -104,16 +248,73 @@
|
||||
return out
|
||||
}
|
||||
|
||||
// A step change re-renders the popover body, dropping focus to <body>, where
|
||||
// keystrokes (Enter, Cmd+Enter) no longer reach the step's key handlers.
|
||||
let contentEl: HTMLElement | undefined = $state()
|
||||
async function focusCurrentStep() {
|
||||
await tick()
|
||||
const target =
|
||||
step === 0
|
||||
? pathEl?.querySelector('input')
|
||||
: contentEl?.querySelector<HTMLElement>('[data-autofocus] input')
|
||||
target?.focus()
|
||||
}
|
||||
|
||||
function goBack() {
|
||||
step -= 1
|
||||
void focusCurrentStep()
|
||||
}
|
||||
|
||||
function scheduleConfig(): PipelineInsertSchedule {
|
||||
return {
|
||||
path: config.schedulePath,
|
||||
schedule: config.schedule.trim(),
|
||||
timezone: config.timezone,
|
||||
cron_version: config.cronVersion,
|
||||
summary: config.summary.trim() || undefined,
|
||||
description: config.description,
|
||||
paused_until: config.pauseUntil ? config.pausedUntil : undefined,
|
||||
is_flow: false,
|
||||
args: {},
|
||||
...scheduleAdvancedCfg($state.snapshot(config.advanced))
|
||||
}
|
||||
}
|
||||
|
||||
function confirm(close: () => void) {
|
||||
if (scheduleFor) {
|
||||
if (!stepValid) return
|
||||
onSchedule?.(scheduleConfig())
|
||||
close()
|
||||
return
|
||||
}
|
||||
const suffix = selected.scriptPath.trim()
|
||||
if (!suffix || !selected.triggerId || !selected.language || !selected.outputId) return
|
||||
if (!stepValid) return
|
||||
if (!isLastStep) {
|
||||
step += 1
|
||||
if (configSteps[step - 1] === 'asset') enterAssetStep()
|
||||
if (configSteps[step - 1] === 'schedule') {
|
||||
loadAdvancedDefaults()
|
||||
if (!config.schedulePathDirty) config.schedulePath = `${pathPrefix}${suffix}_schedule`
|
||||
}
|
||||
void focusCurrentStep()
|
||||
return
|
||||
}
|
||||
const trimmedPrompt = selected.aiPrompt?.trim()
|
||||
onPick({
|
||||
kindId: selected.triggerId,
|
||||
language: selected.language,
|
||||
path: pathPrefix + suffix,
|
||||
outputKind: selected.outputId,
|
||||
aiPrompt: trimmedPrompt && trimmedPrompt.length > 0 ? trimmedPrompt : undefined
|
||||
aiPrompt: trimmedPrompt && trimmedPrompt.length > 0 ? trimmedPrompt : undefined,
|
||||
schedule: configSteps.includes('schedule') ? scheduleConfig() : undefined,
|
||||
outputAsset:
|
||||
configSteps.includes('asset') && config.asset
|
||||
? {
|
||||
kind: config.asset.kind as AssetKind,
|
||||
path: `${config.asset.store}/${config.asset.table.trim()}`
|
||||
}
|
||||
: undefined
|
||||
})
|
||||
close()
|
||||
}
|
||||
@@ -123,9 +324,18 @@
|
||||
enableFlyTransition
|
||||
contentClasses={twMerge(
|
||||
'p-0 bg-surface overflow-hidden relative transition-height',
|
||||
showBottomPanel ? 'h-[26rem]' : 'h-[22rem]'
|
||||
// The first step's columns (w-56 + w-48 + w-80), held across steps.
|
||||
singleKind ? 'w-[32rem]' : 'w-[46rem]',
|
||||
currentConfigStep === 'schedule'
|
||||
? 'h-[27rem]'
|
||||
: currentConfigStep === 'asset'
|
||||
? 'h-[14rem]'
|
||||
: showBottomPanel
|
||||
? 'h-[26rem]'
|
||||
: 'h-[22rem]'
|
||||
)}
|
||||
class="inline-block"
|
||||
bind:isOpen={menuOpen}
|
||||
usePointerDownOutside
|
||||
floatingConfig={{
|
||||
placement,
|
||||
@@ -136,24 +346,30 @@
|
||||
fitViewport: true,
|
||||
overlap: false
|
||||
}}
|
||||
onClose={() => (selected = buildEmptySelected())}
|
||||
onClose={resetWizard}
|
||||
>
|
||||
{#snippet trigger()}
|
||||
{@render triggerSnippet?.()}
|
||||
{@render triggerSnippet?.({ open: menuOpen })}
|
||||
{/snippet}
|
||||
{#snippet content({ close })}
|
||||
<div class="flex flex-col h-full">
|
||||
<div class={'flex flex-row transition-height divide-x overflow-y-scroll'}>
|
||||
{@render topSection()}
|
||||
</div>
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex flex-col gap-5 grow transition-height px-4 border-t',
|
||||
showBottomPanel ? 'h-[14rem] py-4' : 'h-0'
|
||||
)}
|
||||
>
|
||||
{@render bottomSection(close)}
|
||||
</div>
|
||||
<div class="h-full" bind:this={contentEl}>
|
||||
{#if currentConfigStep}
|
||||
{@render configStepSection(currentConfigStep, close)}
|
||||
{:else}
|
||||
<div class="flex flex-col h-full">
|
||||
<div class={'flex flex-row transition-height divide-x overflow-y-scroll'}>
|
||||
{@render topSection()}
|
||||
</div>
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex flex-col gap-5 grow transition-height px-4 border-t',
|
||||
showBottomPanel ? 'h-[14rem] py-4' : 'h-0'
|
||||
)}
|
||||
>
|
||||
{@render bottomSection(close)}
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
{/snippet}
|
||||
</Popover>
|
||||
@@ -170,7 +386,10 @@
|
||||
<Button
|
||||
variant="subtle"
|
||||
btnClasses={'text-left'}
|
||||
onClick={() => (selected.triggerId = k.id)}
|
||||
onClick={() => {
|
||||
selected.triggerId = k.id
|
||||
if (k.id === 'schedule') loadAdvancedDefaults()
|
||||
}}
|
||||
selected={isSelected}
|
||||
>
|
||||
{#if k.icon}
|
||||
@@ -264,14 +483,30 @@
|
||||
{#snippet bottomSection(close: () => void)}
|
||||
<Label label="Path">
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div bind:this={pathEl} class="flex" onkeydown={selectAndAdvanceTo(() => aiPromptEl ?? saveEl)}>
|
||||
<div
|
||||
bind:this={pathEl}
|
||||
class="flex"
|
||||
onkeydown={(e) =>
|
||||
(e.metaKey || e.ctrlKey) && e.key === 'Enter'
|
||||
? (e.preventDefault(), e.stopPropagation(), confirm(close))
|
||||
: selectAndAdvanceTo(() => aiPromptEl ?? saveEl)(e)}
|
||||
>
|
||||
<div
|
||||
class="border rounded-md rounded-r-none border-r-0 text-xs w-fit shrink-0 whitespace-nowrap flex items-center px-2 text-secondary bg-surface-input"
|
||||
>
|
||||
{pathPrefix}
|
||||
</div>
|
||||
<TextInput bind:value={selected.scriptPath} class="rounded-l-none" />
|
||||
<TextInput
|
||||
bind:value={selected.scriptPath}
|
||||
class="rounded-l-none"
|
||||
error={!scriptNameValid && !!selected.scriptPath.trim()}
|
||||
/>
|
||||
</div>
|
||||
{#if !scriptNameValid && selected.scriptPath.trim()}
|
||||
<span class="text-2xs text-red-600 dark:text-red-400">
|
||||
Letters, digits, <code>_</code> and <code>-</code>, in <code>/</code>-separated segments
|
||||
</span>
|
||||
{/if}
|
||||
</Label>
|
||||
{#if !$copilotInfo.workspaceDisabled}
|
||||
<Label label="AI Prompt (optional)">
|
||||
@@ -292,15 +527,217 @@
|
||||
</div>
|
||||
</Label>
|
||||
{/if}
|
||||
{@const hasAiPrompt = !!selected.aiPrompt?.trim()}
|
||||
<div class="ml-auto" bind:this={saveEl}>
|
||||
<Button
|
||||
variant="accent"
|
||||
btnClasses="w-fit"
|
||||
disabled={!selected.scriptPath.trim()}
|
||||
onClick={() => confirm(close)}
|
||||
startIcon={hasAiPrompt ? { icon: Sparkles } : undefined}
|
||||
shortCut={{ Icon: CornerDownLeft }}>{hasAiPrompt ? 'Generate' : 'Create'}</Button
|
||||
>
|
||||
{@render confirmButton(close)}
|
||||
</div>
|
||||
{/snippet}
|
||||
|
||||
{#snippet confirmButton(close: () => void)}
|
||||
{@const hasAiPrompt = !!selected.aiPrompt?.trim()}
|
||||
<Button
|
||||
variant="accent"
|
||||
btnClasses="w-fit"
|
||||
disabled={!stepValid}
|
||||
onClick={() => confirm(close)}
|
||||
startIcon={isLastStep && hasAiPrompt ? { icon: Sparkles } : undefined}
|
||||
shortCut={{ Icon: CornerDownLeft }}
|
||||
>{scheduleFor
|
||||
? 'Create schedule'
|
||||
: !isLastStep
|
||||
? step === 0
|
||||
? 'Configure'
|
||||
: 'Next'
|
||||
: hasAiPrompt
|
||||
? 'Generate'
|
||||
: selectedKind?.configuredAfterCreate
|
||||
? 'Configure'
|
||||
: 'Create'}</Button
|
||||
>
|
||||
{/snippet}
|
||||
|
||||
{#snippet configStepSection(configStep: ConfigStep, close: () => void)}
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div
|
||||
class="flex flex-col h-full"
|
||||
onkeydown={(e) => {
|
||||
if (e.key !== 'Enter') return
|
||||
const target = e.target as HTMLElement | null
|
||||
// Cmd/Ctrl+Enter confirms from anywhere; bare Enter only from the step's own
|
||||
// inputs, not from the Advanced fields (a textarea, pickers with their own Enter).
|
||||
const bareEnterSubmits = target?.tagName === 'INPUT' && !target.closest('[data-advanced]')
|
||||
if (e.metaKey || e.ctrlKey || bareEnterSubmits) {
|
||||
e.preventDefault()
|
||||
e.stopPropagation()
|
||||
confirm(close)
|
||||
}
|
||||
}}
|
||||
>
|
||||
<div class="flex items-center gap-2 px-2 py-2 border-b">
|
||||
{#if !scheduleFor}
|
||||
<Button
|
||||
variant="subtle"
|
||||
unifiedSize="sm"
|
||||
iconOnly
|
||||
startIcon={{ icon: ArrowLeft }}
|
||||
onClick={goBack}
|
||||
title="Back"
|
||||
/>
|
||||
{/if}
|
||||
<span class="text-xs font-semibold text-emphasis {scheduleFor ? 'pl-2' : ''}">
|
||||
{scheduleFor
|
||||
? `Schedule for ${scheduleFor}`
|
||||
: configStep === 'schedule'
|
||||
? 'Schedule'
|
||||
: 'Output asset'}
|
||||
</span>
|
||||
{#if !scheduleFor}
|
||||
<span class="text-2xs text-hint ml-auto pr-2"
|
||||
>Step {step + 1} of {configSteps.length + 1}</span
|
||||
>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex flex-col gap-2 grow overflow-auto p-4">
|
||||
{#if configStep === 'schedule'}
|
||||
<span class="text-2xs text-secondary">
|
||||
Saved as a draft schedule and deployed along with the pipeline.
|
||||
</span>
|
||||
<div data-autofocus class="contents">
|
||||
<CronInput
|
||||
bind:schedule={config.schedule}
|
||||
bind:timezone={config.timezone}
|
||||
bind:validCRON={config.validCron}
|
||||
bind:cronVersion={config.cronVersion}
|
||||
/>
|
||||
</div>
|
||||
<div data-advanced class="mt-4">
|
||||
<Section label="Advanced" collapsable initiallyCollapsed>
|
||||
<div class="flex flex-col gap-6">
|
||||
<Label label="Summary">
|
||||
<TextInput
|
||||
bind:value={config.summary}
|
||||
inputProps={{ placeholder: 'Short summary to be displayed when listed' }}
|
||||
/>
|
||||
</Label>
|
||||
<Label label="Path">
|
||||
<Path
|
||||
bind:path={config.schedulePath}
|
||||
bind:dirty={config.schedulePathDirty}
|
||||
bind:error={config.schedulePathError}
|
||||
initialPath=""
|
||||
checkInitialPathExistence
|
||||
namePlaceholder="schedule"
|
||||
kind="schedule"
|
||||
/>
|
||||
</Label>
|
||||
<Label label="Description">
|
||||
<TextInput
|
||||
underlyingInputEl="textarea"
|
||||
bind:value={config.description}
|
||||
inputProps={{ placeholder: 'What this schedule does and how to use it' }}
|
||||
/>
|
||||
</Label>
|
||||
<div class="flex flex-col gap-1">
|
||||
<Toggle
|
||||
options={{
|
||||
right: 'Pause schedule until...',
|
||||
rightTooltip:
|
||||
'Pausing the schedule will program the next job to run as if the schedule starts at the time the pause is lifted, instead of now.'
|
||||
}}
|
||||
bind:checked={config.pauseUntil}
|
||||
/>
|
||||
{#if config.pauseUntil}
|
||||
<DateTimeInput bind:value={config.pausedUntil} />
|
||||
{/if}
|
||||
</div>
|
||||
<!-- Rendered once the workspace defaults are in, so they never
|
||||
overwrite a handler picked while they load. -->
|
||||
{#if config.advancedDefaults === 'loaded'}
|
||||
<ScheduleAdvancedOptions
|
||||
wsId={$workspaceStore}
|
||||
itemKind="script"
|
||||
canWrite
|
||||
bind:errorHandlerSelected={config.advanced.errorHandlerSelected}
|
||||
bind:errorHandlerPath={config.advanced.errorHandlerPath}
|
||||
bind:errorHandleritemKind={config.advanced.errorHandleritemKind}
|
||||
bind:errorHandlerExtraArgs={config.advanced.errorHandlerExtraArgs}
|
||||
bind:wsErrorHandlerMuted={config.advanced.wsErrorHandlerMuted}
|
||||
bind:failedTimes={config.advanced.failedTimes}
|
||||
bind:failedExact={config.advanced.failedExact}
|
||||
bind:recoveryHandlerSelected={config.advanced.recoveryHandlerSelected}
|
||||
bind:recoveryHandlerPath={config.advanced.recoveryHandlerPath}
|
||||
bind:recoveryHandlerItemKind={config.advanced.recoveryHandlerItemKind}
|
||||
bind:recoveryHandlerExtraArgs={config.advanced.recoveryHandlerExtraArgs}
|
||||
bind:recoveredTimes={config.advanced.recoveredTimes}
|
||||
bind:successHandlerSelected={config.advanced.successHandlerSelected}
|
||||
bind:successHandlerPath={config.advanced.successHandlerPath}
|
||||
bind:successHandlerItemKind={config.advanced.successHandlerItemKind}
|
||||
bind:successHandlerExtraArgs={config.advanced.successHandlerExtraArgs}
|
||||
bind:retry={config.advanced.retry}
|
||||
bind:dynamicSkipPath={config.advanced.dynamicSkipPath}
|
||||
bind:tag={config.advanced.tag}
|
||||
/>
|
||||
{:else}
|
||||
<div class="flex items-center gap-1.5 text-xs text-secondary">
|
||||
<Loader2 size={12} class="animate-spin" />
|
||||
Loading the workspace's default handlers…
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
</Section>
|
||||
</div>
|
||||
{:else if configStep === 'asset' && config.asset}
|
||||
<Label label="Name">
|
||||
<div
|
||||
class={twMerge(
|
||||
inputBaseClass,
|
||||
inputBorderClass({ error: !assetValid }),
|
||||
inputSizeClasses.md,
|
||||
'flex flex-row items-center gap-0 py-0 pl-2 pr-0'
|
||||
)}
|
||||
>
|
||||
<span class="text-xs text-secondary shrink-0">{config.asset.kind}://</span>
|
||||
{#if config.asset.kind === 'datatable'}
|
||||
<DatatablePicker
|
||||
bind:value={config.asset.store}
|
||||
class="shrink-0"
|
||||
selectInputClass="!border-none"
|
||||
placeholder="data table"
|
||||
useContentEditable
|
||||
RightIcon={ChevronDown}
|
||||
/>
|
||||
{:else}
|
||||
<DucklakePicker
|
||||
bind:value={config.asset.store}
|
||||
class="shrink-0"
|
||||
selectInputClass="!border-none"
|
||||
placeholder="catalog"
|
||||
useContentEditable
|
||||
RightIcon={ChevronDown}
|
||||
/>
|
||||
{/if}
|
||||
<span class="text-sm text-secondary">/</span>
|
||||
<div data-autofocus class="grow flex">
|
||||
<TextInput
|
||||
bind:value={config.asset.table}
|
||||
class="!border-none grow"
|
||||
inputProps={{
|
||||
placeholder: 'table',
|
||||
oninput: () => (config.tableEdited = true)
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</Label>
|
||||
<span class="text-2xs text-hint">
|
||||
{config.asset.kind === 'datatable' ? 'Data table' : 'DuckLake catalog'}, then the table
|
||||
name
|
||||
</span>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex px-4 py-3 border-t">
|
||||
<div class="ml-auto">
|
||||
{@render confirmButton(close)}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{/snippet}
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
<!--
|
||||
@component
|
||||
The body every pipeline graph node shares: icon, a kind line ("Script · draft", "Data table"),
|
||||
a title, and trailing chips. Node components add only what is theirs — the icon, the chips,
|
||||
and the controls hung around the card.
|
||||
-->
|
||||
<script lang="ts">
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import {
|
||||
PIPELINE_NODE_EXTRA_ROW,
|
||||
PIPELINE_NODE_HEADER,
|
||||
PIPELINE_NODE_HEADER_CARD_PAD,
|
||||
PIPELINE_NODE_HEIGHT
|
||||
} from './assetGraphLayout'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import type { Snippet } from 'svelte'
|
||||
|
||||
interface Props {
|
||||
icon: Snippet
|
||||
kindLabel: string
|
||||
title: string
|
||||
/** Hover text; the path, when the title is a summary. */
|
||||
tooltip?: string
|
||||
/** Scripts sit on the main surface, assets and triggers a step down; in the
|
||||
* assets-only view, which has no scripts, assets take the main surface. */
|
||||
surface?: 'primary' | 'secondary'
|
||||
selected?: boolean
|
||||
/** Not deployed yet: dashed edge. */
|
||||
draft?: boolean
|
||||
/** Feedback states, which recolor the whole card. `error` is a misconfigured
|
||||
* node, styled like a failed flow step. */
|
||||
tone?: 'danger' | 'error' | 'success' | 'running'
|
||||
/** Makes the whole card a button. */
|
||||
onclick?: (e: MouseEvent) => void
|
||||
/** Between the icon and the text: a live status the eye should find first. */
|
||||
leading?: Snippet
|
||||
trailing?: Snippet
|
||||
/** Defaults to `NODE.width`. */
|
||||
width?: number
|
||||
/** A line under the title, in the text column. */
|
||||
subtitle?: Snippet
|
||||
/** A tab on the card's top edge. Only on a card without `onclick`, since it
|
||||
* may hold buttons of its own. */
|
||||
header?: Snippet
|
||||
/** The header's script is selected: its edge takes the selected colour. */
|
||||
headerSelected?: boolean
|
||||
/** The header's script has an error: its edge goes red. */
|
||||
headerTone?: 'error'
|
||||
}
|
||||
|
||||
let {
|
||||
icon,
|
||||
kindLabel,
|
||||
title,
|
||||
tooltip,
|
||||
surface = 'secondary',
|
||||
selected = false,
|
||||
draft = false,
|
||||
tone,
|
||||
onclick,
|
||||
leading,
|
||||
trailing,
|
||||
subtitle,
|
||||
header,
|
||||
headerSelected = false,
|
||||
headerTone,
|
||||
width = NODE.width
|
||||
}: Props = $props()
|
||||
|
||||
const cardClass = $derived(
|
||||
twMerge(
|
||||
'flex items-center w-full text-left font-normal rounded-md drop-shadow-sm overflow-hidden border transition-colors',
|
||||
surface === 'primary' ? 'bg-surface' : 'bg-surface-secondary',
|
||||
'border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500',
|
||||
// Like a flow step: a card with a background needs no edge, only its shadow,
|
||||
// until one of the states below draws one.
|
||||
surface === 'primary' &&
|
||||
!draft &&
|
||||
!selected &&
|
||||
!tone &&
|
||||
'border-transparent dark:border-transparent hover:border-transparent dark:hover:border-transparent',
|
||||
draft && 'border-dashed border-gray-400 dark:border-gray-500',
|
||||
// Like a selected flow step: the fill and the edge colour change, the edge's
|
||||
// width and dashes stay.
|
||||
// Each dark: variant too: the base edge sets its own dark colour, which a plain
|
||||
// border class does not override.
|
||||
selected &&
|
||||
'bg-surface-accent-selected border-border-selected dark:border-border-selected hover:border-border-selected dark:hover:border-border-selected',
|
||||
tone === 'danger' &&
|
||||
'bg-red-50 dark:bg-red-900/30 border-dashed border-red-400 dark:border-red-500 hover:bg-red-100 dark:hover:bg-red-900/40',
|
||||
tone === 'error' &&
|
||||
(selected
|
||||
? 'bg-red-200 dark:bg-red-600 border-red-500 dark:border-red-500 hover:border-red-500 dark:hover:border-red-500'
|
||||
: 'bg-red-100 dark:bg-red-700 border-transparent dark:border-transparent hover:border-transparent dark:hover:border-transparent'),
|
||||
tone === 'success' &&
|
||||
'bg-green-50 dark:bg-green-900/30 border-green-500 dark:border-green-600',
|
||||
tone === 'running' &&
|
||||
'bg-amber-50 dark:bg-amber-900/30 border-amber-400 dark:border-amber-600 animate-pulse'
|
||||
)
|
||||
)
|
||||
const minHeight = $derived(PIPELINE_NODE_HEIGHT + (subtitle ? PIPELINE_NODE_EXTRA_ROW : 0))
|
||||
const toneText = $derived(
|
||||
tone === 'danger'
|
||||
? 'text-red-700 dark:text-red-400'
|
||||
: tone === 'error'
|
||||
? 'text-red-700 dark:text-red-200'
|
||||
: tone === 'success'
|
||||
? 'text-green-700 dark:text-green-400'
|
||||
: undefined
|
||||
)
|
||||
</script>
|
||||
|
||||
{#snippet body()}
|
||||
<span class="shrink-0 ml-3 mr-2.5 flex items-center">{@render icon()}</span>
|
||||
{@render leading?.()}
|
||||
<span class="flex flex-col min-w-0 flex-1 pr-1 py-1 leading-tight">
|
||||
<span class={twMerge('text-2xs truncate text-secondary', toneText)}>{kindLabel}</span>
|
||||
<span class={twMerge('text-xs truncate text-emphasis', selected && 'text-accent', toneText)}
|
||||
>{title}</span
|
||||
>
|
||||
{#if subtitle}
|
||||
<span class="flex items-center min-w-0 mt-1">{@render subtitle()}</span>
|
||||
{/if}
|
||||
</span>
|
||||
{#if trailing}
|
||||
<span class="flex items-center shrink-0 pr-2">{@render trailing()}</span>
|
||||
{/if}
|
||||
{/snippet}
|
||||
|
||||
{#if onclick}
|
||||
<button
|
||||
type="button"
|
||||
{onclick}
|
||||
class={cardClass}
|
||||
style="width: {width}px; min-height: {minHeight}px;"
|
||||
title={tooltip}
|
||||
>
|
||||
{@render body()}
|
||||
</button>
|
||||
{:else if header}
|
||||
<!-- The header is a tab on the card's top edge, inset from its sides. -->
|
||||
<div class="flex flex-col" style="width: {width}px;">
|
||||
<div
|
||||
class={twMerge(
|
||||
'mx-2 flex items-stretch min-w-0 rounded-t-md overflow-hidden bg-surface border border-b-0 border-gray-200 dark:border-gray-700',
|
||||
// Like an errored card: red, and a stronger red when selected.
|
||||
headerTone === 'error'
|
||||
? headerSelected
|
||||
? 'border-red-500 dark:border-red-500'
|
||||
: 'border-red-300 dark:border-red-600'
|
||||
: headerSelected && 'border-border-selected dark:border-border-selected'
|
||||
)}
|
||||
style="height: {PIPELINE_NODE_HEADER}px;"
|
||||
>
|
||||
{@render header()}
|
||||
</div>
|
||||
<div
|
||||
class={twMerge(cardClass, !selected && !tone && 'bg-surface-tertiary')}
|
||||
style="min-height: {minHeight + PIPELINE_NODE_HEADER_CARD_PAD}px; padding-block: {PIPELINE_NODE_HEADER_CARD_PAD /
|
||||
2}px;"
|
||||
title={tooltip}
|
||||
>
|
||||
{@render body()}
|
||||
</div>
|
||||
</div>
|
||||
{:else}
|
||||
<div
|
||||
class={cardClass}
|
||||
style="width: {width}px; min-height: {minHeight}px;"
|
||||
title={tooltip}
|
||||
>
|
||||
{@render body()}
|
||||
</div>
|
||||
{/if}
|
||||
@@ -1,6 +1,6 @@
|
||||
<script lang="ts">
|
||||
import type { AssetKind, Script } from '$lib/gen'
|
||||
import { CalendarClock, Loader2, Play, Upload, Zap } from 'lucide-svelte'
|
||||
import { CalendarClock, Code2, Loader2, Play, Upload, Zap } from 'lucide-svelte'
|
||||
import Button from '$lib/components/common/button/Button.svelte'
|
||||
import HighlightCode from '$lib/components/HighlightCode.svelte'
|
||||
import PipelineRunForm from './PipelineRunForm.svelte'
|
||||
@@ -44,6 +44,12 @@
|
||||
// "green / ready" state and seeds the whole-pipeline run. `isValid` is the
|
||||
// full-schema validity (all required fields, not just the S3 file).
|
||||
onArgsChange?: (path: string, args: Record<string, any>, isValid: boolean) => void
|
||||
// Just the run form and the runs, without the source: what a data-upload
|
||||
// click asks for. `onShowScript` switches to the script itself.
|
||||
inputsOnly?: boolean
|
||||
onShowScript?: () => void
|
||||
// `onRun` previews a draft's own content, so a draft can run too.
|
||||
runsDrafts?: boolean
|
||||
}
|
||||
|
||||
let {
|
||||
@@ -57,8 +63,12 @@
|
||||
runsPendingJobId,
|
||||
onRunCompleted,
|
||||
initialArgs,
|
||||
onArgsChange
|
||||
onArgsChange,
|
||||
inputsOnly = false,
|
||||
onShowScript,
|
||||
runsDrafts = false
|
||||
}: Props = $props()
|
||||
let runBlockedByDraft = $derived(isDraft && !runsDrafts)
|
||||
|
||||
// Seed once from the persisted args (see initialArgs). Cloned so the run
|
||||
// form mutates its own copy, not the page's stored snapshot.
|
||||
@@ -139,7 +149,7 @@
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col h-full">
|
||||
{#if isDraft}
|
||||
{#if runBlockedByDraft}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-2 px-3 py-1.5 text-2xs bg-amber-50 dark:bg-amber-900/30 border-b border-amber-200 dark:border-amber-900/60 text-amber-700 dark:text-amber-400"
|
||||
>
|
||||
@@ -168,13 +178,24 @@
|
||||
{/if}
|
||||
</span>
|
||||
<div class="flex items-center gap-1.5">
|
||||
{#if inputsOnly && onShowScript}
|
||||
<Button
|
||||
variant="subtle"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: Code2 }}
|
||||
onclick={onShowScript}
|
||||
title="Show the script instead of its inputs"
|
||||
>
|
||||
Open script
|
||||
</Button>
|
||||
{/if}
|
||||
{#if hasCascade}
|
||||
<Button
|
||||
variant="accent-secondary"
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: running ? Loader2 : Zap }}
|
||||
onclick={() => run(true)}
|
||||
disabled={isDraft || running || !runValid}
|
||||
disabled={runBlockedByDraft || running || !runValid}
|
||||
title={`Run this script with these inputs, then run its ${downstreamCount} downstream pipeline script${downstreamCount === 1 ? '' : 's'} in order`}
|
||||
>
|
||||
Run + downstream
|
||||
@@ -185,8 +206,8 @@
|
||||
unifiedSize="sm"
|
||||
startIcon={{ icon: running ? Loader2 : Play }}
|
||||
onclick={() => run(false)}
|
||||
disabled={isDraft || running || !runValid || !onRun}
|
||||
title={isDraft
|
||||
disabled={runBlockedByDraft || running || !runValid || !onRun}
|
||||
title={runBlockedByDraft
|
||||
? 'Deploy this draft to run it for real'
|
||||
: hasCascade
|
||||
? 'Run just this step — downstream scripts are not triggered'
|
||||
@@ -210,9 +231,11 @@
|
||||
{/key}
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex-1 min-h-0 overflow-auto text-xs p-3">
|
||||
<HighlightCode code={script.content ?? ''} language={script.language} />
|
||||
</div>
|
||||
{#if !inputsOnly}
|
||||
<div class="flex-1 min-h-0 overflow-auto text-xs p-3">
|
||||
<HighlightCode code={script.content ?? ''} language={script.language} />
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
</Pane>
|
||||
<Pane size={45} minSize={20}>
|
||||
|
||||
@@ -3,6 +3,8 @@
|
||||
import { WorkspaceService } from '$lib/gen'
|
||||
import { base } from '$lib/base'
|
||||
import { CircleCheck, CircleAlert, Database, HardDrive, ArrowRight } from 'lucide-svelte'
|
||||
import Alert from '$lib/components/common/alert/Alert.svelte'
|
||||
import { Button } from '$lib/components/common'
|
||||
|
||||
interface Props {
|
||||
workspace: string
|
||||
@@ -73,48 +75,43 @@
|
||||
</script>
|
||||
|
||||
{#if show}
|
||||
<div
|
||||
class="flex flex-col gap-3 rounded-lg border border-amber-300 dark:border-amber-900/60 bg-amber-50 dark:bg-amber-950/30 p-4"
|
||||
>
|
||||
<div class="flex items-start gap-2">
|
||||
<CircleAlert size={18} class="text-amber-600 dark:text-amber-400 mt-0.5 shrink-0" />
|
||||
<div class="flex flex-col gap-0.5">
|
||||
<h3 class="text-sm font-semibold text-emphasis">Finish setting up pipelines</h3>
|
||||
<p class="text-xs text-tertiary">
|
||||
Pipelines materialize data into DuckLake tables backed by object storage. Configure the
|
||||
following before your first pipeline can run.
|
||||
</p>
|
||||
</div>
|
||||
<Alert type="warning" title="Finish setting up pipelines" hideIcon>
|
||||
<div class="flex flex-col gap-3">
|
||||
<p>
|
||||
Pipelines materialize data into DuckLake tables backed by object storage. Configure the
|
||||
following before your first pipeline can run.
|
||||
</p>
|
||||
<ul class="flex flex-col gap-2">
|
||||
{#each steps as step (step.title)}
|
||||
{@const Icon = step.icon}
|
||||
<li class="flex items-center gap-3 rounded-md border bg-surface-tertiary px-3 py-2">
|
||||
{#if step.done === true}
|
||||
<CircleCheck size={16} class="text-green-600 dark:text-green-400 shrink-0" />
|
||||
{:else if step.done === false}
|
||||
<CircleAlert size={16} class="text-yellow-600 dark:text-yellow-400 shrink-0" />
|
||||
{:else}
|
||||
<Icon size={16} class="text-secondary shrink-0" />
|
||||
{/if}
|
||||
<div class="flex flex-col min-w-0 flex-1">
|
||||
<span class="text-xs font-semibold text-emphasis">{step.title}</span>
|
||||
<span class="text-2xs text-secondary">{step.description}</span>
|
||||
</div>
|
||||
{#if step.done !== true}
|
||||
<Button
|
||||
variant="default"
|
||||
unifiedSize="sm"
|
||||
href={step.href}
|
||||
endIcon={{ icon: ArrowRight }}
|
||||
wrapperClasses="shrink-0"
|
||||
>
|
||||
{step.cta}
|
||||
</Button>
|
||||
{:else}
|
||||
<span class="shrink-0 text-2xs text-green-700 dark:text-green-400">Configured</span>
|
||||
{/if}
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
</div>
|
||||
|
||||
<ul class="flex flex-col gap-2">
|
||||
{#each steps as step (step.title)}
|
||||
{@const Icon = step.icon}
|
||||
<li class="flex items-center gap-3 rounded-md border bg-surface px-3 py-2">
|
||||
{#if step.done === true}
|
||||
<CircleCheck size={16} class="text-green-600 dark:text-green-400 shrink-0" />
|
||||
{:else if step.done === false}
|
||||
<CircleAlert size={16} class="text-amber-600 dark:text-amber-400 shrink-0" />
|
||||
{:else}
|
||||
<Icon size={16} class="text-tertiary shrink-0" />
|
||||
{/if}
|
||||
<div class="flex flex-col min-w-0 flex-1">
|
||||
<span class="text-xs font-semibold text-primary">{step.title}</span>
|
||||
<span class="text-2xs text-tertiary">{step.description}</span>
|
||||
</div>
|
||||
{#if step.done !== true}
|
||||
<a
|
||||
href={step.href}
|
||||
class="shrink-0 inline-flex items-center gap-1 text-xs text-blue-500 hover:underline whitespace-nowrap"
|
||||
>
|
||||
{step.cta}
|
||||
<ArrowRight size={12} />
|
||||
</a>
|
||||
{:else}
|
||||
<span class="shrink-0 text-2xs text-green-700 dark:text-green-400">Configured</span>
|
||||
{/if}
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
</div>
|
||||
</Alert>
|
||||
{/if}
|
||||
|
||||
@@ -1,18 +1,10 @@
|
||||
<script lang="ts">
|
||||
import { sendUserToast } from '$lib/utils'
|
||||
import { workspaceStore } from '$lib/stores'
|
||||
import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte'
|
||||
import type { NativeTriggerKind } from './types'
|
||||
import {
|
||||
EmailTriggerService,
|
||||
GcpTriggerService,
|
||||
KafkaTriggerService,
|
||||
MqttTriggerService,
|
||||
AmqpTriggerService,
|
||||
NatsTriggerService,
|
||||
PostgresTriggerService,
|
||||
ScheduleService,
|
||||
SqsTriggerService
|
||||
} from '$lib/gen'
|
||||
import type { NativeTriggerKind, PipelineTriggerDraftKind } from './types'
|
||||
import type { Schedule } from '$lib/gen'
|
||||
import { deleteTriggerRow } from './pipelineTriggerDraftDeploy'
|
||||
import KafkaTriggerEditor from '$lib/components/triggers/kafka/KafkaTriggerEditor.svelte'
|
||||
import MqttTriggerEditor from '$lib/components/triggers/mqtt/MqttTriggerEditor.svelte'
|
||||
import AmqpTriggerEditor from '$lib/components/triggers/amqp/AmqpTriggerEditor.svelte'
|
||||
@@ -22,6 +14,10 @@
|
||||
import GcpTriggerEditor from '$lib/components/triggers/gcp/GcpTriggerEditor.svelte'
|
||||
import EmailTriggerEditor from '$lib/components/triggers/email/EmailTriggerEditor.svelte'
|
||||
import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte'
|
||||
import {
|
||||
loadDefaultScheduleAdvanced,
|
||||
scheduleAdvancedCfg
|
||||
} from '$lib/components/triggers/schedules/ScheduleAdvancedOptions.svelte'
|
||||
import WebhookEditor from '$lib/components/triggers/webhook/WebhookEditor.svelte'
|
||||
import { setOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
|
||||
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
|
||||
@@ -110,6 +106,56 @@
|
||||
}
|
||||
}
|
||||
|
||||
/** The kind's full editor over a pipeline-local draft; Save hands the config back,
|
||||
* and the drawer stays open when `onSave` refuses it.
|
||||
* `saved` is the draft being edited (absent for a new one), applied over `defaults`. */
|
||||
export function openTriggerDraft(
|
||||
kind: PipelineTriggerDraftKind,
|
||||
scriptPath: string,
|
||||
defaults: Record<string, any>,
|
||||
saved: Record<string, any> | undefined,
|
||||
onSave: (cfg: Record<string, any>) => boolean
|
||||
) {
|
||||
switch (kind) {
|
||||
case 'schedule': {
|
||||
// A new draft starts from the workspace's default handlers, like a new
|
||||
// schedule does; the drawer only applies them when it opens without one.
|
||||
const open = (advanced: Record<string, any>) =>
|
||||
scheduleEditor?.openDraft(
|
||||
{
|
||||
is_flow: false,
|
||||
args: {},
|
||||
...advanced,
|
||||
...defaults,
|
||||
...saved,
|
||||
script_path: scriptPath
|
||||
} as Schedule,
|
||||
onSave
|
||||
)
|
||||
if (saved || !$workspaceStore) return open({})
|
||||
return loadDefaultScheduleAdvanced($workspaceStore)
|
||||
.then((a) => open(scheduleAdvancedCfg(a)))
|
||||
.catch(() => open({}))
|
||||
}
|
||||
case 'kafka':
|
||||
return kafkaEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'mqtt':
|
||||
return mqttEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'amqp':
|
||||
return amqpEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'nats':
|
||||
return natsEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'postgres':
|
||||
return postgresEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'sqs':
|
||||
return sqsEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'gcp':
|
||||
return gcpEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
case 'email':
|
||||
return emailEditor?.openDraft(scriptPath, defaults, saved, onSave)
|
||||
}
|
||||
}
|
||||
|
||||
// Open the webhook drawer (endpoint URLs + token flow). Caller owns the
|
||||
// draft guard.
|
||||
export function openWebhook(scriptPath: string) {
|
||||
@@ -133,37 +179,7 @@
|
||||
const { kind, path: triggerPath } = triggerDeleteTarget
|
||||
triggerDeleteLoading = true
|
||||
try {
|
||||
switch (kind) {
|
||||
case 'schedule':
|
||||
await ScheduleService.deleteSchedule({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'kafka':
|
||||
await KafkaTriggerService.deleteKafkaTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'mqtt':
|
||||
await MqttTriggerService.deleteMqttTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'amqp':
|
||||
await AmqpTriggerService.deleteAmqpTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'nats':
|
||||
await NatsTriggerService.deleteNatsTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'postgres':
|
||||
await PostgresTriggerService.deletePostgresTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'sqs':
|
||||
await SqsTriggerService.deleteSqsTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'gcp':
|
||||
await GcpTriggerService.deleteGcpTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
case 'email':
|
||||
await EmailTriggerService.deleteEmailTrigger({ workspace, path: triggerPath })
|
||||
break
|
||||
default:
|
||||
return
|
||||
}
|
||||
if (!(await deleteTriggerRow(kind, triggerPath, workspace))) return
|
||||
sendUserToast(`Deleted ${kind} trigger "${triggerPath}"`)
|
||||
triggerDeleteTarget = undefined
|
||||
onUpdate()
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
<script lang="ts">
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import { CheckCircle2, Loader2, XCircle } from 'lucide-svelte'
|
||||
import type { RunnableRunState } from './activeRunnables.svelte'
|
||||
|
||||
let { runState: rs, class: className }: { runState: RunnableRunState; class?: string } = $props()
|
||||
</script>
|
||||
|
||||
<div
|
||||
class={twMerge(
|
||||
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 rounded-sm border',
|
||||
rs.status === 'running'
|
||||
? 'bg-blue-100 dark:bg-blue-900/40 text-blue-700 dark:text-blue-300 border-blue-300 dark:border-blue-700'
|
||||
: rs.status === 'success'
|
||||
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300 border-emerald-300 dark:border-emerald-700'
|
||||
: 'bg-red-100 dark:bg-red-900/40 text-red-700 dark:text-red-300 border-red-300 dark:border-red-700',
|
||||
className
|
||||
)}
|
||||
title={`${
|
||||
rs.status === 'running'
|
||||
? 'Running now'
|
||||
: rs.status === 'success'
|
||||
? 'Last run succeeded'
|
||||
: 'Last run failed'
|
||||
}${rs.runs > 0 ? ` — ran ${rs.runs}× this session` : ''}`}
|
||||
>
|
||||
{#if rs.status === 'running'}
|
||||
<Loader2 size={10} class="animate-spin" />
|
||||
{:else if rs.status === 'success'}
|
||||
<CheckCircle2 size={10} />
|
||||
{:else}
|
||||
<XCircle size={10} />
|
||||
{/if}
|
||||
{#if rs.runs > 0}
|
||||
<span class="text-3xs leading-none tabular-nums">×{rs.runs}</span>
|
||||
{/if}
|
||||
</div>
|
||||
@@ -1,10 +1,8 @@
|
||||
<script lang="ts">
|
||||
import { Handle, Position } from '@xyflow/svelte'
|
||||
import {
|
||||
CheckCircle2,
|
||||
ChevronDown,
|
||||
Code2,
|
||||
EllipsisVertical,
|
||||
GitBranch,
|
||||
Layers,
|
||||
Loader2,
|
||||
@@ -15,17 +13,20 @@
|
||||
Target,
|
||||
Timer,
|
||||
Trash2,
|
||||
XCircle,
|
||||
Zap
|
||||
} from 'lucide-svelte'
|
||||
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import { preventDefault, stopPropagation } from 'svelte/legacy'
|
||||
import type { GraphUsageKind } from './types'
|
||||
import type { RunnableRunState } from './activeRunnables.svelte'
|
||||
import { parseDurationSecs } from './parsePipelineAnnotations'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import DropdownV2 from '$lib/components/DropdownV2.svelte'
|
||||
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
|
||||
import PipelineNodeCard from './PipelineNodeCard.svelte'
|
||||
import NodeActionsMenu from './NodeActionsMenu.svelte'
|
||||
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
|
||||
import NodeOnDot from './NodeOnDot.svelte'
|
||||
import RunStateChip from './RunStateChip.svelte'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import Popover from '$lib/components/meltComponents/Popover.svelte'
|
||||
import type { Item } from '$lib/utils'
|
||||
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
|
||||
@@ -35,8 +36,16 @@
|
||||
|
||||
interface Props {
|
||||
data: {
|
||||
/** What is wrong with the node, and how to fix it: shows the Fix pill. */
|
||||
fix?: NodeFix
|
||||
/** Starts dragging the top dot onto an asset to subscribe the script to it. */
|
||||
onStartOnDrag?: (e: PointerEvent) => void
|
||||
runnable_kind: GraphUsageKind
|
||||
path: string
|
||||
summary?: string
|
||||
language?: ScriptLang
|
||||
// Why the script is misconfigured, when it is: renders it red.
|
||||
error?: string
|
||||
in_pipeline?: boolean
|
||||
partition_kind?: 'daily' | 'hourly' | 'weekly' | 'monthly' | 'dynamic'
|
||||
freshness?: string
|
||||
@@ -101,10 +110,11 @@
|
||||
}
|
||||
let { data, selected = false }: Props = $props()
|
||||
|
||||
// The icon alone conveys "script" vs "flow"; the uppercase kind label was
|
||||
// visually noisy and redundant. Tooltip on hover surfaces the path in
|
||||
// full when truncated.
|
||||
let Icon = $derived(data.runnable_kind === 'flow' ? GitBranch : Code2)
|
||||
let kindLabel = $derived(
|
||||
data.runnable_kind === 'flow' ? 'Flow' : data.dbt ? 'dbt project' : 'Script'
|
||||
)
|
||||
// The title shows the summary when there is one, so the tooltip keeps the path.
|
||||
let nodeTooltip = $derived(
|
||||
data.unsaved
|
||||
? `${data.path} (unsaved draft)`
|
||||
@@ -114,7 +124,6 @@
|
||||
)
|
||||
|
||||
let hover = $state(false)
|
||||
let menuOpen = $state(false)
|
||||
let running = $state(false)
|
||||
|
||||
// Amber "computing now" surface, gated so only the replay player lights it up.
|
||||
@@ -214,131 +223,106 @@
|
||||
<!--
|
||||
Mirrors the flow editor's step styling (getNodeColorClasses): muted
|
||||
surface-tertiary fill with a quiet gray border, accent only for the
|
||||
selected state, dashes for unsaved drafts. The 2px dashed stroke is
|
||||
necessary for the dash gaps to be readable; 1px disappears into the
|
||||
fill.
|
||||
selected state, dashes for unsaved drafts — 1px, like the trigger
|
||||
nodes' dashed outline, so every draft node reads the same.
|
||||
-->
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden border transition-colors',
|
||||
'bg-surface border-gray-400 dark:border-gray-600 hover:border-gray-500 dark:hover:border-gray-500',
|
||||
selected && 'bg-surface-accent-selected border-border-selected',
|
||||
data.unsaved && 'border-2 border-dashed border-gray-400 dark:border-gray-500',
|
||||
computingNow &&
|
||||
'bg-amber-50 dark:bg-amber-900/30 border-amber-400 dark:border-amber-600 animate-pulse'
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={nodeTooltip}
|
||||
>
|
||||
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${selected ? 'text-accent' : 'text-secondary'}`} />
|
||||
<span class="flex-1 min-w-0 pr-1 py-0.5 text-2xs font-mono text-emphasis truncate">
|
||||
{data.path}
|
||||
</span>
|
||||
<!-- Annotation chips share one neutral treatment — the icon carries
|
||||
the meaning (colors are reserved for feedback, per the brand
|
||||
guidelines). Only the freshness chip (when it has a verdict)
|
||||
and the run-state chip below use semantic colors. -->
|
||||
{#if data.partition_kind}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// partitioned ${data.partition_kind}`}
|
||||
>
|
||||
<Layers size={10} />
|
||||
<span class="text-3xs leading-none">{data.partition_kind}</span>
|
||||
</div>
|
||||
{/if}
|
||||
<!-- Freshness is the one annotation chip that carries feedback (a
|
||||
fresh/stale verdict against real run history), so like the
|
||||
run-state chip it uses semantic colors: emerald = within window,
|
||||
amber = stale. Neutral when there's no verdict (drafts, bad
|
||||
window value). -->
|
||||
{#if data.freshness}
|
||||
<div
|
||||
class={twMerge(
|
||||
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
|
||||
freshnessState === 'fresh'
|
||||
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
|
||||
: freshnessState === 'stale'
|
||||
? 'bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300'
|
||||
: 'bg-surface-secondary text-secondary'
|
||||
)}
|
||||
title={freshnessTooltip}
|
||||
>
|
||||
<Timer size={10} />
|
||||
<span class="text-3xs leading-none">{data.freshness}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.tag}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// tag ${data.tag}`}
|
||||
>
|
||||
<Tag size={10} />
|
||||
<span class="text-3xs leading-none">{data.tag}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.retry}
|
||||
{@const r = data.retry}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// retry ${r.count}${r.delay ? ` ${r.delay}` : ''}`}
|
||||
>
|
||||
<RotateCw size={10} />
|
||||
<span class="text-3xs leading-none">×{r.count}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.macros && data.macros.length > 0}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// macros — defines ${data.macros.length} macro${data.macros.length > 1 ? 's' : ''}:\n${data.macros
|
||||
.map((m) => `• ${m.name}(${m.params})${m.is_table ? ' → table' : ''}`)
|
||||
.join('\n')}`}
|
||||
>
|
||||
<SquareFunction size={10} />
|
||||
<span class="text-3xs leading-none">×{data.macros.length}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.dbt}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300"
|
||||
title={`dbt project — materializes ${data.dbt.model_count} model${data.dbt.model_count === 1 ? '' : 's'}`}
|
||||
>
|
||||
<DbtIcon width={10} height={10} />
|
||||
<span class="text-3xs leading-none">×{data.dbt.model_count}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.runState}
|
||||
{@const rs = data.runState}
|
||||
<div
|
||||
class={twMerge(
|
||||
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
|
||||
rs.status === 'running'
|
||||
? 'bg-blue-100 dark:bg-blue-900/40 text-blue-700 dark:text-blue-300'
|
||||
: rs.status === 'success'
|
||||
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
|
||||
: 'bg-red-100 dark:bg-red-900/40 text-red-700 dark:text-red-300'
|
||||
)}
|
||||
title={`${
|
||||
rs.status === 'running'
|
||||
? 'Running now'
|
||||
: rs.status === 'success'
|
||||
? 'Last run succeeded'
|
||||
: 'Last run failed'
|
||||
}${rs.runs > 0 ? ` — ran ${rs.runs}× this session` : ''}`}
|
||||
>
|
||||
{#if rs.status === 'running'}
|
||||
<Loader2 size={10} class="animate-spin" />
|
||||
{:else if rs.status === 'success'}
|
||||
<CheckCircle2 size={10} />
|
||||
<NodeActionsMenu items={menuItems} {hover}>
|
||||
<PipelineNodeCard
|
||||
surface="primary"
|
||||
kindLabel="{kindLabel}{data.unsaved ? ' · draft' : ''}"
|
||||
title={data.summary || data.path}
|
||||
tooltip={data.error ? `${data.path}: ${data.error}` : nodeTooltip}
|
||||
{selected}
|
||||
draft={data.unsaved}
|
||||
tone={computingNow ? 'running' : data.error ? 'error' : undefined}
|
||||
>
|
||||
{#snippet icon()}
|
||||
{#if data.language && data.runnable_kind === 'script'}
|
||||
<LanguageIcon lang={data.language} width={14} height={14} />
|
||||
{:else}
|
||||
<XCircle size={10} />
|
||||
<Icon size={14} class={selected ? 'text-accent' : 'text-secondary'} />
|
||||
{/if}
|
||||
{#if rs.runs > 0}
|
||||
<span class="text-3xs leading-none tabular-nums">×{rs.runs}</span>
|
||||
{/snippet}
|
||||
{#snippet trailing()}
|
||||
<!-- Annotation chips share one neutral treatment — the icon carries
|
||||
the meaning (colors are reserved for feedback, per the brand
|
||||
guidelines). Only the freshness chip (when it has a verdict)
|
||||
and the run-state chip below use semantic colors. -->
|
||||
{#if data.partition_kind}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// partitioned ${data.partition_kind}`}
|
||||
>
|
||||
<Layers size={10} />
|
||||
<span class="text-3xs leading-none">{data.partition_kind}</span>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
<!-- Freshness is the one annotation chip that carries feedback (a
|
||||
fresh/stale verdict against real run history), so like the
|
||||
run-state chip it uses semantic colors: emerald = within window,
|
||||
amber = stale. Neutral when there's no verdict (drafts, bad
|
||||
window value). -->
|
||||
{#if data.freshness}
|
||||
<div
|
||||
class={twMerge(
|
||||
'shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm',
|
||||
freshnessState === 'fresh'
|
||||
? 'bg-emerald-100 dark:bg-emerald-900/40 text-emerald-700 dark:text-emerald-300'
|
||||
: freshnessState === 'stale'
|
||||
? 'bg-amber-100 dark:bg-amber-900/40 text-amber-700 dark:text-amber-300'
|
||||
: 'bg-surface-secondary text-secondary'
|
||||
)}
|
||||
title={freshnessTooltip}
|
||||
>
|
||||
<Timer size={10} />
|
||||
<span class="text-3xs leading-none">{data.freshness}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.tag}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// tag ${data.tag}`}
|
||||
>
|
||||
<Tag size={10} />
|
||||
<span class="text-3xs leading-none">{data.tag}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.retry}
|
||||
{@const r = data.retry}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// retry ${r.count}${r.delay ? ` ${r.delay}` : ''}`}
|
||||
>
|
||||
<RotateCw size={10} />
|
||||
<span class="text-3xs leading-none">×{r.count}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.macros && data.macros.length > 0}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-surface-secondary text-secondary"
|
||||
title={`// macros — defines ${data.macros.length} macro${data.macros.length > 1 ? 's' : ''}:\n${data.macros
|
||||
.map((m) => `• ${m.name}(${m.params})${m.is_table ? ' → table' : ''}`)
|
||||
.join('\n')}`}
|
||||
>
|
||||
<SquareFunction size={10} />
|
||||
<span class="text-3xs leading-none">×{data.macros.length}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.dbt}
|
||||
<div
|
||||
class="shrink-0 flex items-center gap-0.5 px-1 py-0.5 mr-1 rounded-sm bg-orange-100 dark:bg-orange-900/40 text-orange-700 dark:text-orange-300"
|
||||
title={`dbt project — materializes ${data.dbt.model_count} model${data.dbt.model_count === 1 ? '' : 's'}`}
|
||||
>
|
||||
<DbtIcon width={10} height={10} />
|
||||
<span class="text-3xs leading-none">×{data.dbt.model_count}</span>
|
||||
</div>
|
||||
{/if}
|
||||
{#if data.runState}
|
||||
<RunStateChip runState={data.runState} class="mr-1" />
|
||||
{/if}
|
||||
{/snippet}
|
||||
</PipelineNodeCard>
|
||||
</NodeActionsMenu>
|
||||
{#if showRun}
|
||||
<!-- Run button revealed on hover/select. Matches the placement, size,
|
||||
and behaviour of AssetNode's run button so both nodes feel
|
||||
@@ -463,38 +447,12 @@
|
||||
{/if}
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
{#if menuItems.length > 0}
|
||||
<!--
|
||||
Hover-revealed action menu, top-right of the node. Mirrors the
|
||||
FlowModuleSchemaItem pattern: position the trigger button just
|
||||
outside the node frame, only render it on hover (or while the
|
||||
menu is open) so the canvas isn't visually cluttered when idle.
|
||||
The pointerdown stop+preventDefault keeps xyflow from selecting
|
||||
the node when the user is reaching for the menu.
|
||||
-->
|
||||
<div class="absolute -top-2 -right-2 h-7 p-1 min-w-7" style="will-change: transform;">
|
||||
<DropdownV2
|
||||
items={menuItems}
|
||||
placement="bottom-end"
|
||||
bind:open={menuOpen}
|
||||
fixedHeight={false}
|
||||
usePointerDownOutside
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<button
|
||||
class={twMerge(
|
||||
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
|
||||
hover || menuOpen ? 'block' : '!hidden',
|
||||
'shadow-md rounded-md'
|
||||
)}
|
||||
onpointerdown={stopPropagation(preventDefault(() => {}))}
|
||||
title="Actions"
|
||||
>
|
||||
<EllipsisVertical size={12} />
|
||||
</button>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
{#if data.onStartOnDrag}
|
||||
<NodeOnDot onStart={data.onStartOnDrag} visible={hover} />
|
||||
{/if}
|
||||
{#if data.fix}
|
||||
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
|
||||
<NodeFixButton fix={data.fix} />
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -24,46 +24,36 @@
|
||||
type Presentation = {
|
||||
icon: ComponentType
|
||||
label: string
|
||||
// Tailwind class fragments for bg + border + text accent.
|
||||
bg: string
|
||||
border: string
|
||||
borderUnsaved: string
|
||||
iconText: string
|
||||
}
|
||||
|
||||
// One muted treatment for every source kind — colors are meaningful, not
|
||||
// One muted icon color for every source kind — colors are meaningful, not
|
||||
// decorative (brand guidelines), so the kind is carried by the icon and
|
||||
// label while bg/border stay on the surface/gray scale like the flow
|
||||
// editor's nodes. Red stays reserved for the missing-trigger state.
|
||||
const MUTED = {
|
||||
bg: 'bg-surface-secondary',
|
||||
border: 'outline-gray-400 dark:outline-gray-600',
|
||||
borderUnsaved: 'outline-dashed outline-gray-400 dark:outline-gray-500',
|
||||
iconText: 'text-secondary'
|
||||
}
|
||||
// label. Red stays reserved for the missing-trigger state.
|
||||
const MUTED = { iconText: 'text-secondary' }
|
||||
|
||||
export const TRIGGER_NODE_STYLE: Record<TriggerNodeKind, Presentation> = {
|
||||
schedule: { icon: Clock, label: 'schedule', ...MUTED },
|
||||
webhook: { icon: Webhook, label: 'webhook', ...MUTED },
|
||||
email: { icon: Mail, label: 'email', ...MUTED },
|
||||
kafka: { icon: Zap, label: 'kafka', ...MUTED },
|
||||
mqtt: { icon: Radio, label: 'mqtt', ...MUTED },
|
||||
amqp: { icon: Radio, label: 'amqp', ...MUTED },
|
||||
nats: { icon: MessageSquare, label: 'nats', ...MUTED },
|
||||
postgres: { icon: Database, label: 'postgres', ...MUTED },
|
||||
sqs: { icon: Send, label: 'sqs', ...MUTED },
|
||||
gcp: { icon: CloudCog, label: 'gcp', ...MUTED },
|
||||
data_upload: { icon: Upload, label: 'data upload', ...MUTED }
|
||||
schedule: { icon: Clock, label: 'Schedule', ...MUTED },
|
||||
webhook: { icon: Webhook, label: 'Webhook', ...MUTED },
|
||||
email: { icon: Mail, label: 'Email', ...MUTED },
|
||||
kafka: { icon: Zap, label: 'Kafka', ...MUTED },
|
||||
mqtt: { icon: Radio, label: 'MQTT', ...MUTED },
|
||||
amqp: { icon: Radio, label: 'AMQP', ...MUTED },
|
||||
nats: { icon: MessageSquare, label: 'NATS', ...MUTED },
|
||||
postgres: { icon: Database, label: 'Postgres', ...MUTED },
|
||||
sqs: { icon: Send, label: 'SQS', ...MUTED },
|
||||
gcp: { icon: CloudCog, label: 'GCP Pub/Sub', ...MUTED },
|
||||
data_upload: { icon: Upload, label: 'Data upload', ...MUTED }
|
||||
}
|
||||
</script>
|
||||
|
||||
<script lang="ts">
|
||||
import { Handle, Position } from '@xyflow/svelte'
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import { AlertTriangle, CheckCircle2, EllipsisVertical, Target, Trash2 } from 'lucide-svelte'
|
||||
import DropdownV2 from '$lib/components/DropdownV2.svelte'
|
||||
import { stopPropagation, preventDefault } from 'svelte/legacy'
|
||||
import PipelineNodeCard from './PipelineNodeCard.svelte'
|
||||
import NodeActionsMenu from './NodeActionsMenu.svelte'
|
||||
import NodeFixButton, { type NodeFix } from './NodeFixButton.svelte'
|
||||
import { describeSchedule } from '$lib/utils/describeCron'
|
||||
import { AlertTriangle, CheckCircle2, Target, Trash2 } from 'lucide-svelte'
|
||||
import type { Item } from '$lib/utils'
|
||||
|
||||
interface Props {
|
||||
@@ -74,16 +64,22 @@
|
||||
// owning script is in `runnable_path` (used by the title and the
|
||||
// "+ Create trigger" drawer hook passed by the page).
|
||||
data: {
|
||||
/** What is wrong with the node, and how to fix it: shows the Fix pill. */
|
||||
fix?: NodeFix
|
||||
kind: TriggerNodeKind
|
||||
ref: string
|
||||
unsaved?: boolean
|
||||
missing?: boolean
|
||||
// A pipeline-local schedule draft: `ref` is the path it deploys to.
|
||||
// Edit/delete go through the same callbacks, which the page routes
|
||||
// to the draft instead of a trigger row.
|
||||
draft?: boolean
|
||||
// The cron, for a schedule: the node names its cadence instead of "Schedule".
|
||||
schedule?: string
|
||||
timezone?: string
|
||||
// Shown as the title over the trigger's path, when the trigger has one.
|
||||
summary?: string
|
||||
runnable_path?: string
|
||||
// True iff the target script is still a draft (no DB row yet).
|
||||
// Drives the "(after draft save)" hint on the missing
|
||||
// placeholder, since the page-level handler refuses to open the
|
||||
// create drawer until the script is deployed.
|
||||
runnable_unsaved?: boolean
|
||||
// Page-supplied dispatcher that opens the matching native
|
||||
// trigger drawer with `script_path` pre-filled. When absent
|
||||
// (e.g. schedule, or a kind without an editor) the placeholder is
|
||||
@@ -128,7 +124,6 @@
|
||||
let { data }: Props = $props()
|
||||
|
||||
let hover = $state(false)
|
||||
let menuOpen = $state(false)
|
||||
|
||||
let style = $derived(TRIGGER_NODE_STYLE[data.kind])
|
||||
// Webhooks are never genuinely "missing" — every deployed runnable has an
|
||||
@@ -143,7 +138,7 @@
|
||||
let Icon = $derived(displayMissing ? AlertTriangle : style.icon)
|
||||
let missingTitle = $derived(
|
||||
displayMissing
|
||||
? `Missing ${style.label} trigger: ${data.runnable_path ?? ''} declares \`// on ${style.label}\` but no ${style.label} trigger targets it. Click to create one, or remove the annotation.`
|
||||
? `Missing ${style.label} trigger: ${data.runnable_path ?? ''} declares \`// on ${data.kind}\` but no ${style.label} trigger targets it. Click to create one, or remove the annotation.`
|
||||
: undefined
|
||||
)
|
||||
// Webhook gets a drawer (URLs + webhook-specific token creation) rather
|
||||
@@ -200,7 +195,7 @@
|
||||
...(canDelete
|
||||
? [
|
||||
{
|
||||
displayName: 'Delete…',
|
||||
displayName: data.draft ? 'Discard draft' : 'Delete…',
|
||||
icon: Trash2,
|
||||
type: 'delete' as const,
|
||||
action: () => {
|
||||
@@ -212,6 +207,75 @@
|
||||
: [])
|
||||
])
|
||||
|
||||
// A schedule names its cadence ("Every day at 4:00", or the cron itself
|
||||
// when it has no plain reading); every other kind names its kind.
|
||||
let kindName = $derived(
|
||||
data.kind === 'schedule' && data.schedule
|
||||
? (describeSchedule(data.schedule, data.timezone) ?? `Schedule (${data.schedule})`)
|
||||
: style.label
|
||||
)
|
||||
|
||||
// One card for every state: which label, title and click the node gets.
|
||||
let card = $derived.by(
|
||||
(): {
|
||||
kindLabel: string
|
||||
title: string
|
||||
tooltip?: string
|
||||
draft: boolean
|
||||
tone?: 'danger' | 'success'
|
||||
onclick?: () => void
|
||||
} => {
|
||||
const unsaved = !!data.unsaved
|
||||
if (canCreate)
|
||||
return {
|
||||
kindLabel: `${style.label} · missing`,
|
||||
title: 'Click to create',
|
||||
tooltip: missingTitle,
|
||||
draft: false,
|
||||
tone: 'danger',
|
||||
onclick: handleMissingClick
|
||||
}
|
||||
if (canEdit)
|
||||
return {
|
||||
kindLabel: `${kindName}${data.draft ? ' · draft' : unsaved ? ' · unsaved' : ''}`,
|
||||
title: data.summary || data.ref,
|
||||
tooltip: data.draft
|
||||
? `Draft ${style.label}: ${data.ref} — created when you save the pipeline. Click to edit.`
|
||||
: `Edit ${style.label} trigger: ${data.ref}`,
|
||||
draft: unsaved,
|
||||
onclick: handleEditClick
|
||||
}
|
||||
if (canOpenWebhook)
|
||||
return {
|
||||
kindLabel: style.label,
|
||||
title: 'URLs & token',
|
||||
tooltip: `Webhook endpoint for ${data.runnable_path ?? ''} — click to view URLs and create a token`,
|
||||
draft: unsaved,
|
||||
onclick: handleWebhookClick
|
||||
}
|
||||
if (canOpenDataUpload)
|
||||
return {
|
||||
kindLabel: `${style.label}${dataUploadReady ? ' · ready' : ''}`,
|
||||
title: dataUploadReady ? 'File staged' : 'Upload & run',
|
||||
tooltip: dataUploadReady
|
||||
? `Data upload for ${data.runnable_path ?? ''} — a file is staged; the pipeline is ready to run. Click to change it.`
|
||||
: `Data upload for ${data.runnable_path ?? ''} — click to open the run form and upload a file`,
|
||||
draft: !dataUploadReady && unsaved,
|
||||
tone: dataUploadReady ? 'success' : undefined,
|
||||
onclick: handleDataUploadClick
|
||||
}
|
||||
return {
|
||||
kindLabel: `${displayMissing ? style.label : kindName}${displayMissing ? ' · missing' : unsaved ? ' · unsaved' : ''}`,
|
||||
title: displayMissing ? 'no trigger row' : data.summary || data.ref,
|
||||
tooltip:
|
||||
missingTitle ??
|
||||
(unsaved ? `Unsaved ${style.label}: ${data.ref}` : `${style.label}: ${data.ref}`),
|
||||
draft: unsaved && !displayMissing,
|
||||
tone: displayMissing ? 'danger' : undefined
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
function handleMissingClick() {
|
||||
if (!canCreate || !data.runnable_path || !data.onCreateMissingTrigger) return
|
||||
data.onCreateMissingTrigger(data.kind as NativeTriggerKind, data.runnable_path)
|
||||
@@ -235,195 +299,33 @@
|
||||
|
||||
<!-- svelte-ignore a11y_no_static_element_interactions -->
|
||||
<div class="relative" onmouseenter={() => (hover = true)} onmouseleave={() => (hover = false)}>
|
||||
{#if canCreate}
|
||||
<!-- Same affordance as the asset's downstream + button: render the
|
||||
whole node as a button so the cursor + hover + click is obvious
|
||||
without depending on svelte-flow's wrapper-level handlers. -->
|
||||
<button
|
||||
type="button"
|
||||
onclick={handleMissingClick}
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
|
||||
'bg-red-50 dark:bg-red-900/30 outline-dashed outline-red-400 dark:outline-red-500',
|
||||
'hover:bg-red-100 dark:hover:bg-red-900/40 transition-colors'
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={missingTitle}
|
||||
<NodeActionsMenu items={menuItems} {hover}>
|
||||
<PipelineNodeCard
|
||||
kindLabel={card.kindLabel}
|
||||
title={card.title}
|
||||
tooltip={card.tooltip}
|
||||
draft={card.draft}
|
||||
tone={card.tone}
|
||||
onclick={card.onclick}
|
||||
>
|
||||
<Icon size={14} class="shrink-0 ml-2 mr-2 text-red-600 dark:text-red-400" />
|
||||
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
|
||||
<span class="text-3xs uppercase tracking-wide truncate text-red-700 dark:text-red-400">
|
||||
{style.label} · missing
|
||||
</span>
|
||||
<span class="text-2xs font-mono truncate text-red-700 dark:text-red-400">
|
||||
{data.runnable_unsaved ? 'Click to create (after draft save)' : 'Click to create'}
|
||||
</span>
|
||||
</div>
|
||||
</button>
|
||||
{:else if canEdit}
|
||||
<!-- Mirrors the missing-trigger pattern: render the whole node as a
|
||||
button so clicks open the editor drawer reliably (don't rely on
|
||||
svelte-flow's onnodeclick wiring). -->
|
||||
<button
|
||||
type="button"
|
||||
onclick={handleEditClick}
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
|
||||
style.bg,
|
||||
data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
|
||||
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={`Edit ${style.label} trigger: ${data.ref}`}
|
||||
>
|
||||
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${style.iconText}`} />
|
||||
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
|
||||
<span class="text-3xs uppercase tracking-wide truncate text-tertiary">
|
||||
{style.label}{data.unsaved ? ' · unsaved' : ''}
|
||||
</span>
|
||||
<span class="text-2xs font-mono truncate text-emphasis">
|
||||
{data.ref}
|
||||
</span>
|
||||
</div>
|
||||
</button>
|
||||
{:else if canOpenWebhook}
|
||||
<!-- Webhook endpoint: implicit (no trigger row), so the node opens a
|
||||
drawer with the URLs + webhook-specific token creation instead of
|
||||
an editor. Styled like an attached trigger, never the red
|
||||
"missing" state. -->
|
||||
<button
|
||||
type="button"
|
||||
onclick={handleWebhookClick}
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
|
||||
style.bg,
|
||||
data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
|
||||
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={`Webhook endpoint for ${data.runnable_path ?? ''} — click to view URLs and create a token`}
|
||||
>
|
||||
<Icon size={14} class={`shrink-0 ml-2 mr-2 ${style.iconText}`} />
|
||||
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
|
||||
<span class="text-3xs uppercase tracking-wide truncate text-tertiary">
|
||||
{style.label}
|
||||
</span>
|
||||
<span class="text-2xs font-mono truncate text-emphasis"> URLs & token </span>
|
||||
</div>
|
||||
</button>
|
||||
{:else if canOpenDataUpload}
|
||||
<!-- Data upload: UI-first entry point (no trigger row). Clicking the
|
||||
node opens the target script's run form, where the auto-generated
|
||||
S3 picker lets the user upload a file and run the pipeline. Goes
|
||||
green once a file is staged (ready), so "Run pipeline" can proceed;
|
||||
until then it stays neutral with an "upload a file" prompt. Never
|
||||
the red "missing" state. -->
|
||||
<button
|
||||
type="button"
|
||||
onclick={handleDataUploadClick}
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1 w-full text-left',
|
||||
dataUploadReady
|
||||
? 'bg-green-50 dark:bg-green-900/30 outline-green-500 dark:outline-green-600'
|
||||
: style.bg,
|
||||
dataUploadReady ? '' : data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border,
|
||||
'hover:brightness-95 dark:hover:brightness-110 transition-[filter]'
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={dataUploadReady
|
||||
? `Data upload for ${data.runnable_path ?? ''} — a file is staged; the pipeline is ready to run. Click to change it.`
|
||||
: `Data upload for ${data.runnable_path ?? ''} — click to open the run form and upload a file`}
|
||||
>
|
||||
<DataUploadIcon
|
||||
size={14}
|
||||
class={`shrink-0 ml-2 mr-2 ${dataUploadReady ? 'text-green-600 dark:text-green-400' : style.iconText}`}
|
||||
/>
|
||||
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
|
||||
<span
|
||||
class={twMerge(
|
||||
'text-3xs uppercase tracking-wide truncate',
|
||||
dataUploadReady ? 'text-green-700 dark:text-green-400' : 'text-tertiary'
|
||||
)}
|
||||
>
|
||||
{style.label}{dataUploadReady ? ' · ready' : ''}
|
||||
</span>
|
||||
<span
|
||||
class={twMerge(
|
||||
'text-2xs font-mono truncate',
|
||||
dataUploadReady ? 'text-green-700 dark:text-green-400' : 'text-emphasis'
|
||||
)}
|
||||
>
|
||||
{dataUploadReady ? 'File staged' : 'Upload & run'}
|
||||
</span>
|
||||
</div>
|
||||
</button>
|
||||
{:else}
|
||||
<div
|
||||
class={twMerge(
|
||||
'flex items-center rounded-md drop-shadow-sm overflow-hidden outline outline-1',
|
||||
displayMissing
|
||||
? 'bg-red-50 dark:bg-red-900/30 outline-dashed outline-red-400 dark:outline-red-500'
|
||||
: style.bg,
|
||||
displayMissing ? '' : data.unsaved ? `opacity-80 ${style.borderUnsaved}` : style.border
|
||||
)}
|
||||
style="width: {NODE.width}px; min-height: {NODE.height}px;"
|
||||
title={missingTitle ??
|
||||
(data.unsaved ? `Unsaved ${style.label}: ${data.ref}` : `${style.label}: ${data.ref}`)}
|
||||
>
|
||||
<Icon
|
||||
size={14}
|
||||
class={`shrink-0 ml-2 mr-2 ${displayMissing ? 'text-red-600 dark:text-red-400' : style.iconText}`}
|
||||
/>
|
||||
<div class="flex flex-col min-w-0 flex-1 pr-2 py-0.5 leading-tight">
|
||||
<span
|
||||
class={twMerge(
|
||||
'text-3xs uppercase tracking-wide truncate',
|
||||
displayMissing ? 'text-red-700 dark:text-red-400' : 'text-tertiary'
|
||||
)}
|
||||
>
|
||||
{style.label}{displayMissing ? ' · missing' : data.unsaved ? ' · unsaved' : ''}
|
||||
</span>
|
||||
<span
|
||||
class={twMerge(
|
||||
'text-2xs font-mono truncate',
|
||||
displayMissing ? 'text-red-700 dark:text-red-400' : 'text-emphasis'
|
||||
)}
|
||||
>
|
||||
{displayMissing ? 'no trigger row' : data.ref}
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
{#if menuItems.length > 0}
|
||||
<!-- Hover-revealed kebab menu (Delete only for now). Mirrors the
|
||||
RunnableNode pattern: positioned just outside the top-right of
|
||||
the node, rendered only on hover or while the menu is open so
|
||||
the canvas stays clean at rest. `pointerdown` is stopped so
|
||||
svelte-flow doesn't kick off node selection / drag when the
|
||||
user reaches for the menu. -->
|
||||
<div class="absolute -top-2 -right-2 h-7 p-1 min-w-7" style="will-change: transform;">
|
||||
<DropdownV2
|
||||
items={menuItems}
|
||||
placement="bottom-end"
|
||||
bind:open={menuOpen}
|
||||
fixedHeight={false}
|
||||
usePointerDownOutside
|
||||
>
|
||||
{#snippet buttonReplacement()}
|
||||
<button
|
||||
class={twMerge(
|
||||
'center-center p-1 text-secondary shadow-sm bg-surface duration-0 hover:bg-surface-tertiary',
|
||||
hover || menuOpen ? 'block' : '!hidden',
|
||||
'shadow-md rounded-md'
|
||||
)}
|
||||
onpointerdown={stopPropagation(preventDefault(() => {}))}
|
||||
title="Actions"
|
||||
>
|
||||
<EllipsisVertical size={12} />
|
||||
</button>
|
||||
{/snippet}
|
||||
</DropdownV2>
|
||||
{#snippet icon()}
|
||||
{#if canOpenDataUpload}
|
||||
<DataUploadIcon
|
||||
size={14}
|
||||
class={dataUploadReady ? 'text-green-600 dark:text-green-400' : style.iconText}
|
||||
/>
|
||||
{:else}
|
||||
<Icon
|
||||
size={14}
|
||||
class={displayMissing || canCreate ? 'text-red-600 dark:text-red-400' : style.iconText}
|
||||
/>
|
||||
{/if}
|
||||
{/snippet}
|
||||
</PipelineNodeCard>
|
||||
</NodeActionsMenu>
|
||||
{#if data.fix}
|
||||
<div class="absolute left-1/2 -bottom-3 -translate-x-1/2 z-10">
|
||||
<NodeFixButton fix={data.fix} />
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
@@ -147,4 +147,18 @@ describe('layoutAssetGraph (tidy-tree with join breaks)', () => {
|
||||
expect(pos.get(id)!.y).toBeGreaterThan(anchor.y)
|
||||
}
|
||||
})
|
||||
|
||||
it('packs narrow nodes by their own width', () => {
|
||||
const pos = layoutAssetGraph({
|
||||
nodes: [
|
||||
{ ...n('p'), width: 100 },
|
||||
{ ...n('l'), width: 100 },
|
||||
{ ...n('r'), width: 180 }
|
||||
],
|
||||
edges: [e('p', 'l'), e('p', 'r')]
|
||||
})
|
||||
const gap = pos.get('r')!.x - 90 - (pos.get('l')!.x + 50)
|
||||
expect(gap).toBe(NODE.gap.horizontal)
|
||||
expect(pos.get('l')!.x).toBe(50)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -7,7 +7,15 @@ import { NODE } from '$lib/components/graph/util'
|
||||
// breathing room.
|
||||
const NODE_WIDTH = NODE.width
|
||||
const NODE_HEIGHT = NODE.height + 30
|
||||
const LAYER_GAP = NODE.gap.vertical
|
||||
/** Rendered height of a pipeline node: a kind label over a title. Fits inside
|
||||
* the `NODE_HEIGHT` row the layout reserves. */
|
||||
export const PIPELINE_NODE_HEIGHT = 48
|
||||
// A step more than the flow editor's gap: the + on a node's bottom edge sits in it.
|
||||
const LAYER_GAP = NODE.gap.vertical + 16
|
||||
/** Height of the "Add data source" pill the anchor draws, and its gap to the
|
||||
* graph: tighter than LAYER_GAP, since no edge runs through it. */
|
||||
const ANCHOR_HEIGHT = 32
|
||||
const ANCHOR_GAP = 40
|
||||
const SIBLING_GAP = NODE.gap.horizontal
|
||||
// Horizontal gutter between two disjoint subgraphs. Wider than the
|
||||
// within-subgraph sibling gap so visually-unrelated components (e.g. two
|
||||
@@ -15,10 +23,16 @@ const SIBLING_GAP = NODE.gap.horizontal
|
||||
// into each other's columns — but only ~2×, so they don't drift far apart.
|
||||
const COMPONENT_GAP = SIBLING_GAP * 2
|
||||
|
||||
const LAYER_H = NODE_HEIGHT + LAYER_GAP
|
||||
/** Extra height of an assets-only node: the trigger chip under its title. */
|
||||
export const PIPELINE_NODE_EXTRA_ROW = 20
|
||||
/** Height of the header strip naming an assets-only node's producing script. */
|
||||
export const PIPELINE_NODE_HEADER = 22
|
||||
/** Extra vertical padding of a card under a header, top plus bottom. */
|
||||
export const PIPELINE_NODE_HEADER_CARD_PAD = 12
|
||||
|
||||
interface GraphInput {
|
||||
nodes: Array<{ id: string; data: AssetGraphNodeData }>
|
||||
/** `width` defaults to `NODE.width`. */
|
||||
nodes: Array<{ id: string; data: AssetGraphNodeData; width?: number }>
|
||||
edges: Array<{ source: string; target: string }>
|
||||
}
|
||||
|
||||
@@ -52,15 +66,18 @@ interface Band {
|
||||
// intersects theirs.
|
||||
//
|
||||
// y comes from longest-path layering (same top-down orientation as before:
|
||||
// producers above, assets in the middle, consumers below). Returns positions
|
||||
// (band centers) normalized so the component's min x,y = 0. Cyclic input is
|
||||
// handled by dropping feedback edges (see the Kahn step below).
|
||||
// producers above, assets in the middle, consumers below). Returns node
|
||||
// centers, normalized so the component's leftmost edge and top are at 0.
|
||||
// Cyclic input is handled by dropping feedback edges (see the Kahn step below).
|
||||
function layoutComponent(
|
||||
nodes: GraphInput['nodes'],
|
||||
edges: GraphInput['edges']
|
||||
edges: GraphInput['edges'],
|
||||
rowH: number
|
||||
): Map<string, Positioned> {
|
||||
const layerH = rowH + LAYER_GAP
|
||||
const out = new Map<string, Positioned>()
|
||||
const ids = new Set(nodes.map((n) => n.id))
|
||||
const nodeW = new Map(nodes.map((n) => [n.id, n.width ?? NODE_WIDTH]))
|
||||
|
||||
const parents = new Map<string, string[]>()
|
||||
const children = new Map<string, string[]>()
|
||||
@@ -137,7 +154,7 @@ function layoutComponent(
|
||||
const kids = treeChildren.get(id)!
|
||||
const kidsW =
|
||||
kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * Math.max(0, kids.length - 1)
|
||||
W.set(id, Math.max(NODE_WIDTH, kidsW))
|
||||
W.set(id, Math.max(nodeW.get(id)!, kidsW))
|
||||
}
|
||||
|
||||
// Vertical (pixel) extent of a subtree, for band collision checks.
|
||||
@@ -152,14 +169,14 @@ function layoutComponent(
|
||||
if (l > hi) hi = l
|
||||
for (const k of treeChildren.get(cur)!) stack.push(k)
|
||||
}
|
||||
return { top: lo * LAYER_H, bottom: hi * LAYER_H + NODE_HEIGHT }
|
||||
return { top: lo * layerH, bottom: hi * layerH + rowH }
|
||||
}
|
||||
|
||||
// Recursive placement: node centered over its band, children packed
|
||||
// side-by-side and centered within it.
|
||||
function placeTree(id: string, left: number) {
|
||||
const w = W.get(id)!
|
||||
out.set(id, { x: left + w / 2, y: layer.get(id)! * LAYER_H })
|
||||
out.set(id, { x: left + w / 2, y: layer.get(id)! * layerH })
|
||||
const kids = treeChildren.get(id)!
|
||||
if (kids.length === 0) return
|
||||
const kidsW = kids.reduce((acc, k) => acc + W.get(k)!, 0) + SIBLING_GAP * (kids.length - 1)
|
||||
@@ -211,11 +228,11 @@ function layoutComponent(
|
||||
placeAndRecord(id, center - w / 2)
|
||||
}
|
||||
|
||||
// Normalize so the component's min x,y = 0.
|
||||
// Normalize so the component's leftmost edge and top are at 0.
|
||||
let minX = Infinity
|
||||
let minY = Infinity
|
||||
for (const p of out.values()) {
|
||||
if (p.x < minX) minX = p.x
|
||||
for (const [id, p] of out) {
|
||||
if (p.x - nodeW.get(id)! / 2 < minX) minX = p.x - nodeW.get(id)! / 2
|
||||
if (p.y < minY) minY = p.y
|
||||
}
|
||||
if (isFinite(minX) && isFinite(minY)) {
|
||||
@@ -243,7 +260,13 @@ function layoutComponent(
|
||||
//
|
||||
// Falls back to a stable grid if the component layout throws (defensive —
|
||||
// cycles are already absorbed by feedback-edge dropping in layoutComponent).
|
||||
export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<string, Positioned> {
|
||||
export function layoutAssetGraph(
|
||||
graph: GraphInput,
|
||||
anchorId?: string,
|
||||
extraRowHeight = 0
|
||||
): Map<string, Positioned> {
|
||||
const rowH = NODE_HEIGHT + extraRowHeight
|
||||
const layerH = rowH + LAYER_GAP
|
||||
const byId = new Map<string, Positioned>()
|
||||
if (graph.nodes.length === 0) return byId
|
||||
|
||||
@@ -297,19 +320,20 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
|
||||
// 3. Lay out each component and pack side-by-side. xOffset advances by
|
||||
// the laid-out width of each component (max node center + a node width,
|
||||
// since positions are node centers) plus the gutter.
|
||||
const widthOf = new Map(nodes.map((n) => [n.id, n.width ?? NODE_WIDTH]))
|
||||
let xOffset = 0
|
||||
for (let c = 0; c < nComp; c++) {
|
||||
const positions = layoutComponent(compNodes[c], compEdges[c])
|
||||
let maxX = 0
|
||||
for (const p of positions.values()) if (p.x > maxX) maxX = p.x
|
||||
const positions = layoutComponent(compNodes[c], compEdges[c], rowH)
|
||||
let maxRight = 0
|
||||
for (const [id, p] of positions) {
|
||||
maxRight = Math.max(maxRight, p.x + widthOf.get(id)! / 2)
|
||||
byId.set(id, { x: p.x + xOffset, y: p.y })
|
||||
}
|
||||
xOffset += maxX + NODE_WIDTH + COMPONENT_GAP
|
||||
xOffset += maxRight + COMPONENT_GAP
|
||||
}
|
||||
|
||||
// 4. Re-place the anchor centered horizontally over the whole packed
|
||||
// graph, one layer above it; then renormalize so the anchor sits at the
|
||||
// graph, just above it; then renormalize so the anchor sits at the
|
||||
// top (y = 0), pushing the components down a layer to make room (mirrors
|
||||
// the previous "anchor is the parent of every root" behaviour).
|
||||
if (anchorId && graph.nodes.some((n) => n.id === anchorId)) {
|
||||
@@ -324,7 +348,9 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
|
||||
if (p.x > maxX) maxX = p.x
|
||||
if (p.y < minY) minY = p.y
|
||||
}
|
||||
byId.set(anchorId, { x: (minX + maxX) / 2, y: minY - LAYER_H })
|
||||
// One gap above the graph, sized to the pill rather than a full node row:
|
||||
// a row tall enough for an assets-only card leaves it floating high.
|
||||
byId.set(anchorId, { x: (minX + maxX) / 2, y: minY - ANCHOR_HEIGHT - ANCHOR_GAP })
|
||||
let nMinY = Infinity
|
||||
for (const p of byId.values()) if (p.y < nMinY) nMinY = p.y
|
||||
for (const p of byId.values()) p.y -= nMinY
|
||||
@@ -336,7 +362,7 @@ export function layoutAssetGraph(graph: GraphInput, anchorId?: string): Map<stri
|
||||
graph.nodes.forEach((n, i) => {
|
||||
byId.set(n.id, {
|
||||
x: (i % cols) * (NODE_WIDTH + SIBLING_GAP),
|
||||
y: Math.floor(i / cols) * LAYER_H
|
||||
y: Math.floor(i / cols) * layerH
|
||||
})
|
||||
})
|
||||
return byId
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import { NODE } from '$lib/components/graph/util'
|
||||
|
||||
const MIN_WIDTH = 160
|
||||
|
||||
let ctx: CanvasRenderingContext2D | null | undefined
|
||||
|
||||
/** Rendered width of `text` at `rem` in the page font, or a rough guess without a DOM. */
|
||||
function textWidth(text: string, rem: number): number {
|
||||
if (typeof document === 'undefined') return text.length * rem * 16 * 0.6
|
||||
ctx ??= document.createElement('canvas').getContext('2d')
|
||||
const root = parseFloat(getComputedStyle(document.documentElement).fontSize) || 16
|
||||
if (!ctx) return text.length * rem * root * 0.6
|
||||
const body = getComputedStyle(document.body)
|
||||
ctx.font = `${rem * root}px ${body.fontFamily}`
|
||||
const spacing = parseFloat(body.letterSpacing) || 0
|
||||
return ctx.measureText(text).width + spacing * text.length
|
||||
}
|
||||
|
||||
/**
|
||||
* Width of an assets-only asset card, from the pieces `PipelineNodeCard` and
|
||||
* `AssetNode` lay out: icon and the widest of kind / title / trigger chip, or the
|
||||
* header naming the producing script, if wider. The numbers mirror their Tailwind spacing; change one
|
||||
* and the other drifts. Clamped to the regular node width, past which the title
|
||||
* truncates as usual.
|
||||
*/
|
||||
export function assetsOnlyNodeWidth(n: {
|
||||
kind: string
|
||||
title: string
|
||||
chip: string
|
||||
/** The chip leads with an icon (the folded-errors Fix chip). */
|
||||
chipIcon?: boolean
|
||||
/** The chip ends in the edit icon (an editable trigger). */
|
||||
chipEdit?: boolean
|
||||
/** The header tab: its text (a script's summary or path, "N scripts"), icon
|
||||
* count, and whether it ends in the menu chevron (else the edit icon) or has a "draft" mark;
|
||||
* undefined without a header. */
|
||||
header?: { label: string; icons: number; chevron?: boolean; draft?: boolean }
|
||||
}): number {
|
||||
const icon = 12 + 14 + 10
|
||||
const text = Math.max(
|
||||
textWidth(n.kind, 0.7),
|
||||
textWidth(n.title, 0.75),
|
||||
textWidth(n.chip, 0.65) + 12 + 2 + (n.chipIcon ? 14 : 0) + (n.chipEdit ? 4 + 9 : 0)
|
||||
)
|
||||
const body = Math.ceil(icon + text + 4 + 8 + 2 + 8)
|
||||
// Header tab: mx-2 and a 1px border around px-2, then gap-1.5 between its
|
||||
// items: the icons, the label, a "draft" mark, and the chevron or edit icon.
|
||||
// The run chip is left out: it comes and goes with runs, and a width that
|
||||
// followed it would re-lay the graph on a click; the label truncates instead.
|
||||
const h = n.header
|
||||
const header = h
|
||||
? Math.ceil(
|
||||
16 +
|
||||
2 +
|
||||
16 +
|
||||
h.icons * (11 + 6) +
|
||||
textWidth(h.label, 0.65) +
|
||||
(h.draft ? 6 + textWidth('draft', 0.65) : 0) +
|
||||
// The menu chevron, or the edit icon of a single script.
|
||||
6 +
|
||||
(h.chevron ? 11 : 10) +
|
||||
4
|
||||
)
|
||||
: 0
|
||||
return Math.min(NODE.width, Math.max(MIN_WIDTH, body, header))
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
// Pipelines open in the assets-only view. This remembers the ones last left
|
||||
// showing their scripts and triggers, as `<workspace>/<folder>`, so reopening one
|
||||
// shows it the same way. Browser-local: a missing or unreadable list reads as
|
||||
// "none", and storage that throws (private mode, blocked site data) is ignored.
|
||||
const KEY = 'pipeline-full-view-folders'
|
||||
|
||||
function read(): string[] {
|
||||
try {
|
||||
const parsed = JSON.parse(localStorage.getItem(KEY) ?? '[]')
|
||||
return Array.isArray(parsed) ? parsed.filter((f) => typeof f === 'string') : []
|
||||
} catch {
|
||||
return []
|
||||
}
|
||||
}
|
||||
|
||||
export function isAssetsOnlyFolder(folder: string): boolean {
|
||||
return !read().includes(folder)
|
||||
}
|
||||
|
||||
export function setAssetsOnlyFolder(folder: string, on: boolean): void {
|
||||
const rest = read().filter((f) => f !== folder)
|
||||
try {
|
||||
localStorage.setItem(KEY, JSON.stringify(on ? rest : [...rest, folder]))
|
||||
} catch {}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
assetsOnlyView,
|
||||
upstreamDeletion,
|
||||
noAssetNodeId,
|
||||
withoutPassiveReads,
|
||||
type AssetUpstream
|
||||
} from './assetsOnlyView'
|
||||
|
||||
const node = (id: string, type: string, data: any = {}) => ({ id, type, data })
|
||||
const edge = (source: string, target: string, kind: string) => ({
|
||||
id: `${kind}:${source}->${target}`,
|
||||
source,
|
||||
target,
|
||||
kind
|
||||
})
|
||||
|
||||
describe('assetsOnlyView', () => {
|
||||
const nodes = [
|
||||
node('asset:ducklake:raw', 'asset'),
|
||||
node('asset:ducklake:clean', 'asset'),
|
||||
node('asset:ducklake:report', 'asset'),
|
||||
node('script:f/p/clean', 'runnable'),
|
||||
node('script:f/p/report', 'runnable'),
|
||||
node('script:f/p/other', 'runnable'),
|
||||
node('trigger:schedule:f/p/daily', 'trigger', { kind: 'schedule', schedule: '0 0 4 * * *' })
|
||||
]
|
||||
const edges = [
|
||||
edge('trigger:schedule:f/p/daily', 'script:f/p/clean', 'trigger-native'),
|
||||
edge('asset:ducklake:raw', 'script:f/p/clean', 'lineage-read'),
|
||||
edge('script:f/p/clean', 'asset:ducklake:clean', 'lineage-write'),
|
||||
// read-modify-write: no self edge
|
||||
edge('asset:ducklake:clean', 'script:f/p/clean', 'lineage-read'),
|
||||
edge('asset:ducklake:clean', 'script:f/p/report', 'trigger-asset'),
|
||||
edge('asset:ducklake:clean', 'script:f/p/report', 'lineage-read'),
|
||||
edge('script:f/p/report', 'asset:ducklake:report', 'lineage-write'),
|
||||
edge('script:f/p/other', 'asset:ducklake:report', 'lineage-write')
|
||||
]
|
||||
const v = assetsOnlyView(nodes, edges)
|
||||
|
||||
it('joins each script input to its outputs, once', () => {
|
||||
expect(v.edges.map((e) => `${e.source} -> ${e.target}`).sort()).toEqual([
|
||||
'asset:ducklake:clean -> asset:ducklake:report',
|
||||
'asset:ducklake:raw -> asset:ducklake:clean'
|
||||
])
|
||||
expect([...v.nodeIds].sort()).toEqual([
|
||||
'asset:ducklake:clean',
|
||||
'asset:ducklake:raw',
|
||||
'asset:ducklake:report'
|
||||
])
|
||||
})
|
||||
|
||||
it('names the single producer and its trigger, or marks the asset ambiguous', () => {
|
||||
expect(v.upstream.get('asset:ducklake:raw')).toBeUndefined()
|
||||
expect(v.upstream.get('asset:ducklake:clean')).toMatchObject({
|
||||
multiple: false,
|
||||
runnableId: 'script:f/p/clean',
|
||||
trigger: { kind: 'schedule', data: { schedule: '0 0 4 * * *' } }
|
||||
})
|
||||
expect(v.upstream.get('asset:ducklake:report')).toEqual({
|
||||
multiple: true,
|
||||
runnableIds: ['script:f/p/report', 'script:f/p/other']
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('the No asset nodes', () => {
|
||||
it('give each script that builds nothing its own node, fed by what it reads', () => {
|
||||
const v = assetsOnlyView(
|
||||
[
|
||||
node('asset:ducklake:a', 'asset'),
|
||||
node('asset:ducklake:b', 'asset'),
|
||||
node('script:f/p/on_a', 'runnable'),
|
||||
node('script:f/p/on_b', 'runnable'),
|
||||
node('script:f/p/dbt', 'runnable', { dbt: { model_count: 3 } }),
|
||||
node('script:f/p/macros', 'runnable', { macros: [{ name: 'm' }] })
|
||||
],
|
||||
[
|
||||
edge('asset:ducklake:a', 'script:f/p/on_a', 'trigger-asset'),
|
||||
edge('asset:ducklake:b', 'script:f/p/on_b', 'trigger-asset')
|
||||
]
|
||||
)
|
||||
const onA = noAssetNodeId('script:f/p/on_a')
|
||||
const onB = noAssetNodeId('script:f/p/on_b')
|
||||
expect(v.noAssetNodeIds).toEqual([onA, onB])
|
||||
expect(v.upstream.get(onA)).toMatchObject({ runnableId: 'script:f/p/on_a' })
|
||||
expect(v.edges.map((e) => `${e.source} -> ${e.target}`)).toEqual([
|
||||
`asset:ducklake:a -> ${onA}`,
|
||||
`asset:ducklake:b -> ${onB}`
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('upstreamDeletion', () => {
|
||||
const script = { runnable_kind: 'script', path: 'f/p/a' }
|
||||
const schedule = (targets: string[]) => ({
|
||||
kind: 'schedule',
|
||||
nodeId: 'trigger:schedule:f/p/a_schedule',
|
||||
data: { ref: 'f/p/a_schedule', runnable_paths: targets }
|
||||
})
|
||||
const own = (trigger?: any): AssetUpstream & { multiple: false } => ({
|
||||
multiple: false,
|
||||
runnableId: 'script:f/p/a',
|
||||
trigger
|
||||
})
|
||||
|
||||
it('deletes the script and a trigger that fires only it', () => {
|
||||
const u = own(schedule(['f/p/a']))
|
||||
expect(upstreamDeletion(u, script, new Map([['asset:x', u]]))).toEqual({
|
||||
script: { path: 'f/p/a', unsaved: false },
|
||||
trigger: { kind: 'schedule', path: 'f/p/a_schedule', draft: false }
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps a trigger that fires other scripts', () => {
|
||||
const u = own(schedule(['f/p/a', 'f/p/b']))
|
||||
expect(upstreamDeletion(u, script, new Map([['asset:x', u]]))?.trigger).toBeUndefined()
|
||||
})
|
||||
|
||||
it('deletes nothing when the script also co-produces another asset', () => {
|
||||
const u = own(schedule(['f/p/a']))
|
||||
const all = new Map<string, AssetUpstream>([
|
||||
['asset:x', u],
|
||||
['asset:y', { multiple: true, runnableIds: ['script:f/p/a', 'script:f/p/b'] }]
|
||||
])
|
||||
expect(upstreamDeletion(u, script, all)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('deletes nothing when the script builds another asset too', () => {
|
||||
const u = own(schedule(['f/p/a']))
|
||||
const all = new Map([
|
||||
['asset:x', u],
|
||||
['asset:y', u]
|
||||
])
|
||||
expect(upstreamDeletion(u, script, all)).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('withoutPassiveReads', () => {
|
||||
it('drops reads that trigger nothing, and the assets only they drew', () => {
|
||||
const nodes = [
|
||||
node('asset:src', 'asset'),
|
||||
node('asset:lookup', 'asset'),
|
||||
node('asset:out', 'asset'),
|
||||
node('script:a', 'runnable')
|
||||
]
|
||||
const edges = [
|
||||
edge('asset:src', 'script:a', 'lineage-read'),
|
||||
edge('asset:src', 'script:a', 'trigger-asset'),
|
||||
edge('asset:lookup', 'script:a', 'lineage-read'),
|
||||
edge('asset:out', 'script:a', 'lineage-read'),
|
||||
edge('script:a', 'asset:out', 'lineage-write')
|
||||
]
|
||||
const r = withoutPassiveReads(nodes, edges)
|
||||
expect(r.nodes.map((n) => n.id)).toEqual(['asset:src', 'asset:out', 'script:a'])
|
||||
expect(r.edges.map((e) => e.id)).toEqual([
|
||||
'lineage-read:asset:src->script:a',
|
||||
'trigger-asset:asset:src->script:a',
|
||||
'lineage-write:script:a->asset:out'
|
||||
])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,234 @@
|
||||
import type { NativeTriggerKind } from './types'
|
||||
|
||||
/** A node or edge of the canvas model, reduced to what the assets-only view reads. */
|
||||
export type ViewNode = { id: string; type: string; data: any }
|
||||
export type ViewEdge = {
|
||||
id: string
|
||||
source: string
|
||||
target: string
|
||||
kind: string
|
||||
unsaved?: boolean
|
||||
/** Assets-only edges: the folded runnables the data goes through. */
|
||||
via?: string[]
|
||||
/** Assets-only edges: a write to the source reruns one of `via`. */
|
||||
reactive?: boolean
|
||||
}
|
||||
|
||||
/** The assets-only node standing for a script that builds no asset: one per
|
||||
* script, so scripts reacting to different inputs stay apart. */
|
||||
export const noAssetNodeId = (runnableId: string) => `no-asset:${runnableId}`
|
||||
export const isNoAssetNode = (id: string) => id.startsWith('no-asset:')
|
||||
|
||||
/** What an assets-only delete removes: the script, its trigger, or both. */
|
||||
export type AssetUpstreamDelete = {
|
||||
script?: { path: string; unsaved: boolean }
|
||||
trigger?: { kind: NativeTriggerKind; path: string; draft: boolean }
|
||||
}
|
||||
|
||||
/** What an asset shows of the script and trigger that produce it, in the assets-only view. */
|
||||
export type AssetUpstream =
|
||||
/** Several producing scripts, or one with several triggers. */
|
||||
| { multiple: true; runnableIds: string[] }
|
||||
| {
|
||||
multiple: false
|
||||
/** Canvas id of the producing runnable (`script:<path>`). */
|
||||
runnableId: string
|
||||
/** The producer's trigger; undefined when nothing triggers it (run by hand). */
|
||||
trigger?: { nodeId?: string; kind: string; data?: any }
|
||||
}
|
||||
|
||||
/**
|
||||
* The asset → asset view of a pipeline graph: runnables and triggers disappear,
|
||||
* and every read (or asset trigger) of a script is joined to each of its writes,
|
||||
* so an asset points at the assets computed from it. Each produced asset keeps a
|
||||
* reference to its single producer and that producer's single trigger, or is
|
||||
* marked `multiple` when either is ambiguous.
|
||||
*/
|
||||
export function assetsOnlyView<N extends ViewNode, E extends ViewEdge>(
|
||||
nodes: N[],
|
||||
edges: E[]
|
||||
): {
|
||||
nodeIds: Set<string>
|
||||
edges: Array<ViewEdge>
|
||||
upstream: Map<string, AssetUpstream>
|
||||
/** A node per script that builds no asset (`noAssetNodeId`), its upstream
|
||||
* under that id. */
|
||||
noAssetNodeIds: string[]
|
||||
} {
|
||||
const nodeById = new Map(nodes.map((n) => [n.id, n]))
|
||||
const inputsOf = new Map<string, E[]>()
|
||||
const triggersOf = new Map<string, E[]>()
|
||||
const producersOf = new Map<string, E[]>()
|
||||
const push = (m: Map<string, E[]>, k: string, e: E) => m.set(k, [...(m.get(k) ?? []), e])
|
||||
for (const e of edges) {
|
||||
if (e.kind === 'lineage-write') push(producersOf, e.target, e)
|
||||
else if (e.kind === 'lineage-read' || e.kind === 'trigger-asset') {
|
||||
push(inputsOf, e.target, e)
|
||||
if (e.kind === 'trigger-asset') push(triggersOf, e.target, e)
|
||||
} else if (e.kind === 'trigger-native') push(triggersOf, e.target, e)
|
||||
}
|
||||
|
||||
const triggered = new Set(
|
||||
edges.filter((e) => e.kind === 'trigger-asset').map((e) => `${e.source}\n${e.target}`)
|
||||
)
|
||||
const out = new Map<string, ViewEdge>()
|
||||
for (const [assetId, writes] of producersOf) {
|
||||
for (const w of writes) {
|
||||
for (const r of inputsOf.get(w.source) ?? []) {
|
||||
// A read-modify-write of the same table is not a dependency on itself.
|
||||
if (r.source === assetId) continue
|
||||
const id = `flow:${r.source}->${assetId}`
|
||||
const prev = out.get(id)
|
||||
const unsaved = !!(w.unsaved || r.unsaved)
|
||||
const reactive = triggered.has(`${r.source}\n${w.source}`)
|
||||
if (prev) {
|
||||
prev.unsaved = prev.unsaved && unsaved
|
||||
prev.reactive ||= reactive
|
||||
if (!prev.via?.includes(w.source)) prev.via = [...(prev.via ?? []), w.source]
|
||||
} else {
|
||||
out.set(id, {
|
||||
id,
|
||||
source: r.source,
|
||||
target: assetId,
|
||||
kind: 'asset-flow',
|
||||
unsaved,
|
||||
reactive,
|
||||
via: [w.source]
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const e of edges) if (e.kind === 'dbt-ref') out.set(e.id, e)
|
||||
|
||||
function describe(runnableIds: string[]): AssetUpstream {
|
||||
if (runnableIds.length !== 1) return { multiple: true, runnableIds }
|
||||
const runnableId = runnableIds[0]
|
||||
// Every asset trigger of a script is one cause, "an input changed".
|
||||
const triggers = [
|
||||
...new Map(
|
||||
(triggersOf.get(runnableId) ?? []).map((t) => [
|
||||
t.kind === 'trigger-asset' ? 'asset' : t.source,
|
||||
t
|
||||
])
|
||||
).values()
|
||||
]
|
||||
if (triggers.length > 1) return { multiple: true, runnableIds }
|
||||
const t = triggers[0]
|
||||
const tNode = t && nodeById.get(t.source)
|
||||
return {
|
||||
multiple: false,
|
||||
runnableId,
|
||||
trigger: !t
|
||||
? undefined
|
||||
: t.kind === 'trigger-asset'
|
||||
? { kind: 'asset' }
|
||||
: { nodeId: t.source, kind: tNode?.data?.kind, data: tNode?.data }
|
||||
}
|
||||
}
|
||||
|
||||
const upstream = new Map<string, AssetUpstream>()
|
||||
for (const [assetId, writes] of producersOf) {
|
||||
upstream.set(assetId, describe([...new Set(writes.map((w) => w.source))]))
|
||||
}
|
||||
|
||||
// Scripts that build no asset have nowhere to fold into, so each gets a "No
|
||||
// asset" node, fed by what it reads. dbt projects (whose writes are not drawn)
|
||||
// and macro libraries (which build nothing by design) get none.
|
||||
const writers = new Set([...producersOf.values()].flat().map((w) => w.source))
|
||||
const orphans = nodes
|
||||
.filter(
|
||||
(n) => n.type === 'runnable' && !writers.has(n.id) && !n.data?.dbt && !n.data?.macros?.length
|
||||
)
|
||||
.map((n) => n.id)
|
||||
for (const id of orphans) {
|
||||
const nodeId = noAssetNodeId(id)
|
||||
upstream.set(nodeId, describe([id]))
|
||||
for (const r of inputsOf.get(id) ?? []) {
|
||||
const eid = `flow:${r.source}->${nodeId}`
|
||||
const reactive = triggered.has(`${r.source}\n${id}`)
|
||||
const prev = out.get(eid)
|
||||
if (prev) {
|
||||
prev.unsaved = prev.unsaved && !!r.unsaved
|
||||
prev.reactive ||= reactive
|
||||
} else {
|
||||
out.set(eid, {
|
||||
id: eid,
|
||||
source: r.source,
|
||||
target: nodeId,
|
||||
kind: 'asset-flow',
|
||||
unsaved: !!r.unsaved,
|
||||
reactive,
|
||||
via: [id]
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const nodeIds = new Set(nodes.filter((n) => n.type === 'asset').map((n) => n.id))
|
||||
return {
|
||||
nodeIds,
|
||||
edges: [...out.values()],
|
||||
upstream,
|
||||
noAssetNodeIds: orphans.map(noAssetNodeId)
|
||||
}
|
||||
}
|
||||
|
||||
/** What deleting an asset's upstream removes: the script unless another asset
|
||||
* is also built by it, and with it the trigger unless that fires other scripts.
|
||||
* Undefined when nothing can go. */
|
||||
export function upstreamDeletion(
|
||||
u: AssetUpstream & { multiple: false },
|
||||
r: { runnable_kind: string; path: string; unsaved?: boolean },
|
||||
all: Map<string, AssetUpstream>
|
||||
): AssetUpstreamDelete | undefined {
|
||||
if (r.runnable_kind !== 'script') return undefined
|
||||
const producesOthers =
|
||||
[...all.values()].filter((o) =>
|
||||
o.multiple ? o.runnableIds.includes(u.runnableId) : o.runnableId === u.runnableId
|
||||
).length > 1
|
||||
if (producesOthers) return undefined
|
||||
const t = u.trigger
|
||||
const kind = t?.kind as NativeTriggerKind | undefined
|
||||
const targets: string[] = t?.data?.runnable_paths ?? []
|
||||
const ownTrigger =
|
||||
kind &&
|
||||
kind !== 'webhook' &&
|
||||
kind !== 'data_upload' &&
|
||||
t?.data?.ref &&
|
||||
!t.data.missing &&
|
||||
targets.every((p) => p === r.path)
|
||||
return {
|
||||
script: { path: r.path, unsaved: !!r.unsaved },
|
||||
trigger: ownTrigger ? { kind, path: t.data.ref, draft: !!t.data.draft } : undefined
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The graph without its reads that start nothing: a read edge with no asset
|
||||
* trigger along it, and the assets only such reads drew.
|
||||
*/
|
||||
export function withoutPassiveReads<N extends ViewNode, E extends ViewEdge>(
|
||||
nodes: N[],
|
||||
edges: E[]
|
||||
): { nodes: N[]; edges: E[]; dropped: number } {
|
||||
const triggers = new Set(
|
||||
edges.filter((e) => e.kind === 'trigger-asset').map((e) => `${e.source}\n${e.target}`)
|
||||
)
|
||||
const passive = (e: E) =>
|
||||
e.kind === 'lineage-read' && !triggers.has(`${e.source}\n${e.target}`)
|
||||
const kept = edges.filter((e) => !passive(e))
|
||||
const dropped = edges.length - kept.length
|
||||
if (dropped === 0) return { nodes, edges, dropped }
|
||||
const linked = new Set(
|
||||
kept.filter((e) => e.kind !== 'add-anchor').flatMap((e) => [e.source, e.target])
|
||||
)
|
||||
const readOnly = new Set(
|
||||
edges.filter(passive).map((e) => e.source).filter((id) => !linked.has(id))
|
||||
)
|
||||
return {
|
||||
nodes: nodes.filter((n) => !readOnly.has(n.id)),
|
||||
edges: kept.filter((e) => !readOnly.has(e.source) && !readOnly.has(e.target)),
|
||||
dropped
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { describeEdge } from './edgeDescription'
|
||||
|
||||
const names: Record<string, string> = {
|
||||
'asset:ducklake:main/raw': 'raw',
|
||||
'asset:ducklake:main/clean': 'clean',
|
||||
'script:f/p/clean': 'Clean',
|
||||
'trigger:schedule:f/p/daily': 'Daily'
|
||||
}
|
||||
const nameOf = (id: string) => names[id] ?? id
|
||||
const dataOf = (id: string) =>
|
||||
id === 'trigger:schedule:f/p/daily'
|
||||
? { kind: 'schedule', schedule: '0 0 12 * * *', timezone: 'UTC' }
|
||||
: undefined
|
||||
|
||||
describe('describeEdge', () => {
|
||||
it('names an asset trigger by what runs it: any write, changed or not', () => {
|
||||
expect(
|
||||
describeEdge(
|
||||
{ source: 'asset:ducklake:main/raw', target: 'script:f/p/clean', kind: 'trigger-asset' },
|
||||
nameOf,
|
||||
dataOf
|
||||
)
|
||||
).toEqual({ subject: 'Clean', phrase: 'runs after each write to', object: 'raw' })
|
||||
})
|
||||
|
||||
it("gives a schedule's cadence", () => {
|
||||
expect(
|
||||
describeEdge(
|
||||
{ source: 'trigger:schedule:f/p/daily', target: 'script:f/p/clean', kind: 'trigger-native' },
|
||||
nameOf,
|
||||
dataOf
|
||||
)?.phrase
|
||||
).toBe('runs on schedule: Every day at 12:00 UTC')
|
||||
})
|
||||
|
||||
it('names the folded script on an assets-only edge', () => {
|
||||
expect(
|
||||
describeEdge(
|
||||
{
|
||||
source: 'asset:ducklake:main/raw',
|
||||
target: 'asset:ducklake:main/clean',
|
||||
kind: 'asset-flow',
|
||||
via: ['script:f/p/clean']
|
||||
},
|
||||
nameOf,
|
||||
dataOf
|
||||
)
|
||||
).toEqual({ subject: 'clean', phrase: 'is built from', object: 'raw', note: 'By Clean. A write to raw does not rerun it.' })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,110 @@
|
||||
import { describeSchedule } from '$lib/utils/describeCron'
|
||||
import { isNoAssetNode } from './assetsOnlyView'
|
||||
|
||||
/** What a hovered edge says: "`subject` `phrase` `object`", then a note. */
|
||||
export type EdgeDescription = { subject: string; phrase: string; object?: string; note?: string }
|
||||
|
||||
/** The edge fields the description reads (the canvas model's edge). */
|
||||
export type DescribedEdge = {
|
||||
source: string
|
||||
target: string
|
||||
kind: string
|
||||
unsaved?: boolean
|
||||
muted?: boolean
|
||||
missing?: boolean
|
||||
via?: string[]
|
||||
reactive?: boolean
|
||||
macro_names?: string[]
|
||||
via_use?: boolean
|
||||
}
|
||||
|
||||
// What starts a script, per trigger kind: "runs on <event>".
|
||||
const TRIGGER_EVENT: Record<string, string> = {
|
||||
kafka: 'Kafka message',
|
||||
mqtt: 'MQTT message',
|
||||
amqp: 'AMQP message',
|
||||
nats: 'NATS message',
|
||||
sqs: 'SQS message',
|
||||
gcp: 'Pub/Sub message',
|
||||
email: 'incoming email',
|
||||
postgres: 'Postgres event',
|
||||
webhook: 'webhook call',
|
||||
data_upload: 'data upload'
|
||||
}
|
||||
|
||||
/**
|
||||
* The relationship an edge draws, as a sentence about its two ends. `nameOf`
|
||||
* names a node, `dataOf` gives its data (a trigger's kind and cadence).
|
||||
*/
|
||||
export function describeEdge(
|
||||
e: DescribedEdge,
|
||||
nameOf: (id: string) => string,
|
||||
dataOf: (id: string) => any
|
||||
): EdgeDescription | undefined {
|
||||
const src = nameOf(e.source)
|
||||
const tgt = nameOf(e.target)
|
||||
const draft = e.unsaved ? 'Not deployed yet.' : undefined
|
||||
switch (e.kind) {
|
||||
case 'lineage-write':
|
||||
return { subject: src, phrase: 'writes to', object: tgt, note: draft }
|
||||
case 'lineage-read':
|
||||
return {
|
||||
subject: tgt,
|
||||
phrase: 'reads from',
|
||||
object: src,
|
||||
note: e.muted ? 'A write to it does not run the script (muted).' : draft
|
||||
}
|
||||
case 'trigger-asset':
|
||||
return { subject: tgt, phrase: 'runs after each write to', object: src, note: draft }
|
||||
case 'trigger-native': {
|
||||
const t = dataOf(e.source) ?? {}
|
||||
if (e.missing) {
|
||||
return {
|
||||
subject: tgt,
|
||||
phrase: `declares \`// on ${t.kind}\`, but no trigger of that kind targets it`
|
||||
}
|
||||
}
|
||||
const phrase =
|
||||
t.kind === 'schedule' && t.schedule
|
||||
? `runs on schedule: ${describeSchedule(t.schedule, t.timezone) ?? t.schedule}`
|
||||
: `runs on ${TRIGGER_EVENT[t.kind] ?? `${t.kind} trigger`}`
|
||||
return { subject: tgt, phrase, note: t.draft ? 'Draft trigger, created on save.' : draft }
|
||||
}
|
||||
case 'asset-flow': {
|
||||
const via = (e.via ?? []).map(nameOf).join(', ')
|
||||
const rerun = e.reactive
|
||||
? `Each write to ${src} reruns it.`
|
||||
: `A write to ${src} does not rerun it.`
|
||||
// The scripts that build nothing only read: name them, not the node.
|
||||
if (isNoAssetNode(e.target)) {
|
||||
return { subject: src, phrase: 'is read by', object: via || tgt, note: rerun }
|
||||
}
|
||||
return {
|
||||
subject: tgt,
|
||||
phrase: 'is built from',
|
||||
object: src,
|
||||
note: via ? `By ${via}. ${rerun}` : rerun
|
||||
}
|
||||
}
|
||||
case 'dbt-ref':
|
||||
return { subject: tgt, phrase: 'references', object: src, note: 'dbt ref().' }
|
||||
case 'macro':
|
||||
return {
|
||||
subject: tgt,
|
||||
phrase: e.via_use ? 'uses the macro library' : 'uses macros from',
|
||||
object: src,
|
||||
note: e.macro_names?.length ? e.macro_names.join(', ') : undefined
|
||||
}
|
||||
case 'test-dependency':
|
||||
return {
|
||||
subject: src,
|
||||
phrase: 'must run before',
|
||||
object: tgt,
|
||||
note: 'Its data test reads what the first one builds.'
|
||||
}
|
||||
case 'data-test':
|
||||
return { subject: tgt, phrase: 'tests', object: src }
|
||||
default:
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,8 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import type { AssetGraphResponse } from './types'
|
||||
import {
|
||||
hoverLineage,
|
||||
transitivelyImpliedEdges,
|
||||
buildDownstreamMap,
|
||||
computeDownstreamClosure,
|
||||
computeInducedSchedule
|
||||
@@ -225,3 +227,44 @@ describe('computeInducedSchedule', () => {
|
||||
expect(sched.cyclic.sort()).toEqual(['b', 'c'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('hoverLineage', () => {
|
||||
// a → b → c → d, with a side branch x → c and c → y; z is unrelated to b.
|
||||
const edges = [
|
||||
{ source: 'a', target: 'b' },
|
||||
{ source: 'b', target: 'c' },
|
||||
{ source: 'x', target: 'c' },
|
||||
{ source: 'c', target: 'd' },
|
||||
{ source: 'c', target: 'y' },
|
||||
{ source: 'z', target: 'y' }
|
||||
]
|
||||
|
||||
it('reaches everything upstream and downstream, recursively', () => {
|
||||
const l = hoverLineage(edges, 'b')
|
||||
expect([...l.upstream].sort()).toEqual(['a', 'b'])
|
||||
expect([...l.downstream].sort()).toEqual(['b', 'c', 'd', 'y'])
|
||||
})
|
||||
|
||||
it('keeps only the edges on a path through the node', () => {
|
||||
const l = hoverLineage(edges, 'b')
|
||||
expect(edges.filter(l.hasEdge)).toEqual([
|
||||
{ source: 'a', target: 'b' },
|
||||
{ source: 'b', target: 'c' },
|
||||
{ source: 'c', target: 'd' },
|
||||
{ source: 'c', target: 'y' }
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('transitivelyImpliedEdges', () => {
|
||||
it('flags an edge another path already implies, and only that one', () => {
|
||||
const edges = [
|
||||
{ id: 'ab', source: 'a', target: 'b' },
|
||||
{ id: 'bc', source: 'b', target: 'c' },
|
||||
{ id: 'ac', source: 'a', target: 'c' },
|
||||
{ id: 'ad', source: 'a', target: 'd' },
|
||||
{ id: 'ad2', source: 'a', target: 'd' }
|
||||
]
|
||||
expect([...transitivelyImpliedEdges(edges)]).toEqual(['ac'])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -226,3 +226,76 @@ export function computeInducedSchedule(
|
||||
const roots = ordered.filter((n) => (cleanIndegree.get(n) ?? 0) === 0)
|
||||
return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, roots, cyclic }
|
||||
}
|
||||
|
||||
/** What is linked to a node through the drawn graph: everything it depends on
|
||||
* (upstream) and everything depending on it (downstream), both recursive, the
|
||||
* node itself in each. An edge lies on a path through the node exactly when
|
||||
* both its ends are upstream, or both downstream. */
|
||||
export type HoverLineage = {
|
||||
upstream: Set<string>
|
||||
downstream: Set<string>
|
||||
hasEdge: (e: { source: string; target: string }) => boolean
|
||||
}
|
||||
|
||||
export function hoverLineage(
|
||||
edges: ReadonlyArray<{ source: string; target: string }>,
|
||||
id: string
|
||||
): HoverLineage {
|
||||
const walk = (from: 'source' | 'target', to: 'source' | 'target') => {
|
||||
const next = new Map<string, string[]>()
|
||||
for (const e of edges) next.set(e[from], [...(next.get(e[from]) ?? []), e[to]])
|
||||
const seen = new Set([id])
|
||||
const stack = [id]
|
||||
while (stack.length) {
|
||||
for (const n of next.get(stack.pop()!) ?? []) {
|
||||
if (!seen.has(n)) {
|
||||
seen.add(n)
|
||||
stack.push(n)
|
||||
}
|
||||
}
|
||||
}
|
||||
return seen
|
||||
}
|
||||
const upstream = walk('target', 'source')
|
||||
const downstream = walk('source', 'target')
|
||||
return {
|
||||
upstream,
|
||||
downstream,
|
||||
hasEdge: (e) =>
|
||||
(upstream.has(e.source) && upstream.has(e.target)) ||
|
||||
(downstream.has(e.source) && downstream.has(e.target))
|
||||
}
|
||||
}
|
||||
|
||||
/** Ids of the edges another path already implies: an edge A → C when C can be
|
||||
* reached from A through at least one other node (A → B → … → C). Edges joining
|
||||
* the same two nodes don't make each other redundant. */
|
||||
export function transitivelyImpliedEdges(
|
||||
edges: ReadonlyArray<{ id: string; source: string; target: string }>
|
||||
): Set<string> {
|
||||
const next = new Map<string, Set<string>>()
|
||||
for (const e of edges) {
|
||||
if (e.source === e.target) continue
|
||||
next.set(e.source, (next.get(e.source) ?? new Set()).add(e.target))
|
||||
}
|
||||
// Whether `to` is reachable from `from` without taking the direct step.
|
||||
const reachableAround = (from: string, to: string) => {
|
||||
const seen = new Set<string>([from])
|
||||
const stack = [...(next.get(from) ?? [])].filter((n) => n !== to)
|
||||
for (const n of stack) seen.add(n)
|
||||
while (stack.length) {
|
||||
const cur = stack.pop()!
|
||||
for (const n of next.get(cur) ?? []) {
|
||||
if (n === to) return true
|
||||
if (!seen.has(n)) {
|
||||
seen.add(n)
|
||||
stack.push(n)
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
const implied = new Set<string>()
|
||||
for (const e of edges) if (e.source !== e.target && reachableAround(e.source, e.target)) implied.add(e.id)
|
||||
return implied
|
||||
}
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { onDirectiveTargets } from './onDirectiveTargets'
|
||||
|
||||
describe('onDirectiveTargets', () => {
|
||||
it('leaves out what the script feeds, directly or not, and what already triggers it', () => {
|
||||
const nodes = [
|
||||
{ id: 'script:a', type: 'runnable' },
|
||||
{ id: 'script:b', type: 'runnable' },
|
||||
{ id: 'asset:out', type: 'asset' },
|
||||
{ id: 'asset:later', type: 'asset' },
|
||||
{ id: 'asset:in', type: 'asset' },
|
||||
{ id: 'asset:free', type: 'asset' }
|
||||
]
|
||||
const edges = [
|
||||
{ source: 'script:a', target: 'asset:out', kind: 'lineage-write' },
|
||||
{ source: 'asset:out', target: 'script:b', kind: 'trigger-asset' },
|
||||
{ source: 'script:b', target: 'asset:later', kind: 'lineage-write' },
|
||||
{ source: 'asset:in', target: 'script:a', kind: 'trigger-asset' }
|
||||
]
|
||||
expect([...onDirectiveTargets(nodes, edges, 'script:a')]).toEqual(['asset:free'])
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,34 @@
|
||||
/**
|
||||
* The asset nodes a script can be subscribed to (`// on <asset>`) by dragging:
|
||||
* every asset except those downstream of it, whose writes it would then run on
|
||||
* in a loop, and those that already trigger it.
|
||||
*/
|
||||
export function onDirectiveTargets(
|
||||
nodes: Array<{ id: string; type: string }>,
|
||||
edges: Array<{ source: string; target: string; kind: string }>,
|
||||
scriptId: string
|
||||
): Set<string> {
|
||||
const next = new Map<string, string[]>()
|
||||
for (const e of edges) {
|
||||
if (e.kind === 'add-anchor') continue
|
||||
next.set(e.source, [...(next.get(e.source) ?? []), e.target])
|
||||
}
|
||||
const downstream = new Set<string>([scriptId])
|
||||
const stack = [scriptId]
|
||||
while (stack.length) {
|
||||
for (const n of next.get(stack.pop()!) ?? []) {
|
||||
if (!downstream.has(n)) {
|
||||
downstream.add(n)
|
||||
stack.push(n)
|
||||
}
|
||||
}
|
||||
}
|
||||
const triggering = new Set(
|
||||
edges.filter((e) => e.kind === 'trigger-asset' && e.target === scriptId).map((e) => e.source)
|
||||
)
|
||||
return new Set(
|
||||
nodes
|
||||
.filter((n) => n.type === 'asset' && !downstream.has(n.id) && !triggering.has(n.id))
|
||||
.map((n) => n.id)
|
||||
)
|
||||
}
|
||||
@@ -8,7 +8,8 @@ import type { AssetGraphResponse } from './types'
|
||||
function makeHandle(
|
||||
initial: Array<[string, PipelineDraft]> = [],
|
||||
runnables: Array<{ path: string }> = [],
|
||||
folder = 'x'
|
||||
folder = 'x',
|
||||
triggers: unknown[] = []
|
||||
) {
|
||||
let drafts = new Map(initial)
|
||||
let forgotten: string[] = []
|
||||
@@ -16,11 +17,13 @@ function makeHandle(
|
||||
getFolder: () => folder,
|
||||
getWorkspace: () => 'w',
|
||||
getResolvedGraph: () =>
|
||||
({ assets: [], runnables, edges: [], triggers: [] }) as unknown as AssetGraphResponse,
|
||||
({ assets: [], runnables, edges: [], triggers }) as unknown as AssetGraphResponse,
|
||||
getDrafts: () => drafts,
|
||||
setDrafts: (next) => (drafts = next),
|
||||
newDraftLocalId: () => 'id',
|
||||
onForgetPath: (p) => forgotten.push(p)
|
||||
onForgetPath: (p) => forgotten.push(p),
|
||||
getTriggerDrafts: () => new Map(),
|
||||
setTriggerDraft: () => true
|
||||
})
|
||||
return { handle, drafts: () => drafts, forgotten: () => forgotten }
|
||||
}
|
||||
@@ -155,6 +158,30 @@ describe('pipeline AI direct-draft helpers', () => {
|
||||
expect(res.detectedWrites).toEqual(['ducklake://main/out'])
|
||||
})
|
||||
|
||||
// The canvas draws an inactive draft's stored outputs beside its materialize
|
||||
// target, so a seeded default left on a materialize node shows a phantom asset.
|
||||
it('keeps no seeded or stale output on a node with a materialize target', async () => {
|
||||
vi.spyOn(ScriptService, 'getScriptByPath').mockRejectedValue(new Error('404'))
|
||||
const { handle, drafts } = makeHandle()
|
||||
await handle.proposeNode({
|
||||
path: 'f/x/mat',
|
||||
language: 'duckdb' as any,
|
||||
content: '-- pipeline\n-- materialize ducklake://main/abc\nSELECT 1',
|
||||
outputKind: 'ducklake' as any
|
||||
})
|
||||
expect(drafts().get('f/x/mat')?.outputAssets).toBeUndefined()
|
||||
|
||||
await handle.proposeNode({
|
||||
path: 'f/x/later',
|
||||
language: 'duckdb' as any,
|
||||
content: '-- pipeline\nSELECT 1',
|
||||
outputKind: 'ducklake' as any
|
||||
})
|
||||
expect(drafts().get('f/x/later')?.outputAssets?.length).toBeGreaterThan(0)
|
||||
await handle.editNode('f/x/later', '-- pipeline\n-- materialize ducklake://main/abc\nSELECT 1')
|
||||
expect(drafts().get('f/x/later')?.outputAssets).toBeUndefined()
|
||||
})
|
||||
|
||||
it('editNode rejects a path outside the open folder', async () => {
|
||||
const { handle, drafts } = makeHandle()
|
||||
await expect(handle.editNode('f/other/foo', '-- pipeline')).rejects.toThrow(/open folder/)
|
||||
@@ -181,4 +208,24 @@ describe('pipeline AI direct-draft helpers', () => {
|
||||
expect(d?.script.tag).toBe('custom')
|
||||
expect(d?.script.content).toBe('-- pipeline\nSELECT 2')
|
||||
})
|
||||
|
||||
it("does not take a flow's schedule at the same path for the script's", async () => {
|
||||
const flowSchedule = {
|
||||
trigger_kind: 'schedule',
|
||||
runnable_kind: 'flow',
|
||||
runnable_path: 'f/x/ingest',
|
||||
path: 'f/x/flow_schedule'
|
||||
}
|
||||
const content = '# pipeline\n# on schedule\ndef main():\n return 1'
|
||||
const { handle } = makeHandle(
|
||||
[['f/x/ingest', draft({ script: { content, language: 'python3' } as any })]],
|
||||
[],
|
||||
'x',
|
||||
[flowSchedule]
|
||||
)
|
||||
expect(await handle.unconfiguredTriggers('f/x/ingest')).toEqual(['schedule'])
|
||||
await expect(
|
||||
handle.setNodeTrigger('f/x/ingest', 'schedule', { schedule: '0 0 6 * * *', timezone: 'UTC' })
|
||||
).resolves.toMatchObject({ path: 'f/x/ingest_schedule' })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -9,7 +9,12 @@ import {
|
||||
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
|
||||
import { assetUri, autoOutputAsset, type PipelineOutputKind } from './pipelineTemplates'
|
||||
import { parsePipelineAnnotations, scd2CurrentTargetPath } from './parsePipelineAnnotations'
|
||||
import type { AssetGraphResponse } from './types'
|
||||
import type {
|
||||
AssetGraphResponse,
|
||||
PipelineTriggerDraft,
|
||||
PipelineTriggerDraftKind
|
||||
} from './types'
|
||||
import { defaultTriggerPath, isDraftableTriggerKind, triggerDraftKey } from './pipelineTriggerDrafts'
|
||||
import type {
|
||||
PipelineAIChatHelpers,
|
||||
PipelineContext,
|
||||
@@ -62,20 +67,26 @@ export type PipelineAiHelperDeps = {
|
||||
onForgetPath?: (path: string) => void
|
||||
/** Notify the caller a test run started so it can light up its run UI. */
|
||||
onRunStarted?: (jobId: string, path: string) => void
|
||||
/** Whether the folder has trigger drafts, which live beside the script drafts. */
|
||||
hasTriggerDrafts?: () => boolean
|
||||
getTriggerDrafts: () => Map<string, PipelineTriggerDraft>
|
||||
/** False when another trigger draft already holds the config's path. */
|
||||
setTriggerDraft: (d: PipelineTriggerDraft, previousKey?: string) => boolean
|
||||
}
|
||||
|
||||
export function makePipelineScript(
|
||||
language: ScriptLang,
|
||||
scriptPath: string,
|
||||
content: string,
|
||||
createdAt: string
|
||||
createdAt: string,
|
||||
summary = ''
|
||||
): Script {
|
||||
// Cast through unknown: a local draft only needs path/language/content/schema;
|
||||
// the many readonly deployment fields on Script don't matter until createScript.
|
||||
return {
|
||||
hash: '',
|
||||
path: scriptPath,
|
||||
summary: '',
|
||||
summary,
|
||||
description: '',
|
||||
content,
|
||||
schema: emptySchema(),
|
||||
@@ -199,7 +210,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
path: r.path,
|
||||
language: draft?.script.language,
|
||||
unsaved: r.unsaved ?? false,
|
||||
summary: draft?.script.summary || undefined,
|
||||
summary: (draft ? draft.script.summary : r.summary) || undefined,
|
||||
writes: [...new Set(writes)],
|
||||
reads: [...new Set(reads)],
|
||||
triggers: [...new Set(triggers)]
|
||||
@@ -213,7 +224,36 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
}
|
||||
}
|
||||
|
||||
// Declared `on <kind>` bindings of a node that a trigger row or draft fills.
|
||||
// Webhook and data-upload need none, so they never count as unconfigured.
|
||||
function unconfiguredTriggers(path: string, content: string): PipelineTriggerDraftKind[] {
|
||||
const declared = parsePipelineAnnotations(content)
|
||||
.nativeTriggers.map((n) => n.kind)
|
||||
.filter(isDraftableTriggerKind)
|
||||
const drafted = new Set(
|
||||
[...deps.getTriggerDrafts().values()]
|
||||
.filter((d) => d.config.script_path === path)
|
||||
.map((d) => d.kind)
|
||||
)
|
||||
const deployed = new Set(
|
||||
deps
|
||||
.getResolvedGraph()
|
||||
.triggers.filter(
|
||||
(t) =>
|
||||
t.trigger_kind !== 'asset' &&
|
||||
t.runnable_kind === 'script' &&
|
||||
t.runnable_path === path &&
|
||||
t.path &&
|
||||
!t.missing
|
||||
)
|
||||
.map((t) => t.trigger_kind)
|
||||
)
|
||||
return [...new Set(declared)].filter((k) => !drafted.has(k) && !deployed.has(k))
|
||||
}
|
||||
|
||||
const helpers: PipelineAIChatHelpers = {
|
||||
getFolder: folderName,
|
||||
hasDrafts: () => deps.getDrafts().size > 0 || (deps.hasTriggerDrafts?.() ?? false),
|
||||
getPipelineContext: buildContext,
|
||||
getNodeBody: async (path) => {
|
||||
const draft = deps.getDrafts().get(path)
|
||||
@@ -227,7 +267,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
return undefined
|
||||
}
|
||||
},
|
||||
proposeNode: async ({ path, language, content, outputKind }) => {
|
||||
proposeNode: async ({ path, language, content, outputKind, summary }) => {
|
||||
deps.ensureEditable?.()
|
||||
// build_pipeline_node creates a NEW node in the OPEN folder. Reject a path
|
||||
// outside the folder (it would silently stage into this folder's bundle)
|
||||
@@ -267,10 +307,13 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
}
|
||||
const inferred = await inferDraftAssets(language, content)
|
||||
// Fall back to a seeded output (from the declared output_kind) when the
|
||||
// body doesn't yet write anything inferable.
|
||||
// body doesn't yet write anything inferable. A `materialize` target is the
|
||||
// output already (the canvas adds it from the annotation): a seed beside it
|
||||
// would show as a second, default-named output.
|
||||
const materialized = materializeWrites(content).length > 0
|
||||
const seeded =
|
||||
inferred.writes[0] ??
|
||||
(outputKind
|
||||
(outputKind && !materialized
|
||||
? autoOutputAsset(outputKind as PipelineOutputKind, folderName(), language)
|
||||
: undefined)
|
||||
// Effective outputs = what actually becomes an output edge on the canvas:
|
||||
@@ -280,7 +323,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
const next = new Map(drafts)
|
||||
next.set(path, {
|
||||
localId: deps.newDraftLocalId(),
|
||||
script: makePipelineScript(language, path, content, new Date().toISOString()),
|
||||
script: makePipelineScript(language, path, content, new Date().toISOString(), summary),
|
||||
outputAssets,
|
||||
inputAssets: inferred.reads
|
||||
})
|
||||
@@ -298,7 +341,7 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
)
|
||||
}
|
||||
},
|
||||
editNode: async (path, content) => {
|
||||
editNode: async (path, content, summary) => {
|
||||
deps.ensureEditable?.()
|
||||
assertInFolder(path)
|
||||
assertPipelineAnnotation(content)
|
||||
@@ -319,11 +362,19 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
baseScript = await ScriptService.getScriptByPath({ workspace, path })
|
||||
}
|
||||
const inferred = await inferDraftAssets(baseScript.language, content)
|
||||
const outputAssets = inferred.writes.length > 0 ? inferred.writes : existing?.outputAssets
|
||||
// The stored outputs only stand in for a body that writes nothing inferable;
|
||||
// with a `materialize` target the annotation is the output, so stale ones
|
||||
// (a seeded default name, the target before a rename) must not linger.
|
||||
const outputAssets =
|
||||
inferred.writes.length > 0
|
||||
? inferred.writes
|
||||
: materializeWrites(content).length > 0
|
||||
? undefined
|
||||
: existing?.outputAssets
|
||||
const next = new Map(drafts)
|
||||
next.set(path, {
|
||||
localId: existing?.localId ?? deps.newDraftLocalId(),
|
||||
script: { ...baseScript, content },
|
||||
script: { ...baseScript, content, ...(summary !== undefined ? { summary } : {}) },
|
||||
outputAssets,
|
||||
inputAssets: inferred.reads
|
||||
})
|
||||
@@ -332,12 +383,61 @@ export function createPipelineAiHelpers(deps: PipelineAiHelperDeps): PipelineAIC
|
||||
deps.onProposeNode?.(path)
|
||||
// Deployable lineage only (see proposeNode): body writes + materialize target.
|
||||
return {
|
||||
summary: summary ?? baseScript.summary ?? '',
|
||||
detectedReads: inferred.reads.map(assetUri),
|
||||
detectedWrites: dedupeAssets([...inferred.writes, ...materializeWrites(content)]).map(
|
||||
assetUri
|
||||
)
|
||||
}
|
||||
},
|
||||
unconfiguredTriggers: async (path) => {
|
||||
const body = await helpers.getNodeBody(path)
|
||||
return body ? unconfiguredTriggers(path, body.content) : []
|
||||
},
|
||||
setNodeTrigger: async (path, kind, config) => {
|
||||
deps.ensureEditable?.()
|
||||
assertInFolder(path)
|
||||
const body = await helpers.getNodeBody(path)
|
||||
if (!body) throw new Error(`No pipeline node at '${path}'. Build it first.`)
|
||||
const prefix = body.language === 'python3' || body.language === 'bash' ? '#' : body.language === 'duckdb' || body.language === 'postgresql' ? '--' : '//'
|
||||
if (!parsePipelineAnnotations(body.content).nativeTriggers.some((n) => n.kind === kind)) {
|
||||
throw new Error(
|
||||
`'${path}' does not declare \`${prefix} on ${kind}\`. Add that annotation line with edit_pipeline_node first, so the trigger is wired to the node.`
|
||||
)
|
||||
}
|
||||
const row = deps
|
||||
.getResolvedGraph()
|
||||
.triggers.find(
|
||||
(t) =>
|
||||
t.trigger_kind === kind &&
|
||||
t.runnable_kind === 'script' &&
|
||||
t.runnable_path === path &&
|
||||
t.path &&
|
||||
!t.missing &&
|
||||
!(t as { draft?: boolean }).draft
|
||||
)
|
||||
if (row && 'path' in row) {
|
||||
throw new Error(
|
||||
`'${path}' already has a deployed ${kind} trigger at '${row.path}'. Change that one instead of adding a second.`
|
||||
)
|
||||
}
|
||||
const existing = [...deps.getTriggerDrafts().values()].find(
|
||||
(d) => d.kind === kind && d.config.script_path === path
|
||||
)
|
||||
const triggerPath =
|
||||
(typeof config.path === 'string' && config.path) ||
|
||||
existing?.config.path ||
|
||||
defaultTriggerPath(path, kind)
|
||||
const saved = deps.setTriggerDraft(
|
||||
{ kind, config: { ...config, path: triggerPath, script_path: path, is_flow: false } },
|
||||
existing ? triggerDraftKey(kind, existing.config.path) : undefined
|
||||
)
|
||||
if (!saved) {
|
||||
throw new Error(`Another draft trigger already uses the path '${triggerPath}'. Pass a different config.path.`)
|
||||
}
|
||||
deps.onShowDrafts?.()
|
||||
return { path: triggerPath, replaced: !!existing }
|
||||
},
|
||||
removeProposedNode: async (path) => {
|
||||
if (!deps.getDrafts().has(path)) {
|
||||
throw new Error(`No unsaved draft at '${path}' to discard.`)
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
addTriggerDirective,
|
||||
listenToAsset,
|
||||
removeTriggerDirective,
|
||||
stopListeningToAsset
|
||||
} from './pipelineAnnotationEdits'
|
||||
|
||||
const table = { kind: 'datatable' as const, path: 'main/orders' }
|
||||
const lake = { kind: 'ducklake' as const, path: 'main/orders' }
|
||||
|
||||
describe('pipeline annotation edits', () => {
|
||||
it('adds `on` at the end of the header, in its comment style, leaving the body', () => {
|
||||
const sql = '-- pipeline\n-- materialize ducklake://main/x\nSELECT 1; -- on nothing'
|
||||
expect(listenToAsset(sql, table, false)).toBe(
|
||||
'-- pipeline\n-- materialize ducklake://main/x\n-- on datatable://main/orders\nSELECT 1; -- on nothing'
|
||||
)
|
||||
})
|
||||
|
||||
it('adds `on` next to the existing `on` lines', () => {
|
||||
const sql = '-- pipeline\n-- on schedule\n-- materialize ducklake://main/x\nSELECT 1;'
|
||||
expect(listenToAsset(sql, table, false)).toBe(
|
||||
'-- pipeline\n-- on schedule\n-- on datatable://main/orders\n-- materialize ducklake://main/x\nSELECT 1;'
|
||||
)
|
||||
})
|
||||
|
||||
it('removes an `on` line however it is spelled, options included', () => {
|
||||
const py = '# pipeline\n# on datatable://main/orders debounce=60s\nprint(1)'
|
||||
expect(stopListeningToAsset(py, table, true)).toBe('# pipeline\nprint(1)')
|
||||
})
|
||||
|
||||
it('unmutes an auto-triggering read rather than adding `on`, and mutes it to stop', () => {
|
||||
const muted = '// pipeline\n// mute ducklake://main/orders\nexport function main() {}'
|
||||
expect(listenToAsset(muted, lake, true)).toBe('// pipeline\nexport function main() {}')
|
||||
expect(stopListeningToAsset('// pipeline\nexport function main() {}', lake, true)).toBe(
|
||||
'// pipeline\n// mute ducklake://main/orders\nexport function main() {}'
|
||||
)
|
||||
})
|
||||
|
||||
it('adds `on` for an auto-triggering kind the script does not read', () => {
|
||||
expect(listenToAsset('-- pipeline\nSELECT 1;', lake, false)).toBe(
|
||||
'-- pipeline\n-- on ducklake://main/orders\nSELECT 1;'
|
||||
)
|
||||
})
|
||||
|
||||
it('mutes only an auto-triggering asset the script reads', () => {
|
||||
const sub = '-- pipeline\n-- on ducklake://main/orders\nSELECT 1;'
|
||||
expect(stopListeningToAsset(sub, lake, false)).toBe('-- pipeline\nSELECT 1;')
|
||||
})
|
||||
|
||||
it('adds `on` for an auto-triggering kind when every read is muted', () => {
|
||||
const all = '// pipeline\n// mute all\nexport function main() {}'
|
||||
expect(listenToAsset(all, lake, true)).toBe(
|
||||
'// pipeline\n// mute all\n// on ducklake://main/orders\nexport function main() {}'
|
||||
)
|
||||
})
|
||||
|
||||
it('adds a trigger kind once, and drops it', () => {
|
||||
expect(addTriggerDirective('// pipeline\n// on webhook\nx()', 'webhook')).toBe(
|
||||
'// pipeline\n// on webhook\nx()'
|
||||
)
|
||||
expect(removeTriggerDirective('// pipeline\n// on kafka\nx()', 'kafka')).toBe(
|
||||
'// pipeline\nx()'
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,149 @@
|
||||
import type { AssetKind } from '$lib/gen'
|
||||
import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations'
|
||||
|
||||
// Edits to a pipeline script's leading annotation header (`// on …`, `// mute …`),
|
||||
// the comment block `parsePipelineAnnotations` reads. Lines are kept in the
|
||||
// script's own comment style, and nothing past the header is touched.
|
||||
|
||||
/** Asset kind → the prefix of its `// on <ref>` spelling. Mirrors ASSET_KINDS in
|
||||
* backend/parsers/windmill-parser/src/asset_parser.rs. */
|
||||
export const ASSET_REF_PREFIX: Record<AssetKind, string> = {
|
||||
s3object: 's3://',
|
||||
resource: '$res:',
|
||||
ducklake: 'ducklake://',
|
||||
datatable: 'datatable://',
|
||||
volume: 'volume://',
|
||||
dbt: 'dbt://'
|
||||
}
|
||||
|
||||
// A `// pipeline` script reading one of these kinds runs after each write to it
|
||||
// unless muted, with no `// on` line (see AUTO_TRIGGER_KINDS in resolveGraph).
|
||||
const AUTO_TRIGGER_KINDS: ReadonlySet<AssetKind> = new Set(['ducklake', 's3object'])
|
||||
|
||||
export function assetRef(asset: { kind: AssetKind; path: string }): string {
|
||||
return `${ASSET_REF_PREFIX[asset.kind]}${asset.path}`
|
||||
}
|
||||
|
||||
const PREFIXES = ['//', '--', '#']
|
||||
|
||||
type HeaderLine = { index: number; prefix: string; inner: string }
|
||||
|
||||
/** The header: leading comment lines (blank lines allowed), up to the first code. */
|
||||
function header(lines: string[]): HeaderLine[] {
|
||||
const out: HeaderLine[] = []
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const t = lines[i].trimStart()
|
||||
if (t === '') continue
|
||||
const prefix = PREFIXES.find((p) => t.startsWith(p))
|
||||
if (!prefix) break
|
||||
out.push({ index: i, prefix, inner: t.slice(prefix.length).trim() })
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// A header line matched the way the parser reads it, so `on <ref> debounce=60s`,
|
||||
// `res://` vs `$res:` spellings and dbt path quoting all count as the same line.
|
||||
type LineMatch = (parsed: PipelineAnnotations) => boolean
|
||||
const onAsset =
|
||||
(asset: { kind: AssetKind; path: string }): LineMatch =>
|
||||
(p) =>
|
||||
p.triggerAssets.some((a) => a.kind === asset.kind && a.path === asset.path)
|
||||
const muteAsset =
|
||||
(asset: { kind: AssetKind; path: string }): LineMatch =>
|
||||
(p) =>
|
||||
p.muteAssets.some((a) => a.kind === asset.kind && a.path === asset.path)
|
||||
const muteAll: LineMatch = (p) => p.muteAll
|
||||
const onTrigger =
|
||||
(kind: string): LineMatch =>
|
||||
(p) =>
|
||||
p.nativeTriggers.some((t) => t.kind === kind)
|
||||
|
||||
function matches(line: HeaderLine, match: LineMatch): boolean {
|
||||
return match(parsePipelineAnnotations(`${line.prefix} ${line.inner}`))
|
||||
}
|
||||
|
||||
function hasDirective(content: string, match: LineMatch): boolean {
|
||||
return header(content.split('\n')).some((l) => matches(l, match))
|
||||
}
|
||||
|
||||
/** Adds `<prefix> <directive>` in the header's comment style: after the last line
|
||||
* of the same directive (`on`, `mute`), so they stay grouped, else at its end. */
|
||||
function addDirective(content: string, directive: string): string {
|
||||
const lines = content.split('\n')
|
||||
const h = header(lines)
|
||||
const keyword = directive.split(/\s/)[0]
|
||||
const sameKind = h.filter((l) => l.inner.split(/\s/)[0] === keyword)
|
||||
const after = sameKind[sameKind.length - 1] ?? h[h.length - 1]
|
||||
const line = `${after?.prefix ?? '//'} ${directive}`
|
||||
lines.splice(after ? after.index + 1 : 0, 0, line)
|
||||
return lines.join('\n')
|
||||
}
|
||||
|
||||
function removeDirective(content: string, match: LineMatch): string {
|
||||
const lines = content.split('\n')
|
||||
const drop = new Set(
|
||||
header(lines)
|
||||
.filter((l) => matches(l, match))
|
||||
.map((l) => l.index)
|
||||
)
|
||||
return lines.filter((_, i) => !drop.has(i)).join('\n')
|
||||
}
|
||||
|
||||
/** The script runs after each write to the asset: `// on <ref>`, and any
|
||||
* `// mute <ref>` lifted. An auto-triggering kind it `reads` needs no `// on`
|
||||
* once unmuted, unless `// mute all` still silences it. */
|
||||
export function listenToAsset(
|
||||
content: string,
|
||||
asset: { kind: AssetKind; path: string },
|
||||
reads: boolean
|
||||
): string {
|
||||
const ref = assetRef(asset)
|
||||
let out = removeDirective(content, muteAsset(asset))
|
||||
const auto = reads && AUTO_TRIGGER_KINDS.has(asset.kind) && !hasDirective(out, muteAll)
|
||||
if (!auto && !hasDirective(out, onAsset(asset))) out = addDirective(out, `on ${ref}`)
|
||||
return out
|
||||
}
|
||||
|
||||
/** The script no longer runs after writes to the asset: its `// on <ref>` goes,
|
||||
* and an auto-triggering kind it `reads` is muted, since the read alone would
|
||||
* still trigger it. */
|
||||
export function stopListeningToAsset(
|
||||
content: string,
|
||||
asset: { kind: AssetKind; path: string },
|
||||
reads: boolean
|
||||
): string {
|
||||
const ref = assetRef(asset)
|
||||
let out = removeDirective(content, onAsset(asset))
|
||||
if (reads && AUTO_TRIGGER_KINDS.has(asset.kind) && !hasDirective(out, muteAsset(asset))) {
|
||||
out = addDirective(out, `mute ${ref}`)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/** Adds a `// on <trigger kind>` line (`schedule`, `webhook`, …) unless it has one. */
|
||||
export function addTriggerDirective(content: string, kind: string): string {
|
||||
return hasDirective(content, onTrigger(kind)) ? content : addDirective(content, `on ${kind}`)
|
||||
}
|
||||
|
||||
/** Drops a `// on <trigger kind>` line (`kafka`, `schedule`, …). */
|
||||
export function removeTriggerDirective(content: string, kind: string): string {
|
||||
return removeDirective(content, onTrigger(kind))
|
||||
}
|
||||
|
||||
/** What each edit says it changes, for the button that applies it. */
|
||||
export function listenChangeText(
|
||||
asset: { kind: AssetKind; path: string },
|
||||
reads: boolean
|
||||
): string {
|
||||
return reads && AUTO_TRIGGER_KINDS.has(asset.kind)
|
||||
? `Removes any \`mute ${assetRef(asset)}\` line (adds \`on ${assetRef(asset)}\` if all reads are muted)`
|
||||
: `Adds \`on ${assetRef(asset)}\``
|
||||
}
|
||||
export function stopListeningChangeText(
|
||||
asset: { kind: AssetKind; path: string },
|
||||
reads: boolean
|
||||
): string {
|
||||
return reads && AUTO_TRIGGER_KINDS.has(asset.kind)
|
||||
? `Removes \`on ${assetRef(asset)}\` and adds \`mute ${assetRef(asset)}\``
|
||||
: `Removes \`on ${assetRef(asset)}\``
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
import { untrack } from 'svelte'
|
||||
import { ScriptService, type AssetKind, type ScriptLang } from '$lib/gen'
|
||||
import { inferAssets } from '$lib/infer'
|
||||
import {
|
||||
extractReads,
|
||||
extractWrites,
|
||||
type AssetWithAltAccessType
|
||||
} from '$lib/components/assets/lib'
|
||||
import { parsePipelineAnnotations } from './parsePipelineAnnotations'
|
||||
import type { AssetGraphResponse, NativeTriggerKind } from './types'
|
||||
import type { PipelineEditorState } from './pipelineEditorState.svelte'
|
||||
|
||||
type AssetRef = { kind: AssetKind; path: string }
|
||||
|
||||
/**
|
||||
* Reads the body of every deployed script of a pipeline's graph and infers its
|
||||
* asset reads/writes and `// on <native trigger>` annotations, so the canvas
|
||||
* draws the edges of scripts the user never opened. The open script's live
|
||||
* buffer wins over its cached body.
|
||||
*
|
||||
* The caches are only-add and keyed by path: the derived maps iterate the
|
||||
* current graph's scripts, so a renamed or deleted script drops out when the
|
||||
* graph refetches. A stale value under a live path does not, so a save through
|
||||
* the editor must `forget` its path for the sweep to read it again.
|
||||
*/
|
||||
export function usePipelineAssetPrefetch(deps: {
|
||||
getWorkspace: () => string | undefined
|
||||
getGraph: () => AssetGraphResponse | undefined
|
||||
editor: PipelineEditorState
|
||||
}) {
|
||||
const pe = deps.editor
|
||||
let bodies = $state<Map<string, string>>(new Map())
|
||||
// Deployed summary + language from the same fetch, for node titles and icons.
|
||||
let scriptMeta = $state<Map<string, { summary?: string; language: ScriptLang }>>(new Map())
|
||||
let inferredAssets = $state<Map<string, AssetWithAltAccessType[]>>(new Map())
|
||||
let prefetching = $state(false)
|
||||
// Bumped by every sweep so one started for an older graph stops writing.
|
||||
let gen = 0
|
||||
|
||||
$effect(() => {
|
||||
const ws = deps.getWorkspace()
|
||||
const g = deps.getGraph()
|
||||
if (!ws || !g) return
|
||||
const sweep = ++gen
|
||||
const targets = untrack(() =>
|
||||
g.runnables
|
||||
.filter((r) => r.usage_kind === 'script')
|
||||
.map((r) => r.path)
|
||||
.filter((p) => !pe.drafts.has(p) && !bodies.has(p))
|
||||
)
|
||||
if (targets.length === 0) return
|
||||
let i = 0
|
||||
const POOL = 6
|
||||
const worker = async () => {
|
||||
while (i < targets.length && sweep === gen) {
|
||||
const path = targets[i++]
|
||||
try {
|
||||
const s = await ScriptService.getScriptByPath({ workspace: ws, path })
|
||||
if (sweep !== gen) return
|
||||
const content = s.content ?? ''
|
||||
const res = await inferAssets(s.language, content)
|
||||
if (sweep !== gen) return
|
||||
const inferred = (res?.assets ?? []) as AssetWithAltAccessType[]
|
||||
untrack(() => {
|
||||
if (!bodies.has(path)) bodies = new Map(bodies).set(path, content)
|
||||
if (!scriptMeta.has(path)) {
|
||||
scriptMeta = new Map(scriptMeta).set(path, {
|
||||
summary: s.summary || undefined,
|
||||
language: s.language
|
||||
})
|
||||
}
|
||||
if (!inferredAssets.has(path)) {
|
||||
inferredAssets = new Map(inferredAssets).set(path, inferred)
|
||||
}
|
||||
})
|
||||
} catch {
|
||||
// Skip — that node just falls back to base-graph edges.
|
||||
}
|
||||
}
|
||||
}
|
||||
prefetching = true
|
||||
const pool = Array.from({ length: Math.min(POOL, targets.length) }, () => worker())
|
||||
void Promise.all(pool).then(() => {
|
||||
if (sweep === gen) prefetching = false
|
||||
})
|
||||
})
|
||||
|
||||
// Both edge maps in one pass over the graph's scripts; they only differ by
|
||||
// extractWrites vs extractReads over the same source.
|
||||
const edges = $derived.by(() => {
|
||||
const writes = new Map<string, AssetRef[]>()
|
||||
const reads = new Map<string, AssetRef[]>()
|
||||
const g = deps.getGraph()
|
||||
if (!g) return { writes, reads }
|
||||
for (const r of g.runnables) {
|
||||
if (r.usage_kind !== 'script') continue
|
||||
const assets =
|
||||
pe.liveBodyAssets.scriptPath === r.path
|
||||
? pe.liveBodyAssets.assets
|
||||
: inferredAssets.get(r.path)
|
||||
if (!assets) continue
|
||||
const w = extractWrites(assets)
|
||||
if (w.length > 0) writes.set(r.path, w)
|
||||
const rd = extractReads(assets)
|
||||
if (rd.length > 0) reads.set(r.path, rd)
|
||||
}
|
||||
return { writes, reads }
|
||||
})
|
||||
|
||||
const annotatedNativeKinds = $derived.by(() => {
|
||||
const out = new Map<string, Set<NativeTriggerKind>>()
|
||||
const g = deps.getGraph()
|
||||
if (!g) return out
|
||||
const livePath = pe.liveAnnotations.scriptPath
|
||||
for (const r of g.runnables) {
|
||||
if (r.usage_kind !== 'script') continue
|
||||
let kinds: Set<NativeTriggerKind>
|
||||
if (r.path === livePath) {
|
||||
kinds = new Set(pe.liveAnnotations.annotations.nativeTriggers.map((n) => n.kind))
|
||||
} else {
|
||||
const body = bodies.get(r.path)
|
||||
if (!body) continue
|
||||
kinds = new Set(parsePipelineAnnotations(body).nativeTriggers.map((n) => n.kind))
|
||||
}
|
||||
if (kinds.size > 0) out.set(r.path, kinds)
|
||||
}
|
||||
return out
|
||||
})
|
||||
|
||||
return {
|
||||
get bodies(): ReadonlyMap<string, string> {
|
||||
return bodies
|
||||
},
|
||||
get scriptMeta(): ReadonlyMap<string, { summary?: string; language: ScriptLang }> {
|
||||
return scriptMeta
|
||||
},
|
||||
get inferredWritesByPath() {
|
||||
return edges.writes
|
||||
},
|
||||
get inferredReadsByPath() {
|
||||
return edges.reads
|
||||
},
|
||||
get annotatedNativeKindsByPath() {
|
||||
return annotatedNativeKinds
|
||||
},
|
||||
get prefetching() {
|
||||
return prefetching
|
||||
},
|
||||
/** Drop cached paths so the next sweep re-reads them (after a save). */
|
||||
forget(...paths: string[]) {
|
||||
const nextBodies = new Map(bodies)
|
||||
const nextMeta = new Map(scriptMeta)
|
||||
const nextInferred = new Map(inferredAssets)
|
||||
for (const p of paths) {
|
||||
nextBodies.delete(p)
|
||||
nextMeta.delete(p)
|
||||
nextInferred.delete(p)
|
||||
}
|
||||
bodies = nextBodies
|
||||
scriptMeta = nextMeta
|
||||
inferredAssets = nextInferred
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import { untrack } from 'svelte'
|
||||
import type { AssetGraphResponse } from './types'
|
||||
|
||||
type Staged = { args: Record<string, any>; valid: boolean }
|
||||
|
||||
// An S3Object-shaped value (the file picker writes `{ s3: '<path>' }`).
|
||||
function isS3Object(v: any): boolean {
|
||||
return !!v && typeof v === 'object' && !Array.isArray(v) && 's3' in v
|
||||
}
|
||||
|
||||
// Whether a staged value carries actual data. Covers the two shapes a
|
||||
// data-upload entry takes: an S3Object (file picker → `{ s3: '<path>' }`) and
|
||||
// a plain required input (e.g. a JSON array pasted into the run form).
|
||||
function hasMeaningfulValue(v: any): boolean {
|
||||
if (v == null) return false
|
||||
if (typeof v === 'string') return v.length > 0
|
||||
if (Array.isArray(v)) return v.length > 0
|
||||
if (typeof v === 'object')
|
||||
return isS3Object(v) ? typeof v.s3 === 'string' && v.s3.length > 0 : Object.keys(v).length > 0
|
||||
return true // numbers / booleans count as provided
|
||||
}
|
||||
|
||||
/**
|
||||
* Staged run-form input for the graph's data-upload entry scripts, keyed by path.
|
||||
* Lifted out of the (transient, per-selection) run form so the uploaded/entered
|
||||
* data persists across selection changes: it drives each entry node's green
|
||||
* "ready" state and seeds a run with that input. `valid` is the run form's
|
||||
* full-schema validity (all required fields satisfied).
|
||||
*/
|
||||
export function usePipelineDataUploads(getGraph: () => AssetGraphResponse) {
|
||||
let staged = $state<Record<string, Staged>>({})
|
||||
|
||||
// Scripts that are data-upload entry points (a `data_upload` trigger in the
|
||||
// displayed graph). They can't auto-run — they need their data first.
|
||||
const entryPaths = $derived(
|
||||
new Set(
|
||||
getGraph()
|
||||
.triggers.filter((t) => t.trigger_kind === 'data_upload' && t.runnable_kind === 'script')
|
||||
.map((t) => t.runnable_path)
|
||||
)
|
||||
)
|
||||
|
||||
// Ready once the user actually provided its data, not just opened the form. Two
|
||||
// guards: the form's own full-schema `valid` (every required field — including
|
||||
// non-file ones — is satisfied), AND that any declared S3Object file field
|
||||
// actually carries a file (the picker can leave an empty `{ s3: '' }` on a
|
||||
// non-required file field, which `valid` alone wouldn't catch).
|
||||
function isReady(path: string): boolean {
|
||||
const s = staged[path]
|
||||
if (!s || !s.valid) return false
|
||||
const values = Object.values(s.args)
|
||||
const s3s = values.filter(isS3Object)
|
||||
if (s3s.length > 0) return s3s.every(hasMeaningfulValue)
|
||||
return values.some(hasMeaningfulValue)
|
||||
}
|
||||
|
||||
const readyPaths = $derived(new Set([...entryPaths].filter((p) => isReady(p))))
|
||||
|
||||
return {
|
||||
get entryPaths(): ReadonlySet<string> {
|
||||
return entryPaths
|
||||
},
|
||||
/** Entry scripts with their data provided — drives the green node treatment. */
|
||||
get readyPaths(): ReadonlySet<string> {
|
||||
return readyPaths
|
||||
},
|
||||
isReady,
|
||||
argsFor: (path: string): Record<string, any> | undefined => staged[path]?.args,
|
||||
// Persist the run form's args + validity, but only for data-upload entries —
|
||||
// other run forms (partitioned producers) run their own way and must not be
|
||||
// mistaken for a staged upload. Idempotent: the run form re-emits on every
|
||||
// keystroke/validation pass, so bail when the value is unchanged — otherwise
|
||||
// each emit would reassign `staged`, giving `readyPaths` a fresh Set identity
|
||||
// that re-syncs the canvas and re-fires the form's emit effect
|
||||
// (effect_update_depth_exceeded).
|
||||
// Called from the run forms' own effects, so its reads are untracked: a form
|
||||
// must not re-run because the store it just wrote to changed.
|
||||
stage: (path: string, args: Record<string, any>, valid: boolean) =>
|
||||
untrack(() => {
|
||||
if (!entryPaths.has(path)) return
|
||||
const prev = staged[path]
|
||||
if (prev && prev.valid === valid && JSON.stringify(prev.args) === JSON.stringify(args))
|
||||
return
|
||||
staged = { ...staged, [path]: { args, valid } }
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { DraftService, ScriptService } from '$lib/gen'
|
||||
import { planPipelineDelete } from './pipelineDelete'
|
||||
|
||||
afterEach(() => vi.restoreAllMocks())
|
||||
|
||||
describe('planPipelineDelete', () => {
|
||||
it("takes only the pipeline's own triggers and addresses draft-only nodes by storage key", async () => {
|
||||
vi.spyOn(ScriptService, 'listRunnables').mockResolvedValue({
|
||||
items: [
|
||||
{ type: 'script', path: 'f/sales/ingest', auto_kind: 'pipeline' },
|
||||
{ type: 'script', path: 'f/sales/report' },
|
||||
{
|
||||
type: 'script',
|
||||
path: 'u/alice/draft_abc',
|
||||
draft_only: true,
|
||||
draft_path: 'f/sales/unpublished',
|
||||
auto_kind: 'pipeline'
|
||||
}
|
||||
]
|
||||
} as any)
|
||||
vi.spyOn(DraftService, 'getOwnDraft').mockResolvedValue(undefined as any)
|
||||
const trigger = (runnable_kind: string, runnable_path: string, path: string) => ({
|
||||
trigger_kind: 'schedule',
|
||||
runnable_kind,
|
||||
runnable_path,
|
||||
path
|
||||
})
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn().mockResolvedValue({
|
||||
ok: true,
|
||||
json: async () => ({
|
||||
triggers: [
|
||||
trigger('script', 'f/sales/ingest', 'f/sales/ingest_schedule'),
|
||||
trigger('script', 'f/sales/report', 'f/sales/report_schedule'),
|
||||
trigger('flow', 'f/sales/ingest', 'f/sales/flow_schedule')
|
||||
]
|
||||
})
|
||||
})
|
||||
)
|
||||
|
||||
const plan = await planPipelineDelete('w', 'sales')
|
||||
|
||||
expect(plan.triggers).toEqual([{ kind: 'schedule', path: 'f/sales/ingest_schedule' }])
|
||||
expect(plan.scripts).toEqual([
|
||||
{ path: 'f/sales/ingest', displayPath: 'f/sales/ingest', draftOnly: false },
|
||||
{ path: 'u/alice/draft_abc', displayPath: 'f/sales/unpublished', draftOnly: true }
|
||||
])
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,135 @@
|
||||
import { DraftService, OpenAPI, ScriptService, type RunnableItem } from '$lib/gen'
|
||||
import { discardDraft } from '$lib/utils_draft_deploy'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath, pipelineLocalMirrorKey } from '$lib/pipelinePaths'
|
||||
import { deleteTriggerRow } from './pipelineTriggerDraftDeploy'
|
||||
import type { AssetGraphResponse, NativeTriggerKind } from './types'
|
||||
|
||||
export type PipelineDeleteResult = { removed: number; failures: string[] }
|
||||
|
||||
/** The folder's pipeline-member scripts, deployed and draft-only alike: every one of
|
||||
* them makes the home list show the pipeline, so one left behind brings it back. */
|
||||
async function listPipelineScripts(workspace: string, folder: string): Promise<RunnableItem[]> {
|
||||
const out: RunnableItem[] = []
|
||||
let cursor: string | undefined
|
||||
do {
|
||||
const res = await ScriptService.listRunnables({
|
||||
workspace,
|
||||
kinds: 'script',
|
||||
pathStart: `f/${folder}/`,
|
||||
includeDraftOnly: true,
|
||||
perPage: 100,
|
||||
cursor
|
||||
})
|
||||
out.push(...res.items.filter((it) => it.auto_kind === 'pipeline'))
|
||||
cursor = res.next_cursor ?? undefined
|
||||
} while (cursor)
|
||||
return out
|
||||
}
|
||||
|
||||
/** Trigger rows starting one of `scripts`. The folder graph also carries the
|
||||
* triggers of the folder's ordinary scripts and flows, which are not the pipeline's. */
|
||||
async function listNativeTriggers(workspace: string, folder: string, scripts: ReadonlySet<string>) {
|
||||
const res = await fetch(
|
||||
`${OpenAPI.BASE ?? ''}/w/${workspace}/assets/graph?${new URLSearchParams({ folder })}`,
|
||||
{ credentials: 'include' }
|
||||
)
|
||||
if (!res.ok) throw new Error(`GET /assets/graph → ${res.status}`)
|
||||
const graph = (await res.json()) as AssetGraphResponse
|
||||
return graph.triggers.flatMap((t) =>
|
||||
t.trigger_kind !== 'asset' &&
|
||||
t.runnable_kind === 'script' &&
|
||||
scripts.has(t.runnable_path) &&
|
||||
t.trigger_kind !== 'webhook' &&
|
||||
t.trigger_kind !== 'data_upload' &&
|
||||
t.path &&
|
||||
!t.unsaved
|
||||
? [{ kind: t.trigger_kind, path: t.path }]
|
||||
: []
|
||||
)
|
||||
}
|
||||
|
||||
export type PipelineDeletePlan = {
|
||||
/** `path` is what the API addresses (a draft-only node's storage key);
|
||||
* `displayPath` is the path the user typed for it. */
|
||||
scripts: { path: string; displayPath: string; draftOnly: boolean }[]
|
||||
triggers: { kind: NativeTriggerKind; path: string }[]
|
||||
/** Whether the caller has an unsaved pipeline draft for this folder. */
|
||||
hasDraft: boolean
|
||||
}
|
||||
|
||||
/** Everything `deletePipeline` would remove, so it can be listed before it is. */
|
||||
export async function planPipelineDelete(
|
||||
workspace: string,
|
||||
folder: string
|
||||
): Promise<PipelineDeletePlan> {
|
||||
const scripts = await listPipelineScripts(workspace, folder)
|
||||
const deployed = new Set(scripts.filter((s) => !s.draft_only).map((s) => s.path))
|
||||
const [triggers, draft] = await Promise.all([
|
||||
listNativeTriggers(workspace, folder, deployed),
|
||||
DraftService.getOwnDraft({
|
||||
workspace,
|
||||
kind: PIPELINE_DRAFT_KIND,
|
||||
path: pipelineBundlePath(folder)
|
||||
}).catch(() => undefined)
|
||||
])
|
||||
return {
|
||||
scripts: scripts.map((s) => ({
|
||||
path: s.path,
|
||||
displayPath: (s.draft_only && s.draft_path) || s.path,
|
||||
draftOnly: !!s.draft_only
|
||||
})),
|
||||
triggers,
|
||||
hasDraft: !!draft
|
||||
}
|
||||
}
|
||||
|
||||
/** Removes what `plan` lists: the trigger rows, the scripts (deleted when
|
||||
* `hardDelete`, which the backend allows admins only, archived otherwise; a
|
||||
* draft-only node is discarded) and the caller's pipeline draft. The data the
|
||||
* nodes wrote (DuckLake tables, S3 objects) is left alone. Triggers go first: a
|
||||
* script refused afterwards loses its trigger, but no trigger outlives the script
|
||||
* it starts. */
|
||||
export async function deletePipeline(
|
||||
workspace: string,
|
||||
folder: string,
|
||||
plan: PipelineDeletePlan,
|
||||
hardDelete: boolean
|
||||
): Promise<PipelineDeleteResult> {
|
||||
const failures: string[] = []
|
||||
let removed = 0
|
||||
const fail = (what: string, e: any) =>
|
||||
failures.push(`${what}: ${e?.body ?? e?.message ?? String(e)}`)
|
||||
|
||||
for (const t of plan.triggers) {
|
||||
try {
|
||||
if (await deleteTriggerRow(t.kind, t.path, workspace)) removed++
|
||||
} catch (e) {
|
||||
fail(`${t.kind} trigger ${t.path}`, e)
|
||||
}
|
||||
}
|
||||
for (const s of plan.scripts) {
|
||||
try {
|
||||
if (s.draftOnly) {
|
||||
const r = await discardDraft('script', s.path, workspace, true, false, false)
|
||||
if (!r.success) throw new Error(r.error)
|
||||
} else if (hardDelete) {
|
||||
await ScriptService.deleteScriptByPath({ workspace, path: s.path })
|
||||
} else {
|
||||
await ScriptService.archiveScriptByPath({ workspace, path: s.path })
|
||||
}
|
||||
removed++
|
||||
} catch (e) {
|
||||
fail(s.displayPath, e)
|
||||
}
|
||||
}
|
||||
if (plan.hasDraft) {
|
||||
const bundle = await discardDraft(PIPELINE_DRAFT_KIND, pipelineBundlePath(folder), workspace, true)
|
||||
if (!bundle.success) failures.push(`pipeline draft: ${bundle.error}`)
|
||||
}
|
||||
try {
|
||||
localStorage.removeItem(pipelineLocalMirrorKey(folder))
|
||||
} catch {
|
||||
// Storage unavailable: nothing was mirrored either.
|
||||
}
|
||||
return { removed, failures }
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
import { DraftService, ScriptService, type Script } from '$lib/gen'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
|
||||
import type { PipelineTriggerDraft } from './types'
|
||||
import type { Schema } from '$lib/common'
|
||||
import { emptySchema } from '$lib/utils'
|
||||
import { inferArgs, inferAssets } from '$lib/infer'
|
||||
import type { AssetWithAltAccessType } from '$lib/components/assets/lib'
|
||||
import type { PipelineDraft } from './pipelineAiHelpers'
|
||||
import type { PipelineEditorState } from './pipelineEditorState.svelte'
|
||||
import { deployTriggerDraft } from './pipelineTriggerDraftDeploy'
|
||||
|
||||
// Deploying a pipeline's drafts: the scripts first, then the trigger drafts whose
|
||||
// script is now deployed. Shared by the pipeline page's "Save all" and the AI
|
||||
// session's "Deploy pipeline", so both deploy exactly the same way.
|
||||
|
||||
/** Deploy one pipeline script draft; resolves to the new version's hash. */
|
||||
export async function deployPipelineScript(
|
||||
draft: PipelineDraft,
|
||||
workspace: string
|
||||
): Promise<string> {
|
||||
const script = structuredClone($state.snapshot(draft.script) as Script)
|
||||
script.schema = script.schema ?? emptySchema()
|
||||
try {
|
||||
const result = await inferArgs(script.language, script.content, script.schema as Schema)
|
||||
;(script as any).auto_kind = result?.auto_kind || undefined
|
||||
script.has_preprocessor = result?.has_preprocessor ?? false
|
||||
} catch {
|
||||
// Inference failures don't block deploys (the same fallback the per-pane
|
||||
// save uses): createScript is the real validation gate.
|
||||
}
|
||||
// Re-infer the lineage from the CURRENT body. `script.assets` is a snapshot
|
||||
// that isn't refreshed on edit, so a renamed/removed output would otherwise be
|
||||
// re-deployed as a phantom write edge.
|
||||
let assets: AssetWithAltAccessType[] = []
|
||||
try {
|
||||
const inferred = await inferAssets(script.language, script.content)
|
||||
if (inferred?.status !== 'error') assets = (inferred?.assets ?? []) as AssetWithAltAccessType[]
|
||||
} catch {
|
||||
// An unparsable body deploys with no lineage rather than the stale snapshot.
|
||||
}
|
||||
return await ScriptService.createScript({
|
||||
workspace,
|
||||
requestBody: {
|
||||
...script,
|
||||
language: script.language,
|
||||
description: script.description ?? '',
|
||||
// A brand-new draft has an empty hash (the backend rejects an empty hex
|
||||
// string); a draft of a deployed script chains off its hash, since
|
||||
// createScript rejects a parentless deploy to an occupied path.
|
||||
parent_hash: script.hash ? String(script.hash) : undefined,
|
||||
is_template: false,
|
||||
tag: script.tag,
|
||||
kind: script.kind as Script['kind'] | undefined,
|
||||
lock: undefined,
|
||||
assets: assets as any
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Every field a deploy sends: a draft differing from the live script in any of
|
||||
// them still has something to deploy.
|
||||
const DEPLOYED_FIELDS = [
|
||||
'content',
|
||||
'language',
|
||||
'summary',
|
||||
'description',
|
||||
'tag',
|
||||
'kind',
|
||||
'labels'
|
||||
] as const
|
||||
|
||||
/** Whether two versions of a script agree on everything a deploy sends. */
|
||||
function sameDeployedFields(a: Partial<Script>, b: Partial<Script>): boolean {
|
||||
const norm = (v: unknown) => JSON.stringify(Array.isArray(v) && v.length === 0 ? null : (v ?? null))
|
||||
return DEPLOYED_FIELDS.every(
|
||||
(f) =>
|
||||
norm(f === 'description' ? (a[f] ?? '') : a[f]) ===
|
||||
norm(f === 'description' ? (b[f] ?? '') : b[f])
|
||||
)
|
||||
}
|
||||
|
||||
/** Whether a draft equals the live script at its path, i.e. has nothing to deploy. */
|
||||
export async function matchesDeployedScript(
|
||||
path: string,
|
||||
d: PipelineDraft,
|
||||
workspace: string
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
const live = await ScriptService.getScriptByPath({ workspace, path })
|
||||
return sameDeployedFields(live, d.script)
|
||||
} catch {
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A folder's draft bundle against what is deployed, one entry per node (and the
|
||||
* trigger drafts), for a diff viewer: the unit a session reviews and deploys.
|
||||
*/
|
||||
export async function loadPipelineDraftDiff(
|
||||
workspace: string,
|
||||
folder: string
|
||||
): Promise<{ before: unknown; after: unknown }> {
|
||||
const row = await DraftService.getOwnDraft({
|
||||
workspace,
|
||||
kind: PIPELINE_DRAFT_KIND,
|
||||
path: pipelineBundlePath(folder)
|
||||
}).catch(() => undefined)
|
||||
const bundle = (row?.value ?? {}) as {
|
||||
drafts?: Array<[string, PipelineDraft]>
|
||||
triggerDrafts?: PipelineTriggerDraft[]
|
||||
}
|
||||
const drafts = Array.isArray(bundle.drafts) ? bundle.drafts : []
|
||||
const before: Record<string, string> = {}
|
||||
const after: Record<string, string> = {}
|
||||
await Promise.all(
|
||||
drafts.map(async ([path, d]) => {
|
||||
after[path] = d.script?.content ?? ''
|
||||
const live = await ScriptService.getScriptByPath({ workspace, path }).catch(() => undefined)
|
||||
if (live) before[path] = live.content
|
||||
})
|
||||
)
|
||||
const triggers = (bundle.triggerDrafts ?? []).map((t) => ({
|
||||
kind: t.kind,
|
||||
path: t.config.path,
|
||||
script_path: t.config.script_path
|
||||
}))
|
||||
return {
|
||||
before: { nodes: before },
|
||||
after: { nodes: after, ...(triggers.length > 0 ? { new_triggers: triggers } : {}) }
|
||||
}
|
||||
}
|
||||
|
||||
export type PipelineDeployOutcome = {
|
||||
savedPaths: string[]
|
||||
/** Keys of the trigger drafts that deployed. */
|
||||
savedTriggers: string[]
|
||||
/** Failures by script or trigger path. */
|
||||
errors: Map<string, string>
|
||||
}
|
||||
|
||||
/**
|
||||
* Deploy every draft of `editor` into `workspace`, dropping from it what deployed:
|
||||
* failed drafts stay so they can be fixed and retried. The open pane's unsaved
|
||||
* keystrokes are deployed with their draft.
|
||||
*/
|
||||
export async function deployPipelineDrafts(
|
||||
editor: PipelineEditorState,
|
||||
workspace: string
|
||||
): Promise<PipelineDeployOutcome> {
|
||||
// Closing the pane lands an open deployed script's edits as its draft, so they
|
||||
// deploy too. It reopens on that script once the deploy is done, unless another
|
||||
// one was opened meanwhile.
|
||||
const reopen =
|
||||
editor.liveEditPath != undefined || editor.activeDraftPath != undefined
|
||||
? await editor.closePane()
|
||||
: undefined
|
||||
const entries = [...editor.drafts.entries()]
|
||||
const errors = new Map<string, string>()
|
||||
const savedPaths: string[] = []
|
||||
const hashes = new Map<string, string>()
|
||||
// Parallel: every createScript is independent, and one bad body must not block
|
||||
// the others.
|
||||
const results = await Promise.allSettled(
|
||||
entries.map(([, d]) => deployPipelineScript(d, workspace))
|
||||
)
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
const r = results[i]
|
||||
const [path, d] = entries[i]
|
||||
if (r.status === 'fulfilled') {
|
||||
savedPaths.push(path)
|
||||
if (r.value) hashes.set(path, r.value)
|
||||
continue
|
||||
}
|
||||
// A refused draft that equals the deployed script has nothing left to deploy.
|
||||
// A duplicate refusal alone does not prove it: it is checked against every
|
||||
// past version, not just the live one.
|
||||
if (await matchesDeployedScript(path, d, workspace)) {
|
||||
savedPaths.push(path)
|
||||
continue
|
||||
}
|
||||
const msg = String((r.reason as any)?.body ?? (r.reason as any)?.message ?? r.reason)
|
||||
errors.set(
|
||||
path,
|
||||
/same hash/i.test(msg)
|
||||
? 'This is the content of an earlier version, which cannot be deployed again as is. Change anything in it (a comment will do) to deploy it.'
|
||||
: msg
|
||||
)
|
||||
}
|
||||
|
||||
// A trigger's `script_path` must already exist, so triggers whose script is
|
||||
// still an undeployed draft stay drafts.
|
||||
const ready = [...editor.triggerDrafts].filter(
|
||||
([, d]) => !editor.drafts.has(d.config.script_path) || savedPaths.includes(d.config.script_path)
|
||||
)
|
||||
const triggerResults = await Promise.all(ready.map(([, d]) => deployTriggerDraft(d, workspace)))
|
||||
const savedTriggers: string[] = []
|
||||
triggerResults.forEach((ok, i) => {
|
||||
const [key, d] = ready[i]
|
||||
if (ok) savedTriggers.push(key)
|
||||
else errors.set(d.config.path, `Could not create the ${d.kind} trigger, see the notification`)
|
||||
})
|
||||
for (const key of savedTriggers) editor.discardTriggerDraft(key)
|
||||
|
||||
if (savedPaths.length > 0) {
|
||||
// A draft edited again while it deployed (its node reopened from the canvas)
|
||||
// keeps the newer edits, now on top of the version just deployed; only a draft
|
||||
// still holding exactly what was sent is done.
|
||||
const sent = new Map(entries.map(([p, d]) => [p, d.script]))
|
||||
const next = new Map<string, PipelineDraft>()
|
||||
for (const [k, d] of editor.drafts) {
|
||||
if (!savedPaths.includes(k)) {
|
||||
next.set(k, d)
|
||||
continue
|
||||
}
|
||||
const open = editor.liveContent.scriptPath === k ? editor.liveContent.content : undefined
|
||||
const now = open === undefined ? d.script : { ...d.script, content: open }
|
||||
const was = sent.get(k)
|
||||
if (was && sameDeployedFields(now, was)) continue
|
||||
const hash = hashes.get(k)
|
||||
next.set(k, hash ? { ...d, script: { ...d.script, hash } } : d)
|
||||
}
|
||||
editor.drafts = next
|
||||
}
|
||||
if (reopen && editor.activeDraftPath === undefined && editor.selection === undefined) {
|
||||
if (editor.drafts.has(reopen)) editor.activeDraftPath = reopen
|
||||
else editor.selection = { kind: 'runnable', runnable_kind: 'script', path: reopen }
|
||||
}
|
||||
return { savedPaths, savedTriggers, errors }
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const createScript = vi.fn()
|
||||
vi.mock('$lib/gen', () => ({
|
||||
ScriptService: { createScript: (a: unknown) => createScript(a) },
|
||||
DraftService: {}
|
||||
}))
|
||||
vi.mock('$lib/infer', () => ({
|
||||
inferArgs: async () => undefined,
|
||||
inferAssets: async () => ({ status: 'ok', assets: [] })
|
||||
}))
|
||||
vi.mock('./pipelineTriggerDraftDeploy', () => ({ deployTriggerDraft: async () => true }))
|
||||
|
||||
const { deployPipelineDrafts } = await import('./pipelineDeploy.svelte')
|
||||
const { PipelineEditorState } = await import('./pipelineEditorState.svelte')
|
||||
|
||||
const draft = (content: string) => ({
|
||||
localId: 'pe-1',
|
||||
script: { path: 'f/x/n', language: 'duckdb', content, hash: 'aaaa' } as any
|
||||
})
|
||||
|
||||
describe('deployPipelineDrafts', () => {
|
||||
it('drops a draft once what it holds is deployed', async () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
|
||||
createScript.mockResolvedValueOnce('bbbb')
|
||||
await deployPipelineDrafts(pe, 'ws')
|
||||
expect(pe.drafts.has('f/x/n')).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps edits made while the deploy was in flight, on top of the new version', async () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
|
||||
createScript.mockImplementationOnce(async () => {
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 3')]])
|
||||
return 'bbbb'
|
||||
})
|
||||
await deployPipelineDrafts(pe, 'ws')
|
||||
expect(pe.drafts.get('f/x/n')?.script.content).toBe('SELECT 3')
|
||||
expect(pe.drafts.get('f/x/n')?.script.hash).toBe('bbbb')
|
||||
})
|
||||
})
|
||||
|
||||
describe('deployPipelineDrafts metadata', () => {
|
||||
it('keeps a summary edited while the deploy was in flight', async () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
|
||||
createScript.mockImplementationOnce(async () => {
|
||||
const d = draft('SELECT 2')
|
||||
pe.drafts = new Map([['f/x/n', { ...d, script: { ...d.script, summary: 'Renamed' } }]])
|
||||
return 'bbbb'
|
||||
})
|
||||
await deployPipelineDrafts(pe, 'ws')
|
||||
expect(pe.drafts.get('f/x/n')?.script.summary).toBe('Renamed')
|
||||
})
|
||||
})
|
||||
@@ -1,6 +1,8 @@
|
||||
import { tick } from 'svelte'
|
||||
import type { AssetKind, Script } from '$lib/gen'
|
||||
import type { AssetWithAltAccessType } from '$lib/components/assets/lib'
|
||||
import type { AssetGraphSelection } from './types'
|
||||
import type { AssetGraphSelection, PipelineTriggerDraft } from './types'
|
||||
import { defaultTriggerPath, triggerDraftKey } from './pipelineTriggerDrafts'
|
||||
import {
|
||||
parsePipelineAnnotations,
|
||||
type ColumnLineage,
|
||||
@@ -26,11 +28,18 @@ type LiveBodyAssets = {
|
||||
assets: AssetWithAltAccessType[]
|
||||
columnLineage?: ColumnLineage[]
|
||||
}
|
||||
type LiveContent = { scriptPath: string | undefined; content: string }
|
||||
type LiveContent = {
|
||||
scriptPath: string | undefined
|
||||
content: string
|
||||
/** The deployed script the buffer was opened from; unset for a draft. */
|
||||
base?: Script
|
||||
}
|
||||
|
||||
export class PipelineEditorState {
|
||||
/** In-flight drafts keyed by script path (manual + AI-staged). */
|
||||
drafts = $state<Map<string, PipelineDraft>>(new Map())
|
||||
/** Undeployed triggers keyed by `triggerDraftKey`, persisted in the same bundle. */
|
||||
triggerDrafts = $state<Map<string, PipelineTriggerDraft>>(new Map())
|
||||
/** Draft open in the details pane (mutually exclusive with `selection`). */
|
||||
activeDraftPath = $state<string | undefined>(undefined)
|
||||
/** The persisted node/asset selected on the canvas. */
|
||||
@@ -51,23 +60,22 @@ export class PipelineEditorState {
|
||||
* autosave hydrate when persistence is enabled. */
|
||||
loadedFromDbDraft = $state(false)
|
||||
|
||||
/** Folder this state is scoped to. Used by the in-session preview (where one
|
||||
* instance is reused across editor hide/show) to detect a retarget to a
|
||||
* different folder and reset, so stale drafts don't bleed across folders. */
|
||||
/** Folder this state is scoped to. The route page compares it on a folder
|
||||
* switch and resets, so one folder's drafts don't bleed into the next. */
|
||||
folder = $state<string | undefined>(undefined)
|
||||
|
||||
/** True once the DB draft bundle for the current folder has been hydrated
|
||||
* into this instance. Gated per-instance (not per component mount) so the
|
||||
* in-session preview hydrates ONCE when its runtime is fresh and then keeps
|
||||
* the in-memory drafts across editor hide/show — re-reading the DB on every
|
||||
* remount would race a not-yet-flushed autosave and drop a just-staged draft.
|
||||
* Reset to false on a folder retarget so the new folder re-hydrates. */
|
||||
* into this instance. Gated per-instance (not per component mount) so an AI
|
||||
* session's folder hydrates ONCE and then keeps its in-memory drafts across
|
||||
* editor remounts — re-reading the DB on every remount would race a
|
||||
* not-yet-flushed autosave and drop a just-staged draft. `reset` clears it
|
||||
* so a new folder re-hydrates. */
|
||||
hydratedFromDb = $state(false)
|
||||
|
||||
/** Clear all in-flight state. Used when the session preview retargets a
|
||||
* different pipeline folder (a same-folder remount keeps the drafts). */
|
||||
/** Clear all in-flight state, for the route page switching folders. */
|
||||
reset = () => {
|
||||
this.drafts = new Map()
|
||||
this.triggerDrafts = new Map()
|
||||
this.activeDraftPath = undefined
|
||||
this.selection = undefined
|
||||
this.clearLiveOverlays()
|
||||
@@ -84,6 +92,18 @@ export class PipelineEditorState {
|
||||
return `pe-${this.#nextDraftLocalId}`
|
||||
}
|
||||
|
||||
// A deployed script with unsaved edits is autosaved as a draft before the pane
|
||||
// promotes it, so both must give it the same id.
|
||||
#promotedLocalIds = new Map<string, string>()
|
||||
promotedDraftLocalId = (path: string): string => {
|
||||
let id = this.#promotedLocalIds.get(path)
|
||||
if (!id) {
|
||||
id = this.newDraftLocalId()
|
||||
this.#promotedLocalIds.set(path, id)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
handleAnnotationsChange = (scriptPath: string | undefined, annotations: PipelineAnnotations) => {
|
||||
this.liveAnnotations = { scriptPath, annotations }
|
||||
}
|
||||
@@ -94,8 +114,34 @@ export class PipelineEditorState {
|
||||
) => {
|
||||
this.liveBodyAssets = { scriptPath, assets, columnLineage }
|
||||
}
|
||||
handleContentChange = (scriptPath: string | undefined, content: string) => {
|
||||
this.liveContent = { scriptPath, content }
|
||||
handleContentChange = (scriptPath: string | undefined, content: string, base?: Script) => {
|
||||
this.liveContent = { scriptPath, content, base }
|
||||
}
|
||||
|
||||
/** The open deployed script's unsaved edits, which live only in `liveContent`
|
||||
* until its pane closes: the path, or undefined when there are none. */
|
||||
get liveEditPath(): string | undefined {
|
||||
const { scriptPath, content, base } = this.liveContent
|
||||
if (scriptPath == undefined || !base || this.drafts.has(scriptPath)) return undefined
|
||||
return content !== (base.content ?? '') ? scriptPath : undefined
|
||||
}
|
||||
|
||||
/** Closes the open pane and waits for its edits to land as a draft, as closing
|
||||
* it by hand does: what is deployed or discarded next then includes them, and
|
||||
* no pane is left holding a stale copy. Returns the path the pane was on. */
|
||||
closePane = async (): Promise<string | undefined> => {
|
||||
const path =
|
||||
this.activeDraftPath ?? (this.selection?.kind === 'runnable' ? this.selection.path : undefined)
|
||||
if (this.activeDraftPath === undefined && this.selection === undefined) return undefined
|
||||
this.activeDraftPath = undefined
|
||||
this.selection = undefined
|
||||
// The pane unmounts on the next flush; its save-back commits a microtask later.
|
||||
await tick()
|
||||
await new Promise<void>((resolve) => queueMicrotask(resolve))
|
||||
// Its buffer now lives in the draft; left here, autosave would serialize it
|
||||
// again once that draft is deployed or discarded.
|
||||
this.clearLiveOverlays()
|
||||
return path
|
||||
}
|
||||
|
||||
clearLiveOverlays = () => {
|
||||
@@ -125,6 +171,72 @@ export class PipelineEditorState {
|
||||
next.delete(path)
|
||||
this.drafts = next
|
||||
this.forgetPath(path)
|
||||
this.discardTriggerDraftsFor(path)
|
||||
}
|
||||
|
||||
/** Summary and labels of a script draft, edited from the pane header. */
|
||||
setDraftMeta = (path: string, meta: { summary: string; labels: string[] | undefined }) => {
|
||||
const d = this.drafts.get(path)
|
||||
if (!d) return
|
||||
this.drafts = new Map(this.drafts).set(path, {
|
||||
...d,
|
||||
script: { ...d.script, summary: meta.summary, labels: meta.labels }
|
||||
})
|
||||
}
|
||||
|
||||
/** Add or replace a trigger draft; `previousKey` is the entry being edited, which
|
||||
* a rename drops. False, and nothing changes, when the path is another draft's:
|
||||
* both would deploy to one trigger. */
|
||||
setTriggerDraft = (d: PipelineTriggerDraft, previousKey?: string): boolean => {
|
||||
const key = triggerDraftKey(d.kind, d.config.path)
|
||||
if (key !== previousKey && this.triggerDrafts.has(key)) return false
|
||||
const next = new Map(this.triggerDrafts)
|
||||
if (previousKey) next.delete(previousKey)
|
||||
next.set(key, d)
|
||||
this.triggerDrafts = next
|
||||
return true
|
||||
}
|
||||
|
||||
discardTriggerDraft = (key: string) => {
|
||||
if (!this.triggerDrafts.has(key)) return
|
||||
const next = new Map(this.triggerDrafts)
|
||||
next.delete(key)
|
||||
this.triggerDrafts = next
|
||||
}
|
||||
|
||||
/** A trigger draft cannot outlive the script draft it was configured with:
|
||||
* deploying it would target a path nothing is deployed at. */
|
||||
discardTriggerDraftsFor = (scriptPath: string) => {
|
||||
const next = new Map(
|
||||
[...this.triggerDrafts].filter(([, d]) => d.config.script_path !== scriptPath)
|
||||
)
|
||||
if (next.size !== this.triggerDrafts.size) this.triggerDrafts = next
|
||||
}
|
||||
|
||||
/** Follow a script draft rename, keeping default `<script>_<kind>` names in step. */
|
||||
/** Returns the trigger path another draft already holds, and changes nothing,
|
||||
* when a renamed default name would land on it. */
|
||||
retargetTriggerDrafts = (oldScriptPath: string, newScriptPath: string): string | undefined => {
|
||||
const moved = new Map<string, PipelineTriggerDraft>()
|
||||
const kept = new Map<string, PipelineTriggerDraft>()
|
||||
for (const [k, d] of this.triggerDrafts) {
|
||||
if (d.config.script_path !== oldScriptPath) {
|
||||
kept.set(k, d)
|
||||
continue
|
||||
}
|
||||
const path =
|
||||
d.config.path === defaultTriggerPath(oldScriptPath, d.kind)
|
||||
? defaultTriggerPath(newScriptPath, d.kind)
|
||||
: d.config.path
|
||||
moved.set(triggerDraftKey(d.kind, path), {
|
||||
...d,
|
||||
config: { ...d.config, path, script_path: newScriptPath }
|
||||
})
|
||||
}
|
||||
if (moved.size === 0) return undefined
|
||||
for (const [k, d] of moved) if (kept.has(k)) return d.config.path
|
||||
this.triggerDrafts = new Map([...kept, ...moved])
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** Commit body edits + inferred outputs back into the drafts Map on pane
|
||||
@@ -147,7 +259,7 @@ export class PipelineEditorState {
|
||||
if (!snapshot.script) return
|
||||
const next = new Map(this.drafts)
|
||||
next.set(p, {
|
||||
localId: this.newDraftLocalId(),
|
||||
localId: this.promotedDraftLocalId(p),
|
||||
script: snapshot.script,
|
||||
outputAssets: snapshot.writes.length > 0 ? snapshot.writes : undefined,
|
||||
inputAssets: snapshot.reads
|
||||
|
||||
@@ -104,3 +104,72 @@ describe('PipelineEditorState.handleDraftPersist — read capture', () => {
|
||||
expect(pe.drafts.get('f/x/n')?.inputAssets).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('PipelineEditorState trigger drafts', () => {
|
||||
it('follow their script draft: renamed with it, discarded with it', () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 1')]])
|
||||
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/n_kafka', script_path: 'f/x/n' } })
|
||||
pe.setTriggerDraft({ kind: 'schedule', config: { path: 'f/x/nightly', script_path: 'f/x/n' } })
|
||||
|
||||
pe.retargetTriggerDrafts('f/x/n', 'f/x/m')
|
||||
// A default `<script>_<kind>` name follows the script; a chosen one stays.
|
||||
expect([...pe.triggerDrafts.keys()].sort()).toEqual([
|
||||
'kafka:f/x/m_kafka',
|
||||
'schedule:f/x/nightly'
|
||||
])
|
||||
expect([...pe.triggerDrafts.values()].every((d) => d.config.script_path === 'f/x/m')).toBe(true)
|
||||
|
||||
pe.drafts = new Map([['f/x/m', draft('SELECT 1')]])
|
||||
pe.discardDraft('f/x/m')
|
||||
expect(pe.triggerDrafts.size).toBe(0)
|
||||
})
|
||||
|
||||
it('refuse a path another draft holds, but let a draft keep its own', () => {
|
||||
const pe = new PipelineEditorState()
|
||||
const a = { kind: 'kafka' as const, config: { path: 'f/x/k', script_path: 'f/x/a' } }
|
||||
expect(pe.setTriggerDraft(a)).toBe(true)
|
||||
expect(
|
||||
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/k', script_path: 'f/x/b' } })
|
||||
).toBe(false)
|
||||
expect(pe.triggerDrafts.get('kafka:f/x/k')?.config.script_path).toBe('f/x/a')
|
||||
expect(pe.setTriggerDraft({ ...a, config: { ...a.config, topics: ['t'] } }, 'kafka:f/x/k')).toBe(
|
||||
true
|
||||
)
|
||||
})
|
||||
|
||||
it('refuse a script rename that moves a default name onto another draft', () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/a_kafka', script_path: 'f/x/a' } })
|
||||
pe.setTriggerDraft({ kind: 'kafka', config: { path: 'f/x/b_kafka', script_path: 'f/x/c' } })
|
||||
expect(pe.retargetTriggerDrafts('f/x/a', 'f/x/b')).toBe('f/x/b_kafka')
|
||||
expect([...pe.triggerDrafts.keys()].sort()).toEqual(['kafka:f/x/a_kafka', 'kafka:f/x/b_kafka'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('PipelineEditorState.liveEditPath', () => {
|
||||
const base = { path: 'f/x/n', language: 'duckdb', content: 'SELECT 1' } as PipelineDraft['script']
|
||||
|
||||
it('counts an open deployed script with unsaved edits, until it has a draft', () => {
|
||||
const pe = new PipelineEditorState()
|
||||
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 1', base }
|
||||
expect(pe.liveEditPath).toBeUndefined()
|
||||
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 2', base }
|
||||
expect(pe.liveEditPath).toBe('f/x/n')
|
||||
pe.drafts = new Map([['f/x/n', draft('SELECT 2')]])
|
||||
expect(pe.liveEditPath).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('PipelineEditorState.closePane', () => {
|
||||
it("leaves no open buffer behind for autosave to bring back once the draft is gone", async () => {
|
||||
const pe = new PipelineEditorState()
|
||||
const base = { path: 'f/x/n', language: 'duckdb', content: 'SELECT 1' } as PipelineDraft['script']
|
||||
pe.selection = { kind: 'runnable', runnable_kind: 'script', path: 'f/x/n' }
|
||||
pe.liveContent = { scriptPath: 'f/x/n', content: 'SELECT 2', base }
|
||||
expect(await pe.closePane()).toBe('f/x/n')
|
||||
expect(pe.selection).toBeUndefined()
|
||||
expect(pe.liveEditPath).toBeUndefined()
|
||||
expect(pe.liveContent.scriptPath).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
import { usedTriggerKinds } from '$lib/stores'
|
||||
import { formatCron } from '$lib/utils'
|
||||
import { saveScheduleFromCfg } from '$lib/components/flows/scheduleUtils'
|
||||
import { saveEmailTriggerFromCfg } from '$lib/components/triggers/email/utils'
|
||||
import { saveKafkaTriggerFromCfg } from '$lib/components/triggers/kafka/utils'
|
||||
import { saveMqttTriggerFromCfg } from '$lib/components/triggers/mqtt/utils'
|
||||
import { saveAmqpTriggerFromCfg } from '$lib/components/triggers/amqp/utils'
|
||||
import { saveNatsTriggerFromCfg } from '$lib/components/triggers/nats/utils'
|
||||
import { savePostgresTriggerFromCfg } from '$lib/components/triggers/postgres/utils'
|
||||
import { saveSqsTriggerFromCfg } from '$lib/components/triggers/sqs/utils'
|
||||
import { saveGcpTriggerFromCfg } from '$lib/components/triggers/gcp/utils'
|
||||
import {
|
||||
AmqpTriggerService,
|
||||
EmailTriggerService,
|
||||
GcpTriggerService,
|
||||
KafkaTriggerService,
|
||||
MqttTriggerService,
|
||||
NatsTriggerService,
|
||||
PostgresTriggerService,
|
||||
ScheduleService,
|
||||
SqsTriggerService
|
||||
} from '$lib/gen'
|
||||
import type { NativeTriggerKind, PipelineTriggerDraft } from './types'
|
||||
|
||||
/** Creates the trigger through the same helper its editor saves with; that helper
|
||||
* reports the outcome as a toast and resolves to whether it succeeded. */
|
||||
export function deployTriggerDraft(d: PipelineTriggerDraft, workspace: string): Promise<boolean> {
|
||||
const cfg = d.config
|
||||
switch (d.kind) {
|
||||
case 'schedule':
|
||||
return saveScheduleFromCfg({ ...cfg, schedule: formatCron(cfg.schedule) }, false, workspace)
|
||||
case 'email':
|
||||
return saveEmailTriggerFromCfg('', cfg, false, workspace, false, usedTriggerKinds)
|
||||
case 'kafka':
|
||||
return saveKafkaTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'mqtt':
|
||||
return saveMqttTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'amqp':
|
||||
return saveAmqpTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'nats':
|
||||
return saveNatsTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'postgres':
|
||||
return savePostgresTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'sqs':
|
||||
return saveSqsTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
case 'gcp':
|
||||
return saveGcpTriggerFromCfg('', cfg, false, workspace, usedTriggerKinds)
|
||||
}
|
||||
}
|
||||
|
||||
/** Deletes a deployed trigger row. False for the kinds that have none (webhook,
|
||||
* data upload). */
|
||||
export async function deleteTriggerRow(
|
||||
kind: NativeTriggerKind,
|
||||
path: string,
|
||||
workspace: string
|
||||
): Promise<boolean> {
|
||||
switch (kind) {
|
||||
case 'schedule':
|
||||
await ScheduleService.deleteSchedule({ workspace, path })
|
||||
return true
|
||||
case 'kafka':
|
||||
await KafkaTriggerService.deleteKafkaTrigger({ workspace, path })
|
||||
return true
|
||||
case 'mqtt':
|
||||
await MqttTriggerService.deleteMqttTrigger({ workspace, path })
|
||||
return true
|
||||
case 'amqp':
|
||||
await AmqpTriggerService.deleteAmqpTrigger({ workspace, path })
|
||||
return true
|
||||
case 'nats':
|
||||
await NatsTriggerService.deleteNatsTrigger({ workspace, path })
|
||||
return true
|
||||
case 'postgres':
|
||||
await PostgresTriggerService.deletePostgresTrigger({ workspace, path })
|
||||
return true
|
||||
case 'sqs':
|
||||
await SqsTriggerService.deleteSqsTrigger({ workspace, path })
|
||||
return true
|
||||
case 'gcp':
|
||||
await GcpTriggerService.deleteGcpTrigger({ workspace, path })
|
||||
return true
|
||||
case 'email':
|
||||
await EmailTriggerService.deleteEmailTrigger({ workspace, path })
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
import type { PipelineTriggerDraftKind } from './types'
|
||||
|
||||
export const DRAFTABLE_TRIGGER_KINDS: readonly PipelineTriggerDraftKind[] = [
|
||||
'schedule',
|
||||
'email',
|
||||
'kafka',
|
||||
'mqtt',
|
||||
'amqp',
|
||||
'nats',
|
||||
'postgres',
|
||||
'sqs',
|
||||
'gcp'
|
||||
]
|
||||
|
||||
export function isDraftableTriggerKind(kind: string): kind is PipelineTriggerDraftKind {
|
||||
return (DRAFTABLE_TRIGGER_KINDS as readonly string[]).includes(kind)
|
||||
}
|
||||
|
||||
/** Drafts are keyed by kind too: two kinds may share a trigger path. */
|
||||
export function triggerDraftKey(kind: string, path: string): string {
|
||||
return `${kind}:${path}`
|
||||
}
|
||||
|
||||
export function defaultTriggerPath(scriptPath: string, kind: PipelineTriggerDraftKind): string {
|
||||
return `${scriptPath}_${kind}`
|
||||
}
|
||||
@@ -2,6 +2,7 @@ import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
resolveGraph,
|
||||
computeMutedReadKeys,
|
||||
pipelineNodeErrors,
|
||||
dbtAssociations,
|
||||
type ResolveGraphInput
|
||||
} from './resolveGraph'
|
||||
@@ -492,6 +493,49 @@ describe('resolveGraph', () => {
|
||||
expect(r.edges.some((e) => e.asset_path === 'main/out' && e.access_type === 'w')).toBe(true)
|
||||
})
|
||||
|
||||
it('drops a persisted asset the open script renamed away, unless something still uses it', () => {
|
||||
const write = (runnable_path: string, asset_path: string) => ({
|
||||
runnable_path,
|
||||
runnable_kind: 'script' as const,
|
||||
asset_kind: 's3object' as const,
|
||||
asset_path,
|
||||
access_type: 'w' as const
|
||||
})
|
||||
const base = baseGraph({
|
||||
assets: [
|
||||
{ kind: 's3object', path: '/old.json' },
|
||||
{ kind: 's3object', path: '/shared.json' },
|
||||
{ kind: 's3object', path: '/unreferenced.json' }
|
||||
],
|
||||
runnables: [
|
||||
{ path: 'f/x/prod', usage_kind: 'script' },
|
||||
{ path: 'f/x/other', usage_kind: 'script' }
|
||||
],
|
||||
edges: [write('f/x/prod', '/old.json'), write('f/x/prod', '/shared.json')],
|
||||
triggers: [
|
||||
{
|
||||
trigger_kind: 'asset',
|
||||
asset_kind: 's3object',
|
||||
asset_path: '/shared.json',
|
||||
runnable_kind: 'script',
|
||||
runnable_path: 'f/x/other'
|
||||
}
|
||||
]
|
||||
})
|
||||
const r = resolveGraph(
|
||||
input({
|
||||
base,
|
||||
liveAnnotations: { scriptPath: 'f/x/prod', annotations: ann() },
|
||||
liveBodyAssets: { scriptPath: 'f/x/prod', assets: [s3('/new.json', 'w')] }
|
||||
})
|
||||
)
|
||||
expect(r.assets.map((a) => a.path).sort()).toEqual([
|
||||
'/new.json',
|
||||
'/shared.json',
|
||||
'/unreferenced.json'
|
||||
])
|
||||
})
|
||||
|
||||
it('editing a saved scd2 producer keeps both the base and _current persisted write edges', () => {
|
||||
// Deploy persists a write to both `main/dim` and `main/dim_current`.
|
||||
// Opening the producer for editing must not judge the companion `_current`
|
||||
@@ -656,6 +700,31 @@ describe('resolveGraph', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('a trigger draft stands in for its draft script’s missing trigger of that kind', () => {
|
||||
const drafts = new Map([['f/x/new', { script: { content: '// on schedule\n// on kafka' } }]])
|
||||
const r = resolveGraph(
|
||||
input({
|
||||
drafts,
|
||||
triggerDrafts: [
|
||||
{ kind: 'schedule', config: { path: 'f/x/new_schedule', script_path: 'f/x/new' } }
|
||||
]
|
||||
})
|
||||
)
|
||||
expect(r.triggers.filter((t) => t.trigger_kind === 'kafka')).toEqual([
|
||||
expect.objectContaining({ runnable_path: 'f/x/new', missing: true })
|
||||
])
|
||||
expect(r.triggers.filter((t) => t.trigger_kind === 'schedule')).toEqual([
|
||||
{
|
||||
trigger_kind: 'schedule',
|
||||
path: 'f/x/new_schedule',
|
||||
runnable_kind: 'script',
|
||||
runnable_path: 'f/x/new',
|
||||
unsaved: true,
|
||||
draft: true
|
||||
}
|
||||
])
|
||||
})
|
||||
|
||||
it('macro edges: base passes through; live `// use` adds an unsaved via_use edge', () => {
|
||||
const base = baseGraph({
|
||||
runnables: [
|
||||
@@ -884,11 +953,11 @@ describe('live buffer overlays (open script)', () => {
|
||||
access_type: 'w',
|
||||
unsaved: true
|
||||
})
|
||||
// …the stale write edge is dropped, but the deployed dataset node stays.
|
||||
// …the stale write edge is dropped, and with nothing else using it, so is the node.
|
||||
expect(
|
||||
r.edges.filter((e) => e.asset_path === 'main/orders' && e.runnable_path === 'f/x/prod')
|
||||
).toEqual([])
|
||||
expect(r.assets).toContainEqual({ kind: 'ducklake', path: 'main/orders' })
|
||||
expect(r.assets).not.toContainEqual({ kind: 'ducklake', path: 'main/orders' })
|
||||
})
|
||||
|
||||
it('open saved script: unchanged materialize target dedups against the persisted write edge', () => {
|
||||
@@ -1083,9 +1152,7 @@ describe('dbtAssociations', () => {
|
||||
it('counts only what the project materializes as its transforms', () => {
|
||||
const { writesByOwner } = dbtAssociations(runnables, edges)
|
||||
// Hovering the project highlights the models it builds, not its inputs.
|
||||
expect([...(writesByOwner.get('script:f/a/dbtproj') ?? [])]).toEqual([
|
||||
'asset:dbt:wh/s/model_a'
|
||||
])
|
||||
expect([...(writesByOwner.get('script:f/a/dbtproj') ?? [])]).toEqual(['asset:dbt:wh/s/model_a'])
|
||||
})
|
||||
|
||||
it('ignores runnables that are not dbt', () => {
|
||||
@@ -1162,8 +1229,160 @@ describe('dbtAssociations with two writers of one relation', () => {
|
||||
writeBy('f/a/downstream')
|
||||
])
|
||||
expect(writesByOwner.get('script:f/a/upstream')).toEqual(new Set(['asset:dbt:wh/s/shared']))
|
||||
expect(writesByOwner.get('script:f/a/downstream')).toEqual(
|
||||
new Set(['asset:dbt:wh/s/shared'])
|
||||
expect(writesByOwner.get('script:f/a/downstream')).toEqual(new Set(['asset:dbt:wh/s/shared']))
|
||||
})
|
||||
})
|
||||
|
||||
describe('pipelineNodeErrors', () => {
|
||||
const edge = (runnable_path: string, asset_path: string, access_type: 'r' | 'w') => ({
|
||||
runnable_path,
|
||||
runnable_kind: 'script' as const,
|
||||
asset_kind: 'ducklake' as const,
|
||||
asset_path,
|
||||
access_type
|
||||
})
|
||||
const graph = baseGraph({
|
||||
assets: [
|
||||
{ kind: 'ducklake', path: 'main/orphan' },
|
||||
{ kind: 'ducklake', path: 'main/produced' },
|
||||
{ kind: 'ducklake', path: 'main/external' }
|
||||
],
|
||||
runnables: [
|
||||
{ path: 'f/x/prod', usage_kind: 'script' },
|
||||
{ path: 'f/x/cons', usage_kind: 'script' }
|
||||
],
|
||||
edges: [
|
||||
edge('f/x/prod', 'main/produced', 'w'),
|
||||
edge('f/x/cons', 'main/produced', 'r'),
|
||||
edge('f/x/cons', 'main/external', 'r')
|
||||
]
|
||||
})
|
||||
|
||||
it('flags an unused asset, and a `// on` with no producer and its script, but not a plain read', () => {
|
||||
const onRefs = (...paths: string[]) =>
|
||||
paths.map((path) => ({ kind: 'ducklake' as const, path }))
|
||||
const none = pipelineNodeErrors(graph, new Map())
|
||||
expect([...none.assets.keys()]).toEqual(['ducklake:main/orphan'])
|
||||
expect(none.scripts.size).toBe(0)
|
||||
const errors = pipelineNodeErrors(
|
||||
graph,
|
||||
new Map([['f/x/cons', onRefs('main/produced', 'main/external')]])
|
||||
)
|
||||
// The subscriber is at fault, not the asset nothing writes.
|
||||
expect([...errors.assets.keys()]).toEqual(['ducklake:main/orphan'])
|
||||
expect([...errors.scripts.keys()]).toEqual(['f/x/cons'])
|
||||
expect([...errors.unwritten.keys()]).toEqual(['ducklake:main/external'])
|
||||
})
|
||||
|
||||
it('flags a pipeline script nothing starts, but not a macro library or a triggered one', () => {
|
||||
const g = baseGraph({
|
||||
runnables: [
|
||||
{ path: 'f/x/stray', usage_kind: 'script', in_pipeline: true },
|
||||
{ path: 'f/x/lib', usage_kind: 'script', in_pipeline: true, macros: [{ name: 'm' }] },
|
||||
{ path: 'f/x/timed', usage_kind: 'script', in_pipeline: true }
|
||||
] as any,
|
||||
triggers: [
|
||||
{
|
||||
trigger_kind: 'schedule',
|
||||
path: 'f/x/timed_schedule',
|
||||
runnable_kind: 'script',
|
||||
runnable_path: 'f/x/timed'
|
||||
}
|
||||
]
|
||||
})
|
||||
expect([...pipelineNodeErrors(g, new Map()).untriggered.keys()]).toEqual(['f/x/stray'])
|
||||
})
|
||||
|
||||
it('flags a script only started by reads of assets nothing writes', () => {
|
||||
const g = baseGraph({
|
||||
assets: [{ kind: 'ducklake', path: 'main/raw' }],
|
||||
runnables: [{ path: 'f/x/reader', usage_kind: 'script', in_pipeline: true }] as any,
|
||||
edges: [
|
||||
{
|
||||
runnable_path: 'f/x/reader',
|
||||
runnable_kind: 'script',
|
||||
asset_kind: 'ducklake',
|
||||
asset_path: 'main/raw',
|
||||
access_type: 'r'
|
||||
}
|
||||
],
|
||||
triggers: [
|
||||
{
|
||||
trigger_kind: 'asset',
|
||||
asset_kind: 'ducklake',
|
||||
asset_path: 'main/raw',
|
||||
runnable_kind: 'script',
|
||||
runnable_path: 'f/x/reader'
|
||||
}
|
||||
]
|
||||
})
|
||||
expect(pipelineNodeErrors(g, new Map()).untriggered.get('f/x/reader')).toBe('unwritten-inputs')
|
||||
expect(
|
||||
pipelineNodeErrors(g, new Map(), new Set(['ducklake:main/raw'])).untriggered.size
|
||||
).toBe(0)
|
||||
})
|
||||
|
||||
it('does not count a flow write as starting an asset-triggered script', () => {
|
||||
const g = baseGraph({
|
||||
assets: [{ kind: 'ducklake', path: 'main/raw' }],
|
||||
runnables: [
|
||||
{ path: 'f/x/reader', usage_kind: 'script', in_pipeline: true },
|
||||
{ path: 'f/x/loader', usage_kind: 'flow' }
|
||||
] as any,
|
||||
edges: [
|
||||
{
|
||||
runnable_path: 'f/x/loader',
|
||||
runnable_kind: 'flow',
|
||||
asset_kind: 'ducklake',
|
||||
asset_path: 'main/raw',
|
||||
access_type: 'w'
|
||||
}
|
||||
],
|
||||
triggers: [
|
||||
{
|
||||
trigger_kind: 'asset',
|
||||
asset_kind: 'ducklake',
|
||||
asset_path: 'main/raw',
|
||||
runnable_kind: 'script',
|
||||
runnable_path: 'f/x/reader'
|
||||
}
|
||||
]
|
||||
} as any)
|
||||
expect(pipelineNodeErrors(g, new Map()).untriggered.get('f/x/reader')).toBe('unwritten-inputs')
|
||||
})
|
||||
|
||||
it('clears a subscription the workspace writes outside the pipeline', () => {
|
||||
const errors = pipelineNodeErrors(
|
||||
graph,
|
||||
new Map([['f/x/cons', [{ kind: 'ducklake' as const, path: 'main/external' }]]]),
|
||||
new Set(['ducklake:main/external'])
|
||||
)
|
||||
expect(errors.scripts.size).toBe(0)
|
||||
expect(errors.unwritten.size).toBe(0)
|
||||
})
|
||||
|
||||
it('treats dbt models as used and written by their (hidden) project, but not dbt sources', () => {
|
||||
const dbt = (path: string, resource_type: string) => ({
|
||||
kind: 'dbt' as const,
|
||||
path,
|
||||
dbt: { unique_id: `x.${path}`, resource_type } as any
|
||||
})
|
||||
const errors = pipelineNodeErrors(
|
||||
baseGraph({
|
||||
assets: [dbt('mart', 'model'), dbt('raw', 'source')],
|
||||
runnables: [{ path: 'f/x/s', usage_kind: 'script' }]
|
||||
}),
|
||||
new Map([
|
||||
[
|
||||
'f/x/s',
|
||||
[
|
||||
{ kind: 'dbt', path: 'mart' },
|
||||
{ kind: 'dbt', path: 'raw' }
|
||||
]
|
||||
]
|
||||
])
|
||||
)
|
||||
expect([...errors.unwritten.keys()]).toEqual(['dbt:raw'])
|
||||
expect([...errors.scripts.keys()]).toEqual(['f/x/s'])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,4 +1,9 @@
|
||||
import type { AssetGraphMacroEdge, AssetGraphResponse, NativeTriggerKind } from './types'
|
||||
import type {
|
||||
AssetGraphMacroEdge,
|
||||
AssetGraphResponse,
|
||||
NativeTriggerKind,
|
||||
PipelineTriggerDraft
|
||||
} from './types'
|
||||
import {
|
||||
mergeColumnLineage,
|
||||
parsePipelineAnnotations,
|
||||
@@ -46,6 +51,8 @@ export type ResolveGraphInput = {
|
||||
* annotation has no matching trigger row in `base.triggers`.
|
||||
*/
|
||||
annotatedNativeKindsByPath: Map<string, Set<NativeTriggerKind>>
|
||||
/** Undeployed triggers; each stands in for its script's missing trigger of that kind. */
|
||||
triggerDrafts?: Iterable<PipelineTriggerDraft>
|
||||
}
|
||||
|
||||
/** Mutable bag the sequential passes accumulate the resolved graph into. */
|
||||
@@ -264,16 +271,200 @@ export function resolveGraph(input: ResolveGraphInput): AssetGraphResponse {
|
||||
// there is nothing to show (also keeps the no-macros response shape
|
||||
// byte-identical to before the feature).
|
||||
const macroEdges = resolveMacroEdges(input)
|
||||
const triggers = withTriggerDrafts(
|
||||
[...baseTriggers, ...acc.extraTriggers],
|
||||
input.triggerDrafts ?? []
|
||||
)
|
||||
return {
|
||||
...base,
|
||||
assets: acc.assets,
|
||||
assets: dropOrphanedAssets(acc.assets, base, acc.edges, triggers),
|
||||
runnables: acc.runnables,
|
||||
edges: acc.edges,
|
||||
triggers: [...baseTriggers, ...acc.extraTriggers],
|
||||
triggers,
|
||||
...(macroEdges.length > 0 || base.macro_edges ? { macro_edges: macroEdges } : {})
|
||||
}
|
||||
}
|
||||
|
||||
function withTriggerDrafts(
|
||||
triggers: AssetGraphResponse['triggers'],
|
||||
triggerDrafts: Iterable<PipelineTriggerDraft>
|
||||
): AssetGraphResponse['triggers'] {
|
||||
const drafts = [...triggerDrafts]
|
||||
if (drafts.length === 0) return triggers
|
||||
const drafted = new Set(drafts.map((d) => `${d.kind}:${d.config.script_path}`))
|
||||
return [
|
||||
...triggers.filter(
|
||||
(t) =>
|
||||
!(
|
||||
t.trigger_kind !== 'asset' &&
|
||||
t.missing &&
|
||||
t.runnable_kind === 'script' &&
|
||||
drafted.has(`${t.trigger_kind}:${t.runnable_path}`)
|
||||
)
|
||||
),
|
||||
...drafts.map((d) => ({
|
||||
trigger_kind: d.kind,
|
||||
path: d.config.path,
|
||||
runnable_kind: 'script' as const,
|
||||
runnable_path: d.config.script_path,
|
||||
unsaved: true,
|
||||
draft: true,
|
||||
...(d.kind === 'schedule'
|
||||
? { schedule: d.config.schedule as string, timezone: d.config.timezone as string }
|
||||
: {}),
|
||||
...(d.config.summary ? { summary: d.config.summary as string } : {})
|
||||
}))
|
||||
]
|
||||
}
|
||||
|
||||
/** Why a node is red, and what the Fix actions act on. */
|
||||
export type PipelineNodeErrors = {
|
||||
/** `kind:path` → reason. */
|
||||
assets: Map<string, string>
|
||||
/** Script path → reason. */
|
||||
scripts: Map<string, string>
|
||||
/** Subscribed assets nothing writes: `kind:path` → the asset and its subscribers. */
|
||||
unwritten: Map<string, { asset: { kind: AssetKind; path: string }; subscribers: string[] }>
|
||||
/** Script path → the unwritten asset it runs on (its first, when several). */
|
||||
waitsOn: Map<string, { kind: AssetKind; path: string }>
|
||||
/** Pipeline scripts nothing starts → why: no trigger at all, or only asset
|
||||
* triggers on assets nothing writes (an auto-triggering read included). */
|
||||
untriggered: Map<string, 'no-trigger' | 'unwritten-inputs'>
|
||||
/** Assets some asset trigger waits on that nothing in the pipeline writes: the
|
||||
* ones to look up across the workspace for `writtenElsewhere`. */
|
||||
unwrittenTriggerAssets: Set<string>
|
||||
}
|
||||
|
||||
/**
|
||||
* Nodes the pipeline cannot work with, with the reason. Assets (keyed
|
||||
* `kind:path`): one nothing uses at all. Scripts (keyed by path): one
|
||||
* subscribed (`// on <asset>`) to an asset nothing writes, in the pipeline or,
|
||||
* per `writtenElsewhere`, by a script anywhere else in the workspace, since the subscription can never fire; the asset itself is
|
||||
* fine, and is reported in `unwritten`. A body read of an external asset is fine and is not passed here:
|
||||
* only the explicit `// on` annotations, per script path, count as subscriptions.
|
||||
*/
|
||||
export function pipelineNodeErrors(
|
||||
graph: AssetGraphResponse,
|
||||
explicitOnByPath: Map<string, Array<{ kind: AssetKind; path: string }>>,
|
||||
writtenElsewhere: ReadonlySet<string> = new Set()
|
||||
): PipelineNodeErrors {
|
||||
const referenced = referencedAssetKeys(graph)
|
||||
// Only a script's write fires an asset trigger: a flow's never does.
|
||||
const written = new Set(
|
||||
graph.edges
|
||||
.filter(
|
||||
(e) => e.runnable_kind === 'script' && (e.access_type === 'w' || e.access_type === 'rw')
|
||||
)
|
||||
.map((e) => `${e.asset_kind}:${e.asset_path}`)
|
||||
)
|
||||
for (const k of writtenElsewhere) written.add(k)
|
||||
// The dbt project's own edges are hidden from the pipeline graph (see
|
||||
// `hideDbtRunnables`), so its relations are judged by their provenance: the
|
||||
// project uses every one, and materializes all but its sources.
|
||||
for (const a of graph.assets) {
|
||||
if (!a.dbt) continue
|
||||
referenced.add(`${a.kind}:${a.path}`)
|
||||
if (a.dbt.resource_type !== 'source') written.add(`${a.kind}:${a.path}`)
|
||||
}
|
||||
const errors = new Map<string, string>()
|
||||
const scriptErrors = new Map<string, string>()
|
||||
const unwritten: PipelineNodeErrors['unwritten'] = new Map()
|
||||
const waitsOn: PipelineNodeErrors['waitsOn'] = new Map()
|
||||
for (const a of graph.assets) {
|
||||
const key = `${a.kind}:${a.path}`
|
||||
if (!referenced.has(key)) errors.set(key, 'nothing in the pipeline uses it')
|
||||
}
|
||||
const scripts = new Set(
|
||||
graph.runnables.filter((r) => r.usage_kind === 'script').map((r) => r.path)
|
||||
)
|
||||
for (const [path, refs] of explicitOnByPath) {
|
||||
if (!scripts.has(path)) continue
|
||||
for (const ref of refs) {
|
||||
const key = `${ref.kind}:${ref.path}`
|
||||
if (written.has(key)) continue
|
||||
const entry = unwritten.get(key) ?? { asset: ref, subscribers: [] }
|
||||
if (!entry.subscribers.includes(path)) entry.subscribers.push(path)
|
||||
unwritten.set(key, entry)
|
||||
if (!scriptErrors.has(path)) {
|
||||
scriptErrors.set(path, `runs on ${ref.path}, which nothing writes`)
|
||||
waitsOn.set(path, ref)
|
||||
}
|
||||
}
|
||||
}
|
||||
// A pipeline script nothing starts runs only by hand. An asset trigger starts
|
||||
// it only if something writes that asset, which covers the automatic trigger
|
||||
// of a ducklake/s3 read too. dbt projects and macro libraries are never
|
||||
// started on their own, so they are left out.
|
||||
const assetKey = (t: { asset_kind: AssetKind; asset_path: string }) =>
|
||||
`${t.asset_kind}:${t.asset_path}`
|
||||
const unwrittenTriggerAssets = new Set<string>()
|
||||
const hasTrigger = new Set<string>()
|
||||
const started = new Set<string>()
|
||||
for (const t of graph.triggers) {
|
||||
hasTrigger.add(t.runnable_path)
|
||||
if (t.trigger_kind !== 'asset') started.add(t.runnable_path)
|
||||
else if (written.has(assetKey(t))) started.add(t.runnable_path)
|
||||
else unwrittenTriggerAssets.add(assetKey(t))
|
||||
}
|
||||
for (const k of unwritten.keys()) unwrittenTriggerAssets.add(k)
|
||||
const untriggered: PipelineNodeErrors['untriggered'] = new Map()
|
||||
for (const r of graph.runnables) {
|
||||
if (r.usage_kind !== 'script' || !r.in_pipeline || r.dbt || r.macros?.length) continue
|
||||
if (started.has(r.path) || scriptErrors.has(r.path)) continue
|
||||
const why = hasTrigger.has(r.path) ? 'unwritten-inputs' : 'no-trigger'
|
||||
untriggered.set(r.path, why)
|
||||
scriptErrors.set(
|
||||
r.path,
|
||||
why === 'no-trigger'
|
||||
? 'nothing starts it: it has no trigger'
|
||||
: 'nothing starts it: nothing writes the assets it runs after'
|
||||
)
|
||||
}
|
||||
return {
|
||||
assets: errors,
|
||||
scripts: scriptErrors,
|
||||
unwritten,
|
||||
waitsOn,
|
||||
untriggered,
|
||||
unwrittenTriggerAssets
|
||||
}
|
||||
}
|
||||
|
||||
function referencedAssetKeys(
|
||||
graph: Pick<AssetGraphResponse, 'edges' | 'triggers' | 'test_edges' | 'dbt_edges'>
|
||||
): Set<string> {
|
||||
const keys = new Set<string>()
|
||||
for (const e of graph.edges) keys.add(`${e.asset_kind}:${e.asset_path}`)
|
||||
for (const t of graph.triggers)
|
||||
if (t.trigger_kind === 'asset') keys.add(`${t.asset_kind}:${t.asset_path}`)
|
||||
for (const e of graph.test_edges ?? []) keys.add(`${e.asset_kind}:${e.asset_path}`)
|
||||
for (const e of graph.dbt_edges ?? []) {
|
||||
keys.add(`dbt:${e.from_asset_path}`)
|
||||
keys.add(`dbt:${e.to_asset_path}`)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
/**
|
||||
* The backend only returns assets something references, so a deployed asset
|
||||
* whose every edge and trigger the draft/live overlays stripped — an output the
|
||||
* open script renamed — would linger as an edgeless node until the deploy
|
||||
* lands. Assets that were already unreferenced in `base` are left alone.
|
||||
*/
|
||||
function dropOrphanedAssets(
|
||||
assets: AssetGraphResponse['assets'],
|
||||
base: AssetGraphResponse,
|
||||
edges: AssetGraphResponse['edges'],
|
||||
triggers: AssetGraphResponse['triggers']
|
||||
): AssetGraphResponse['assets'] {
|
||||
const before = referencedAssetKeys(base)
|
||||
const after = referencedAssetKeys({ ...base, edges, triggers })
|
||||
return assets.filter((a) => {
|
||||
const key = `${a.kind}:${a.path}`
|
||||
return !before.has(key) || after.has(key)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Macro-library → consumer edges: the deployed base edges, with `// use`
|
||||
* declarations of overlaid scripts (drafts + the open buffer) taking over
|
||||
@@ -848,7 +1039,6 @@ function overlayInferredLineage(acc: Accumulator, input: ResolveGraphInput) {
|
||||
overlayLineage(inferredReadsByPath, 'r')
|
||||
}
|
||||
|
||||
|
||||
/** dbt project ↔ relation association, for a graph that does NOT draw it.
|
||||
*
|
||||
* A dbt script owns every relation of its project, so drawing one edge per
|
||||
|
||||
@@ -6,6 +6,12 @@ export type GraphUsageKind = 'script' | 'flow'
|
||||
export interface AssetGraphAssetNode {
|
||||
kind: AssetKind
|
||||
path: string
|
||||
// Filled in client-side: the node is misconfigured (see `pipelineNodeErrors`).
|
||||
error?: string
|
||||
// Filled in client-side: no script in the workspace writes it, so a step
|
||||
// downstream of it would never run on its writes (a flow's write fires no
|
||||
// asset trigger).
|
||||
never_written?: boolean
|
||||
// Fork workspaces only: 'fork' when this ducklake asset was materialized in
|
||||
// the fork itself, 'deferred' when reads fall back to the parent workspace's
|
||||
// current table via a defer view. Absent outside forks / for other kinds /
|
||||
@@ -70,6 +76,12 @@ export interface DbtDataTest {
|
||||
export interface AssetGraphRunnableNode {
|
||||
path: string
|
||||
usage_kind: GraphUsageKind
|
||||
// Filled in client-side (not sent by the backend): the canvas titles the
|
||||
// node with the summary over the path, and shows the language's icon.
|
||||
summary?: string
|
||||
language?: import('$lib/gen').ScriptLang
|
||||
// Why the script is misconfigured, when it is (see `pipelineNodeErrors`).
|
||||
error?: string
|
||||
// Script has `// pipeline` annotation. Drives the pipeline-member
|
||||
// visual state; unrelated to what the script actually writes (that's
|
||||
// parsed separately into lineage edges).
|
||||
@@ -170,6 +182,17 @@ export type NativeTriggerKind =
|
||||
| 'gcp'
|
||||
| 'data_upload'
|
||||
|
||||
export type PipelineTriggerDraftKind = Exclude<NativeTriggerKind, 'webhook' | 'data_upload'>
|
||||
|
||||
/** A trigger configured in the pipeline editor but not deployed yet. It lives only
|
||||
* in the pipeline's draft bundle, never as a workspace trigger draft, and is
|
||||
* created when the pipeline is saved. `config` is what the kind's editor saves
|
||||
* (for a schedule, a `NewSchedule`). */
|
||||
export type PipelineTriggerDraft = {
|
||||
kind: PipelineTriggerDraftKind
|
||||
config: Record<string, any> & { path: string; script_path: string }
|
||||
}
|
||||
|
||||
export type AssetGraphTrigger =
|
||||
| {
|
||||
trigger_kind: 'asset'
|
||||
@@ -192,6 +215,13 @@ export type AssetGraphTrigger =
|
||||
// the canvas renders a red placeholder with a "Create trigger"
|
||||
// affordance instead of a fully-wired source.
|
||||
missing?: boolean
|
||||
// A pipeline-local `PipelineTriggerDraft`: `path` is the trigger it
|
||||
// will be deployed as, and no row exists yet.
|
||||
draft?: boolean
|
||||
// A schedule's cron, zone and summary, filled in client-side, for the node's labels.
|
||||
schedule?: string
|
||||
timezone?: string
|
||||
summary?: string
|
||||
}
|
||||
|
||||
// Macro-library → consumer edge: the consumer calls `macro_names` of
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { Item } from '$lib/utils'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import LanguageIcon from '$lib/components/common/languageIcons/LanguageIcon.svelte'
|
||||
import { Code2 } from 'lucide-svelte'
|
||||
|
||||
export type UpstreamScript = { path: string; summary?: string; language?: ScriptLang }
|
||||
|
||||
/** A menu of the scripts that produce an asset, to pick the one to open. */
|
||||
export function upstreamScriptItems(
|
||||
scripts: UpstreamScript[],
|
||||
onPick: (path: string) => void
|
||||
): Item[] {
|
||||
return scripts.map((s) => ({
|
||||
displayName: s.summary || s.path,
|
||||
description: s.summary ? s.path : undefined,
|
||||
...(s.language
|
||||
? { icon: LanguageIcon, iconProps: { lang: s.language, width: 14, height: 14 } }
|
||||
: { icon: Code2 }),
|
||||
action: () => onPick(s.path)
|
||||
}))
|
||||
}
|
||||
|
||||
/** An upstream trigger the asset pane can open, and how. */
|
||||
export type UpstreamTriggerAction = { label: string; detail?: string; onOpen: () => void }
|
||||
|
||||
export function upstreamTriggerItems(triggers: UpstreamTriggerAction[]): Item[] {
|
||||
return triggers.map((t) => ({
|
||||
displayName: t.label,
|
||||
description: t.detail,
|
||||
action: () => t.onOpen()
|
||||
}))
|
||||
}
|
||||
@@ -17,6 +17,8 @@
|
||||
bgClass?: string | undefined
|
||||
bgStyle?: string | undefined
|
||||
iconClass?: string | undefined
|
||||
/** Drop the type's icon; the text starts at the alert's edge. */
|
||||
hideIcon?: boolean
|
||||
iconStyle?: string | undefined
|
||||
titleClass?: string | undefined
|
||||
titleStyle?: string | undefined
|
||||
@@ -41,6 +43,7 @@
|
||||
bgClass = undefined,
|
||||
bgStyle = undefined,
|
||||
iconClass = undefined,
|
||||
hideIcon = false,
|
||||
iconStyle = undefined,
|
||||
titleClass = undefined,
|
||||
titleStyle = undefined,
|
||||
@@ -78,14 +81,16 @@
|
||||
<div class="flex flex-row items-center">
|
||||
<!-- shrink-0: the badge's min-content width is the 16px icon, so without it a
|
||||
narrow container squeezes the circle into an oval. -->
|
||||
<div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-full">
|
||||
<SvelteComponent
|
||||
class={twMerge(classes[type].iconClass, iconClass)}
|
||||
style={iconStyle}
|
||||
size={16}
|
||||
/>
|
||||
</div>
|
||||
<div class={twMerge('ml-1 w-full')}>
|
||||
{#if !hideIcon}
|
||||
<div class="flex h-8 w-8 shrink-0 items-center justify-center rounded-full">
|
||||
<SvelteComponent
|
||||
class={twMerge(classes[type].iconClass, iconClass)}
|
||||
style={iconStyle}
|
||||
size={16}
|
||||
/>
|
||||
</div>
|
||||
{/if}
|
||||
<div class={twMerge(!hideIcon && 'ml-1', 'w-full')}>
|
||||
{#if hasTitleRow}
|
||||
<div class={twMerge('w-full flex flex-row items-center justify-between')}>
|
||||
<span
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
import type { Item } from '$lib/utils'
|
||||
import type { ContextMenuItem } from './ContextMenu.svelte'
|
||||
|
||||
/** A dropdown's items as the same entries of a right-click menu. */
|
||||
export function contextMenuItemsFromMenu(items: Item[]): ContextMenuItem[] {
|
||||
return items.flatMap((item) => [
|
||||
...(item.separatorTop ? [{ id: `${item.displayName}-divider`, label: '', divider: true }] : []),
|
||||
{
|
||||
id: item.displayName,
|
||||
label: item.displayName,
|
||||
icon: item.icon,
|
||||
disabled: item.disabled,
|
||||
type: item.type,
|
||||
shortcut: item.shortcut,
|
||||
onClick: item.action as (() => void) | undefined
|
||||
}
|
||||
])
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
<script lang="ts">
|
||||
import { base } from '$lib/base'
|
||||
import Dropdown from '$lib/components/DropdownV2.svelte'
|
||||
import ConfirmationModal from '../confirmationModal/ConfirmationModal.svelte'
|
||||
import {
|
||||
deletePipeline,
|
||||
planPipelineDelete,
|
||||
type PipelineDeletePlan
|
||||
} from '$lib/components/assets/AssetGraph/pipelineDelete'
|
||||
import { userStore, workspaceStore } from '$lib/stores'
|
||||
import { sendUserToast } from '$lib/toast'
|
||||
import { Loader2, Trash } from 'lucide-svelte'
|
||||
import Row from './Row.svelte'
|
||||
|
||||
interface Props {
|
||||
folder: string
|
||||
depth?: number
|
||||
/** Shown instead of `f/<folder>`, e.g. under the folder's own tree node. */
|
||||
label?: string
|
||||
onDeleted?: () => void
|
||||
}
|
||||
|
||||
let { folder, depth = 0, label, onDeleted }: Props = $props()
|
||||
|
||||
let confirmOpen = $state(false)
|
||||
let deleting = $state(false)
|
||||
let plan = $state<PipelineDeletePlan | undefined>(undefined)
|
||||
let planError = $state<string | undefined>(undefined)
|
||||
// The backend deletes scripts for admins only; everyone else archives them.
|
||||
let hardDelete = $derived(!!($userStore?.is_admin || $userStore?.is_super_admin))
|
||||
|
||||
async function openConfirm() {
|
||||
const ws = $workspaceStore
|
||||
if (!ws) return
|
||||
plan = undefined
|
||||
planError = undefined
|
||||
confirmOpen = true
|
||||
try {
|
||||
plan = await planPipelineDelete(ws, folder)
|
||||
} catch (e: any) {
|
||||
planError = e?.body ?? e?.message ?? String(e)
|
||||
}
|
||||
}
|
||||
|
||||
async function confirmDelete() {
|
||||
const ws = $workspaceStore
|
||||
if (!ws || !plan) return
|
||||
deleting = true
|
||||
try {
|
||||
const { removed, failures } = await deletePipeline(ws, folder, plan, hardDelete)
|
||||
// Loaded on demand: the home list should not pull in the session runtime.
|
||||
const { forgetDeletedPipeline } = await import(
|
||||
'$lib/components/sessions/sessionRuntime.svelte'
|
||||
)
|
||||
forgetDeletedPipeline(ws, folder)
|
||||
if (failures.length) {
|
||||
sendUserToast(`Pipeline f/${folder} was only partly removed`, true, [], failures.join('\n'))
|
||||
} else {
|
||||
sendUserToast(`Removed pipeline f/${folder} (${removed} item${removed === 1 ? '' : 's'})`)
|
||||
}
|
||||
onDeleted?.()
|
||||
} catch (e: any) {
|
||||
sendUserToast(`Could not remove pipeline f/${folder}: ${e?.body ?? e?.message ?? e}`, true)
|
||||
} finally {
|
||||
deleting = false
|
||||
confirmOpen = false
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<Row
|
||||
kind="data_pipeline"
|
||||
marked={undefined}
|
||||
href="{base}/pipeline/{encodeURIComponent(folder)}"
|
||||
workspaceId={$workspaceStore ?? ''}
|
||||
path={`f/${folder}`}
|
||||
summary={label ?? `Pipeline · ${folder}`}
|
||||
canFavorite={false}
|
||||
{depth}
|
||||
>
|
||||
{#snippet actions()}
|
||||
{#if !$userStore?.operator}
|
||||
<Dropdown
|
||||
items={[
|
||||
{
|
||||
displayName: 'Delete',
|
||||
icon: Trash,
|
||||
type: 'delete',
|
||||
action: () => void openConfirm()
|
||||
}
|
||||
]}
|
||||
/>
|
||||
{/if}
|
||||
{/snippet}
|
||||
</Row>
|
||||
|
||||
<ConfirmationModal
|
||||
open={confirmOpen}
|
||||
title="Delete pipeline f/{folder}"
|
||||
confirmationText={hardDelete ? 'Delete' : 'Archive'}
|
||||
loading={deleting}
|
||||
confirmDisabled={!plan}
|
||||
trashbin={hardDelete}
|
||||
on:canceled={() => (confirmOpen = false)}
|
||||
on:confirmed={confirmDelete}
|
||||
>
|
||||
<div class="flex flex-col gap-3 text-xs">
|
||||
{#if planError}
|
||||
<span class="text-red-600 dark:text-red-400">Could not list the pipeline: {planError}</span>
|
||||
{:else if !plan}
|
||||
<span class="flex items-center gap-2 text-secondary">
|
||||
<Loader2 size={14} class="animate-spin" /> Listing what this removes…
|
||||
</span>
|
||||
{:else}
|
||||
{#if plan.scripts.some((s) => !s.draftOnly)}
|
||||
{@render group(
|
||||
hardDelete ? 'Scripts deleted' : 'Scripts archived',
|
||||
plan.scripts.filter((s) => !s.draftOnly).map((s) => s.displayPath)
|
||||
)}
|
||||
{/if}
|
||||
{#if plan.scripts.some((s) => s.draftOnly)}
|
||||
{@render group(
|
||||
'Draft scripts discarded',
|
||||
plan.scripts.filter((s) => s.draftOnly).map((s) => s.displayPath)
|
||||
)}
|
||||
{/if}
|
||||
{#if plan.triggers.length}
|
||||
{@render group(
|
||||
'Triggers deleted',
|
||||
plan.triggers.map((t) => `${t.path} (${t.kind})`)
|
||||
)}
|
||||
{/if}
|
||||
{#if plan.hasDraft}
|
||||
{@render group('Unsaved pipeline changes discarded', ['Your pipeline draft'])}
|
||||
{/if}
|
||||
{#if !plan.scripts.length && !plan.triggers.length && !plan.hasDraft}
|
||||
<span class="text-secondary">Nothing left to remove in this pipeline.</span>
|
||||
{/if}
|
||||
<span class="text-secondary">
|
||||
The tables and files the pipeline wrote are kept, and so is the folder f/{folder}.
|
||||
</span>
|
||||
{/if}
|
||||
</div>
|
||||
</ConfirmationModal>
|
||||
|
||||
{#snippet group(title: string, entries: string[])}
|
||||
<div class="flex flex-col gap-1">
|
||||
<span class="font-semibold text-emphasis">{title} ({entries.length})</span>
|
||||
<ul class="max-h-40 overflow-auto rounded-md border bg-surface-secondary px-3 py-1.5">
|
||||
{#each entries as entry (entry)}
|
||||
<li class="font-mono text-2xs text-primary truncate">{entry}</li>
|
||||
{/each}
|
||||
</ul>
|
||||
</div>
|
||||
{/snippet}
|
||||
@@ -501,11 +501,17 @@ export class AIChatManager implements ChatViewHost {
|
||||
skipResponsesApi = $state(false)
|
||||
|
||||
mode = $state<AIMode>(AIMode.NAVIGATOR)
|
||||
pipelineAiChatHelpers = $state<PipelineAIChatHelpers | undefined>(undefined)
|
||||
// One per mounted pipeline editor; a session can have several folders open.
|
||||
#pipelineEditors = new Set<PipelineAIChatHelpers>()
|
||||
// Resolved when a pipeline editor registers its tools. open_preview(pipeline)
|
||||
// awaits this so the model's next build_pipeline_node call can't race ahead of
|
||||
// the async canvas mount and hit "Unknown tool call".
|
||||
#pipelineHelpersWaiters = new Set<() => void>()
|
||||
#pipelineHelpersWaiters = new Set<(helpers: PipelineAIChatHelpers) => void>()
|
||||
// A session's pipeline editor unmounts whenever another tab takes the preview
|
||||
// panel. Its tools stay offered for every folder the session opened, and a call
|
||||
// reopens the tab through this, so a page shown in between costs no failed call.
|
||||
#pipelineReopener: ((folder: string) => void) | undefined = undefined
|
||||
#openedPipelineFolders = new Set<string>()
|
||||
readonly isOpen = $derived(chatState.size > 0)
|
||||
savedSize = $state<number>(0)
|
||||
instructions = $state<string>('')
|
||||
@@ -900,6 +906,15 @@ export class AIChatManager implements ChatViewHost {
|
||||
await this.#persistModifiedItems()
|
||||
}
|
||||
|
||||
// Record the items deploying a chat-modified item produced — a pipeline's
|
||||
// scripts — so a fork review selects exactly them. Persisted at once, like a
|
||||
// discard: the deploy can run outside any turn.
|
||||
async recordDeployedItems(itemKind: UserDraftItemKind, storagePaths: string[]) {
|
||||
if (!this.modifiedItems || storagePaths.length === 0) return
|
||||
for (const p of storagePaths) this.modifiedItems.add(maskKey(itemKind, p))
|
||||
await this.#persistModifiedItems()
|
||||
}
|
||||
|
||||
// Move a mask entry to the path a draft actually deployed to. A draft-only
|
||||
// flow/app parks at a synthetic `draft_{uuid}` storage path and deploys to
|
||||
// its chosen path — without the move, the existence check at the synthetic
|
||||
@@ -2492,8 +2507,27 @@ export class AIChatManager implements ChatViewHost {
|
||||
|
||||
// Public because it is purely local, unlike `changeMode(GLOBAL)`, which also
|
||||
// fires the three network refreshes.
|
||||
setPipelineReopener = (reopen: (folder: string) => void) => {
|
||||
this.#pipelineReopener = reopen
|
||||
}
|
||||
|
||||
#pipelineFolders = (): string[] => [
|
||||
...new Set([
|
||||
...[...this.#pipelineEditors].map((p) => p.getFolder()),
|
||||
...(this.#pipelineReopener ? this.#openedPipelineFolders : [])
|
||||
])
|
||||
]
|
||||
|
||||
#ensurePipeline = async (folder: string): Promise<PipelineAIChatHelpers | undefined> => {
|
||||
const find = () => [...this.#pipelineEditors].find((p) => p.getFolder() === folder)
|
||||
const live = find()
|
||||
if (live || !this.#pipelineReopener) return live
|
||||
this.#pipelineReopener(folder)
|
||||
return (await this.waitForPipelineHelpers(folder)) ? find() : undefined
|
||||
}
|
||||
|
||||
configureGlobalMode = () => {
|
||||
const pipeline = this.pipelineAiChatHelpers
|
||||
const hasPipelines = this.#pipelineFolders().length > 0
|
||||
const opts = this.globalAssemblyOpts()
|
||||
const baseHelpers: GlobalToolHelpers = {
|
||||
// A session targets its own fixed (possibly forked) workspace, so capture it for
|
||||
@@ -2525,7 +2559,14 @@ export class AIChatManager implements ChatViewHost {
|
||||
}
|
||||
}
|
||||
this.#assembledTools = assembleGlobalTools(opts)
|
||||
this.helpers = pipeline ? { ...baseHelpers, pipeline } : baseHelpers
|
||||
this.helpers = hasPipelines
|
||||
? {
|
||||
...baseHelpers,
|
||||
pipelines: () => [...this.#pipelineEditors],
|
||||
pipelineFolders: this.#pipelineFolders,
|
||||
ensurePipeline: this.#ensurePipeline
|
||||
}
|
||||
: baseHelpers
|
||||
this.systemMessage = assembleGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), opts)
|
||||
this.syncArtifactsSession()
|
||||
}
|
||||
@@ -2538,7 +2579,7 @@ export class AIChatManager implements ChatViewHost {
|
||||
mcpServers: this.mcpServers,
|
||||
access: this.sessionAccess,
|
||||
sessionContext: this.sessionContextResolver?.(),
|
||||
pipelineContext: this.pipelineAiChatHelpers?.getPipelineContext()
|
||||
pipelineFolders: this.#pipelineFolders()
|
||||
})
|
||||
|
||||
// Fetch the workspace's AI skills and, if GLOBAL mode is still active, rebuild
|
||||
@@ -4870,19 +4911,18 @@ export class AIChatManager implements ChatViewHost {
|
||||
// AI edits apply directly as drafts, so there is nothing to auto-accept.
|
||||
// Returns a cleanup that tears the registration back down.
|
||||
setPipelineHelpers = (pipelineHelpers: PipelineAIChatHelpers) => {
|
||||
this.pipelineAiChatHelpers = pipelineHelpers
|
||||
this.#pipelineEditors.add(pipelineHelpers)
|
||||
this.#openedPipelineFolders.add(pipelineHelpers.getFolder())
|
||||
untrack(() => {
|
||||
if (this.mode === AIMode.GLOBAL) {
|
||||
this.configureGlobalMode()
|
||||
}
|
||||
})
|
||||
// The pipeline tools are now registered — release anything awaiting them.
|
||||
const waiters = [...this.#pipelineHelpersWaiters]
|
||||
this.#pipelineHelpersWaiters.clear()
|
||||
waiters.forEach((resolve) => resolve())
|
||||
;[...this.#pipelineHelpersWaiters].forEach((notify) => notify(pipelineHelpers))
|
||||
|
||||
return () => {
|
||||
this.pipelineAiChatHelpers = undefined
|
||||
this.#pipelineEditors.delete(pipelineHelpers)
|
||||
untrack(() => {
|
||||
if (this.mode === AIMode.GLOBAL) {
|
||||
this.configureGlobalMode()
|
||||
@@ -4892,15 +4932,16 @@ export class AIChatManager implements ChatViewHost {
|
||||
}
|
||||
|
||||
/**
|
||||
* Await the pipeline editor's tool registration. Resolves `true` immediately
|
||||
* when a pipeline editor is already mounted, or when the next one registers;
|
||||
* resolves `false` after `timeoutMs` if none registers (e.g. a backgrounded
|
||||
* session whose preview tab is not mounted, or a closed tab). Callers must
|
||||
* treat `false` as "tools NOT available" rather than silently reporting
|
||||
* success — the open_preview handler surfaces that to the model.
|
||||
* Await the tool registration of the pipeline editor for `folder` (any folder
|
||||
* when unset). Resolves `true` immediately when one is already mounted, or when
|
||||
* it registers; resolves `false` after `timeoutMs` if none registers (e.g. a
|
||||
* session whose runtime is not mounted, or a closed tab). Callers must treat
|
||||
* `false` as "tools NOT available" rather than silently reporting success —
|
||||
* the open_preview handler surfaces that to the model.
|
||||
*/
|
||||
waitForPipelineHelpers = (timeoutMs = 8000): Promise<boolean> => {
|
||||
if (this.pipelineAiChatHelpers) return Promise.resolve(true)
|
||||
waitForPipelineHelpers = (folder?: string, timeoutMs = 8000): Promise<boolean> => {
|
||||
const matches = (p: PipelineAIChatHelpers) => folder === undefined || p.getFolder() === folder
|
||||
if ([...this.#pipelineEditors].some(matches)) return Promise.resolve(true)
|
||||
return new Promise<boolean>((resolve) => {
|
||||
let settled = false
|
||||
const finish = (registered: boolean) => {
|
||||
@@ -4909,7 +4950,9 @@ export class AIChatManager implements ChatViewHost {
|
||||
this.#pipelineHelpersWaiters.delete(onRegister)
|
||||
resolve(registered)
|
||||
}
|
||||
const onRegister = () => finish(true)
|
||||
const onRegister = (p: PipelineAIChatHelpers) => {
|
||||
if (matches(p)) finish(true)
|
||||
}
|
||||
this.#pipelineHelpersWaiters.add(onRegister)
|
||||
setTimeout(() => finish(false), timeoutMs)
|
||||
})
|
||||
|
||||
@@ -4329,11 +4329,15 @@ describe('DOM selector chips scoped by app path', () => {
|
||||
describe('AIChatManager.waitForPipelineHelpers', () => {
|
||||
function fakePipelineHelpers(): PipelineAIChatHelpers {
|
||||
return {
|
||||
getFolder: () => 'f',
|
||||
hasDrafts: () => false,
|
||||
getPipelineContext: () => ({ folder: 'f', mode: 'edit', nodes: [], assets: [] }),
|
||||
getNodeBody: async () => undefined,
|
||||
proposeNode: async () => ({ path: '', detectedReads: [], detectedWrites: [] }),
|
||||
editNode: async () => ({ detectedReads: [], detectedWrites: [] }),
|
||||
removeProposedNode: async () => {},
|
||||
unconfiguredTriggers: async () => [],
|
||||
setNodeTrigger: async () => ({ path: '', replaced: false }),
|
||||
testNode: async () => undefined
|
||||
}
|
||||
}
|
||||
@@ -4341,13 +4345,13 @@ describe('AIChatManager.waitForPipelineHelpers', () => {
|
||||
it('resolves true immediately when a pipeline editor is already registered', async () => {
|
||||
const manager = new AIChatManager()
|
||||
manager.setPipelineHelpers(fakePipelineHelpers())
|
||||
await expect(manager.waitForPipelineHelpers(1000)).resolves.toBe(true)
|
||||
await expect(manager.waitForPipelineHelpers(undefined, 1000)).resolves.toBe(true)
|
||||
})
|
||||
|
||||
it('resolves true once a pipeline editor registers', async () => {
|
||||
const manager = new AIChatManager()
|
||||
let outcome: boolean | undefined
|
||||
const wait = manager.waitForPipelineHelpers(1000).then((v) => (outcome = v))
|
||||
const wait = manager.waitForPipelineHelpers(undefined, 1000).then((v) => (outcome = v))
|
||||
await Promise.resolve()
|
||||
expect(outcome).toBeUndefined()
|
||||
manager.setPipelineHelpers(fakePipelineHelpers())
|
||||
@@ -4355,12 +4359,24 @@ describe('AIChatManager.waitForPipelineHelpers', () => {
|
||||
expect(outcome).toBe(true)
|
||||
})
|
||||
|
||||
it('waits for the editor of the requested folder, not any editor', async () => {
|
||||
const manager = new AIChatManager()
|
||||
manager.setPipelineHelpers(fakePipelineHelpers())
|
||||
let outcome: boolean | undefined
|
||||
const wait = manager.waitForPipelineHelpers('sales', 1000).then((v) => (outcome = v))
|
||||
await Promise.resolve()
|
||||
expect(outcome).toBeUndefined()
|
||||
manager.setPipelineHelpers({ ...fakePipelineHelpers(), getFolder: () => 'sales' })
|
||||
await wait
|
||||
expect(outcome).toBe(true)
|
||||
})
|
||||
|
||||
// The false result is the signal the open_preview handler needs: a backgrounded
|
||||
// session's editor never mounts, so it must report "tools unavailable" rather
|
||||
// than silently claim success.
|
||||
it('resolves false after the timeout when no editor ever registers', async () => {
|
||||
const manager = new AIChatManager()
|
||||
await expect(manager.waitForPipelineHelpers(10)).resolves.toBe(false)
|
||||
await expect(manager.waitForPipelineHelpers(undefined, 10)).resolves.toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -19,6 +19,7 @@ import { applyReasoningToConfig } from '../reasoningRegistry'
|
||||
import {
|
||||
appendPendingToolImages,
|
||||
processToolCall,
|
||||
type LiveToolSet,
|
||||
queuedToolStatus,
|
||||
type Tool,
|
||||
type ToolCallbacks,
|
||||
@@ -197,7 +198,11 @@ export async function parseAnthropicCompletion(
|
||||
tools: Tool<any>[],
|
||||
helpers: any,
|
||||
abortController?: AbortController,
|
||||
options?: { workspace?: string; onTokenUsage?: (usage: ChatTokenUsage) => void }
|
||||
options?: {
|
||||
workspace?: string
|
||||
onTokenUsage?: (usage: ChatTokenUsage) => void
|
||||
live?: LiveToolSet
|
||||
}
|
||||
): Promise<ParsedCompletionResult> {
|
||||
let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = []
|
||||
let error = null
|
||||
@@ -447,7 +452,8 @@ export async function parseAnthropicCompletion(
|
||||
helpers,
|
||||
toolCallbacks: callbacks,
|
||||
workspace: options?.workspace,
|
||||
messages
|
||||
messages,
|
||||
live: options?.live
|
||||
})
|
||||
messages.push(messageToAdd)
|
||||
addedMessages.push(messageToAdd)
|
||||
|
||||
@@ -453,7 +453,8 @@ export async function runChatLoop(config: ChatLoopConfig): Promise<ChatLoopResul
|
||||
const parseOptions = {
|
||||
workspace,
|
||||
provider: modelProvider.provider,
|
||||
onTokenUsage: reportUsage
|
||||
onTokenUsage: reportUsage,
|
||||
live: () => ({ tools: config.tools, helpers: config.helpers })
|
||||
}
|
||||
|
||||
if (isOpenAI) {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { randomUUID } from '$lib/utils/uuid'
|
||||
import {
|
||||
AppService,
|
||||
AssetService,
|
||||
AzureTriggerService,
|
||||
EmailTriggerService,
|
||||
FlowService,
|
||||
@@ -67,6 +68,7 @@ import {
|
||||
stripFileArgs
|
||||
} from '$lib/components/job_args'
|
||||
import { processSecretArgs } from '$lib/components/secretArgUtils'
|
||||
import { parsePipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations'
|
||||
import { PLAN_MODE_MESSAGES } from '../planModeMessages'
|
||||
import {
|
||||
DEFAULT_DATA as DEFAULT_RAW_APP_DATA,
|
||||
@@ -423,6 +425,8 @@ const updateUserInstructionsSchema = z.object({
|
||||
)
|
||||
})
|
||||
|
||||
const listPipelinesSchema = z.object({})
|
||||
|
||||
const listWorkspaceItemsSchema = z.object({
|
||||
types: z
|
||||
.array(itemTypeSchema)
|
||||
@@ -1424,7 +1428,7 @@ const buildGlobalSystemPrompt = (
|
||||
const activePreviewRule = previewTools
|
||||
? '\n- If the user message includes an ACTIVE PREVIEW section, that is the page the side panel is showing — resolve "this page", "here" and "it" against it, and against `open` (the item the user has open in its editor) when there is one. It already tells you what get_preview_status would, so do not call that tool to learn what is on screen; call it only to check the panel\'s *other* tabs.'
|
||||
: ''
|
||||
const pipelineBullet = `- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on <ref>\` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow.${pipelineAlphaNote}`
|
||||
const pipelineBullet = `- A "data pipeline" is NOT a flow: it is a DAG of independent scripts in one folder, wired by storage assets (DuckLake/data tables/S3) and triggers via top-of-file \`pipeline\` / \`on <ref>\` annotation comments written in each script's comment syntax (\`--\` for SQL, \`#\` for Python/Bash, \`//\` for TS — a \`//\` line in a SQL node is a syntax error). When the user asks for a data pipeline (or to ingest/transform/materialize data across steps), call get_instructions with subject "pipeline" and build annotated script drafts — do not build a flow. list_pipelines lists the existing ones by folder.${pipelineAlphaNote}`
|
||||
// Hosting and edition come from the hostname and a store the app populates at init, so
|
||||
// they are knowable only in the browser: a non-browser caller reads false for both and
|
||||
// would be told "self-hosted Community Edition" whatever it targets. No base URL here
|
||||
@@ -1524,7 +1528,7 @@ ${pipelineBullet}`
|
||||
canWriteDraft,
|
||||
`
|
||||
- After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited.
|
||||
- Building a data pipeline: call open_preview(kind="pipeline", path="<folder>") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty ${when(canCreateFolder, 'or not-yet-created ')}folder is fine${when(canCreateFolder, ' (create_folder first if needed, then open it)')}. Opening it registers build_pipeline_node / edit_pipeline_node — use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor.`
|
||||
- Building a data pipeline: call open_preview(kind="pipeline", path="<folder>") as the FIRST step, before creating any node — this opens the pipeline editor the user reviews in. path is the folder, not an item; an empty ${when(canCreateFolder, 'or not-yet-created ')}folder is fine${when(canCreateFolder, ' (create_folder first if needed, then open it)')}. Opening it adds build_pipeline_node / edit_pipeline_node to your tools from the next step on — they are not in your tool list before open_preview returns, so call open_preview on its own, wait for its result, then build. They then stay available for the rest of the session even while another tab (a page, another preview) is shown: a call brings the pipeline editor back by itself, so do not re-open it first. Use ONLY those to add or change pipeline nodes, never write_script for a pipeline node — they apply directly as unsaved drafts on the canvas (no separate accept/reject step) that the user reviews and deploys. Do not write pipeline scripts without first opening the editor.`
|
||||
)}
|
||||
- When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app").
|
||||
- Writing an app file does not compile it: the open preview rebuilds it afterwards. After editing a raw app's frontend files with its preview open, call get_app_runtime_logs to check the build — when it failed, it returns the build errors (e.g. syntax or import errors) and bundler logs to fix.
|
||||
@@ -3654,6 +3658,23 @@ export const globalTools: SessionTool<{}>[] = [
|
||||
return JSON.stringify(results, null, 2)
|
||||
}
|
||||
},
|
||||
{
|
||||
requires: NONE,
|
||||
def: createToolDef(
|
||||
listPipelinesSchema,
|
||||
'list_pipelines',
|
||||
'List the data pipelines of the workspace: each folder holding at least one pipeline script, with its number of scripts. A pipeline is named by its folder, not by an item path.'
|
||||
),
|
||||
planModeSafe: true,
|
||||
fn: async ({ workspace, toolId, toolCallbacks }) => {
|
||||
toolCallbacks.setToolStatus(toolId, { content: 'Listing data pipelines...' })
|
||||
const pipelines = await AssetService.listPipelineFolders({ workspace })
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Listed ${pipelines.length} data pipeline(s)`
|
||||
})
|
||||
return JSON.stringify(pipelines)
|
||||
}
|
||||
},
|
||||
{
|
||||
requires: NONE,
|
||||
def: createToolDef(
|
||||
@@ -4291,7 +4312,7 @@ export const globalTools: SessionTool<{}>[] = [
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Found ${results.length} resource type(s) for "${parsed.query}"`
|
||||
})
|
||||
return JSON.stringify(
|
||||
const listing = JSON.stringify(
|
||||
results.map((rt) => ({
|
||||
name: rt.name,
|
||||
schema: rt.schema
|
||||
@@ -4299,6 +4320,15 @@ export const globalTools: SessionTool<{}>[] = [
|
||||
null,
|
||||
2
|
||||
)
|
||||
// The search is semantic and always returns its closest types, so a miss reads
|
||||
// like a hit unless the absence is spelled out.
|
||||
const words = (parsed.query.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter(
|
||||
(w) => w.length >= 3
|
||||
)
|
||||
const named = results.some((rt) => words.some((w) => rt.name.toLowerCase().includes(w)))
|
||||
return named || words.length === 0
|
||||
? listing
|
||||
: `${listing}\n\nNo resource type name contains "${parsed.query}": these are only the nearest matches, so no dedicated type exists. Searching again with synonyms will not find one — use a fitting generic type above, or create a custom c_<name> type.`
|
||||
}
|
||||
},
|
||||
createDbSchemaTool<{}>({
|
||||
@@ -5471,6 +5501,17 @@ function writeScriptDraft(
|
||||
ctx: WriteDraftCtx,
|
||||
codeDiff?: ToolCodeDiff
|
||||
): Promise<string> {
|
||||
// A session builds pipeline nodes into the folder's pipeline draft; a script draft
|
||||
// written here never reaches the pipeline graph, and saving the pipeline drops it.
|
||||
if (
|
||||
(ctx.sessionId ?? sessionIdFromCtx(ctx)) &&
|
||||
parsePipelineAnnotations(args.content).inPipeline
|
||||
) {
|
||||
const folder = args.path.match(/^f\/([^/]+)\//)?.[1] ?? '<folder>'
|
||||
throw new Error(
|
||||
`"${args.path}" is a data pipeline node (its source starts with the pipeline annotation). In a session, pipeline nodes are not script drafts: call open_preview(kind="pipeline", path="${folder}") if that folder's pipeline editor is not open yet, then use build_pipeline_node (new node) or edit_pipeline_node (existing node). Nothing was written.`
|
||||
)
|
||||
}
|
||||
return writeDraft(SCRIPT_SPEC, 'script', args.path, args, ctx, {
|
||||
override: args.override,
|
||||
codeDiff:
|
||||
|
||||
@@ -15,22 +15,17 @@ vi.mock('@codingame/monaco-vscode-languages-service-override', () => ({ default:
|
||||
vi.mock('$lib/components/vscode', () => ({}))
|
||||
|
||||
import { getSessionContextPromptSection, type SessionPromptContext } from './core'
|
||||
import { getPipelinePromptSection, type PipelineContext } from '../pipeline/core'
|
||||
import { getPipelinePromptSection } from '../pipeline/core'
|
||||
import { assembleGlobalSystemMessage, assembleGlobalTools } from './globalAssembly'
|
||||
|
||||
const sessionContext: SessionPromptContext = { workspaceId: 'ws', parentWorkspaceId: 'parent' }
|
||||
const pipelineContext: PipelineContext = {
|
||||
folder: 'myfolder',
|
||||
mode: 'edit',
|
||||
nodes: [],
|
||||
assets: []
|
||||
}
|
||||
const pipelineFolders = ['myfolder']
|
||||
|
||||
describe('assembleGlobalTools', () => {
|
||||
it('includes the pipeline tools only when a pipeline editor is active', () => {
|
||||
const names = (opts: Parameters<typeof assembleGlobalTools>[0]) =>
|
||||
assembleGlobalTools(opts).map((t) => t.def.function.name)
|
||||
expect(names({ pipelineContext })).toContain('get_pipeline_graph')
|
||||
expect(names({ pipelineFolders })).toContain('get_pipeline_graph')
|
||||
expect(names({})).not.toContain('get_pipeline_graph')
|
||||
})
|
||||
})
|
||||
@@ -41,13 +36,13 @@ describe('assembleGlobalSystemMessage', () => {
|
||||
it('carries both optional sections when both contexts are given', () => {
|
||||
const content = assembleGlobalSystemMessage(undefined, {
|
||||
sessionContext,
|
||||
pipelineContext
|
||||
pipelineFolders
|
||||
}).content as string
|
||||
expect(content).toContain(getSessionContextPromptSection(sessionContext))
|
||||
expect(content).toContain(getPipelinePromptSection(pipelineContext))
|
||||
expect(content).toContain(getPipelinePromptSection(pipelineFolders))
|
||||
// Order is part of the assembled output, so it is pinned too.
|
||||
expect(content.indexOf(getSessionContextPromptSection(sessionContext))).toBeLessThan(
|
||||
content.indexOf(getPipelinePromptSection(pipelineContext))
|
||||
content.indexOf(getPipelinePromptSection(pipelineFolders))
|
||||
)
|
||||
})
|
||||
|
||||
@@ -58,6 +53,6 @@ describe('assembleGlobalSystemMessage', () => {
|
||||
// shares, so a stray section built from any other context still trips them.
|
||||
const heading = (s: string) => s.trim().split('\n')[0]
|
||||
expect(content).not.toContain(heading(getSessionContextPromptSection(sessionContext)))
|
||||
expect(content).not.toContain(heading(getPipelinePromptSection(pipelineContext)))
|
||||
expect(content).not.toContain(heading(getPipelinePromptSection(pipelineFolders)))
|
||||
})
|
||||
})
|
||||
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
type SessionPromptContext
|
||||
} from './core'
|
||||
import { createMcpTools } from './mcpTools'
|
||||
import { getPipelinePromptSection, pipelineTools, type PipelineContext } from '../pipeline/core'
|
||||
import { getPipelinePromptSection, pipelineTools } from '../pipeline/core'
|
||||
import { getFolderInstructionsPromptSection, type FolderInstruction } from '../folderInstructions'
|
||||
|
||||
// Derived, not retyped: an option added to prepareGlobalSystemMessage is reachable
|
||||
@@ -21,7 +21,8 @@ import { getFolderInstructionsPromptSection, type FolderInstruction } from '../f
|
||||
// directly — would get it.
|
||||
export type GlobalAssemblyOpts = NonNullable<Parameters<typeof prepareGlobalSystemMessage>[1]> & {
|
||||
sessionContext?: SessionPromptContext
|
||||
pipelineContext?: PipelineContext
|
||||
/** Folders of the open pipeline editors; the pipeline tools and prompt come with any. */
|
||||
pipelineFolders?: readonly string[]
|
||||
folderInstructions?: readonly FolderInstruction[]
|
||||
}
|
||||
|
||||
@@ -34,8 +35,8 @@ export function assembleGlobalSystemMessage(
|
||||
if (opts.sessionContext) {
|
||||
systemMessage.content += getSessionContextPromptSection(opts.sessionContext, opts.access)
|
||||
}
|
||||
if (opts.pipelineContext) {
|
||||
systemMessage.content += getPipelinePromptSection(opts.pipelineContext, opts.access)
|
||||
if (opts.pipelineFolders?.length) {
|
||||
systemMessage.content += getPipelinePromptSection(opts.pipelineFolders, opts.access)
|
||||
}
|
||||
return systemMessage
|
||||
}
|
||||
@@ -43,7 +44,7 @@ export function assembleGlobalSystemMessage(
|
||||
export function assembleGlobalTools(opts: GlobalAssemblyOpts): SessionTool<any>[] {
|
||||
return [
|
||||
...globalToolsFor({ sessionPreview: opts.previewTools ?? false }),
|
||||
...(opts.pipelineContext ? pipelineTools : []),
|
||||
...(opts.pipelineFolders?.length ? pipelineTools : []),
|
||||
...createMcpTools(opts.mcpServers ?? [])
|
||||
]
|
||||
}
|
||||
|
||||
@@ -32,7 +32,7 @@ import type { SessionAccess, SessionCapability, SessionTool } from '../sessionCa
|
||||
|
||||
const ASSEMBLY_OPTS = {
|
||||
previewTools: true,
|
||||
pipelineContext: { folder: 'my_pipeline', mode: 'edit', nodes: [], assets: [] },
|
||||
pipelineFolders: ['my_pipeline'],
|
||||
mcpServers: [{ path: 'f/test/server' } as any]
|
||||
}
|
||||
|
||||
@@ -168,7 +168,7 @@ describe('session tool policies', () => {
|
||||
user: { username: 'alex', folders: ['shared'], folders_read: ['shared'] },
|
||||
access,
|
||||
sessionContext: ctx,
|
||||
pipelineContext: { folder: 'my_pipeline', mode: 'edit', nodes: [], assets: [] }
|
||||
pipelineFolders: ['my_pipeline']
|
||||
})
|
||||
// Both decoration variants: the escalation one adds its own tool mentions.
|
||||
for (const blocks of [0, 9]) {
|
||||
|
||||
@@ -18,6 +18,7 @@ import { applyReasoningToConfig } from '../reasoningRegistry'
|
||||
import {
|
||||
appendPendingToolImages,
|
||||
processToolCall,
|
||||
type LiveToolSet,
|
||||
queuedToolStatus,
|
||||
type Tool,
|
||||
type ToolCallbacks,
|
||||
@@ -413,7 +414,11 @@ export async function parseOpenAIResponsesCompletion(
|
||||
addedMessages: ChatCompletionMessageParam[],
|
||||
tools: Tool<any>[],
|
||||
helpers: any,
|
||||
options?: { workspace?: string; onTokenUsage?: (usage: ChatTokenUsage) => void }
|
||||
options?: {
|
||||
workspace?: string
|
||||
onTokenUsage?: (usage: ChatTokenUsage) => void
|
||||
live?: LiveToolSet
|
||||
}
|
||||
): Promise<ParsedCompletionResult> {
|
||||
let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = []
|
||||
let error: OpenAIError | ResponseErrorEvent | null = null
|
||||
@@ -620,7 +625,8 @@ export async function parseOpenAIResponsesCompletion(
|
||||
helpers,
|
||||
toolCallbacks: callbacks,
|
||||
workspace: options?.workspace,
|
||||
messages
|
||||
messages,
|
||||
live: options?.live
|
||||
})
|
||||
messages.push(messageToAdd)
|
||||
addedMessages.push(messageToAdd)
|
||||
|
||||
@@ -39,7 +39,7 @@ const sampleContext: PipelineContext = {
|
||||
}
|
||||
|
||||
function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
|
||||
helpers: { pipeline: PipelineAIChatHelpers }
|
||||
helpers: { pipelines: () => PipelineAIChatHelpers[] }
|
||||
calls: Record<string, any[]>
|
||||
} {
|
||||
const calls: Record<string, any[]> = {}
|
||||
@@ -50,6 +50,8 @@ function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
|
||||
return ret
|
||||
}
|
||||
const pipeline: PipelineAIChatHelpers = {
|
||||
getFolder: () => sampleContext.folder,
|
||||
hasDrafts: () => false,
|
||||
getPipelineContext: () => sampleContext,
|
||||
getNodeBody: async (path: string) => {
|
||||
calls.getNodeBody = [...(calls.getNodeBody ?? []), [path]]
|
||||
@@ -59,15 +61,20 @@ function makeHelpers(overrides: Partial<PipelineAIChatHelpers> = {}): {
|
||||
calls.proposeNode = [...(calls.proposeNode ?? []), [input]]
|
||||
return { path: input.path, detectedReads: [], detectedWrites: [] }
|
||||
},
|
||||
editNode: async (path, content) => {
|
||||
calls.editNode = [...(calls.editNode ?? []), [path, content]]
|
||||
return { detectedReads: [], detectedWrites: [] }
|
||||
editNode: async (path, content, summary) => {
|
||||
calls.editNode = [...(calls.editNode ?? []), [path, content, summary]]
|
||||
return { summary: summary ?? 'Existing summary', detectedReads: [], detectedWrites: [] }
|
||||
},
|
||||
removeProposedNode: record('removeProposedNode'),
|
||||
unconfiguredTriggers: async () => [],
|
||||
setNodeTrigger: async (path, kind, config) => {
|
||||
calls.setNodeTrigger = [...(calls.setNodeTrigger ?? []), [path, kind, config]]
|
||||
return { path: `${path}_${kind}`, replaced: false }
|
||||
},
|
||||
testNode: async () => 'job-123',
|
||||
...overrides
|
||||
}
|
||||
return { helpers: { pipeline }, calls }
|
||||
return { helpers: { pipelines: () => [pipeline] }, calls }
|
||||
}
|
||||
|
||||
describe('pipeline tools', () => {
|
||||
@@ -78,6 +85,7 @@ describe('pipeline tools', () => {
|
||||
'get_pipeline_graph',
|
||||
'read_pipeline_node',
|
||||
'remove_pipeline_node',
|
||||
'set_pipeline_trigger',
|
||||
'test_pipeline_node'
|
||||
])
|
||||
})
|
||||
@@ -96,18 +104,25 @@ describe('pipeline tools', () => {
|
||||
|
||||
it('build_pipeline_node forwards to proposeNode and does not deploy', async () => {
|
||||
const { helpers, calls } = makeHelpers()
|
||||
const modified: string[] = []
|
||||
const out = await toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/analytics/clean',
|
||||
language: 'bun',
|
||||
summary: 'Test node',
|
||||
content: '// pipeline\nexport async function main() {}',
|
||||
output_kind: 'ducklake'
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolCallbacks: {
|
||||
...noopCallbacks(),
|
||||
onItemModified: (kind, path) => modified.push(`${kind}:${path}`)
|
||||
},
|
||||
toolId: 't'
|
||||
})
|
||||
// The session tracks the folder's draft bundle, the unit it deploys.
|
||||
expect(modified).toEqual(['data_pipeline:f/analytics/data_pipeline'])
|
||||
expect(calls.proposeNode?.[0]?.[0]).toMatchObject({
|
||||
path: 'f/analytics/clean',
|
||||
language: 'bun',
|
||||
@@ -116,6 +131,99 @@ describe('pipeline tools', () => {
|
||||
expect(out).toContain('not deployed')
|
||||
})
|
||||
|
||||
it('build_pipeline_node attaches declared triggers, and flags one left unset', async () => {
|
||||
const { helpers, calls } = makeHelpers({ unconfiguredTriggers: async () => ['kafka'] })
|
||||
const out = await toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/analytics/ingest',
|
||||
language: 'python3',
|
||||
summary: 'Test node',
|
||||
content: '# pipeline\n# on schedule\n# on kafka\ndef main():\n return 1',
|
||||
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
expect(calls.setNodeTrigger?.[0]).toEqual([
|
||||
'f/analytics/ingest',
|
||||
'schedule',
|
||||
{ args: {}, enabled: true, schedule: '0 0 6 * * *', timezone: 'UTC' }
|
||||
])
|
||||
expect(out).toContain("schedule 'f/analytics/ingest_schedule'")
|
||||
expect(out).toContain('ACTION REQUIRED')
|
||||
expect(out).toContain('set_pipeline_trigger')
|
||||
})
|
||||
|
||||
it('build_pipeline_node rejects an invalid trigger config before staging the node', async () => {
|
||||
const { helpers, calls } = makeHelpers()
|
||||
await expect(
|
||||
toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/analytics/ingest',
|
||||
language: 'python3',
|
||||
summary: 'Test node',
|
||||
content: '# pipeline\n# on schedule\ndef main():\n return 1',
|
||||
triggers: [{ kind: 'schedule', config: { timezone: 'UTC' } }]
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
).rejects.toThrow(/schedule/)
|
||||
expect(calls.proposeNode).toBeUndefined()
|
||||
})
|
||||
|
||||
it('build_pipeline_node keeps and records the node when a trigger fails to attach', async () => {
|
||||
const { helpers, calls } = makeHelpers({
|
||||
setNodeTrigger: async () => {
|
||||
throw new Error('path taken')
|
||||
}
|
||||
})
|
||||
const modified: string[] = []
|
||||
const out = await toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/analytics/ingest',
|
||||
language: 'python3',
|
||||
content: '# pipeline\n# on schedule\ndef main():\n return 1',
|
||||
summary: 'Ingest',
|
||||
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: {
|
||||
...noopCallbacks(),
|
||||
onItemModified: (kind, path) => modified.push(`${kind}:${path}`)
|
||||
},
|
||||
toolId: 't'
|
||||
})
|
||||
expect(calls.proposeNode).toHaveLength(1)
|
||||
expect(modified).toEqual(['data_pipeline:f/analytics/data_pipeline'])
|
||||
expect(out).toContain('path taken')
|
||||
})
|
||||
|
||||
it('build_pipeline_node refuses a trigger the content does not declare, before staging', async () => {
|
||||
const { helpers, calls } = makeHelpers()
|
||||
await expect(
|
||||
toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/analytics/ingest',
|
||||
language: 'python3',
|
||||
content: '# pipeline\ndef main():\n return 1',
|
||||
summary: 'Ingest',
|
||||
triggers: [{ kind: 'schedule', config: { schedule: '0 0 6 * * *', timezone: 'UTC' } }]
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
).rejects.toThrow(/on schedule/)
|
||||
expect(calls.proposeNode).toBeUndefined()
|
||||
})
|
||||
|
||||
it('build_pipeline_node reports the inferred asset lineage', async () => {
|
||||
const { helpers } = makeHelpers({
|
||||
proposeNode: async (input) => ({
|
||||
@@ -125,7 +233,12 @@ describe('pipeline tools', () => {
|
||||
})
|
||||
})
|
||||
const out = await toolByName('build_pipeline_node').fn({
|
||||
args: { path: 'f/analytics/clean', language: 'duckdb', content: '-- pipeline' },
|
||||
args: {
|
||||
path: 'f/analytics/clean',
|
||||
language: 'duckdb',
|
||||
content: '-- pipeline',
|
||||
summary: 'Test node'
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
@@ -140,7 +253,12 @@ describe('pipeline tools', () => {
|
||||
proposeNode: async (input) => ({ path: input.path, detectedReads: [], detectedWrites: [] })
|
||||
})
|
||||
const out = await toolByName('build_pipeline_node').fn({
|
||||
args: { path: 'f/analytics/clean', language: 'duckdb', content: '-- pipeline' },
|
||||
args: {
|
||||
path: 'f/analytics/clean',
|
||||
language: 'duckdb',
|
||||
content: '-- pipeline',
|
||||
summary: 'Test node'
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
@@ -164,6 +282,34 @@ describe('pipeline tools', () => {
|
||||
expect(calls.editNode?.[0]?.[1]).toContain('const x = 42')
|
||||
})
|
||||
|
||||
it('edit_pipeline_node renames a node by summary alone, and asks for one when missing', async () => {
|
||||
const body = { language: 'bun' as const, content: 'const x = 1\n' }
|
||||
const { helpers, calls } = makeHelpers({ getNodeBody: async () => body })
|
||||
const run = (args: Record<string, unknown>) =>
|
||||
toolByName('edit_pipeline_node').fn({
|
||||
args: { path: 'f/analytics/orders', ...args },
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
await run({ summary: 'Clean orders' })
|
||||
expect(calls.editNode?.[0]).toEqual(['f/analytics/orders', body.content, 'Clean orders'])
|
||||
|
||||
const bare = makeHelpers({
|
||||
getNodeBody: async () => body,
|
||||
editNode: async () => ({ summary: '', detectedReads: [], detectedWrites: [] })
|
||||
})
|
||||
const out = await toolByName('edit_pipeline_node').fn({
|
||||
args: { path: 'f/analytics/orders', old_string: 'x = 1', new_string: 'x = 2' },
|
||||
workspace: 'w',
|
||||
helpers: bare.helpers,
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
expect(out).toContain('has no summary')
|
||||
})
|
||||
|
||||
it('edit_pipeline_node surfaces a clear error when old_string is absent', async () => {
|
||||
const { helpers } = makeHelpers({
|
||||
getNodeBody: async () => ({ language: 'bun', content: 'const x = 1\n' })
|
||||
@@ -182,7 +328,7 @@ describe('pipeline tools', () => {
|
||||
it('mutation tools fail clearly when no pipeline editor is registered', async () => {
|
||||
await expect(
|
||||
toolByName('build_pipeline_node').fn({
|
||||
args: { path: 'f/a/b', language: 'bun', content: 'x' },
|
||||
args: { path: 'f/a/b', language: 'bun', content: 'x', summary: 'Test node' },
|
||||
workspace: 'w',
|
||||
helpers: {},
|
||||
toolCallbacks: noopCallbacks(),
|
||||
@@ -191,14 +337,114 @@ describe('pipeline tools', () => {
|
||||
).rejects.toThrow(/No pipeline editor is open/)
|
||||
})
|
||||
|
||||
it('remove_pipeline_node drops the pipeline from the changes once its last draft goes', async () => {
|
||||
const run = async (hasDrafts: boolean) => {
|
||||
const { helpers } = makeHelpers({ hasDrafts: () => hasDrafts })
|
||||
const log: string[] = []
|
||||
await toolByName('remove_pipeline_node').fn({
|
||||
args: { path: 'f/analytics/clean' },
|
||||
workspace: 'w',
|
||||
helpers,
|
||||
toolCallbacks: {
|
||||
...noopCallbacks(),
|
||||
onItemModified: (kind, path) => log.push(`modified ${kind}:${path}`),
|
||||
onItemDiscarded: (kind, path) => log.push(`discarded ${kind}:${path}`)
|
||||
},
|
||||
toolId: 't'
|
||||
})
|
||||
return log
|
||||
}
|
||||
expect(await run(false)).toEqual(['discarded data_pipeline:f/analytics/data_pipeline'])
|
||||
expect(await run(true)).toEqual(['modified data_pipeline:f/analytics/data_pipeline'])
|
||||
})
|
||||
|
||||
it('test_pipeline_node requires confirmation', () => {
|
||||
expect(toolByName('test_pipeline_node').requiresConfirmation).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('pipeline tools with several editors open', () => {
|
||||
function editorFor(folder: string, built: string[]): PipelineAIChatHelpers {
|
||||
return {
|
||||
...makeHelpers().helpers.pipelines()[0],
|
||||
getFolder: () => folder,
|
||||
getPipelineContext: () => ({ ...sampleContext, folder }),
|
||||
proposeNode: async (input) => {
|
||||
built.push(`${folder}:${input.path}`)
|
||||
return { path: input.path, detectedReads: [], detectedWrites: [] }
|
||||
}
|
||||
}
|
||||
}
|
||||
const call = (name: string, args: any, editors: PipelineAIChatHelpers[]) =>
|
||||
toolByName(name).fn({
|
||||
args,
|
||||
workspace: 'w',
|
||||
helpers: { pipelines: () => editors },
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
|
||||
it('remounts an opened folder whose editor another tab replaced', async () => {
|
||||
const built: string[] = []
|
||||
const mounted: PipelineAIChatHelpers[] = []
|
||||
const ensured: string[] = []
|
||||
await toolByName('build_pipeline_node').fn({
|
||||
args: {
|
||||
path: 'f/sales/clean',
|
||||
language: 'bun',
|
||||
content: '// pipeline',
|
||||
summary: 'Test node'
|
||||
},
|
||||
workspace: 'w',
|
||||
helpers: {
|
||||
pipelines: () => mounted,
|
||||
pipelineFolders: () => ['sales'],
|
||||
ensurePipeline: async (folder: string) => {
|
||||
ensured.push(folder)
|
||||
mounted.push(editorFor(folder, built))
|
||||
return mounted[0]
|
||||
}
|
||||
},
|
||||
toolCallbacks: noopCallbacks(),
|
||||
toolId: 't'
|
||||
})
|
||||
expect(ensured).toEqual(['sales'])
|
||||
expect(built).toEqual(['sales:f/sales/clean'])
|
||||
})
|
||||
|
||||
it('routes a node to the editor of its folder', async () => {
|
||||
const built: string[] = []
|
||||
const editors = [editorFor('crm', built), editorFor('sales', built)]
|
||||
await call(
|
||||
'build_pipeline_node',
|
||||
{ path: 'f/sales/clean', language: 'bun', content: '// pipeline', summary: 'Test node' },
|
||||
editors
|
||||
)
|
||||
expect(built).toEqual(['sales:f/sales/clean'])
|
||||
await expect(
|
||||
call(
|
||||
'build_pipeline_node',
|
||||
{ path: 'f/other/clean', language: 'bun', content: '// pipeline', summary: 'Test node' },
|
||||
editors
|
||||
)
|
||||
).rejects.toThrow(/open_preview\(kind="pipeline", path="other"\)/)
|
||||
})
|
||||
|
||||
it('get_pipeline_graph needs the folder only when it is ambiguous', async () => {
|
||||
const editors = [editorFor('crm', []), editorFor('sales', [])]
|
||||
await expect(call('get_pipeline_graph', {}, editors)).rejects.toThrow(/pass the folder/)
|
||||
expect(
|
||||
JSON.parse(await call('get_pipeline_graph', { folder: 'f/sales' }, editors))
|
||||
).toMatchObject({ folder: 'sales' })
|
||||
expect(JSON.parse(await call('get_pipeline_graph', {}, [editors[0]]))).toMatchObject({
|
||||
folder: 'crm'
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('getPipelinePromptSection', () => {
|
||||
it('names the active folder and the direct-draft workflow', () => {
|
||||
const section = getPipelinePromptSection(sampleContext)
|
||||
const section = getPipelinePromptSection([sampleContext.folder])
|
||||
expect(section).toContain('/pipeline/analytics')
|
||||
expect(section).toContain('build_pipeline_node')
|
||||
expect(section).toContain('directly as unsaved drafts')
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { z } from 'zod'
|
||||
import { $ScriptLang } from '$lib/gen/schemas.gen'
|
||||
import type { ScriptLang } from '$lib/gen'
|
||||
import { createToolDef, executeTestRun, findAndReplace } from '../shared'
|
||||
import { createToolDef, executeTestRun, findAndReplace, type ToolCallbacks } from '../shared'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
|
||||
import {
|
||||
NONE,
|
||||
RUN_PREVIEW,
|
||||
@@ -10,6 +11,11 @@ import {
|
||||
type SessionTool
|
||||
} from '../sessionCapabilities'
|
||||
import type { PipelineOutputKind } from '$lib/components/assets/AssetGraph/pipelineTemplates'
|
||||
import type { PipelineTriggerDraftKind } from '$lib/components/assets/AssetGraph/types'
|
||||
import { DRAFTABLE_TRIGGER_KINDS } from '$lib/components/assets/AssetGraph/pipelineTriggerDrafts'
|
||||
import { scheduleRequestSchema, triggerRequestSchemas } from '../workspaceToolsZod.gen'
|
||||
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
|
||||
import { parsePipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations'
|
||||
|
||||
// ============================================================================
|
||||
// Pipeline AI chat tools.
|
||||
@@ -57,6 +63,10 @@ export type PipelineContext = {
|
||||
* small — the page owns the draft Map and canvas rendering.
|
||||
*/
|
||||
export interface PipelineAIChatHelpers {
|
||||
/** Folder name (no `f/` prefix) of the pipeline this editor shows. */
|
||||
getFolder: () => string
|
||||
/** Whether the folder has any script or trigger draft left. */
|
||||
hasDrafts: () => boolean
|
||||
getPipelineContext: () => PipelineContext
|
||||
/** Read a node's source (the in-flight draft body if one exists, else deployed). */
|
||||
getNodeBody: (path: string) => Promise<{ language: ScriptLang; content: string } | undefined>
|
||||
@@ -68,29 +78,108 @@ export interface PipelineAIChatHelpers {
|
||||
language: ScriptLang
|
||||
content: string
|
||||
outputKind?: PipelineOutputKind
|
||||
summary?: string
|
||||
}) => Promise<{ path: string; detectedReads: string[]; detectedWrites: string[] }>
|
||||
/** Replace an existing node's body, applied as an unsaved draft. Returns the
|
||||
* re-inferred asset lineage (URIs) so the caller sees the effect of the edit. */
|
||||
editNode: (
|
||||
path: string,
|
||||
content: string
|
||||
) => Promise<{ detectedReads: string[]; detectedWrites: string[] }>
|
||||
content: string,
|
||||
summary?: string
|
||||
) => Promise<{ summary: string; detectedReads: string[]; detectedWrites: string[] }>
|
||||
/** Discard the unsaved draft at a path (undo a build_pipeline_node). */
|
||||
removeProposedNode: (path: string) => Promise<void>
|
||||
/** Trigger kinds the node declares (`on <kind>`) that no trigger row or draft fills. */
|
||||
unconfiguredTriggers: (path: string) => Promise<PipelineTriggerDraftKind[]>
|
||||
/** Attach (or replace) the node's trigger draft of that kind, deployed with the
|
||||
* pipeline. `config` is the kind's trigger config; path/script_path are filled in. */
|
||||
setNodeTrigger: (
|
||||
path: string,
|
||||
kind: PipelineTriggerDraftKind,
|
||||
config: Record<string, any>
|
||||
) => Promise<{ path: string; replaced: boolean }>
|
||||
/** Preview-run a node (draft body preferred). Returns the started job id. */
|
||||
testNode: (path: string, args?: Record<string, any>) => Promise<string | undefined>
|
||||
}
|
||||
|
||||
/** Helper bag the pipeline tools receive from the manager in global mode. */
|
||||
export type PipelineToolHelpers = { pipeline?: PipelineAIChatHelpers }
|
||||
/** Helper bag the pipeline tools receive from the manager in global mode: every
|
||||
* pipeline editor currently open, one per folder. */
|
||||
export type PipelineToolHelpers = {
|
||||
pipelines?: () => readonly PipelineAIChatHelpers[]
|
||||
/** Folders the chat may work on, including ones whose editor is not mounted now. */
|
||||
pipelineFolders?: () => readonly string[]
|
||||
/** Remounts the folder's editor (a session reopens its preview tab) and resolves
|
||||
* to it, or `undefined` when it cannot be brought back. */
|
||||
ensurePipeline?: (folder: string) => Promise<PipelineAIChatHelpers | undefined>
|
||||
}
|
||||
|
||||
function requirePipeline(helpers: PipelineToolHelpers): PipelineAIChatHelpers {
|
||||
if (!helpers?.pipeline) {
|
||||
/** The folder's editor, remounted when the chat opened it before and something else
|
||||
* has taken its tab since. */
|
||||
async function remount(
|
||||
helpers: PipelineToolHelpers,
|
||||
folder: string | undefined
|
||||
): Promise<PipelineAIChatHelpers | undefined> {
|
||||
if (!folder || !helpers?.ensurePipeline) return undefined
|
||||
if (!(helpers.pipelineFolders?.() ?? []).includes(folder)) return undefined
|
||||
return helpers.ensurePipeline(folder)
|
||||
}
|
||||
|
||||
function openPipelines(helpers: PipelineToolHelpers): readonly PipelineAIChatHelpers[] {
|
||||
const open = helpers?.pipelines?.() ?? []
|
||||
if (open.length === 0) {
|
||||
throw new Error(
|
||||
'No pipeline editor is open. Pipeline tools only work on a /pipeline/<folder> page in edit mode.'
|
||||
)
|
||||
}
|
||||
return helpers.pipeline
|
||||
return open
|
||||
}
|
||||
|
||||
function notOpenError(folder: string, open: readonly PipelineAIChatHelpers[]): Error {
|
||||
const names = open.map((p) => `'${p.getFolder()}'`).join(', ')
|
||||
return new Error(
|
||||
`No pipeline editor is open for folder '${folder}' (open: ${names}). Open it with open_preview(kind="pipeline", path="${folder}") first.`
|
||||
)
|
||||
}
|
||||
|
||||
/** The editor of the pipeline a node path belongs to (`f/<folder>/<node>`). With a
|
||||
* single editor open, a path outside it still goes to that editor, whose own check
|
||||
* names the corrected path. */
|
||||
async function pipelineForPath(
|
||||
helpers: PipelineToolHelpers,
|
||||
path: string
|
||||
): Promise<PipelineAIChatHelpers> {
|
||||
const folder = path.match(/^f\/([^/]+)\//)?.[1]
|
||||
const live = folder ? helpers?.pipelines?.().find((p) => p.getFolder() === folder) : undefined
|
||||
const match = live ?? (await remount(helpers, folder))
|
||||
if (match) return match
|
||||
const open = openPipelines(helpers)
|
||||
if (open.length === 1) return open[0]
|
||||
throw folder
|
||||
? notOpenError(folder, open)
|
||||
: new Error(
|
||||
`'${path}' is not a pipeline node path — nodes live at f/<folder>/<node_name>, with <folder> one of: ${open.map((p) => `'${p.getFolder()}'`).join(', ')}.`
|
||||
)
|
||||
}
|
||||
|
||||
async function pipelineForFolder(
|
||||
helpers: PipelineToolHelpers,
|
||||
folder: string | undefined
|
||||
): Promise<PipelineAIChatHelpers> {
|
||||
const known = helpers?.pipelineFolders?.() ?? []
|
||||
const name = folder ? normalizePipelineFolder(folder) : known.length === 1 ? known[0] : undefined
|
||||
const live = name ? helpers?.pipelines?.().find((p) => p.getFolder() === name) : undefined
|
||||
const remounted = live ?? (await remount(helpers, name))
|
||||
if (remounted) return remounted
|
||||
const open = openPipelines(helpers)
|
||||
if (folder) {
|
||||
const match = open.find((p) => p.getFolder() === name)
|
||||
if (!match) throw notOpenError(normalizePipelineFolder(folder), open)
|
||||
return match
|
||||
}
|
||||
if (open.length === 1) return open[0]
|
||||
throw new Error(
|
||||
`Several pipeline editors are open (${open.map((p) => `'${p.getFolder()}'`).join(', ')}) — pass the folder to read.`
|
||||
)
|
||||
}
|
||||
|
||||
const scriptLangSchema = z.enum($ScriptLang.enum)
|
||||
@@ -105,12 +194,20 @@ const outputKindSchema = z
|
||||
// Read tools
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
const getPipelineGraphSchema = z.object({})
|
||||
const getPipelineGraphSchema = z.object({
|
||||
folder: z
|
||||
.string()
|
||||
.optional()
|
||||
.describe(
|
||||
'Folder of the pipeline to read (e.g. `analytics`). Required when several pipeline editors are open; may be omitted when only one is.'
|
||||
)
|
||||
})
|
||||
|
||||
const getPipelineGraphToolDef = createToolDef(
|
||||
getPipelineGraphSchema,
|
||||
'get_pipeline_graph',
|
||||
"Read the live pipeline graph for the open /pipeline/<folder> editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG."
|
||||
"Read the live pipeline graph of an open /pipeline/<folder> editor: its nodes (scripts), each node's language, asset reads/writes, declared triggers, and whether it has an unsaved draft edit. Call this before building or editing nodes so you reuse existing assets/paths and understand the current DAG.",
|
||||
{ strict: false }
|
||||
)
|
||||
|
||||
const readPipelineNodeSchema = z.object({
|
||||
@@ -127,11 +224,75 @@ const readPipelineNodeToolDef = createToolDef(
|
||||
// Mutation tools (apply directly as unsaved drafts; never deploy)
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
const triggerKindEnum = z.enum(DRAFTABLE_TRIGGER_KINDS as [PipelineTriggerDraftKind, ...PipelineTriggerDraftKind[]])
|
||||
|
||||
const nodeTriggerSchema = z.object({
|
||||
kind: triggerKindEnum.describe('Trigger kind; the node must declare `on <kind>`.'),
|
||||
config: z
|
||||
.record(z.string(), z.any())
|
||||
.describe(
|
||||
'The trigger configuration. schedule: { schedule (6-field cron, e.g. "0 0 6 * * *"), timezone (IANA, e.g. "UTC"), args? (the node\'s inputs), summary? }. Any other kind: the fields get_trigger_schema returns for it. path, script_path and is_flow are filled in; pass path only to name the trigger yourself.'
|
||||
)
|
||||
})
|
||||
|
||||
const setPipelineTriggerSchema = nodeTriggerSchema.extend({
|
||||
path: z.string().describe('Workspace path of the node the trigger runs.')
|
||||
})
|
||||
|
||||
const setPipelineTriggerToolDef = createToolDef(
|
||||
setPipelineTriggerSchema,
|
||||
'set_pipeline_trigger',
|
||||
'Attach the trigger a node declares with `on <schedule|email|kafka|mqtt|amqp|nats|postgres|sqs|gcp>` (its cron, topic, queue…) as an unsaved trigger draft, deployed together with the pipeline. Calling it again for the same node and kind replaces that draft. Webhook and data_upload need no trigger.',
|
||||
{ strict: false }
|
||||
)
|
||||
|
||||
/** Validates a node trigger's config against the kind's request schema, with the
|
||||
* fields the pipeline fills in present so they are not reported missing. */
|
||||
function validateTriggerConfig(
|
||||
kind: PipelineTriggerDraftKind,
|
||||
config: Record<string, any>,
|
||||
nodePath: string
|
||||
): Record<string, any> {
|
||||
const filled = { path: `${nodePath}_${kind}`, script_path: nodePath, is_flow: false, ...config }
|
||||
if (kind === 'schedule') {
|
||||
const parsed = scheduleRequestSchema.safeParse({ args: {}, enabled: true, ...filled })
|
||||
if (!parsed.success) {
|
||||
throw new Error(
|
||||
`Invalid schedule config: ${parsed.error.issues.map((i) => `${i.path.join('.') || '<root>'}: ${i.message}`).join('; ')}`
|
||||
)
|
||||
}
|
||||
return { args: {}, enabled: true, ...config }
|
||||
}
|
||||
const parsed = triggerRequestSchemas[kind].safeParse(filled)
|
||||
if (!parsed.success) {
|
||||
throw new Error(
|
||||
`Invalid config for a "${kind}" trigger. Call get_trigger_schema with kind "${kind}" for its exact fields. Issues: ${parsed.error.issues.map((i) => `${i.path.join('.') || '<root>'}: ${i.message}`).join('; ')}`
|
||||
)
|
||||
}
|
||||
return config
|
||||
}
|
||||
|
||||
/** The note every node write ends with while a declared trigger is still unset: the
|
||||
* node would deploy with nothing to start it. */
|
||||
function unconfiguredTriggersNote(path: string, kinds: readonly string[]): string {
|
||||
if (kinds.length === 0) return ''
|
||||
return ` ACTION REQUIRED: '${path}' declares \`on ${kinds.join('`, `on ')}\` but has no ${kinds.join('/')} trigger yet, so nothing would start it once deployed. Call set_pipeline_trigger for it now (a schedule needs its cron and timezone; ask the user only if the cadence cannot be inferred).`
|
||||
}
|
||||
|
||||
const summaryField = z
|
||||
.string()
|
||||
.trim()
|
||||
.min(1)
|
||||
.max(80)
|
||||
.describe(
|
||||
'Succinct summary of what the node does, a few words in sentence case with no trailing period (e.g. "Ingest Stripe charges", "Daily revenue by region"). It is the title the node shows in the workspace.'
|
||||
)
|
||||
|
||||
const buildPipelineNodeSchema = z.object({
|
||||
path: z
|
||||
.string()
|
||||
.describe(
|
||||
"Workspace path for the new node, e.g. f/<folder>/<name>. Use the open pipeline's folder. Must not collide with an existing node."
|
||||
'Workspace path for the new node, e.g. f/<folder>/<name>, in the folder of an open pipeline editor. Must not collide with an existing node.'
|
||||
),
|
||||
language: scriptLangSchema.describe(
|
||||
'Script language. SQL-shaped data work uses duckdb (DuckLake/S3) or postgresql (data tables); bun/python3 for general transforms.'
|
||||
@@ -141,7 +302,14 @@ const buildPipelineNodeSchema = z.object({
|
||||
.describe(
|
||||
"Full script source. Start it with the `pipeline` annotation as a top-of-file comment in the LANGUAGE'S comment syntax — `-- pipeline` for SQL (duckdb/postgresql), `# pipeline` for python3/bash, `// pipeline` for bun/TS — to mark it a pipeline member; declare inputs the same way (e.g. `-- on <asset-uri|schedule|webhook|...>`), and write outputs via the wmill SDK / SQL so the lineage edges are inferred. A `// pipeline` line in a SQL node is a syntax error. Read existing node bodies first to match conventions."
|
||||
),
|
||||
output_kind: outputKindSchema.optional()
|
||||
summary: summaryField,
|
||||
output_kind: outputKindSchema.optional(),
|
||||
triggers: z
|
||||
.array(nodeTriggerSchema)
|
||||
.optional()
|
||||
.describe(
|
||||
'Triggers for the `on <kind>` lines the content declares (schedule, email, kafka, …), attached as trigger drafts in the same step. Every such declaration needs one; webhook and data_upload need none.'
|
||||
)
|
||||
})
|
||||
|
||||
const buildPipelineNodeToolDef = createToolDef(
|
||||
@@ -153,8 +321,17 @@ const buildPipelineNodeToolDef = createToolDef(
|
||||
|
||||
const editPipelineNodeSchema = z.object({
|
||||
path: z.string().describe('Workspace path of the node to edit.'),
|
||||
old_string: z.string().min(1).describe("Exact text to find in the node's current source."),
|
||||
new_string: z.string().describe('Replacement text.'),
|
||||
old_string: z
|
||||
.string()
|
||||
.min(1)
|
||||
.optional()
|
||||
.describe("Exact text to find in the node's current source. Omit to change only the summary."),
|
||||
new_string: z.string().optional().describe('Replacement text.'),
|
||||
summary: summaryField
|
||||
.optional()
|
||||
.describe(
|
||||
'New succinct summary (a few words, no trailing period). Pass it whenever the node has none or the edit changes what the node does.'
|
||||
),
|
||||
replace_all: z
|
||||
.boolean()
|
||||
.optional()
|
||||
@@ -197,6 +374,12 @@ const testPipelineNodeToolDef = createToolDef(
|
||||
{ strict: false }
|
||||
)
|
||||
|
||||
// A pipeline's drafts are one bundle for the whole folder, so that bundle is the
|
||||
// item a node edit marks as modified — it is what the session deploys as a unit.
|
||||
function recordPipelineModified(toolCallbacks: ToolCallbacks, pipeline: PipelineAIChatHelpers) {
|
||||
toolCallbacks.onItemModified?.(PIPELINE_DRAFT_KIND, pipelineBundlePath(pipeline.getFolder()))
|
||||
}
|
||||
|
||||
// Summarize the asset lineage the parser inferred from a just-applied node so the
|
||||
// model gets same-turn feedback on whether its intended edges formed. An empty
|
||||
// result is the useful signal: a write/read expressed via a variable or dynamic
|
||||
@@ -217,8 +400,9 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
requires: NONE,
|
||||
def: getPipelineGraphToolDef,
|
||||
planModeSafe: true,
|
||||
fn: async ({ helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const { folder } = getPipelineGraphSchema.parse(args ?? {})
|
||||
const pipeline = await pipelineForFolder(helpers, folder)
|
||||
toolCallbacks.setToolStatus(toolId, { content: 'Reading pipeline graph...' })
|
||||
const ctx = pipeline.getPipelineContext()
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
@@ -233,8 +417,8 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
def: readPipelineNodeToolDef,
|
||||
planModeSafe: true,
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
const { path } = readPipelineNodeSchema.parse(args)
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
toolCallbacks.setToolStatus(toolId, { content: `Reading node '${path}'...` })
|
||||
const node = await pipeline.getNodeBody(path)
|
||||
if (!node) {
|
||||
@@ -251,20 +435,51 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
showDetails: true,
|
||||
showFade: true,
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
const { path, language, content, output_kind } = buildPipelineNodeSchema.parse(args)
|
||||
const { path, language, content, summary, output_kind, triggers } =
|
||||
buildPipelineNodeSchema.parse(args)
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
// Checked before the node is staged, so a bad trigger leaves nothing half-built.
|
||||
const declared = new Set(parsePipelineAnnotations(content).nativeTriggers.map((n) => n.kind))
|
||||
const configs = (triggers ?? []).map((t) => {
|
||||
if (!declared.has(t.kind)) {
|
||||
throw new Error(
|
||||
`The content declares no \`on ${t.kind}\`, so a ${t.kind} trigger has nothing to bind to. Add the annotation line or drop that trigger. Nothing was built.`
|
||||
)
|
||||
}
|
||||
return { kind: t.kind, config: validateTriggerConfig(t.kind, t.config, path) }
|
||||
})
|
||||
toolCallbacks.setToolStatus(toolId, { content: `Building node '${path}'...` })
|
||||
const { detectedReads, detectedWrites } = await pipeline.proposeNode({
|
||||
path,
|
||||
language: language as ScriptLang,
|
||||
content,
|
||||
summary,
|
||||
outputKind: output_kind as PipelineOutputKind | undefined
|
||||
})
|
||||
// Recorded as soon as it is staged: a trigger that fails below leaves the node.
|
||||
recordPipelineModified(toolCallbacks, pipeline)
|
||||
const attached: string[] = []
|
||||
const failed: string[] = []
|
||||
for (const t of configs) {
|
||||
try {
|
||||
const { path: triggerPath } = await pipeline.setNodeTrigger(path, t.kind, t.config)
|
||||
attached.push(`${t.kind} '${triggerPath}'`)
|
||||
} catch (e) {
|
||||
failed.push(`${t.kind}: ${e instanceof Error ? e.message : String(e)}`)
|
||||
}
|
||||
}
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Added draft node '${path}'`,
|
||||
result: 'Success'
|
||||
})
|
||||
return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.${inferredLineageNote(detectedReads, detectedWrites)}`
|
||||
const attachedNote =
|
||||
(attached.length
|
||||
? ` Attached trigger draft${attached.length > 1 ? 's' : ''}: ${attached.join(', ')}.`
|
||||
: '') +
|
||||
(failed.length
|
||||
? ` The node was built, but these triggers were not attached — fix them with set_pipeline_trigger: ${failed.join('; ')}.`
|
||||
: '')
|
||||
return `Pipeline node '${path}' added as an unsaved draft on the canvas. It is not deployed — the user deploys it.${attachedNote}${inferredLineageNote(detectedReads, detectedWrites)}${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -274,36 +489,74 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
showDetails: true,
|
||||
showFade: true,
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
const { path, old_string, new_string, replace_all } = editPipelineNodeSchema.parse(args)
|
||||
const { path, old_string, new_string, replace_all, summary } =
|
||||
editPipelineNodeSchema.parse(args)
|
||||
if (old_string === undefined && summary === undefined) {
|
||||
throw new Error('Pass old_string/new_string to change the source, summary to rename it, or both.')
|
||||
}
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
const node = await pipeline.getNodeBody(path)
|
||||
if (!node) {
|
||||
return `No pipeline node found at '${path}'. Call get_pipeline_graph to list the available nodes.`
|
||||
}
|
||||
toolCallbacks.setToolStatus(toolId, { content: `Editing node '${path}'...` })
|
||||
const updated = findAndReplace(
|
||||
node.content,
|
||||
old_string,
|
||||
new_string,
|
||||
replace_all ?? false,
|
||||
'node source'
|
||||
)
|
||||
const { detectedReads, detectedWrites } = await pipeline.editNode(path, updated)
|
||||
const updated =
|
||||
old_string === undefined
|
||||
? node.content
|
||||
: findAndReplace(
|
||||
node.content,
|
||||
old_string,
|
||||
new_string ?? '',
|
||||
replace_all ?? false,
|
||||
'node source'
|
||||
)
|
||||
const edited = await pipeline.editNode(path, updated, summary)
|
||||
const { detectedReads, detectedWrites } = edited
|
||||
const summaryNote = edited.summary.trim()
|
||||
? ''
|
||||
: ` '${path}' has no summary: call edit_pipeline_node with just \`summary\` (a few words saying what it does).`
|
||||
recordPipelineModified(toolCallbacks, pipeline)
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Edited draft '${path}'`,
|
||||
result: 'Success'
|
||||
})
|
||||
return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).${inferredLineageNote(detectedReads, detectedWrites)}`
|
||||
return `Pipeline node '${path}' updated as an unsaved draft on the canvas (not deployed).${inferredLineageNote(detectedReads, detectedWrites)}${summaryNote}${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
|
||||
}
|
||||
},
|
||||
{
|
||||
requires: WRITE_DRAFT,
|
||||
def: setPipelineTriggerToolDef,
|
||||
showDetails: true,
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const { path, kind, config } = setPipelineTriggerSchema.parse(args)
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
toolCallbacks.setToolStatus(toolId, { content: `Setting the ${kind} trigger of '${path}'...` })
|
||||
const valid = validateTriggerConfig(kind, config, path)
|
||||
const { path: triggerPath, replaced } = await pipeline.setNodeTrigger(path, kind, valid)
|
||||
recordPipelineModified(toolCallbacks, pipeline)
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `${replaced ? 'Updated' : 'Attached'} ${kind} trigger '${triggerPath}'`,
|
||||
result: 'Success'
|
||||
})
|
||||
return `${replaced ? 'Replaced' : 'Attached'} the ${kind} trigger draft '${triggerPath}' on '${path}'. It deploys with the pipeline.${unconfiguredTriggersNote(path, await pipeline.unconfiguredTriggers(path))}`
|
||||
}
|
||||
},
|
||||
{
|
||||
requires: WRITE_DRAFT,
|
||||
def: removePipelineNodeToolDef,
|
||||
fn: async ({ args, helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
const { path } = removePipelineNodeSchema.parse(args)
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
toolCallbacks.setToolStatus(toolId, { content: `Discarding draft '${path}'...` })
|
||||
await pipeline.removeProposedNode(path)
|
||||
// With the last draft gone there is nothing left to deploy: the pipeline
|
||||
// is no longer this chat's change, and listing it would show it deployed.
|
||||
if (pipeline.hasDrafts()) recordPipelineModified(toolCallbacks, pipeline)
|
||||
else
|
||||
toolCallbacks.onItemDiscarded?.(
|
||||
PIPELINE_DRAFT_KIND,
|
||||
pipelineBundlePath(pipeline.getFolder())
|
||||
)
|
||||
toolCallbacks.setToolStatus(toolId, {
|
||||
content: `Discarded draft '${path}'`,
|
||||
result: 'Success'
|
||||
@@ -319,8 +572,8 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
showDetails: true,
|
||||
autoCollapseDetails: false,
|
||||
fn: async ({ args, workspace, helpers, toolId, toolCallbacks }) => {
|
||||
const pipeline = requirePipeline(helpers)
|
||||
const { path, args: runArgs } = testPipelineNodeSchema.parse(args)
|
||||
const pipeline = await pipelineForPath(helpers, path)
|
||||
return executeTestRun({
|
||||
jobStarter: async () => {
|
||||
const jobId = await pipeline.testNode(path, runArgs ?? undefined)
|
||||
@@ -350,22 +603,35 @@ export const pipelineTools: SessionTool<PipelineToolHelpers>[] = [
|
||||
* since the tools it names are withheld from that session; the annotation model stays,
|
||||
* as it is what lets the model read and explain an existing pipeline.
|
||||
*/
|
||||
export function getPipelinePromptSection(ctx: PipelineContext, access?: SessionAccess): string {
|
||||
export function getPipelinePromptSection(
|
||||
folders: readonly string[],
|
||||
access?: SessionAccess
|
||||
): string {
|
||||
const canWriteDraft = !access || access.has('write_draft')
|
||||
const [first] = folders
|
||||
const openLine =
|
||||
folders.length === 1
|
||||
? `The user has the /pipeline/${first} editor open.`
|
||||
: `The user has ${folders.length} pipeline editors open: ${folders.map((f) => `/pipeline/${f}`).join(', ')}. Each tool acts on the pipeline its node path's folder names; pass \`folder\` to get_pipeline_graph.`
|
||||
const pathLine =
|
||||
folders.length === 1
|
||||
? `Every node of this pipeline lives at \`f/${first}/<node_name>\` — \`${first}\` is the folder name and \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`).`
|
||||
: `Every node of a pipeline lives at \`f/<folder>/<node_name>\` (e.g. \`f/${first}/<node_name>\`) — \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`). A node can only be built in the folder of an open pipeline editor.`
|
||||
return `
|
||||
|
||||
Data Pipeline editor (ACTIVE):
|
||||
- The user has the /pipeline/${ctx.folder} editor open. A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers.
|
||||
- ${openLine} A pipeline is a DAG of scripts (nodes) connected by storage assets (DuckLake tables, data tables, S3 objects, volumes, resources) and execution triggers.
|
||||
- Annotations are top-of-file comments in the NODE'S OWN comment syntax: \`--\` for SQL (duckdb/postgresql), \`#\` for python3/bash, \`//\` for bun/TS. The \`//\` shown below is the TS form — translate it (a \`// pipeline\` line in a SQL node is a syntax error that won't deploy).
|
||||
- A script becomes a pipeline node when its source starts with the \`// pipeline\` annotation. Declare execution-DAG inputs with \`// on <asset-uri | schedule | webhook | email | kafka | mqtt | amqp | nats | postgres | sqs | gcp | data_upload>\` (e.g. \`// on ducklake://main/orders\`). Outputs are inferred from what the body writes (wmill SDK calls / SQL CREATE TABLE / writeS3File); declare a managed output with \`// materialize <asset-uri>\`. Optional badges: \`// partitioned <daily|hourly|weekly|monthly|dynamic>\`, \`// freshness <duration>\`, \`// tag <name>\`, \`// retry <count> [delay]\`, \`// data_test <kind> ...\` (managed DuckLake targets only — deploy rejects it beside a \`dbt://\` target), \`// measure <name> = <agg> [where <pred>]\`, \`// dimension <name> = <expr>\`.
|
||||
- \`materialize\` (the managed output): a managed \`// materialize ducklake://<name>/<table>\` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. The \`dbt://\` target below is the opposite: the node writes its own DDL and none of the write strategies apply to it. IMPORTANT: a MANAGED \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake://<name>/<table>\`) — deploy rejects a \`ducklake://\` target on any other language. For a \`python3\`/\`bun\`/\`postgresql\` node writing the lake, do NOT use \`// materialize\`; write the output via the SDK instead (e.g. \`wmill.writeS3File(...)\`, a \`CREATE TABLE\` in postgresql, or \`wmill.databaseUrlFromResource\`/ducklake helpers) and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. The one target any language BUT DBT'S OWN may declare (a dbt project's writes come from its manifest, so \`// materialize\` on a dbt script is rejected at deploy) is a WAREHOUSE RELATION: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, with \`<warehouse>\` a warehouse the workspace configures under Settings → dbt. \`manual\` is its only mode — nothing generates warehouse DDL, so the node issues its own write and the annotation records the outcome. Use it on an ingestion node a dbt project reads as a \`source\`: the declared relation and the dbt model become ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when that node completes. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize <uri> append\` INSERT-appends rows (incremental); \`// materialize <uri> key=<col>\` merges/upserts on \`<col>\`. \`// materialize manual <uri>\` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned <daily|hourly|weekly|monthly|dynamic>\` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token — usable in any asset URI AND in the body SQL — is substituted with the current partition's IDENTITY string at run time. To filter the source to the active slice on a time grain, use the runtime-injected macro: \`WHERE wm_partition(<ts_col>) = {partition}\`. \`wm_partition(ts)\` buckets a timestamp with the exact identity format the runtime used (daily/hourly/weekly/monthly), so it always matches and you never hand-write a \`strftime\` format. Do NOT write \`= TIMESTAMP {partition}\`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at runtime. For \`dynamic\` partitioning the identity is your caller-supplied key (not a timestamp, no macro), so filter on it directly: \`WHERE <your_key_col> = {partition}\`. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run".
|
||||
- \`materialize\` (the managed output): a managed \`// materialize ducklake://<name>/<table>\` means the runtime writes the node's output table FOR you — write the body as a single SELECT and the runtime wraps it in the create/replace, so do NOT also write your own CREATE TABLE / INSERT. The \`dbt://\` target below is the opposite: the node writes its own DDL and none of the write strategies apply to it. IMPORTANT: a MANAGED \`// materialize\` is **DuckDB-only** and its target MUST be a DuckLake table (\`ducklake://<name>/<table>\`) — deploy rejects a \`ducklake://\` target on any other language. For a \`python3\`/\`bun\`/\`postgresql\` node writing the lake, do NOT use \`// materialize\`; write the table straight into DuckLake with \`wmill.ducklake()\` (or a data table with \`wmill.datatable()\`), or a \`CREATE TABLE\` in postgresql and let the output be inferred. Reach for \`duckdb\` when a node should materialize a DuckLake table. The one target any language BUT DBT'S OWN may declare (a dbt project's writes come from its manifest, so \`// materialize\` on a dbt script is rejected at deploy) is a WAREHOUSE RELATION: \`// materialize manual dbt://<warehouse>/<schema>/<name>\`, with \`<warehouse>\` a warehouse the workspace configures under Settings → dbt. \`manual\` is its only mode — nothing generates warehouse DDL, so the node issues its own write and the annotation records the outcome. Use it on an ingestion node a dbt project reads as a \`source\`: the declared relation and the dbt model become ONE graph node, and a downstream \`// on dbt://<warehouse>/<schema>/<name>\` fires when that node completes. Write strategy: with no option it REPLACES the whole table each run (full refresh; the only mode whose output columns may change); \`// materialize <uri> append\` INSERT-appends rows (incremental); \`// materialize <uri> key=<col>\` merges/upserts on \`<col>\`. \`// materialize manual <uri>\` opts OUT of managed writes — the script writes its own DDL and the annotation only records the output asset for lineage. \`materialize\` is paired with partitioning for incremental pipelines: a \`// partitioned <daily|hourly|weekly|monthly|dynamic>\` node runs once per partition (append/merge into a fixed-schema table), and the \`{partition}\` token — usable in any asset URI AND in the body SQL — is substituted with the current partition's IDENTITY string at run time. To filter the source to the active slice on a time grain, use the runtime-injected macro: \`WHERE wm_partition(<ts_col>) = {partition}\`. \`wm_partition(ts)\` buckets a timestamp with the exact identity format the runtime used (daily/hourly/weekly/monthly), so it always matches and you never hand-write a \`strftime\` format. Do NOT write \`= TIMESTAMP {partition}\`: the identity string is not a valid timestamp literal for hourly/weekly/monthly and errors at runtime. For \`dynamic\` partitioning the identity is your caller-supplied key (not a timestamp, no macro), so filter on it directly: \`WHERE <your_key_col> = {partition}\`. \`materialize\` is an output DECLARATION on the node — it is not a command; there is no "materialize run".
|
||||
- \`measure\` / \`dimension\` (declared metrics): on a node that materializes a DuckLake table, \`// measure <name> = <aggregate> [where <predicate>]\` names the canonical way to aggregate that table (e.g. \`// measure revenue = sum(amount) where not is_refund\`), and \`// dimension <name> = <expr>\` names a way to slice it (e.g. \`// dimension region = region\`, \`// dimension month = date_trunc('month', ordered_at)\`). They execute nothing: they are catalogued at deploy so the editor and other agents can reuse the definition instead of re-deriving it and silently disagreeing. Keep the predicate in the \`where\` clause rather than folding it into the aggregate: it is rendered as \`<agg> FILTER (WHERE <pred>)\`, which is what lets two measures with different predicates sit under one GROUP BY. DuckLake-only, and only meaningful next to \`// materialize\`. Declare one when a number carries a judgement call someone else would get wrong (refunds excluded, test rows dropped, which column is the amount); do NOT blanket every table with measures, an obvious \`count(*)\` earns nothing. To USE a metric another node declares, read that node with read_pipeline_node and reuse its exact expression rather than guessing it.
|
||||
- Use get_pipeline_graph to see the current nodes/assets/triggers, and read_pipeline_node before editing one.
|
||||
- Every node of this pipeline lives at \`f/${ctx.folder}/<node_name>\` — \`${ctx.folder}\` is the folder name and \`f/\` is the owner prefix every workspace path carries, so write it exactly once (never \`f/f/…\`, and never a bare \`<node_name>\`).${
|
||||
- ${pathLine}${
|
||||
canWriteDraft
|
||||
? `
|
||||
- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open.
|
||||
- Build new nodes with build_pipeline_node and edit existing ones with edit_pipeline_node. Give every node a succinct \`summary\` (a few words saying what it does, e.g. "Ingest Stripe charges"), and update it when an edit changes what the node does. These apply directly as unsaved drafts on the canvas (like the flow/script editor applies AI edits) — they DO NOT deploy. There is no separate Accept/Reject step. Prefer these over the generic write_script/edit_script draft tools while a pipeline is open.
|
||||
- Reuse existing asset paths from the graph when wiring a downstream node to an upstream one (read the upstream's write asset, then \`// on\` that same URI).
|
||||
- TRIGGERS: an ingestion node (one with no upstream asset to react to) must declare how it starts, e.g. \`// on schedule\`, and every \`on <schedule|email|kafka|mqtt|amqp|nats|postgres|sqs|gcp>\` line needs its trigger: pass it in build_pipeline_node's \`triggers\` (or call set_pipeline_trigger) in the same step you build the node. A schedule needs a 6-field cron and an IANA timezone — pick a sensible cadence from the request (e.g. daily at 06:00 UTC for a daily sync) rather than leaving it empty; for the other kinds call get_trigger_schema(kind) for the config fields and ask the user for any connection/resource you cannot find. Never leave a declared trigger without one: the node would deploy with nothing to start it. \`webhook\` and \`data_upload\` need no trigger.
|
||||
- Only deploy when the user explicitly asks; the user deploys drafts from the canvas.`
|
||||
: ''
|
||||
}`
|
||||
|
||||
@@ -395,6 +395,30 @@ describe('processToolCall', () => {
|
||||
expect(result.content).toBe(`Error while calling tool: ${expectedError}`)
|
||||
})
|
||||
|
||||
it('resolves a tool registered earlier in the same batch from the live set', async () => {
|
||||
const { createToolDef, processToolCall } = await import('./shared')
|
||||
const fn = vi.fn().mockResolvedValue('built')
|
||||
const registered = {
|
||||
def: createToolDef(z.object({}), 'build_pipeline_node', 'Build node'),
|
||||
fn
|
||||
}
|
||||
const result = await processToolCall({
|
||||
tools: [],
|
||||
live: () => ({ tools: [registered], helpers: { pipelines: true } }),
|
||||
toolCall: {
|
||||
id: 'call_live',
|
||||
type: 'function',
|
||||
function: { name: 'build_pipeline_node', arguments: '{}' }
|
||||
},
|
||||
helpers: {},
|
||||
workspace: 'test-workspace',
|
||||
toolCallbacks: { setToolStatus: vi.fn(), removeToolStatus: vi.fn() }
|
||||
})
|
||||
|
||||
expect(result.content).toBe('built')
|
||||
expect(fn).toHaveBeenCalledWith(expect.objectContaining({ helpers: { pipelines: true } }))
|
||||
})
|
||||
|
||||
it('continues to confirmation when pre-confirmation validation passes', async () => {
|
||||
const { createToolDef, processToolCall } = await import('./shared')
|
||||
const fn = vi.fn().mockResolvedValue('ok')
|
||||
|
||||
@@ -834,6 +834,13 @@ async function callTool<T>({
|
||||
}): Promise<string> {
|
||||
const tool = tools.find((t) => t.def.function.name === functionName)
|
||||
if (!tool) {
|
||||
// The pipeline tools exist only while a pipeline editor is mounted, and change_mode
|
||||
// cannot mount one, so the generic hint would send the model probing.
|
||||
if (/_pipeline_node$|^get_pipeline_graph$/.test(functionName)) {
|
||||
throw new Error(
|
||||
`${functionName} is not available yet: no pipeline editor is open for this chat. Call open_preview(kind="pipeline", path="<folder>") first and wait for its result; once it reports the editor open, the pipeline tools are callable. Do not fall back to write_script.`
|
||||
)
|
||||
}
|
||||
throw new Error(
|
||||
`Unknown tool call: ${functionName}. Probably not in the correct mode, use the change_mode tool to switch to the correct mode.`
|
||||
)
|
||||
@@ -924,22 +931,33 @@ type ToolCallStatus =
|
||||
| 'blocked_plan_mode'
|
||||
| 'held_for_instructions'
|
||||
|
||||
/** Re-reads the chat loop's current tools and helpers. A call earlier in the same
|
||||
* response can register tools (open_preview mounting the pipeline editor, change_mode),
|
||||
* so a later call in that batch must resolve against the live set, not the snapshot the
|
||||
* response was requested with. */
|
||||
export type LiveToolSet = () => { tools: Tool<any>[]; helpers: any }
|
||||
|
||||
export async function processToolCall<T>({
|
||||
tools,
|
||||
tools: snapshotTools,
|
||||
toolCall,
|
||||
helpers,
|
||||
helpers: snapshotHelpers,
|
||||
toolCallbacks,
|
||||
workspace,
|
||||
messages = []
|
||||
messages = [],
|
||||
live
|
||||
}: {
|
||||
tools: Tool<T>[]
|
||||
toolCall: ChatCompletionMessageFunctionToolCall
|
||||
helpers: T
|
||||
live?: LiveToolSet
|
||||
toolCallbacks: ToolCallbacks
|
||||
workspace?: string
|
||||
/** The conversation so far, for which folder instructions it already carries. */
|
||||
messages?: readonly ChatCompletionMessageParam[]
|
||||
}): Promise<ChatCompletionMessageParam> {
|
||||
const current = live?.()
|
||||
const tools: Tool<T>[] = current?.tools ?? snapshotTools
|
||||
const helpers: T = current ? current.helpers : snapshotHelpers
|
||||
const tool = tools.find((t) => t.def.function.name === toolCall.function.name)
|
||||
const workspaceId = workspace ?? get(workspaceStore) ?? ''
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@ import { formatResourceTypes } from './utils'
|
||||
import {
|
||||
appendPendingToolImages,
|
||||
processToolCall,
|
||||
type LiveToolSet,
|
||||
queuedToolStatus,
|
||||
type Tool,
|
||||
type ToolCallbacks
|
||||
@@ -1372,6 +1373,7 @@ export async function parseOpenAICompletion(
|
||||
workspace?: string
|
||||
provider?: string
|
||||
onTokenUsage?: (usage: ChatTokenUsage) => void
|
||||
live?: LiveToolSet
|
||||
}
|
||||
): Promise<{ shouldContinue: boolean; tokenUsage: ChatTokenUsage }> {
|
||||
const finalToolCalls: Record<number, ChatCompletionChunk.Choice.Delta.ToolCall> = {}
|
||||
@@ -1583,7 +1585,8 @@ export async function parseOpenAICompletion(
|
||||
helpers,
|
||||
toolCallbacks: callbacks,
|
||||
workspace: options?.workspace,
|
||||
messages
|
||||
messages,
|
||||
live: options?.live
|
||||
})
|
||||
messages.push(messageToAdd)
|
||||
addedMessages.push(messageToAdd)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import { Handle, Position } from '@xyflow/svelte'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import ContextMenu, { type ContextMenuItem } from '../../../common/contextmenu/ContextMenu.svelte'
|
||||
import { contextMenuItemsFromMenu } from '../../../common/contextmenu/fromMenuItems'
|
||||
import { getGraphContext } from '../../graphContext'
|
||||
import type { Item } from '$lib/utils'
|
||||
|
||||
@@ -28,21 +29,7 @@
|
||||
}: Props = $props()
|
||||
|
||||
let resolvedContextMenuItems: ContextMenuItem[] | undefined = $derived(
|
||||
contextMenuItems ??
|
||||
menuItems?.flatMap((item) => [
|
||||
...(item.separatorTop
|
||||
? [{ id: `${item.displayName}-divider`, label: '', divider: true }]
|
||||
: []),
|
||||
{
|
||||
id: item.displayName,
|
||||
label: item.displayName,
|
||||
icon: item.icon,
|
||||
disabled: item.disabled,
|
||||
type: item.type,
|
||||
shortcut: item.shortcut,
|
||||
onClick: item.action as (() => void) | undefined
|
||||
}
|
||||
])
|
||||
contextMenuItems ?? (menuItems && contextMenuItemsFromMenu(menuItems))
|
||||
)
|
||||
|
||||
// NodeWrapper is reused outside the flow graph (e.g. the app decision-tree
|
||||
|
||||
@@ -20,6 +20,7 @@
|
||||
PanelLeftClose
|
||||
} from 'lucide-svelte'
|
||||
import BarsStaggered from '$lib/components/icons/BarsStaggered.svelte'
|
||||
import DbtIcon from '$lib/components/icons/DbtIcon.svelte'
|
||||
import { PythonIcon, TypeScriptIcon } from '$lib/components/common/languageIcons'
|
||||
import { HOME_SHOW_CREATE_FLOW, HOME_SHOW_CREATE_APP } from '$lib/consts'
|
||||
import { importFlowStore } from '$lib/components/flows/flowStore.svelte'
|
||||
@@ -55,6 +56,23 @@
|
||||
label: string
|
||||
icon: typeof PythonIcon
|
||||
onSelect: () => void
|
||||
badge?: { label: string; class: string }
|
||||
/** opened in a new tab from a link beside the variant */
|
||||
docHref?: string
|
||||
/** shown in the doc panel while the variant is hovered, instead of its option's */
|
||||
doc?: Doc
|
||||
}
|
||||
|
||||
/** what the doc panel shows */
|
||||
type Doc = {
|
||||
label: string
|
||||
/** a lucide icon or a brand mark; rendered at 26px through both size APIs */
|
||||
icon: any
|
||||
accent: string
|
||||
tagline: string
|
||||
description: string
|
||||
bullets: string[]
|
||||
badge?: { label: string; class: string }
|
||||
}
|
||||
|
||||
/** an importable artifact kind handled by the shared YAML/JSON import drawer */
|
||||
@@ -87,6 +105,9 @@
|
||||
const badgeAdvanced = 'bg-gray-100 text-gray-600 dark:bg-gray-700 dark:text-gray-300'
|
||||
const badgeLegacy = 'bg-amber-100 text-amber-700 dark:bg-amber-900/40 dark:text-amber-300'
|
||||
const badgeAlpha = 'bg-emerald-100 text-emerald-700 dark:bg-emerald-900/40 dark:text-emerald-300'
|
||||
const badgeRecommended = 'bg-blue-100 text-blue-700 dark:bg-blue-900/40 dark:text-blue-300'
|
||||
|
||||
const DBT_DOCS = 'https://www.windmill.dev/docs/getting_started/scripts_quickstart/dbt'
|
||||
|
||||
const allOptions: Option[] = [
|
||||
{
|
||||
@@ -200,7 +221,34 @@
|
||||
'Asset-aware lineage'
|
||||
],
|
||||
onSelect: () => goto(`${base}/pipeline`),
|
||||
badge: { label: 'Alpha', class: badgeAlpha }
|
||||
badge: { label: 'Alpha', class: badgeAlpha },
|
||||
variants: [
|
||||
{
|
||||
label: 'Windmill pipelines',
|
||||
icon: Workflow,
|
||||
badge: { label: 'Recommended', class: badgeRecommended },
|
||||
onSelect: () => goto(`${base}/pipeline`)
|
||||
},
|
||||
{
|
||||
label: 'dbt',
|
||||
icon: DbtIcon,
|
||||
docHref: DBT_DOCS,
|
||||
onSelect: () => goto(`${base}/scripts/add?lang=dbt`),
|
||||
doc: {
|
||||
label: 'dbt',
|
||||
icon: DbtIcon,
|
||||
accent: 'orange',
|
||||
tagline: 'Run an existing dbt project',
|
||||
description:
|
||||
'A pipeline is Windmill’s own abstraction; an existing dbt project stays a dbt project, and runs here unchanged as its own kind of script, with dbt still owning its models, refs and tests. All it needs is a warehouse configured under Settings → dbt. Its models show up in the asset graph as dbt:// nodes, next to whatever pipelines you do build, so you can adopt pipelines later, or never.',
|
||||
bullets: [
|
||||
'Your dbt project, unchanged',
|
||||
'Needs a warehouse under Settings → dbt',
|
||||
'Models appear as dbt:// assets'
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
] as Option[])
|
||||
: []),
|
||||
@@ -301,18 +349,25 @@
|
||||
// per-row fan-out submenus: open to the right when there's room. The popover hugs
|
||||
// the right viewport edge on most screens, so the default flip (right → left) would
|
||||
// cover the doc panel — fall back below the trigger row instead.
|
||||
const variantSubmenu = () =>
|
||||
builders.createSubmenu({
|
||||
positioning: {
|
||||
placement: 'right-start',
|
||||
gutter: 4,
|
||||
flip: { fallbackPlacements: ['bottom-end', 'bottom-start', 'top-end', 'top-start'] },
|
||||
fitViewport: true,
|
||||
overflowPadding: 8
|
||||
}
|
||||
})
|
||||
const {
|
||||
elements: { subTrigger: wacSubTrigger, subMenu: wacSubMenu },
|
||||
states: { subOpen: wacSubOpen }
|
||||
} = builders.createSubmenu({
|
||||
positioning: {
|
||||
placement: 'right-start',
|
||||
gutter: 4,
|
||||
flip: { fallbackPlacements: ['bottom-end', 'bottom-start', 'top-end', 'top-start'] },
|
||||
fitViewport: true,
|
||||
overflowPadding: 8
|
||||
}
|
||||
})
|
||||
} = variantSubmenu()
|
||||
const {
|
||||
elements: { subTrigger: pipelineSubTrigger, subMenu: pipelineSubMenu },
|
||||
states: { subOpen: pipelineSubOpen }
|
||||
} = variantSubmenu()
|
||||
let variantSubOpen = $derived($wacSubOpen || $pipelineSubOpen)
|
||||
const {
|
||||
elements: { subTrigger: importSubTrigger, subMenu: importSubMenu },
|
||||
states: { subOpen: importSubOpen }
|
||||
@@ -378,7 +433,10 @@
|
||||
if (isOpen && !wasOpen) {
|
||||
logFeatureUsage('home', 'new_menu_open', { key: source })
|
||||
}
|
||||
if (!isOpen) activeKey = undefined
|
||||
if (!isOpen) {
|
||||
activeKey = undefined
|
||||
activeVariantDoc = undefined
|
||||
}
|
||||
wasOpen = isOpen
|
||||
})
|
||||
|
||||
@@ -389,17 +447,25 @@
|
||||
// only persist the non-default (hidden) state, so a cleared key means "shown"
|
||||
storeLocalSetting(SHOW_DOC_SETTING, value ? undefined : 'false')
|
||||
}
|
||||
// The pointer crosses the gutter outside the menu on its way into the Workflow-as-Code
|
||||
// submenu, so leaving the menu keeps the panel while that submenu is open; it clears
|
||||
// once the submenu closes with neither pointer nor focus left on the menu.
|
||||
// The pointer crosses the gutter outside the menu on its way into a variant submenu,
|
||||
// so leaving the menu keeps the panel while that submenu is open; it clears once the
|
||||
// submenu closes with neither pointer nor focus left on the menu.
|
||||
let menuEl: HTMLDivElement | undefined = $state(undefined)
|
||||
$effect(() => {
|
||||
if ($wacSubOpen || !menuEl) return
|
||||
if (variantSubOpen || !menuEl) return
|
||||
if (!menuEl.matches(':hover') && !menuEl.contains(document.activeElement)) {
|
||||
activeKey = undefined
|
||||
}
|
||||
})
|
||||
let active = $derived(options.find((o) => o.key === activeKey))
|
||||
// A hovered variant with its own doc (dbt under Data pipelines) takes over the panel.
|
||||
let activeVariantDoc: Doc | undefined = $state(undefined)
|
||||
function highlightOption(key: string | undefined) {
|
||||
activeKey = key
|
||||
activeVariantDoc = undefined
|
||||
}
|
||||
let active: Doc | undefined = $derived(
|
||||
activeVariantDoc ?? options.find((o) => o.key === activeKey)
|
||||
)
|
||||
let activeAc = $derived(active ? accentClasses[active.accent] : undefined)
|
||||
|
||||
// shared YAML/JSON import drawer, reused by every "Import …" extra
|
||||
@@ -476,7 +542,7 @@
|
||||
style={showDoc ? 'width: 780px;' : ''}
|
||||
bind:this={menuEl}
|
||||
onpointerleave={() => {
|
||||
if (!$wacSubOpen) activeKey = undefined
|
||||
if (!variantSubOpen) highlightOption(undefined)
|
||||
}}
|
||||
>
|
||||
<div
|
||||
@@ -492,7 +558,7 @@
|
||||
<div
|
||||
class="w-12 h-12 rounded-xl flex items-center justify-center shrink-0 {activeAc.tile}"
|
||||
>
|
||||
<active.icon size={26} class={activeAc.iconText} />
|
||||
<active.icon size={26} width={26} height={26} class={activeAc.iconText} />
|
||||
</div>
|
||||
<div class="min-w-0">
|
||||
<div class="flex flex-row items-center gap-2">
|
||||
@@ -551,48 +617,92 @@
|
||||
</span>
|
||||
{/if}
|
||||
{/snippet}
|
||||
{#snippet variantRow(
|
||||
option: Option,
|
||||
ac: (typeof accentClasses)[string],
|
||||
rowClass: string,
|
||||
subTrigger: any,
|
||||
subMenu: any,
|
||||
subOpen: boolean
|
||||
)}
|
||||
<button
|
||||
use:melt={subTrigger}
|
||||
class={rowClass}
|
||||
onfocusin={() => highlightOption(option.key)}
|
||||
onpointerenter={() => highlightOption(option.key)}
|
||||
>
|
||||
{@render rowBody(option, ac)}
|
||||
<ChevronRight size={14} class="shrink-0 text-tertiary" />
|
||||
</button>
|
||||
{#if subOpen}
|
||||
<!-- The invisible `before:` margin bridges the gutter to the trigger row: when
|
||||
the submenu falls below its row, that gutter lies over the next row, and
|
||||
crossing it highlights that item, which closes the submenu. -->
|
||||
<div
|
||||
use:melt={subMenu}
|
||||
use:hugViewportRight
|
||||
class="pointer-events-auto z-[6001] flex flex-col gap-0.5 p-1 min-w-52 w-max rounded-lg border border-gray-200 dark:border-gray-700 bg-surface shadow-xl focus:outline-none before:absolute before:-inset-2 before:-z-10 before:content-['']"
|
||||
>
|
||||
{#each option.variants ?? [] as variant (variant.label)}
|
||||
{@const VariantIcon = variant.icon}
|
||||
<div class="flex flex-row items-center gap-1">
|
||||
<button
|
||||
use:melt={$item}
|
||||
class="flex flex-row flex-1 items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
|
||||
onclick={() => variant.onSelect()}
|
||||
onfocusin={() => (activeVariantDoc = variant.doc)}
|
||||
onpointerenter={() => (activeVariantDoc = variant.doc)}
|
||||
>
|
||||
<VariantIcon width={14} height={14} size={14} />
|
||||
<span class="text-xs font-medium text-primary whitespace-nowrap">
|
||||
{variant.label}
|
||||
</span>
|
||||
{#if variant.badge}
|
||||
<span
|
||||
class="shrink-0 rounded px-1.5 py-0.5 text-[10px] font-semibold uppercase tracking-wide {variant
|
||||
.badge.class}"
|
||||
>
|
||||
{variant.badge.label}
|
||||
</span>
|
||||
{/if}
|
||||
</button>
|
||||
{#if variant.docHref}
|
||||
<a
|
||||
href={variant.docHref}
|
||||
target="_blank"
|
||||
rel="noreferrer"
|
||||
class="px-2 text-[10px] text-tertiary hover:text-secondary hover:underline whitespace-nowrap"
|
||||
onpointerenter={() => (activeVariantDoc = variant.doc)}
|
||||
>
|
||||
See documentation
|
||||
</a>
|
||||
{/if}
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
{/snippet}
|
||||
{#each options as option (option.key)}
|
||||
{@const ac = accentClasses[option.accent]}
|
||||
{@const rowClass =
|
||||
'w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover'}
|
||||
{#if option.variants}
|
||||
<button
|
||||
use:melt={$wacSubTrigger}
|
||||
class={rowClass}
|
||||
onfocusin={() => (activeKey = option.key)}
|
||||
onpointerenter={() => (activeKey = option.key)}
|
||||
>
|
||||
{@render rowBody(option, ac)}
|
||||
<ChevronRight size={14} class="shrink-0 text-tertiary" />
|
||||
</button>
|
||||
{#if $wacSubOpen}
|
||||
<!-- The invisible `before:` margin bridges the gutter to the trigger row: when
|
||||
the submenu falls below its row, that gutter lies over the next row, and
|
||||
crossing it highlights that item, which closes the submenu. -->
|
||||
<div
|
||||
use:melt={$wacSubMenu}
|
||||
use:hugViewportRight
|
||||
class="pointer-events-auto z-[6001] flex flex-col gap-0.5 p-1 w-52 rounded-lg border border-gray-200 dark:border-gray-700 bg-surface shadow-xl focus:outline-none before:absolute before:-inset-2 before:-z-10 before:content-['']"
|
||||
>
|
||||
{#each option.variants ?? [] as variant (variant.label)}
|
||||
{@const VariantIcon = variant.icon}
|
||||
<button
|
||||
use:melt={$item}
|
||||
class="flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
|
||||
onclick={() => variant.onSelect()}
|
||||
>
|
||||
<VariantIcon width={14} height={14} />
|
||||
<span class="text-xs font-medium text-primary">{variant.label}</span>
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
{#if option.variants && option.key === 'pipeline'}
|
||||
{@render variantRow(
|
||||
option,
|
||||
ac,
|
||||
rowClass,
|
||||
$pipelineSubTrigger,
|
||||
$pipelineSubMenu,
|
||||
$pipelineSubOpen
|
||||
)}
|
||||
{:else if option.variants}
|
||||
{@render variantRow(option, ac, rowClass, $wacSubTrigger, $wacSubMenu, $wacSubOpen)}
|
||||
{:else}
|
||||
<button
|
||||
use:melt={$item}
|
||||
class={rowClass}
|
||||
onfocusin={() => (activeKey = option.key)}
|
||||
onpointerenter={() => (activeKey = option.key)}
|
||||
onfocusin={() => highlightOption(option.key)}
|
||||
onpointerenter={() => highlightOption(option.key)}
|
||||
onclick={() => option.onSelect()}
|
||||
>
|
||||
{@render rowBody(option, ac)}
|
||||
@@ -605,8 +715,8 @@
|
||||
<button
|
||||
use:melt={$importSubTrigger}
|
||||
class="w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover"
|
||||
onfocusin={() => (activeKey = undefined)}
|
||||
onpointerenter={() => (activeKey = undefined)}
|
||||
onfocusin={() => highlightOption(undefined)}
|
||||
onpointerenter={() => highlightOption(undefined)}
|
||||
>
|
||||
<div
|
||||
class="w-6 h-6 rounded-md flex items-center justify-center shrink-0 bg-gray-100 dark:bg-gray-700"
|
||||
|
||||
@@ -60,8 +60,7 @@
|
||||
import { effectivePath, type ItemType } from './treeViewUtils'
|
||||
import { tick, untrack } from 'svelte'
|
||||
import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte'
|
||||
import { NetworkIcon } from 'lucide-svelte'
|
||||
import { base } from '$lib/base'
|
||||
import PipelineRow from '../common/table/PipelineRow.svelte'
|
||||
import BulkActionsBar from './BulkActionsBar.svelte'
|
||||
import { HomeSelection, setHomeSelection, toBulkItem } from './homeSelection.svelte'
|
||||
|
||||
@@ -835,6 +834,8 @@
|
||||
// keeping a dead path. `tick` lets the reloaded rows re-register first.
|
||||
const renderedBefore = homeSelection.renderedKeys
|
||||
void ownerCountsRes.refetch()
|
||||
// Deleting a pipeline removes it from this list, which no item reload covers.
|
||||
void pipelineFoldersRes.refetch()
|
||||
await reloadItems()
|
||||
await tick()
|
||||
homeSelection.dropVanished(renderedBefore)
|
||||
@@ -2168,13 +2169,7 @@
|
||||
<div class="border rounded-md bg-surface-tertiary" class:wm-imported={justImported}>
|
||||
{#if filter === ''}
|
||||
{#each [...visiblePipelineFolders].sort() as folder (folder)}
|
||||
<a
|
||||
href="{base}/pipeline/{encodeURIComponent(folder)}"
|
||||
class="w-full inline-flex items-center gap-4 px-4 py-3 border-b last:border-b-0 hover:bg-surface-hover transition-colors text-sm first-of-type:rounded-t-md"
|
||||
>
|
||||
<NetworkIcon size={16} class="text-emerald-600 dark:text-emerald-400" />
|
||||
<span class="text-xs font-medium text-emphasis truncate">Pipeline · f/{folder}</span>
|
||||
</a>
|
||||
<PipelineRow {folder} onDeleted={reloadItemsAndCounts} />
|
||||
{/each}
|
||||
{/if}
|
||||
{#each displayedItems as item, i (item.type + '/' + item.path + (item.hash ? '/' + item.hash : ''))}
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
<script lang="ts">
|
||||
import TreeView from './TreeView.svelte'
|
||||
import { onDestroy, untrack, type Snippet } from 'svelte'
|
||||
import { createEventDispatcher, onDestroy, untrack, type Snippet } from 'svelte'
|
||||
import ResizeTransitionWrapper from '$lib/components/common/ResizeTransitionWrapper.svelte'
|
||||
|
||||
import { ChevronDown, ChevronUp, Folder, FolderTree, NetworkIcon, User } from 'lucide-svelte'
|
||||
import { ChevronDown, ChevronUp, Folder, FolderTree, User } from 'lucide-svelte'
|
||||
import PipelineRow from '../common/table/PipelineRow.svelte'
|
||||
import Item from './Item.svelte'
|
||||
import { countLeaves, type FolderItem, type ItemType, type UserItem } from './treeViewUtils'
|
||||
import { twMerge } from 'tailwind-merge'
|
||||
import { pluralize } from '$lib/utils'
|
||||
import { base } from '$lib/base'
|
||||
import { Button } from '$lib/components/common'
|
||||
|
||||
interface Props {
|
||||
@@ -74,6 +74,8 @@
|
||||
|
||||
let visualDepth = $derived(depth + indent)
|
||||
|
||||
const dispatch = createEventDispatcher<{ reload: void }>()
|
||||
|
||||
// Bounds the request burst from "expand all": however many root owners the tree
|
||||
// renders (its slice grows as you scroll), it fetches at most this many. Lazy owners
|
||||
// past the cap stay collapsed and load on a single click (see the effect).
|
||||
@@ -316,17 +318,12 @@
|
||||
{#if opened || isSearching}
|
||||
<div>
|
||||
{#if hasPipeline && isFolder(item)}
|
||||
<!-- py-3 matches common/table/Row.svelte so this row sits at
|
||||
the same height as the script/flow/app rows that follow
|
||||
it under the same folder; py-2 was visibly shorter. -->
|
||||
<a
|
||||
href="{base}/pipeline/{encodeURIComponent(item.folderName)}"
|
||||
class="flex items-center gap-4 px-4 py-3 border-b text-sm hover:bg-surface-hover transition-colors"
|
||||
style="padding-left: {(visualDepth + 1) * 16}px;"
|
||||
>
|
||||
<NetworkIcon size={16} class="text-emerald-600 dark:text-emerald-400" />
|
||||
<span class="text-xs font-medium text-emphasis">Pipeline</span>
|
||||
</a>
|
||||
<PipelineRow
|
||||
folder={item.folderName}
|
||||
depth={visualDepth + 1}
|
||||
label="Pipeline"
|
||||
onDeleted={() => dispatch('reload')}
|
||||
/>
|
||||
{/if}
|
||||
{#each item.items.slice(0, effectiveMax) as subItem, index ((subItem['path'] ? subItem['type'] + '__' + subItem['path'] + '__' + index : undefined) ?? 'folder__' + subItem['folderName'] + '__' + index)}
|
||||
<TreeView
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
<script lang="ts">
|
||||
import { resource } from 'runed'
|
||||
import { tick, untrack } from 'svelte'
|
||||
import { Loader2, Workflow } from 'lucide-svelte'
|
||||
import { onDestroy, tick, untrack } from 'svelte'
|
||||
import { Loader2, Save, Workflow } from 'lucide-svelte'
|
||||
import PipelineGraphEditor from '$lib/components/assets/AssetGraph/PipelineGraphEditor.svelte'
|
||||
import PipelineTriggerEditors from '$lib/components/assets/AssetGraph/PipelineTriggerEditors.svelte'
|
||||
import { usePipelineDataUploads } from '$lib/components/assets/AssetGraph/pipelineDataUploads.svelte'
|
||||
import { resolveGraph } from '$lib/components/assets/AssetGraph/resolveGraph'
|
||||
import { useActiveRunnableIds } from '$lib/components/assets/AssetGraph/activeRunnables.svelte'
|
||||
import type {
|
||||
@@ -11,10 +12,26 @@
|
||||
AssetGraphSelection,
|
||||
NativeTriggerKind
|
||||
} from '$lib/components/assets/AssetGraph/types'
|
||||
import { AssetService, JobService, type AssetKind } from '$lib/gen'
|
||||
import { AssetService, JobService, type ScriptLang } from '$lib/gen'
|
||||
import { DATA_ASSET_KINDS } from '$lib/components/assets/AssetGraph/cascadeRun'
|
||||
import { usePipelineAssetPrefetch } from '$lib/components/assets/AssetGraph/pipelineAssetPrefetch.svelte'
|
||||
import { sendUserToast } from '$lib/utils'
|
||||
import { createPipelineAiHelpers } from '$lib/components/assets/AssetGraph/pipelineAiHelpers'
|
||||
import { deployPipelineDrafts } from '$lib/components/assets/AssetGraph/pipelineDeploy.svelte'
|
||||
import {
|
||||
diffDeployedGraph,
|
||||
extractCascadeFacts,
|
||||
formatDrift
|
||||
} from '$lib/components/assets/AssetGraph/deployGraphDiff'
|
||||
import PipelineDeployErrors from '$lib/components/assets/AssetGraph/PipelineDeployErrors.svelte'
|
||||
import PipelineDeployTriggersModal from '$lib/components/assets/AssetGraph/PipelineDeployTriggersModal.svelte'
|
||||
import AutosaveIndicator from '$lib/components/AutosaveIndicator.svelte'
|
||||
import { Button } from '$lib/components/common'
|
||||
import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
|
||||
import type { DeployResult } from '$lib/utils_workspace_deploy'
|
||||
import type { SessionRuntime } from './sessionRuntime.svelte'
|
||||
import { maskKey } from './modifiedItemsMask'
|
||||
|
||||
let {
|
||||
runtime,
|
||||
@@ -29,7 +46,7 @@
|
||||
/** Folder name the pipeline graph is scoped to (not a workspace item path). */
|
||||
path: string
|
||||
workspaceId: string
|
||||
/** Only the visible session registers the pipeline tools on its manager. */
|
||||
/** Only the visible session polls the live run badges. */
|
||||
isActiveSession?: boolean
|
||||
/** Whether this is the foreground preview tab. */
|
||||
active?: boolean
|
||||
@@ -43,50 +60,36 @@
|
||||
// wrong chat — leaving this session's chat unable to build canvas nodes.
|
||||
const aiChatManager = runtime.manager
|
||||
|
||||
// Only the foreground tab of the foreground session owns the chat's pipeline
|
||||
// tools and runs the live-badge poll — a background tab (another preview tab is
|
||||
// showing, or another session is active) must not shadow that context or poll.
|
||||
// Only the foreground tab of the foreground session runs the live-badge poll.
|
||||
const engaged = $derived(isActiveSession && active)
|
||||
|
||||
// Externalized editor state — lives on the runtime so the drafts persist across
|
||||
// hide/show of the preview pane (the pane unmounts on hide).
|
||||
const pe = runtime.pipelineEditorState
|
||||
|
||||
// The reused `pe` is scoped to a folder. A same-folder remount (hide→show)
|
||||
// keeps the drafts; a retarget to a different folder resets so stale drafts
|
||||
// don't bleed across folders. untrack the writes so this can't self-loop.
|
||||
$effect(() => {
|
||||
const folder = path
|
||||
untrack(() => {
|
||||
if (pe.folder !== folder) {
|
||||
if (pe.folder !== undefined) {
|
||||
pe.reset()
|
||||
// A retarget without remount re-scopes the poll, so the release
|
||||
// effect could never match the old folder's job — drop the hint.
|
||||
activeRunnable = undefined
|
||||
activeRunnableJobId = undefined
|
||||
// Re-scope the Global pipeline prompt to the new folder (the helper
|
||||
// methods already read the reactive path, but the system message
|
||||
// string was built for the old one). Only when this tab is engaged —
|
||||
// its helpers are the registered set; a background tab reconfigures
|
||||
// when it next becomes the foreground one.
|
||||
if (engaged) aiChatManager.rebuildGlobalSystemMessage()
|
||||
}
|
||||
pe.folder = folder
|
||||
}
|
||||
})
|
||||
})
|
||||
// This folder's editor state lives on the runtime, so its drafts survive a
|
||||
// remount. The host remounts this view when the tab moves to another folder, so
|
||||
// `path` is fixed for its lifetime.
|
||||
const pe = untrack(() => runtime.pipelineEditor(path))
|
||||
|
||||
const EMPTY_GRAPH: AssetGraphResponse = { assets: [], runnables: [], edges: [], triggers: [] }
|
||||
const EMPTY_PATH_MAP = new Map<string, Array<{ kind: AssetKind; path: string }>>()
|
||||
const EMPTY_NATIVE_MAP = new Map<string, Set<any>>()
|
||||
|
||||
// Data assets only, as on the pipeline page: variables and resources are
|
||||
// config most scripts reference, and would swamp the layout as hub nodes.
|
||||
const graphRes = resource(
|
||||
() => ({ workspace: workspaceId, folder: path }),
|
||||
async ({ workspace, folder }) =>
|
||||
workspace && folder ? await AssetService.getAssetsGraph({ workspace, folder }) : EMPTY_GRAPH
|
||||
workspace && folder
|
||||
? await AssetService.getAssetsGraph({
|
||||
workspace,
|
||||
folder,
|
||||
assetKinds: DATA_ASSET_KINDS.join(',')
|
||||
})
|
||||
: EMPTY_GRAPH
|
||||
)
|
||||
|
||||
const assetPrefetch = usePipelineAssetPrefetch({
|
||||
getWorkspace: () => workspaceId,
|
||||
getGraph: () => graphRes.current as AssetGraphResponse | undefined,
|
||||
editor: pe
|
||||
})
|
||||
|
||||
// Folder whose graph is actually rendered — `graphRes.current` is stale-
|
||||
// while-revalidate on a folder retarget, so the canvas's one-shot initial
|
||||
// fit is keyed on the folder captured when a graph lands, not on `path`
|
||||
@@ -97,9 +100,7 @@
|
||||
})
|
||||
|
||||
// Deployed graph + the in-flight draft overlay (AI-built nodes render as plain
|
||||
// dashed unsaved drafts, same as manual drafts). The session
|
||||
// skips the route page's folder-wide asset prefetch (empty inferred maps); the
|
||||
// open script's live overlays still feed the graph. (resolveGraph's base is the
|
||||
// dashed unsaved drafts, same as manual drafts). (resolveGraph's base is the
|
||||
// pipeline runnables subset; the 'job' usage_kind of the wire type never appears.)
|
||||
let resolvedGraph = $derived.by<AssetGraphResponse>(() =>
|
||||
resolveGraph({
|
||||
@@ -107,9 +108,10 @@
|
||||
drafts: pe.drafts,
|
||||
liveBodyAssets: pe.liveBodyAssets,
|
||||
liveAnnotations: pe.liveAnnotations,
|
||||
inferredWritesByPath: EMPTY_PATH_MAP,
|
||||
inferredReadsByPath: EMPTY_PATH_MAP,
|
||||
annotatedNativeKindsByPath: EMPTY_NATIVE_MAP
|
||||
inferredWritesByPath: assetPrefetch.inferredWritesByPath,
|
||||
inferredReadsByPath: assetPrefetch.inferredReadsByPath,
|
||||
annotatedNativeKindsByPath: assetPrefetch.annotatedNativeKindsByPath,
|
||||
triggerDrafts: pe.triggerDrafts.values()
|
||||
})
|
||||
)
|
||||
|
||||
@@ -120,7 +122,13 @@
|
||||
getDrafts: () => pe.drafts,
|
||||
setDrafts: (next) => (pe.drafts = next),
|
||||
newDraftLocalId: pe.newDraftLocalId,
|
||||
onForgetPath: pe.forgetPath,
|
||||
hasTriggerDrafts: () => pe.triggerDrafts.size > 0,
|
||||
getTriggerDrafts: () => pe.triggerDrafts,
|
||||
setTriggerDraft: pe.setTriggerDraft,
|
||||
onForgetPath: (p) => {
|
||||
pe.forgetPath(p)
|
||||
pe.discardTriggerDraftsFor(p)
|
||||
},
|
||||
// Open the staged node in the details pane so its code is visible.
|
||||
onProposeNode: (p) => {
|
||||
pe.activeDraftPath = p
|
||||
@@ -135,6 +143,7 @@
|
||||
})
|
||||
|
||||
function handleCanvasSelect(s: AssetGraphSelection | undefined) {
|
||||
uploadInputsPath = undefined
|
||||
if (s && s.kind === 'runnable' && s.runnable_kind === 'script' && pe.drafts.has(s.path)) {
|
||||
pe.activeDraftPath = s.path
|
||||
pe.selection = undefined
|
||||
@@ -145,6 +154,7 @@
|
||||
}
|
||||
|
||||
async function afterSaved(savedPath: string) {
|
||||
assetPrefetch.forget(savedPath)
|
||||
const next = new Map(pe.drafts)
|
||||
next.delete(savedPath)
|
||||
pe.drafts = next
|
||||
@@ -194,7 +204,11 @@
|
||||
|
||||
// Data upload has no trigger row — open the target in the details pane and pulse
|
||||
// the signal so its auto-generated run form focuses the S3 input.
|
||||
// A data-upload click opens the node on its inputs alone; any other selection
|
||||
// (or "Open script") goes back to the editor.
|
||||
let uploadInputsPath = $state<string | undefined>(undefined)
|
||||
function openDataUploadRun(scriptPath: string) {
|
||||
uploadInputsPath = scriptPath
|
||||
if (pe.drafts.has(scriptPath)) {
|
||||
pe.activeDraftPath = scriptPath
|
||||
pe.selection = undefined
|
||||
@@ -236,29 +250,40 @@
|
||||
}
|
||||
})
|
||||
|
||||
const dataUploads = usePipelineDataUploads(() => resolvedGraph)
|
||||
|
||||
function runProducer(producer: { kind: 'script' | 'flow'; path: string; cascade?: boolean }) {
|
||||
// Pipeline nodes are scripts; a flow producer can't be preview/by-path run.
|
||||
if (producer.kind !== 'script') return Promise.resolve(undefined)
|
||||
return runNode(producer.path, {}, producer.cascade ?? false)
|
||||
// A data-upload entry runs on the file staged in its run form.
|
||||
return runNode(producer.path, dataUploads.argsFor(producer.path) ?? {}, producer.cascade ?? false)
|
||||
}
|
||||
|
||||
// Tell the open node's runs list a job started, so it shows it right away.
|
||||
let runsRefreshKey = $state(0)
|
||||
let runsPendingJobId = $state<string | undefined>(undefined)
|
||||
|
||||
// `live` is an open editor's body, newer than what is stored: it is previewed,
|
||||
// a deployed node included.
|
||||
async function runNode(
|
||||
nodePath: string,
|
||||
args: Record<string, any> = {},
|
||||
cascade = false
|
||||
cascade = false,
|
||||
live?: { content: string; language: ScriptLang }
|
||||
): Promise<string | undefined> {
|
||||
const draft = pe.drafts.get(nodePath)
|
||||
const body = live ?? (draft && { content: draft.script.content, language: draft.script.language })
|
||||
activeRunnables.arm(`script:${nodePath}`)
|
||||
try {
|
||||
let jobId: string
|
||||
if (draft) {
|
||||
// Preview-run the draft content; a preview never dispatches downstream.
|
||||
if (body) {
|
||||
// Preview-run the unsaved body; a preview never dispatches downstream.
|
||||
jobId = await JobService.runScriptPreview({
|
||||
workspace: workspaceId,
|
||||
requestBody: {
|
||||
path: nodePath,
|
||||
content: draft.script.content,
|
||||
language: draft.script.language,
|
||||
content: body.content,
|
||||
language: body.language,
|
||||
args
|
||||
}
|
||||
})
|
||||
@@ -275,6 +300,8 @@
|
||||
}
|
||||
activeRunnable = { kind: 'script', path: nodePath }
|
||||
activeRunnableJobId = jobId
|
||||
runsPendingJobId = jobId
|
||||
runsRefreshKey++
|
||||
return jobId
|
||||
} catch (e: any) {
|
||||
sendUserToast(`Run failed: ${e?.body ?? e?.message ?? e}`, true)
|
||||
@@ -282,14 +309,126 @@
|
||||
}
|
||||
}
|
||||
|
||||
// Register the pipeline tools on this session's manager while this tab is the
|
||||
// engaged (foreground) one. setPipelineHelpers rebuilds the global tool set to
|
||||
// include the pipeline tools and tears them down on cleanup — so switching to
|
||||
// another preview tab or session releases them.
|
||||
$effect(() => {
|
||||
if (!engaged) return
|
||||
return aiChatManager.setPipelineHelpers(helpers)
|
||||
})
|
||||
// Register this folder's pipeline tools on the session's own manager for as long
|
||||
// as the view is mounted — background tabs included, since each folder's tools
|
||||
// act on its own state — and release them on unmount.
|
||||
$effect(() => aiChatManager.setPipelineHelpers(helpers))
|
||||
|
||||
// ── Deploy all ───────────────────────────────────────────────────────────
|
||||
// The same deploy as the pipeline page's "Save all", reached from this tab's
|
||||
// button and from the session's changes list (through the runtime).
|
||||
const pendingCount = $derived(
|
||||
pe.drafts.size + pe.triggerDrafts.size + (pe.liveEditPath != undefined ? 1 : 0)
|
||||
)
|
||||
let deploying = $state(false)
|
||||
let deployErrors = $state<Map<string, string>>(new Map())
|
||||
let deployErrorsOpen = $state(false)
|
||||
|
||||
let triggerConfirmOpen = $state(false)
|
||||
let resolveTriggerConfirm: ((ok: boolean) => void) | undefined
|
||||
function confirmTriggers(): Promise<boolean> {
|
||||
triggerConfirmOpen = true
|
||||
return new Promise((resolve) => (resolveTriggerConfirm = resolve))
|
||||
}
|
||||
function answerTriggerConfirm(ok: boolean) {
|
||||
triggerConfirmOpen = false
|
||||
resolveTriggerConfirm?.(ok)
|
||||
resolveTriggerConfirm = undefined
|
||||
}
|
||||
|
||||
// A deploy asked for from the changes list can arrive while the tab is still
|
||||
// loading this folder's drafts.
|
||||
async function hydrated(timeoutMs = 10000): Promise<boolean> {
|
||||
for (let waited = 0; !pe.hydratedFromDb && waited < timeoutMs; waited += 100) {
|
||||
await new Promise((resolve) => setTimeout(resolve, 100))
|
||||
}
|
||||
return pe.hydratedFromDb
|
||||
}
|
||||
|
||||
async function deployAll(): Promise<DeployResult> {
|
||||
// Taken before any await: a second caller (the tab's button and the changes
|
||||
// list both reach here) would otherwise replace the pending confirmation and
|
||||
// leave the first one waiting forever.
|
||||
if (deploying) return { success: false, error: 'This pipeline is already deploying.' }
|
||||
deploying = true
|
||||
try {
|
||||
if (!(await hydrated())) return { success: false, error: 'The pipeline drafts did not load.' }
|
||||
if (pendingCount === 0) {
|
||||
// Nothing left to deploy, but an earlier deploy may have failed to save
|
||||
// that: the server would still hold the deployed drafts.
|
||||
const unsaved = await saveRemaining()
|
||||
return unsaved ? { success: false, error: unsaved } : { success: true }
|
||||
}
|
||||
if (pe.triggerDrafts.size > 0 && !(await confirmTriggers())) {
|
||||
return { success: false, error: 'Deploy cancelled.' }
|
||||
}
|
||||
// What the canvas promises for each draft, checked against what the
|
||||
// backend derived once deployed.
|
||||
const predicted = new Map(
|
||||
[...pe.drafts.keys()].map((p) => [p, extractCascadeFacts(resolvedGraph, p)])
|
||||
)
|
||||
const { savedPaths, savedTriggers, errors } = await deployPipelineDrafts(pe, workspaceId)
|
||||
assetPrefetch.forget(...savedPaths)
|
||||
const bundleKey = maskKey(PIPELINE_DRAFT_KIND, pipelineBundlePath(path))
|
||||
if (aiChatManager.modifiedItems?.has(bundleKey)) {
|
||||
await aiChatManager.recordDeployedItems('script', savedPaths)
|
||||
}
|
||||
if (savedPaths.length > 0 || savedTriggers.length > 0) {
|
||||
await graphRes.refetch()
|
||||
const deployed = new Map([...predicted].filter(([p]) => savedPaths.includes(p)))
|
||||
const drift = formatDrift(
|
||||
diffDeployedGraph(deployed, (graphRes.current ?? EMPTY_GRAPH) as AssetGraphResponse)
|
||||
)
|
||||
if (drift) sendUserToast(drift, true)
|
||||
}
|
||||
// The session's changes list re-reads the draft once this resolves, so
|
||||
// what is left of it has to be saved by then.
|
||||
const unsaved = await saveRemaining()
|
||||
deployErrors = errors
|
||||
if (unsaved) return { success: false, error: `Deployed, but ${unsaved}` }
|
||||
if (errors.size > 0) {
|
||||
deployErrorsOpen = true
|
||||
return {
|
||||
success: false,
|
||||
error: `${errors.size} of the pipeline's drafts failed to deploy, see its tab.`
|
||||
}
|
||||
}
|
||||
return { success: true }
|
||||
} finally {
|
||||
deploying = false
|
||||
}
|
||||
}
|
||||
|
||||
// Saves what is left of the folder's draft and says why if that failed. A
|
||||
// flush resolves even when the save failed, and the server would then keep
|
||||
// drafts already deployed — deploying them again would recreate their
|
||||
// triggers. A failed save stays queued until one succeeds, so the flush
|
||||
// retries exactly what failed, an open script's unsaved edits included.
|
||||
async function saveRemaining(): Promise<string | undefined> {
|
||||
await tick()
|
||||
const draft = {
|
||||
workspace: workspaceId,
|
||||
itemKind: PIPELINE_DRAFT_KIND,
|
||||
path: pipelineBundlePath(path)
|
||||
}
|
||||
await UserDraftDbSyncer.flush(draft)
|
||||
if (UserDraftDbSyncer.getConflict(draft).conflict)
|
||||
return 'the remaining drafts conflict with a newer version.'
|
||||
const sync = UserDraftDbSyncer.getState(draft)
|
||||
if (sync.state === 'failed')
|
||||
return `the remaining drafts could not be saved: ${sync.failureMessage ?? 'unknown error'}.`
|
||||
return undefined
|
||||
}
|
||||
|
||||
async function deployAllFromButton() {
|
||||
const count = pendingCount
|
||||
const res = await deployAll()
|
||||
if (res.success) sendUserToast(`Deployed ${count} draft${count === 1 ? '' : 's'}`)
|
||||
}
|
||||
|
||||
$effect(() => runtime.registerPipelineView(path, { deployAll }))
|
||||
// A deploy waiting on the trigger confirmation must still settle when the tab goes.
|
||||
onDestroy(() => answerTriggerConfirm(false))
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col h-full w-full bg-surface">
|
||||
@@ -299,6 +438,31 @@
|
||||
<Workflow size={14} />
|
||||
<span class="font-mono text-emphasis truncate">f/{path}</span>
|
||||
<span class="text-tertiary">· data pipeline</span>
|
||||
{#if workspaceId}
|
||||
<AutosaveIndicator
|
||||
workspace={workspaceId}
|
||||
itemKind={PIPELINE_DRAFT_KIND}
|
||||
path={pipelineBundlePath(path)}
|
||||
draftOnly
|
||||
loadedFromDraft={pe.loadedFromDbDraft}
|
||||
/>
|
||||
{/if}
|
||||
<div class="flex-1"></div>
|
||||
{#if deployErrors.size > 0}
|
||||
<PipelineDeployErrors errors={deployErrors} bind:open={deployErrorsOpen} />
|
||||
{/if}
|
||||
{#if pendingCount > 0}
|
||||
<Button
|
||||
variant="accent"
|
||||
unifiedSize="xs"
|
||||
startIcon={{ icon: deploying ? Loader2 : Save }}
|
||||
onclick={deployAllFromButton}
|
||||
disabled={deploying}
|
||||
title={deploying ? 'Deploying drafts…' : `Deploy all ${pendingCount} drafts`}
|
||||
>
|
||||
{deploying ? 'Deploying…' : `Deploy all (${pendingCount})`}
|
||||
</Button>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="flex-1 min-h-0">
|
||||
<!-- Only block on the deployed-graph fetch when there's nothing to show yet.
|
||||
@@ -328,6 +492,8 @@
|
||||
folder={path}
|
||||
viewportFitKey={viewportFitFolder}
|
||||
persistDrafts={true}
|
||||
modalPanel
|
||||
prefetchingAssets={assetPrefetch.prefetching}
|
||||
displayGraph={resolvedGraph}
|
||||
mode="edit"
|
||||
workspace={workspaceId}
|
||||
@@ -338,12 +504,21 @@
|
||||
onOpenWebhook={openWebhookDrawer}
|
||||
onOpenDataUpload={openDataUploadRun}
|
||||
focusUploadSignal={focusDataUploadSignal}
|
||||
readyDataUploadPaths={dataUploads.readyPaths}
|
||||
runFormInitialArgs={pe.openScriptPath ? dataUploads.argsFor(pe.openScriptPath) : undefined}
|
||||
onRunFormArgsChange={dataUploads.stage}
|
||||
{activeRunnable}
|
||||
{runsRefreshKey}
|
||||
{runsPendingJobId}
|
||||
activeRunnableIds={activeRunnables.ids}
|
||||
runStates={activeRunnables.states}
|
||||
eventLogEvents={activeRunnables.events}
|
||||
onRunProducer={runProducer}
|
||||
onRunByPath={(path, args) => runNode(path, args)}
|
||||
inputsOnly={uploadInputsPath !== undefined &&
|
||||
(pe.openScriptPath === undefined || uploadInputsPath === pe.openScriptPath)}
|
||||
onRunInputs={(path, args, live) => runNode(path, args, false, live)}
|
||||
onShowScript={() => (uploadInputsPath = undefined)}
|
||||
canRunByPath
|
||||
onTestStateChange={(running) => {
|
||||
const openPath = pe.openScriptPath
|
||||
@@ -370,6 +545,7 @@
|
||||
await graphRes.refetch()
|
||||
}}
|
||||
onScriptRenamed={async (oldPath, newPath) => {
|
||||
assetPrefetch.forget(oldPath, newPath)
|
||||
// Repoint the selection so the canvas follows the renamed node instead
|
||||
// of staying on the now-gone old path until an unrelated refetch.
|
||||
if (pe.selection?.kind === 'runnable' && pe.selection.path === oldPath) {
|
||||
@@ -402,3 +578,10 @@
|
||||
workspace={workspaceId}
|
||||
onUpdate={() => graphRes.refetch()}
|
||||
/>
|
||||
|
||||
<PipelineDeployTriggersModal
|
||||
open={triggerConfirmOpen}
|
||||
triggerDrafts={pe.triggerDrafts}
|
||||
onConfirmed={() => answerTriggerConfirm(true)}
|
||||
onCanceled={() => answerTriggerConfirm(false)}
|
||||
/>
|
||||
|
||||
@@ -16,8 +16,9 @@ section or the flat layout.
|
||||
|
||||
<script lang="ts">
|
||||
import { untrack } from 'svelte'
|
||||
import { Compass, FileText, Pen } from 'lucide-svelte'
|
||||
import { Compass, FileText, NetworkIcon, Pen } from 'lucide-svelte'
|
||||
import { resource } from 'runed'
|
||||
import { AssetService } from '$lib/gen'
|
||||
import { workspaceStore } from '$lib/stores'
|
||||
import RowIcon from '$lib/components/common/table/RowIcon.svelte'
|
||||
import {
|
||||
@@ -228,9 +229,38 @@ section or the flat layout.
|
||||
: undefined
|
||||
)
|
||||
|
||||
// Data pipelines are folders, not items, so they get a branch of their own
|
||||
// rather than a leaf in the workspace tree.
|
||||
const pipelinesResource = resource(
|
||||
() => effectiveWorkspace,
|
||||
async (ws) => (ws ? await AssetService.listPipelineFolders({ workspace: ws }) : [])
|
||||
)
|
||||
const pipelinesBranch = $derived<DrillBranch<PreviewTarget> | undefined>(
|
||||
pipelinesResource.current && pipelinesResource.current.length > 0
|
||||
? {
|
||||
type: 'branch',
|
||||
key: 'pipelines',
|
||||
label: 'Data pipelines',
|
||||
icon: NetworkIcon,
|
||||
searchGroup: true,
|
||||
children: pipelinesResource.current.map(({ folder }) => {
|
||||
const path = `/pipeline/${encodeURIComponent(folder)}`
|
||||
return {
|
||||
type: 'leaf' as const,
|
||||
key: pageKey(path),
|
||||
label: `f/${folder}`,
|
||||
icon: NetworkIcon,
|
||||
data: { type: 'page', href: pageHref(path), label: folder }
|
||||
}
|
||||
})
|
||||
}
|
||||
: undefined
|
||||
)
|
||||
|
||||
const tree = $derived<DrillNode<PreviewTarget>[]>([
|
||||
pagesBranch,
|
||||
...(artifactsBranch ? [artifactsBranch] : []),
|
||||
...(pipelinesBranch ? [pipelinesBranch] : []),
|
||||
...tagItems(
|
||||
buildWorkspaceTree({
|
||||
loaded: loader.loaded,
|
||||
|
||||
@@ -354,7 +354,10 @@
|
||||
{#await import('./PipelineEditorView.svelte')}
|
||||
{@render editorLoading()}
|
||||
{:then Module}
|
||||
<Module.default {runtime} path={itemPath} {workspaceId} {isActiveSession} {active} />
|
||||
<!-- Keyed: the view binds one folder's state for its lifetime. -->
|
||||
{#key itemPath}
|
||||
<Module.default {runtime} path={itemPath} {workspaceId} {isActiveSession} {active} />
|
||||
{/key}
|
||||
{/await}
|
||||
{:else}
|
||||
{#await import('./RawAppEditorView.svelte')}
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
import { badgeCounts, badgeOf, buildDeployItems, type DeployItem } from './sessionDeployModel'
|
||||
import { useExistingMaskKeys } from './sessionDeployModel.svelte'
|
||||
import { previewTargetForDeployKind } from './sessionPreviewTabs.svelte'
|
||||
import { pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
|
||||
import type { PreviewTarget } from './previewRouter'
|
||||
import JobsSegment from '$lib/components/copilot/chat/JobsSegment.svelte'
|
||||
import RowIcon from '$lib/components/common/table/RowIcon.svelte'
|
||||
@@ -378,6 +378,11 @@
|
||||
onDataChanged={refreshDock}
|
||||
onItemDeployed={(item) =>
|
||||
void runtime?.manager.renameModifiedItem(item.draftKind, item.path, item.displayPath)}
|
||||
onItemDiscarded={(item) => void runtime?.manager.removeModifiedItem(item.draftKind, item.path)}
|
||||
onItemDiscarded={(item) => {
|
||||
void runtime?.manager.removeModifiedItem(item.draftKind, item.path)
|
||||
const folder = pipelineFolderFromBundlePath(item.path)
|
||||
if (item.draftKind === PIPELINE_DRAFT_KIND && folder) void runtime?.forgetPipelineDrafts(folder)
|
||||
}}
|
||||
deployPipeline={runtime?.deployPipeline}
|
||||
/>
|
||||
{/if}
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
import DevWorkspaceBadge from '$lib/components/DevWorkspaceBadge.svelte'
|
||||
import { useSessionDeployModel } from './sessionDeployModel.svelte'
|
||||
import type { DeployItem } from './sessionDeployModel'
|
||||
import type { DeployResult } from '$lib/utils_workspace_deploy'
|
||||
|
||||
// Session "Edits" drawer. Builds the deploy model over the session
|
||||
// workspace's drafts and hands it to WorkspaceDiffDrawer, which renders the
|
||||
@@ -18,7 +19,8 @@
|
||||
keys,
|
||||
onDataChanged,
|
||||
onItemDeployed,
|
||||
onItemDiscarded
|
||||
onItemDiscarded,
|
||||
deployPipeline
|
||||
}: {
|
||||
workspaceId: string
|
||||
parentWorkspaceId?: string
|
||||
@@ -35,6 +37,8 @@
|
||||
* drops it. */
|
||||
onItemDeployed?: (item: DeployItem) => void
|
||||
onItemDiscarded?: (item: DeployItem) => void
|
||||
/** Forwarded to the deploy model: deploys a pipeline folder's draft bundle. */
|
||||
deployPipeline?: (folder: string) => Promise<DeployResult>
|
||||
} = $props()
|
||||
|
||||
const isFork = $derived(!!parentWorkspaceId)
|
||||
@@ -50,7 +54,8 @@
|
||||
mask: keys,
|
||||
onDataChanged,
|
||||
onItemDeployed,
|
||||
onItemDiscarded
|
||||
onItemDiscarded,
|
||||
deployPipeline
|
||||
}))
|
||||
|
||||
// Editor URL for a row (every item lives in the session workspace).
|
||||
|
||||
@@ -1338,8 +1338,8 @@
|
||||
bind:open={seedOfferOpen}
|
||||
kind="X"
|
||||
enterConfirms={false}
|
||||
title="Keep this {seedOffer?.route.kind ?? 'item'} in the new session?"
|
||||
description="You came here from {seedOffer?.route.itemPath ?? ''}."
|
||||
title="Keep this {seedOffer?.kind ?? 'item'} in the new session?"
|
||||
description="You came here from {seedOffer?.path ?? ''}."
|
||||
>
|
||||
<p class="text-sm text-secondary">
|
||||
Keeping it opens it in the preview, so the chat starts with it as context.
|
||||
|
||||
@@ -62,6 +62,12 @@ describe('diffInMask', () => {
|
||||
expect(diffInMask(diff('script', 'u/me/s'), mask)).toBe(true)
|
||||
})
|
||||
|
||||
it('selects only the scripts a masked pipeline deployed, not the rest of its folder', () => {
|
||||
const mask = new Set(['data_pipeline:f/crm/data_pipeline', 'script:f/crm/clean'])
|
||||
expect(diffInMask(diff('script', 'f/crm/clean'), mask)).toBe(true)
|
||||
expect(diffInMask(diff('script', 'f/crm/unrelated'), mask)).toBe(false)
|
||||
})
|
||||
|
||||
it('matches a legacy app diff under both its identity and bridged mask keys', () => {
|
||||
expect(diffInMask(diff('app', 'u/me/legacy'), new Set(['app:u/me/legacy']))).toBe(true)
|
||||
expect(diffInMask(diff('app', 'u/me/legacy'), new Set(['raw_app:u/me/legacy']))).toBe(true)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { UserDraftItemKind, WorkspaceItemDiff } from '$lib/gen'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineBundlePath } from '$lib/pipelinePaths'
|
||||
|
||||
// The "modified items mask" tracks which workspace items an AI chat touched via
|
||||
// tool calls. Each key is `${UserDraftItemKind}:${storagePath}`. UserDraftItemKind
|
||||
@@ -45,6 +46,21 @@ export function forkDiffKindToUserDraftKind(kind: ForkDiffKind): UserDraftItemKi
|
||||
return FORK_DIFF_KIND_TO_USER_DRAFT_KIND[kind]
|
||||
}
|
||||
|
||||
// A pipeline the chat edited is masked as its folder's draft bundle, and each
|
||||
// script that bundle deploys is masked too, so a fork review selects exactly
|
||||
// those. The session's changes list shows deployed scripts of a masked
|
||||
// pipeline's folder as that pipeline's one row — any of them, since a script's
|
||||
// mask entry does not say whether a pipeline deploy or the chat put it there.
|
||||
export function foldedIntoPipelineRow(
|
||||
mask: ReadonlySet<string>,
|
||||
kind: UserDraftItemKind,
|
||||
path: string
|
||||
): boolean {
|
||||
if (kind !== 'script') return false
|
||||
const folder = path.match(/^f\/([^/]+)\//)?.[1]
|
||||
return !!folder && mask.has(maskKey(PIPELINE_DRAFT_KIND, pipelineBundlePath(folder)))
|
||||
}
|
||||
|
||||
// True when a fork-comparison diff names an item present in the chat-modified mask.
|
||||
export function diffInMask(diff: WorkspaceItemDiff, mask: Set<string>): boolean {
|
||||
const kind = forkDiffKindToUserDraftKind(diff.kind)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { getContext, onDestroy, setContext } from 'svelte'
|
||||
import type { OpenInSessionSource } from './OpenInSessionButton.svelte'
|
||||
import type { PreviewItemRoute } from './previewPaths'
|
||||
import type { SessionTarget } from './sessionState.svelte'
|
||||
|
||||
// The "Open in AI session" hand-off, published by the component that owns the
|
||||
// item being edited (FlowBuilder, RawAppEditor) for AI entry points too deep in
|
||||
@@ -38,20 +38,17 @@ export function getOpenInSessionHandoff(): OpenInSessionHandoff | undefined {
|
||||
return getContext<OpenInSessionHandoff | undefined>(KEY)
|
||||
}
|
||||
|
||||
/** The mounted editor's hand-off for the item `route` names, or undefined when
|
||||
* no editor on screen publishes one for it (a legacy app, a detail page).
|
||||
* Matched on the item rather than taken as "the latest registered": a script
|
||||
* editor mounted in a flow's drawer registers too, and the rail wants the
|
||||
* page's own item. */
|
||||
/** The mounted editor's hand-off for `target`, or undefined when no editor on
|
||||
* screen publishes one for it (a detail page). Matched on the item rather than
|
||||
* taken as "the latest registered": a script editor mounted in a flow's drawer
|
||||
* registers too, and the rail wants the page's own item. */
|
||||
export function findMountedOpenInSessionSource(
|
||||
route: PreviewItemRoute
|
||||
target: SessionTarget
|
||||
): OpenInSessionSource | undefined {
|
||||
const kind = route.kind === 'app' ? (route.raw_app ? 'raw_app' : undefined) : route.kind
|
||||
if (!kind) return undefined
|
||||
for (const handoff of mounted) {
|
||||
const source = handoff.source()
|
||||
const target = source?.target
|
||||
if (target && target.kind === kind && target.path === route.itemPath) return source
|
||||
const t = source?.target
|
||||
if (t && t.kind === target.kind && t.path === target.path) return source
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { base } from '$lib/base'
|
||||
import type { WorkspaceItemKind } from '$lib/components/workspacePicker'
|
||||
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
|
||||
|
||||
// The paths a preview location can point at, and the base handling around them. Kept apart
|
||||
// from `previewRouter`, which reads a location's *view* from each page's filter schema and
|
||||
@@ -186,3 +187,11 @@ export function parseHistoricalScriptEdit(url: string): { path: string; hash: st
|
||||
const hash = new URL(url, 'http://x').searchParams.get('hash')
|
||||
return hash ? { path: route.itemPath, hash } : null
|
||||
}
|
||||
|
||||
// A `/pipeline/<folder>` route is the data-pipeline graph editor for that folder
|
||||
// (the folder is a single path segment, not a workspace item path). The bare
|
||||
// `/pipeline` list page is not an editor. Returns the folder name, or null.
|
||||
export function parsePipelineRoute(fullPath: string): string | null {
|
||||
const m = stripBase(fullPath).match(/^\/pipeline\/([^/?#]+)/)
|
||||
return m ? normalizePipelineFolder(decodeURIComponent(m[1])) : null
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@ import {
|
||||
pageKey,
|
||||
pageHref,
|
||||
parsePageItemRoute,
|
||||
parsePipelineRoute,
|
||||
parseHistoricalScriptEdit,
|
||||
parsePreviewItemRoute,
|
||||
RESOURCES_PATH,
|
||||
@@ -35,6 +36,7 @@ export {
|
||||
pageHref,
|
||||
parseHistoricalScriptEdit,
|
||||
parsePageItemRoute,
|
||||
parsePipelineRoute,
|
||||
parsePreviewItemRoute,
|
||||
stripBase,
|
||||
TRIGGER_PAGES,
|
||||
@@ -63,7 +65,6 @@ import { buildVariablesFilterSchema } from '$lib/components/variables/variablesF
|
||||
import { buildResourcesFilterSchema } from '$lib/components/resources/resourcesFilter'
|
||||
import { buildAssetsFilterSchema } from '$lib/components/assets/assetsFilter'
|
||||
import { COMPARE_ITEMS_PARAM } from './modifiedItemsMask'
|
||||
import { normalizePipelineFolder } from '$lib/utils/pipelineFolder'
|
||||
import type { WorkspaceItem } from '$lib/components/workspacePicker'
|
||||
import type { SessionTargetKind } from './sessionRuntime.svelte'
|
||||
|
||||
@@ -502,13 +503,6 @@ export function parsePreviewSelectedId(url: string): string | undefined {
|
||||
}
|
||||
}
|
||||
|
||||
// A `/pipeline/<folder>` route is the data-pipeline graph editor for that folder
|
||||
// (the folder is a single path segment, not a workspace item path). The bare
|
||||
// `/pipeline` list page is not an editor. Returns the folder name, or null.
|
||||
export function parsePipelineRoute(fullPath: string): string | null {
|
||||
const m = stripBase(fullPath).match(/^\/pipeline\/([^/?#]+)/)
|
||||
return m ? normalizePipelineFolder(decodeURIComponent(m[1])) : null
|
||||
}
|
||||
|
||||
// The id (before the query) is the artifact's stable routing identity; the name rides in the
|
||||
// hash so the tab strip labels it without a store lookup, and the version in the query so the
|
||||
@@ -562,9 +556,9 @@ export const isArtifactKey = (key: string) => key.startsWith('artifact:')
|
||||
// How a preview tab should render: as an in-process live editor, the deployed
|
||||
// item's view page, or an iframe fallback. An item of a wrappable kind (script,
|
||||
// flow, raw app) mounts its per-(kind,path) cell editor on `/edit/` and its viewer
|
||||
// on `/get/`; a `/pipeline/<folder>` route mounts the data-pipeline graph editor
|
||||
// (single, shared runtime.pipelineEditorState — `path` is the folder); the list page
|
||||
// of a page item kind mounts that list; everything
|
||||
// on `/get/`; a `/pipeline/<folder>` route mounts the data-pipeline graph editor of
|
||||
// that folder (`path` is the folder); the list page of a page item kind mounts that
|
||||
// list; everything
|
||||
// else (static pages, regular drag-and-drop apps, any other route) stays an iframe.
|
||||
export type PreviewSlot =
|
||||
| { kind: 'editor'; editorKind: SessionTargetKind | 'pipeline'; path: string }
|
||||
|
||||
@@ -27,6 +27,8 @@ import {
|
||||
import { maskKey } from './modifiedItemsMask'
|
||||
import { sessionState } from './sessionState.svelte'
|
||||
import { logFeatureUsage } from '$lib/utils/featureUsage'
|
||||
import { AssetService } from '$lib/gen'
|
||||
import { PIPELINE_DRAFT_KIND, pipelineFolderFromBundlePath } from '$lib/pipelinePaths'
|
||||
|
||||
export type DeploymentStatus = { status: 'loading' | 'failed'; error?: string }
|
||||
|
||||
@@ -52,6 +54,9 @@ export interface SessionDeployModelArgs {
|
||||
onItemDiscarded?: (item: DeployItem) => void
|
||||
/** Chat-modified-items mask; undefined shows every draft. */
|
||||
mask?: Set<string>
|
||||
/** Deploys a pipeline folder's draft bundle, which no per-item deploy can: it
|
||||
* expands into the folder's scripts and triggers. */
|
||||
deployPipeline?: (folder: string) => Promise<DeployResult>
|
||||
}
|
||||
|
||||
export interface DiffValues {
|
||||
@@ -59,6 +64,15 @@ export interface DiffValues {
|
||||
after: unknown
|
||||
}
|
||||
|
||||
// A pipeline has no row of its own: it exists while its folder holds pipeline scripts.
|
||||
async function itemExists(kind: Kind, path: string, workspace: string): Promise<boolean> {
|
||||
if (kind !== PIPELINE_DRAFT_KIND) return checkItemExists(kind, path, workspace)
|
||||
const folder = pipelineFolderFromBundlePath(path)
|
||||
if (!folder) return false
|
||||
const pipelines = await AssetService.listPipelineFolders({ workspace })
|
||||
return pipelines.some((p) => p.folder === folder)
|
||||
}
|
||||
|
||||
/** Resolves which mask-only candidates still exist in the session workspace
|
||||
* (deployed) vs are gone (discarded drafts / deleted items) — the
|
||||
* `existingKeys` input of `buildDeployItems`. Undefined while resolving, so
|
||||
@@ -100,7 +114,7 @@ export function useExistingMaskKeys(
|
||||
// workspace; a discarded or deleted one doesn't.
|
||||
void Promise.all(
|
||||
cands.map((c) =>
|
||||
checkItemExists(c.deployKind, c.path, workspaceId)
|
||||
itemExists(c.deployKind, c.path, workspaceId)
|
||||
.then((exists) => (exists ? c.key : null))
|
||||
.catch(() => null)
|
||||
)
|
||||
@@ -246,11 +260,18 @@ export function useSessionDeployModel(getArgs: () => SessionDeployModelArgs) {
|
||||
async function runPlan(entry: DeployPlanEntry): Promise<DeployResult> {
|
||||
const cur = getArgs().workspaceId
|
||||
switch (entry.op) {
|
||||
case 'deploy_draft':
|
||||
case 'deploy_draft': {
|
||||
const folder =
|
||||
entry.draftKind === PIPELINE_DRAFT_KIND
|
||||
? pipelineFolderFromBundlePath(entry.path)
|
||||
: undefined
|
||||
const deployPipeline = getArgs().deployPipeline
|
||||
if (folder && deployPipeline) return deployPipeline(folder)
|
||||
return deployDraft(entry.draftKind, entry.path, cur, {
|
||||
draftOnly: entry.draftOnly,
|
||||
rawApp: entry.rawApp
|
||||
})
|
||||
}
|
||||
case 'discard':
|
||||
return discardDraft(entry.draftKind, entry.path, cur, entry.draftOnly, entry.legacy)
|
||||
}
|
||||
@@ -314,6 +335,16 @@ export function useSessionDeployModel(getArgs: () => SessionDeployModelArgs) {
|
||||
|
||||
// ── Diff values (one resolver for tree + column) ─────────────────────────
|
||||
async function loadDiffValues(item: DeployItem): Promise<DiffValues> {
|
||||
const pipelineFolder =
|
||||
item.draftKind === PIPELINE_DRAFT_KIND ? pipelineFolderFromBundlePath(item.path) : undefined
|
||||
if (pipelineFolder) {
|
||||
// Dynamic: the pipeline deploy module reaches every trigger kind's save utils.
|
||||
const { loadPipelineDraftDiff } = await import(
|
||||
'$lib/components/assets/AssetGraph/pipelineDeploy.svelte'
|
||||
)
|
||||
if (item.hasDraft) return loadPipelineDraftDiff(getArgs().workspaceId, pipelineFolder)
|
||||
return { before: undefined, after: undefined }
|
||||
}
|
||||
const base = diffBaseFor(item, getArgs().workspaceId)
|
||||
if (base.kind === 'draft') {
|
||||
const { deployed, draft } = await getDraftDiffValues(
|
||||
|
||||
@@ -101,6 +101,15 @@ describe('buildDeployItems', () => {
|
||||
expect(items[0].hasDraft).toBe(true)
|
||||
})
|
||||
|
||||
it("a pipeline's deployed scripts fold into its one row", () => {
|
||||
const bundle = maskKey('data_pipeline', 'f/crm/data_pipeline')
|
||||
const node = maskKey('script', 'f/crm/clean')
|
||||
const mask = new Set([bundle, node])
|
||||
const items = buildDeployItems({ draftItems: [], mask, existingKeys: new Set([bundle, node]) })
|
||||
expect(items.map((i) => i.key)).toEqual([bundle])
|
||||
expect(maskOnlyCandidates({ draftItems: [], mask }).map((c) => c.key)).toEqual([bundle])
|
||||
})
|
||||
|
||||
it('maskOnlyCandidates returns keys not covered by a draft', () => {
|
||||
const cands = maskOnlyCandidates({
|
||||
draftItems: [draft('script', 'u/a/draft')],
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user