diff --git a/cli/src/guidance/skills.gen.ts b/cli/src/guidance/skills.gen.ts index e5b3619a27..eb5d46ef57 100644 --- a/cli/src/guidance/skills.gen.ts +++ b/cli/src/guidance/skills.gen.ts @@ -7915,10 +7915,48 @@ A **data pipeline** is NOT a flow. A flow is one runnable that orchestrates step A pipeline node that produces a table should almost always be a **\`duckdb\`** node that materializes its output into a **DuckLake** table with \`-- materialize ducklake:///\` (in a DuckDB node the annotation uses SQL \`--\` comment syntax; write the body as a bare \`SELECT\` and let the runtime do the write). DuckLake is the default lakehouse store for pipelines and is the shape the pipeline editor is built around, so prefer it unless the work specifically calls for something else: - \`postgresql\` / data tables — only for row-level, OLTP-style mutations against an existing Postgres data table (frequent single-row upserts/updates, transactional reads that an app queries live). -- \`bun\` / \`python3\` — only for non-tabular work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node still produces tabular data for downstream steps, land it in DuckLake (write it with the wmill SDK / ducklake helpers) rather than inventing a parallel store. +- \`bun\` / \`python3\` — only for work that doesn't map to SQL: calling an external API, wrangling files, arbitrary glue. When such a node produces tabular data for downstream steps, write it **straight into DuckLake** (or a data table) from the same script — see "Ingesting from Python / TypeScript" below — rather than inventing a parallel store. Do not spread a pipeline across postgres, S3, and DuckLake when one DuckLake lake would do; a consistent DuckLake lakehouse is the goal. +## Ingesting from Python / TypeScript + +A node that fetches data in code (an API, a SaaS export, a scraper) writes the rows **directly** into their DuckLake table with \`wmill.ducklake()\`, or into a data table with \`wmill.datatable()\`. DuckLake already stores every table as parquet files in the workspace object storage, so **never stage the raw payload first** — no intermediate JSON / CSV / parquet file in S3 and no second "load the raw file" node. One ingestion node, one table. + +Pass the rows as a query argument (a list of dicts / array of objects is sent as JSON) and let SQL unnest them; declare the column types so the table gets a real schema: + +\`\`\`python +# pipeline +# on schedule +import wmill + +def main(): + rows = fetch_charges() # list[dict] + lake = wmill.ducklake() + lake.query(""" + CREATE OR REPLACE TABLE stripe_charges AS + SELECT unnest(from_json($rows, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true) + """, rows=rows).fetch() +\`\`\` + +\`\`\`ts +// pipeline +// on schedule +import * as wmill from "windmill-client" + +export async function main() { + const rows = await fetchCharges() + const lake = wmill.ducklake() + await lake\`CREATE OR REPLACE TABLE stripe_charges AS + SELECT unnest(from_json(\${rows}, '[{"id":"VARCHAR","amount":"BIGINT","created":"BIGINT"}]'), recursive := true)\`.fetch() +} +\`\`\` + +- Use \`INSERT INTO
SELECT ...\` instead of \`CREATE OR REPLACE\` to append (an incremental pull), with a one-time \`CREATE TABLE IF NOT EXISTS\` first. +- Assign the client to a variable (\`lake = wmill.ducklake()\`) and keep the SQL a **string literal** (Python) or a template without \`sql.raw\` (TS): that is what lets the graph record the node's write to \`ducklake://main/
\`. A chained \`wmill.ducklake().query(...)\` or SQL built in a variable records no output, so the downstream edge never forms. +- \`wmill.ducklake("")\` targets a non-default catalog; \`wmill.datatable("")\` writes a Postgres data table the same way (Python \`db.query(sql, *args)\` with \`$1\` placeholders, TS the same template form) — use it only when the data belongs in Postgres (see above). +- The downstream node triggers on the table itself: \`-- on ducklake://main/stripe_charges\`. + ## Storage prerequisites A DuckLake pipeline only runs once the workspace has **object storage** (S3 / Azure Blob / GCS) **and a DuckLake catalog** configured — DuckLake tables and \`s3://\` assets can't be materialized or read without it. Check which DuckLake catalogs the workspace has before you build. @@ -7962,7 +8000,7 @@ The key must be a **string literal** — the graph parser is static and cannot f ## Materialize (the managed output) -> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake:///
\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the output via the SDK (\`wmill.writeS3File(...)\`, ducklake helpers, …) and let it be inferred. Use \`duckdb\` when a node should materialize a DuckLake table. +> **A MANAGED \`// materialize\` is DuckDB-only**, and its target must be a DuckLake table (\`ducklake:///
\`). Deploy **rejects** a \`ducklake://\` \`// materialize\` on any other language (\`python3\`, \`bun\`, \`postgresql\`). For a non-DuckDB node writing the lake, do **not** use \`// materialize\` — write the table itself with \`wmill.ducklake()\` (see "Ingesting from Python / TypeScript") and let the output be inferred. Use \`duckdb\` when a node should materialize a DuckLake table. > > The one target ANY language may declare (except a dbt script, whose writes come from its manifest) is a **warehouse relation**: \`// materialize manual dbt:////\`, where \`\` is a warehouse the workspace configures under Settings → dbt. \`manual\` is the only mode it has — nothing generates warehouse DDL, so the node issues its own write (a postgresql \`CREATE TABLE\` / \`INSERT\`, an SDK load, …) and the annotation records the outcome. Use it on an ingestion node whose output a dbt project reads as a \`source\`: the declared relation and the dbt model land on ONE graph node, and a downstream \`// on dbt:////\` fires when the ingestion node completes. diff --git a/frontend/src/lib/components/AutosaveIndicator.svelte b/frontend/src/lib/components/AutosaveIndicator.svelte index 76dc030c58..1e318ed9b2 100644 --- a/frontend/src/lib/components/AutosaveIndicator.svelte +++ b/frontend/src/lib/components/AutosaveIndicator.svelte @@ -142,7 +142,13 @@ let hintTimer: ReturnType | undefined const kindLabel = $derived( - itemKind === 'flow' ? 'flow' : itemKind === 'app' || itemKind === 'raw_app' ? 'app' : 'script' + itemKind === 'flow' + ? 'flow' + : itemKind === 'app' || itemKind === 'raw_app' + ? 'app' + : itemKind === 'data_pipeline' + ? 'pipeline' + : 'script' ) // Each false → true transition fires a hint (others > loaded). Seeded to diff --git a/frontend/src/lib/components/CronBuilder.svelte b/frontend/src/lib/components/CronBuilder.svelte index bcf8508c12..3bceabb417 100644 --- a/frontend/src/lib/components/CronBuilder.svelte +++ b/frontend/src/lib/components/CronBuilder.svelte @@ -15,7 +15,7 @@ contentClasses="p-4" > {#snippet trigger()} - {/snippet} diff --git a/frontend/src/lib/components/DatatablePicker.svelte b/frontend/src/lib/components/DatatablePicker.svelte index 53617acf36..17ad6f3361 100644 --- a/frontend/src/lib/components/DatatablePicker.svelte +++ b/frontend/src/lib/components/DatatablePicker.svelte @@ -17,6 +17,9 @@ selectInputClass?: string class?: string onClear?: () => void + /** Size the input to its content, like Path's folder picker. */ + useContentEditable?: boolean + RightIcon?: any } let { @@ -27,7 +30,9 @@ placeholder = undefined, selectInputClass = '', class: className = '', - onClear = undefined + onClear = undefined, + useContentEditable = false, + RightIcon = undefined }: Props = $props() let datatables = usePromise(() => @@ -47,6 +52,8 @@ {placeholder} inputClass={selectInputClass} {onClear} + {useContentEditable} + {RightIcon} /> {#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'datatable', path: value })} diff --git a/frontend/src/lib/components/DucklakePicker.svelte b/frontend/src/lib/components/DucklakePicker.svelte index 3a6f2bf739..d6ed4d0cdf 100644 --- a/frontend/src/lib/components/DucklakePicker.svelte +++ b/frontend/src/lib/components/DucklakePicker.svelte @@ -17,6 +17,9 @@ selectInputClass?: string class?: string onClear?: () => void + /** Size the input to its content, like Path's folder picker. */ + useContentEditable?: boolean + RightIcon?: any } let { @@ -27,7 +30,9 @@ placeholder = undefined, selectInputClass = '', class: className = '', - onClear = undefined + onClear = undefined, + useContentEditable = false, + RightIcon = undefined }: Props = $props() let ducklakes = usePromise(() => @@ -45,6 +50,8 @@ {placeholder} inputClass={selectInputClass} {onClear} + {useContentEditable} + {RightIcon} /> {#if showSchemaExplorer && value && assetCanBeExplored({ kind: 'ducklake', path: value })} diff --git a/frontend/src/lib/components/Editor.svelte b/frontend/src/lib/components/Editor.svelte index f9df8adcef..03bf5ce87e 100644 --- a/frontend/src/lib/components/Editor.svelte +++ b/frontend/src/lib/components/Editor.svelte @@ -1523,6 +1523,9 @@ } catch (e) { console.log('error initializing services', e) } + // Unmounted while the services loaded: the container is detached, and + // creating the editor on it throws. + if (destroyed || !divEl?.isConnected) return // vscode.languages.registerDefinitionProvider('*', { // provideDefinition(document, position, token) { diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index 8990f70ae5..8e60d40f72 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -2101,43 +2101,15 @@ {/if} {#if customUi?.previewPanel?.hideArgs} - -
- {#if testJob?.id && testJob.type === 'CompletedJob' && opWs} - -
- - {testJob.id.slice(0, 8)}… ↗ - - -
- {/if} -
+ +
+ +
+
{#if testIsLoading} {@render cancelTestButton('sm', 'shadow-md')} {:else if (customUi?.previewPanel?.downstreamSubscribers ?? 0) > 0 || customUi?.previewPanel?.onBoundedRun} @@ -2271,6 +2243,31 @@ {@render runTestButton('sm', 'shadow-md')} {/if}
+ {#if testJob?.id && testJob.type === 'CompletedJob' && opWs} + +
+ + {testJob.id.slice(0, 8)}… ↗ + + +
+ {/if} +
{#if customUi?.previewPanel?.argsAboveLogs && schema?.properties && Object.keys(schema.properties).length > 0}
string | undefined /** Header variant: no path line (the host shows it) and a lighter summary. */ compact?: boolean /** Rendered inside the trigger in place of the summary, for a host that hangs a second copy @@ -51,6 +58,7 @@ editable = false, onSaved, kind = 'flow', + saveOverride = undefined, compact = false, label, triggerClass = 'block min-w-0 max-w-full px-1 py-0.5 rounded text-left cursor-pointer hover:bg-surface-hover transition-colors', @@ -67,7 +75,9 @@ let popoverOpen = $state(false) // Resolved before the popover opens, not on opening it: the path field only exists for an owner, // and `openFocus` looks for it in the same flush the popover renders in. - const own = $derived(isOwner(path ?? '', actingUser, $operatingWorkspace)) + const own = $derived( + saveOverride !== undefined || isOwner(path ?? '', actingUser, $operatingWorkspace) + ) let onBehalfOfEmail = $state(undefined) let summaryInput: ReturnType | undefined = $state() let labelsDirty = $state(false) @@ -101,6 +111,17 @@ async function save(close: () => void) { const initialPath = path ?? '' const newPath = own ? editPath : initialPath + if (saveOverride) { + const error = saveOverride({ path: newPath, summary: editSummary, labels }) + if (error) { + sendUserToast(error, true) + return + } + labelsDirty = false + close() + onSaved?.(newPath) + return + } try { await updateItemPathAndSummary({ diff --git a/frontend/src/lib/components/assets/AssetGraph/AddDataSourceMenu.svelte b/frontend/src/lib/components/assets/AssetGraph/AddDataSourceMenu.svelte new file mode 100644 index 0000000000..49a222148b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/AddDataSourceMenu.svelte @@ -0,0 +1,150 @@ + + + + diff --git a/frontend/src/lib/components/assets/AssetGraph/AddDownstreamMenu.svelte b/frontend/src/lib/components/assets/AssetGraph/AddDownstreamMenu.svelte new file mode 100644 index 0000000000..2a54fe11d4 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/AddDownstreamMenu.svelte @@ -0,0 +1,73 @@ + + + + + + diff --git a/frontend/src/lib/components/assets/AssetGraph/AddDownstreamPill.svelte b/frontend/src/lib/components/assets/AssetGraph/AddDownstreamPill.svelte new file mode 100644 index 0000000000..8ebd24455c --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/AddDownstreamPill.svelte @@ -0,0 +1,23 @@ + + + + + + + Add downstream step + + diff --git a/frontend/src/lib/components/assets/AssetGraph/AddNode.svelte b/frontend/src/lib/components/assets/AssetGraph/AddNode.svelte index 880abb685a..57a31e34d8 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AddNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AddNode.svelte @@ -1,145 +1,36 @@ - - {#snippet trigger()} - - - {/snippet} - + +
+ + {#snippet trigger()} + + + {/snippet} + +
diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte index 27f6ce6be7..87b085593c 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte @@ -1,8 +1,11 @@ -
+
onNodePointerEnter(node.id)} + onnodepointerleave={onNodePointerLeave} + onedgepointerenter={onEdgeEnter} + onedgepointerleave={onEdgeLeave} --background-color={false} >
- + + {#if assetsOnlyToggle} + + { + assetsOnly = !e.detail + if (assetsOnlyFolder != undefined) setAssetsOnlyFolder(assetsOnlyFolder, assetsOnly) + }} + /> + {#if withoutPassive.dropped > 0} + { + showPassiveReads = e.detail + try { + localStorage.setItem(SHOW_PASSIVE_READS_KEY, String(e.detail)) + } catch {} + }} + /> + {/if} + + {/if} {#if showMinimap} + +{/if} + +{#if scheduleWizard} + +{/if} + +{#if onAddPipelineScript} + +{/if} +