Files
windmill/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte
T
Ruben FiszelandClaude Opus 4.8 f6998ec54c feat: data tests for ducklake pipeline materialization (#9708)
* feat: data tests for ducklake pipeline materialization

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): data_test count badge on pipeline graph nodes

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: surface annotation badges (incl. data_test) on deployed pipeline nodes

Backend graph endpoint now parses each pipeline member's deployed body and returns partition/freshness/tag/retry/data_test, so badges render on deployed nodes, not only live drafts. Aligns the TS DataTest.relationships fields to snake_case to match the Rust serde wire shape (the type is now populated from both the parser and the backend JSON).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): keep materialize output edge when editing the producer in the pipeline graph

The live-edit overlay re-derived a selected/edited script's lineage from // on inputs + body-inferred assets only, so the // materialize <asset> output (an annotation, not body SQL) was judged stale and its write-edge dropped on select — leaving the materialized asset unlinked (and the node's annotation badges hidden). Include the parsed materialize target in liveRefKeys and the draft writeOuts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: run all data tests in one pass with a structured per-test result

Replace the raise-on-first-violation probes with a single materialize summary that embeds every test's violating-row count in a data_tests column (computed in a CTE, since DuckDB rejects subqueries inside struct literals). The worker reads the breakdown and decides pass/fail: a clean run returns the per-test summary in the result; a failing run errors with the FULL list (every test, ✓/✗ + counts), not just the first failure. Verified live (EE) for built-ins + custom, pass and multi-failure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): data-test pass/fail checklist in the job result

DisplayResult renders a per-test checklist (✓/✗ + violation counts) above the raw result for managed materialize runs — from the structured data_tests on success, and parsed from the worker's breakdown message on failure. Shows in the script editor Test panel, the runs page, and the pipeline asset run pane.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): move data-test badge onto the producer→asset edge with run status

The test badge now sits on the write-edge (the transformation link) rather than the producer node, since the tests assert on what the transformation produces. It's tinted by the producer's last-run status (green = passed, red = a test failed) and its hover title lists every declared test. Removes the now-redundant node badge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(frontend): render custom data-test scripts as their own clickable graph nodes

A // data_test <script_path> custom test now appears as its own node below the asset it validates, joined by a dashed 'tests' edge. Clicking it opens the test script in the detail pane (dispatched like any runnable). Built-in tests stay folded into the edge badge; only script-backed tests become nodes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): type data-test edge field via AssetGraphResponse, not in-scope g

BuiltEdge is declared at component scope, outside build(g), so referencing typeof g.runnables in its type failed CI's svelte-check (Cannot find name 'g'). Use the imported AssetGraphResponse type instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): anchor edge badge on routed path + a11y text on test icons

Address review: the data-test edge badge anchored on the straight-line midpoint, floating off detoured edges — anchor it at detourX when the edge is routed through a gutter lane. Add sr-only pass/fail text so the checklist icons are distinguishable to screen readers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: close data-test enforcement bypass + gate badges to scripts + reject multi-stmt custom tests

Address review (cubic) findings:
- P1: managed materialize generates its own summary row carrying data_tests, and enforcement reads that column — but a // result_collection annotation (e.g. a scalar mode) could reshape the row and drop data_tests, silently bypassing a failing test. Force LastStatementAllRows for managed materialize runs so the summary row is always intact.
- P2: asset-graph annotation badges were keyed by path only, so a flow sharing a path with a pipeline script inherited its badges. Gate the lookup on usage_kind == Script.
- P2: a custom test body is embedded as a subquery, so a multi-statement body produced invalid SQL with an opaque DuckDB error. Validate single-statement up front with an actionable error; align docs/comments.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: fail loud if fewer data-test outcomes recovered than declared

Defense-in-depth from the fresh-context review: enforcement reads per-test outcomes off the materialize summary row, but if the data_tests column were ever dropped/reshaped at the FFI boundary, extract_data_tests would return fewer (or zero) outcomes and the run would silently pass unverified tests. Track the embedded test count on MaterializeExec and abort with a clear error when recovered < declared. Verified: normal run (4==4) unaffected; the scalar-result_collection bypass already fails.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: relationships data test same-lake reuse + schema-qualified target quoting

Address Codex/Pi review (two P1s in the relationships codegen):
- A relationship into the same ducklake as the materialize target minted a second ATTACH of that lake under _wm_ref_N while _wm_target already held it — DuckDB forbids attaching one database twice, so the test failed before it could run. Reuse _wm_target for same-lake references.
- A schema-qualified target (ducklake://warehouse/main.dim_products.sku) emitted FROM _wm_ref_0."main.dim_products" — one quoted identifier with a literal dot — silently querying a nonexistent table. Quote each dotted segment so the dot stays a schema separator.
Adds tests for both.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(frontend): refresh data_test badge on deployed-script drafts + scope to materialize target

Address Codex review nits (both P2):
- resolveGraph: the existing-runnable draft-overlay branch kept the deployed data_tests, so adding/removing // data_test lines on an already-deployed script left the badge stale until redeploy. Refresh it from the live parse like the new-runnable branch.
- AssetGraphCanvas: data tests were attached to every write-edge from a producer. They assert on the // materialize target (always a ducklake asset in v1), so only the ducklake write-edge now carries the badge and custom-test nodes — a producer's other (S3/datatable) outputs no longer show them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 23:47:02 +02:00

170 lines
6.3 KiB
Svelte
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<script lang="ts">
import { BaseEdge, getBezierPath, type EdgeProps } from '@xyflow/svelte'
import { NODE } from '$lib/components/graph/util'
import { FlaskConical } from 'lucide-svelte'
import type { DataTest } from './parsePipelineAnnotations'
let {
sourceX,
sourceY,
sourcePosition,
targetX,
targetY,
targetPosition,
markerEnd,
style,
data
}: EdgeProps = $props()
// Data-test badge on the producer→asset write-edge: the edge *is* the
// transformation, so the tests that assert on its output belong here. The
// producer's last-run status tints it (the script fails if any test fails).
let tests = $derived((data as { data_tests?: DataTest[] } | undefined)?.data_tests)
let testsStatus = $derived(
(data as { testsRunStatus?: 'running' | 'success' | 'failure' } | undefined)?.testsRunStatus
)
// Badge position on the link. A detoured edge runs its vertical segment in
// the gutter lane at `detourX`, so anchor the badge there (not on the
// straight-line midpoint, which would float off the routed path).
let detourX = $derived((data as { detourX?: number } | undefined)?.detourX)
let labelX = $derived(detourX ?? (sourceX + targetX) / 2)
let labelY = $derived((sourceY + targetY) / 2)
function fmtTest(t: DataTest): string {
switch (t.type) {
case 'unique':
return `unique(${t.column})`
case 'not_null':
return `not_null(${t.column})`
case 'accepted_values':
return `accepted_values(${t.column} = ${t.values.join(',')})`
case 'relationships':
return `relationships(${t.column} → ${t.to_path}.${t.to_column})`
case 'custom':
return `custom(${t.path})`
}
}
let badgeTitle = $derived(
tests && tests.length > 0
? `${tests.length} data test${tests.length > 1 ? 's' : ''}${
testsStatus === 'success'
? ' · last run passed'
: testsStatus === 'failure'
? ' · last run failed'
: ''
}:\n${tests.map((t) => `• ${fmtTest(t)}`).join('\n')}`
: ''
)
// An edge that skips at least one full layer (source-bottom → target-top
// gap larger than gap + node row) while staying near-vertical runs
// straight under the nodes in between — the child-and-grandchild-of-the-
// same-node case, where the tidy-tree layout puts all three on one
// column. Detour those through the gutter beside the column.
const SKIP_DY = NODE.gap.vertical * 2 + NODE.height
const NEAR_VERTICAL_DX = NODE.width / 4
// Lane offset: half a node plus a margin that stays inside the
// inter-column gutter (gap.horizontal / 2).
const LANE = NODE.width / 2 + NODE.gap.horizontal / 2
const CORNER = 14
// Rounded-orthogonal detour: drop out of the source, swing into the
// gutter lane at `midX`, run it down past the intermediate rows, swing
// back into the target.
function gutterPath(midX: number): string {
const yTop = sourceY + 24
const yBot = targetY - 24
const outDir = midX > sourceX ? 1 : -1
const inDir = targetX > midX ? 1 : -1
return [
`M ${sourceX} ${sourceY}`,
`L ${sourceX} ${yTop - CORNER}`,
`Q ${sourceX} ${yTop} ${sourceX + outDir * CORNER} ${yTop}`,
`L ${midX - outDir * CORNER} ${yTop}`,
`Q ${midX} ${yTop} ${midX} ${yTop + CORNER}`,
`L ${midX} ${yBot - CORNER}`,
`Q ${midX} ${yBot} ${midX + inDir * CORNER} ${yBot}`,
`L ${targetX - inDir * CORNER} ${yBot}`,
`Q ${targetX} ${yBot} ${targetX} ${yBot + CORNER}`,
`L ${targetX} ${targetY}`
].join(' ')
}
// Long edges bend out of the source within the layer gap and descend in
// the *target* column. The flow editor's pattern (descend at the source,
// bend at the bottom) assumed the source column below was free — in the
// banded tree layout it's where the source's own children live, so the
// straight segment ran under them (e.g. fx_rates → daily_revenue under
// fx_rates' other subtree). Long edges only target joins here (a single-
// parent child is always exactly one layer below its parent), and the
// space above a join is the seam between its parents' bands — clear by
// construction.
let edgePath = $derived.by(() => {
const dy = targetY - sourceY
// Obstacle-aware detour: the canvas detected this edge's straight run
// would pass over an unrelated node and chose a clear lane. Route there.
const detourX = (data as { detourX?: number } | undefined)?.detourX
if (detourX != undefined) {
return gutterPath(detourX)
}
if (dy > SKIP_DY && Math.abs(targetX - sourceX) < NEAR_VERTICAL_DX) {
// Near-vertical same-column skip: lane just beside the column, past
// both endpoints so the horizontal runs never degenerate.
const side = targetX >= sourceX ? 1 : -1
return gutterPath(
side > 0 ? Math.max(sourceX, targetX) + LANE : Math.min(sourceX, targetX) - LANE
)
}
const bendY = sourceY + Math.min(100, NODE.gap.vertical)
const long = dy > 100
const [bezier] = getBezierPath({
sourceX,
sourceY,
sourcePosition,
targetX,
targetY: long ? bendY : targetY,
targetPosition,
curvature: 0.25
})
// The bezier ends at (targetX, bendY); continue straight down the
// target column into the node.
return long ? `${bezier} L${targetX},${targetY}` : bezier
})
</script>
<BaseEdge
path={edgePath}
{markerEnd}
{style}
interactionWidth={0}
label={undefined}
labelStyle={undefined}
/>
{#if tests && tests.length > 0}
<!-- Badge centered on the link midpoint. Edges render in the SVG layer, so
the HTML badge is wrapped in a foreignObject (no EdgeLabelRenderer in
this @xyflow/svelte version). -->
<foreignObject x={labelX - 28} y={labelY - 9} width="56" height="18" class="overflow-visible">
<div
xmlns="http://www.w3.org/1999/xhtml"
class="w-full h-full flex items-center justify-center"
style="pointer-events: none;"
>
<div
class="flex items-center gap-0.5 px-1 py-0.5 rounded-sm border shadow-sm text-3xs leading-none font-mono cursor-default {testsStatus ===
'failure'
? 'bg-red-50 dark:bg-red-950/50 border-red-300 dark:border-red-900/60 text-red-700 dark:text-red-300'
: testsStatus === 'success'
? 'bg-emerald-50 dark:bg-emerald-950/50 border-emerald-300 dark:border-emerald-900/60 text-emerald-700 dark:text-emerald-300'
: 'bg-surface border-border text-secondary'}"
style="pointer-events: all;"
title={badgeTitle}
>
<FlaskConical size={10} />
<span>×{tests.length}</span>
</div>
</div>
</foreignObject>
{/if}