windmill/docs/pipelines-vs-dbt.md

Pipelines vs. dbt

Positioning analysis and architectural notes for the data-pipeline abstraction currently landing on feat/asset-graph-view. Covers what we're building, how it differs from dbt, which dbt features are real gaps vs. TODO, and a focused deep-dive on incremental materialization — including a recommendation to collapse it into partitioning rather than ship it as a separate concept.

What we're building

Asset-centric, polyglot, annotation-driven, event-aware:

• Assets (datatable, ducklake, s3object, volume) are graph nodes; scripts are edges that produce/consume them. See backend/parsers/windmill-parser/src/asset_parser.rs:25.
• Lineage comes from two sources: parsed annotations (// pipeline, // on datatable://..., // partitioned daily, // freshness 1h, // trigger any, // debounce, // tag, // retry) and body-inferred reads/writes via the asset parser.
• Triggers are first-class: schedule, webhook, email, kafka, mqtt, nats, postgres, sqs, gcp — all wired into the same DAG view (frontend/src/lib/components/assets/AssetGraph/types.ts:56).
• Per-language scaffolds (DuckDB ATTACH, Postgres, Python, TS, Bash) generate starter code per PipelineOutputKind (datatable | ducklake | s3_parquet | s3_object | none). See frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts:11.

Differentiators vs. dbt

1. Event-driven + batch in one DAG. dbt is batch-on-warehouse. Kafka → Python normalize → DuckDB aggregate → Postgres view → Slack notify is native here; in dbt land it's "use Airflow/Prefect for the non-SQL parts."
2. Polyglot, not SQL+Jinja. Python/TS/Bash/Duck/PG transformations live in the same graph. No Jinja templating language; annotations are real comments parsed strictly.
3. Multi-substrate by design. datatable (Postgres), ducklake (lakehouse), s3_parquet, s3_object are peers. dbt's universe is "tables in your warehouse."
4. One platform. Same runtime as workflows, internal apps, background jobs, RBAC, secrets, schedules. dbt is single-purpose.
5. Inferred lineage from code. Body parser picks up CREATE TABLE / S3 writes — annotations are not strictly required to get edges. dbt requires explicit {{ ref() }} everywhere.

Where dbt wins today

Gap	Architectural blocker?	Verdict
Data tests	No	Shipped (// data_test)
Incremental materializations	No, but pick a philosophy	TODO with design decision
Column lineage	No	Shipped (// column); docs site still TODO
Snapshots / SCD2	Yes (key=… history)	Managed strategy
Selective execution grammar	No	UI/CLI surface
Schema contracts	No, but design metadata model	TODO with design work
Packages / community / macros	No	Shipped (// macros workspace macro libraries)
Semantic layer / metrics	No	Large additive scope

The three items where the current abstraction needs deliberate decisions before more weight lands on it: incremental philosophy, schema metadata, and annotation extensibility. The rest is execution.

1. Data tests

dbt: unique, not_null, accepted_values, custom generic tests, plus singular tests. Run as SELECT statements that pass when they return 0 rows.

Shipped via the // data_test annotation (built on materialization): // data_test unique <col>, not_null, accepted_values <col> = a,b,c, relationships <col> -> <asset>.<col>, and // data_test <script_path> for the custom escape hatch (dbt's singular test). Each compiles to a SQL verifier probe that runs against the freshly-materialized asset and raises on violation, riding the existing failure-propagation path. This is also the first extensible annotation — see ducklake-materialization.md §"Data tests" for the annotation→verifier pattern that column-lineage will reuse. The keyword is data_test, not test, to stay clear of the unrelated // test: CI-test annotation.

2. Incremental materializations

See Incremental deep-dive below.

3. Column lineage + docs

dbt: SQL-AST parsing for column-level deps; dbt docs serve produces a static lineage site with descriptions.

Shipped, inferred-first. For DuckDB scripts the column lineage is derived automatically from the SQL AST — windmill-parser-sql-asset walks each output-producing query's projection and maps every output column to the source columns its expression reads (passthroughs and computed columns like amount + tax AS total), resolving each input to its asset via the same ATTACH/alias machinery the asset parser already uses. This is the dbt-style AST lineage, and it needs no annotation.

The // column <out_col> <- <asset-uri>.<col>[, …] annotation is the override / escape hatch, for the cases inference can't reach: polyglot transforms (Python/TS/Bash — no SQL AST), dynamic SQL (${sql.raw(...)}, flagged by SqlQueryDetails.has_raw_interpolation), or correcting a mis-inferred edge. Inferred and annotated lineage are merged per output column with the annotation winning (merge_column_lineage). The annotation is the second extensible annotation family after // data_test — same head-then-tail parse shape (see ColumnLineage/ColumnRef in asset_parser.rs) — and is pure metadata: it drives the graph surface, never a runtime probe.

Surfaced two ways in the asset graph: a count badge on the producer→materialized-asset write-edge, and a transitive column-lineage trace (ColumnLineageTrace.svelte, over the cross-script graph built by columnLineageGraph.ts) in the asset details pane — select an asset to see its columns and their full upstream/downstream lineage, click any column to highlight its complete impact set across the pipeline (forward + backward).

SQL-AST inference runs both server-side (the graph endpoint + deploy via parse_assets, for deployed members) and in the live editor — the same parser compiled to WASM (windmill-parser-wasm-asset) runs on the open draft's buffer, and resolveGraph merges its column_lineage with the buffer's // column annotations under the same annotation-wins precedence, so the draft preview matches what deploys. The dbt docs serve-style static lineage site is still TODO.

4. Snapshots / SCD2 — shipped

dbt: {% snapshot %} blocks with strategy='timestamp' or 'check'. Shipped as a modifier on the keyed merge: // materialize ducklake://<lake>/<dim> key=<natural_key> history [track=<c1,c2,…>] (the leading keyword scd2 is an alias). The SELECT returns the current snapshot (one row per key); the runtime adds valid_from/valid_to/is_current and, on each run, diffs against the live current rows to close changed versions and open new ones (dbt's strategy='check' shape — track= is the check-columns list; empty ⇒ all non-key columns). Because it's a managed strategy (not a new output kind), it inherits // data_test and schema capture, and each run (re)creates a <dim>_current view; consumers get effective-dated joins via native ASOF JOIN … >= valid_from. Deletes follow dbt: soft by default (absent keys stay current), or deletes=close for hard_deletes=close. v1 is non-partitioned; partitioned history is a follow-up. See ducklake-materialization.md.

5. Selective execution grammar

dbt: --select tag:nightly+ state:modified+ +my_model+. Today: requestRunCascadeSignal in the canvas, // tag annotation parsed. Graph + tags + last-run state has all the inputs. UI/CLI surface, not abstraction work.

6. Schema contracts

dbt: contract: enforced + columns: [{name, data_type}]. Compile-time check that model output matches the declaration.

Today: // on datatable://users/active is a string. Rename a column upstream → downstream breaks at runtime, silently.

This is the item where the current asset abstraction is thinnest. To do contracts well: capture output schemas after a run (substrate-specific DESCRIBE), persist them as asset metadata, validate consumer references at save time. The asset-as-typed-node model accommodates it — but where schemas live (asset row, sidecar?), when they're captured (post-run? edit-time?), and how versioning works are non-trivial design choices. Worth doing intentionally now while the asset surface is still young.

7. Packages / community / macros — shipped (macro libraries)

dbt: dbt deps, dbt-utils, dbt-expectations. Whole ecosystem on Jinja macros — text templating that exists largely to paper over weak warehouse SQL and absent code reuse.

Shipped as workspace DuckDB macro libraries — engine-native CREATE MACRO (scalar + table-valued) instead of templating, resolved and injected by Windmill ("resolve-and-inject"):

• A DuckDB script annotated // macros is a library: its body is CREATE OR REPLACE MACRO statements (plus plain setup). Deploy parses each macro into the macro_definition registry (workspace-unique names) — available to consumers the moment the deploy transaction commits.
• Any DuckDB script that calls a registered macro gets CREATE OR REPLACE TEMP MACRO … blocks injected into its statement list at job time — dependency-topo-ordered (DuckDB bind-checks macro bodies at CREATE) and placed after the setup/ATTACH prefix. Each provider library's own setup statements are injected ahead of the definitions (deduped across libraries), so a macro body that references its lib's ATTACH binds on the implicit path too. Late-bound: a lib redeploy applies to subsequent runs — workers cache the registry per workspace and evict it via a transactional notify_macro_registry_change event (worst-case staleness = the notify poll interval, ~10s; 60s TTL as backstop) — dbt's semantics with a deploy-time registry instead of a compile step.
• // use <lib_path> force-injects a whole library (definitions + its setup statements) — the escape hatch for dynamic SQL (e.g. calls inside query('…') strings) that lexical detection can't see. A library's own // use declarations are honored transitively: consuming a macro from lib B pulls in whatever B // uses, so a dynamic dependency stays encapsulated in the library instead of leaking to every consumer.
• Deploy-time validation: builtin-shadow rejection (DuckDB silently allows shadowing concat!), within-file forward references, cross-lib name collisions, non-setup statements, managed-ATTACH setup.
• Graph surface: the library renders as a node ("defines N macros" badge + signature strip in the details pane) with violet lib→consumer edges from deploy-recorded call detection (macro_usage) and // use annotations.

Trust model (deliberate, dbt-package-like). Macro libraries are workspace-trusted code: anyone who can deploy a script can publish macros, and implicit call detection injects them into other users' DuckDB jobs with the consuming job's credentials — the same trust boundary as a dbt package in the project (any committer changes every model). Two guardrails bound it: a script's own macro definitions always win (a library deploy can never replace a local CREATE MACRO — the name is excluded from injection entirely), and names are workspace-unique with built-in shadowing rejected at deploy. A call to a macro that doesn't exist yet is the residual surface (whoever first deploys a lib defining that name "captures" the call) — if workspaces ever need finer trust, per-consumer opt-in/opt-out or ACL-filtered injection can be layered on the same seam.

Table macros + query_table(src) cover most of dbt's model factory pattern (one parameterized transform, N thin per-asset scripts); DuckDB's richer SQL (COLUMNS(*), native PIVOT, lambdas) absorbs much of dbt-utils outright. Parameterized custom data tests (dbt's generic tests) remain a follow-up on the // data_test <script> seam, as does the broader annotation-extensibility question (hooks-as-scripts etc.) — macros closed the reuse gap without opening the parser vocabulary.

8. Semantic layer / metrics

dbt: metrics: blocks, MetricFlow, BI-tool query API. Large scope, additive. Lowest priority of the eight.

Incremental deep-dive

How dbt incremental works

-- models/marts/orders_daily.sql
{{ config(
    materialized='incremental',
    unique_key='order_id',
    incremental_strategy='merge',
    on_schema_change='append_new_columns'
) }}

SELECT order_id, user_id, amount, created_at
FROM {{ ref('orders_raw') }}
{% if is_incremental() %}
WHERE created_at > (SELECT MAX(created_at) FROM {{ this }})
{% endif %}

• First run (target doesn't exist): CREATE TABLE orders_daily AS SELECT ... — full build, no WHERE.
• Subsequent runs: stage to temp table, then MERGE on unique_key.

Knobs: incremental_strategy ∈ {merge, append, delete+insert, insert_overwrite, microbatch}. on_schema_change ∈ {fail, ignore, append_new_columns, sync_all_columns}. --full-refresh forces rebuild.

Pain points: watermark + unique_key interaction is subtle (late-arriving rows past the watermark are silently dropped); on_schema_change defaults to ignore (silent column drop); Jinja is_incremental() runs at compile, not runtime — debugging requires dbt compile; cross-warehouse MERGE dialect is dbt's biggest internal complexity.

Where Windmill stands today

• // partitioned daily|hourly|weekly|monthly|dynamic key=... parsed into PartitionSpec at backend/parsers/windmill-parser/src/asset_parser.rs:172.
• // freshness 1h parsed.
• Templates emit CREATE TABLE IF NOT EXISTS ... AS SELECT * — full refresh, every run, no partition substitution.
• No WM_PARTITION_* context flowing into scripts.
• No materialized-partition state per asset.

Annotations are present but metadata-only. Nothing actually executes incrementally yet.

Path A — Literal templates ("script is the truth")

Philosophy: WYSIWYG. Windmill never wraps. Templates scaffold boilerplate, partition context is injected as bind / env vars, the user owns the SQL.

-- pipeline
-- on datatable://prod/orders_raw
-- partitioned daily
-- unique_key order_id

ATTACH 'datatable://prod' AS pg;

CREATE TABLE IF NOT EXISTS pg.orders_daily (
  order_id    BIGINT PRIMARY KEY,
  user_id     BIGINT,
  amount      NUMERIC,
  created_at  TIMESTAMPTZ
);

CREATE OR REPLACE TEMP TABLE _stage AS
SELECT order_id, user_id, amount, created_at
FROM pg.orders_raw
WHERE created_at >= $WM_PARTITION_START
  AND created_at <  $WM_PARTITION_END;

BEGIN;
DELETE FROM pg.orders_daily
 WHERE created_at >= $WM_PARTITION_START
   AND created_at <  $WM_PARTITION_END;
INSERT INTO pg.orders_daily SELECT * FROM _stage;
COMMIT;

Runtime: resolve (value, start, end) from scheduler tick / trigger event / backfill range → bind as SQL params → execute script as-is → record (asset_path, partition_value) on success.

Pros: no compile step; backfill is trivial (idempotent DELETE+INSERT); late-arriving data → just re-run the affected partition; no dialect rewriting in core; Python/TS/Bash/SQL all fit the same model.

Cons: boilerplate per script; materialization changes require script edits; user owns dialect specifics.

Path B — dbt-style wrapping

Philosophy: separate intent (SELECT) from execution (DDL). User declares what; Windmill compiles to per-substrate DDL.

User writes:

SELECT order_id, user_id, amount, created_at
FROM pg.orders_raw
WHERE created_at >= $WM_PARTITION_START
  AND created_at <  $WM_PARTITION_END

Runtime parses, looks up target schema, wraps per output kind + strategy + first-run/subsequent-run state.

Pros: concise; materialization is a config flip; automatic schema-drift handling; cross-substrate consistency.

Cons: two-layer execution ("what ran?" needs a compile-output view); SELECT-only restricts pre/post-statement work (dbt's answer: pre_hook / post_hook — more surface); doesn't generalize to Python/TS (you end up with two execution models); cross-substrate MERGE dialect is where dbt has burned the most engineering — we'd inherit that tax forever; schema introspection per substrate is its own project.

Path C — Hybrid (recommended)

• Literal-by-default: scaffolds emit full DDL with WM_PARTITION_* substitution. WYSIWYG for all languages.
• Helper library (e.g. wmll.partition, wmll.datatable.upsert_partition): lifts boilerplate into library calls without hiding semantics — readable source.
• Opt-in wrapping for single-SELECT SQL scripts via // materialized incremental wrap=true. Limit to DuckDB first; add others as needed. Always log the compiled SQL.
• State + backfill UI: persist materialized partitions per asset; UI to backfill a range with concurrency cap.

Ships A's 80% case first without committing to B's dialect-rewriting tax. Wrapping becomes opt-in convenience for users who want dbt-style ergonomics.

See ducklake-materialization.md for the DuckLake-native realization of this path: how snapshots make the assets versioned/reproducible for free, the executor codegen seam, and the materialization-metadata schema.

Decisions either path forces

1. Partition window provenance. Scheduler tick? Trigger event time (Kafka event_time header)? Explicit backfill? Default = "now's bucket"?
2. Surface. Bind params ($WM_PARTITION_START), env vars (WM_PARTITION_START), helper library — probably all three for different languages, but pick canonical names.
3. First-run bootstrap. Template scaffolds CREATE TABLE IF NOT EXISTS (A), or runtime detects "table missing → full refresh" (B).
4. State tracking. materialized_partitions keyed by (workspace, asset_kind, asset_path). Drives "run stale," backfill gap detection, downstream waiting.
5. Backfill execution. N partitions → serial? Parallel with concurrency cap per asset?
6. Idempotency contract. // partitioned should imply "re-running the same partition is safe." Templates and helpers must enforce.

Partitioning vs. incremental: the reframing

Partitioning covers ~80% of what dbt's incremental does. What it gives for free:

• Unit of work (one partition per run)
• Idempotency (DELETE-by-partition + INSERT is safe to rerun)
• State (track which partitions are materialized)
• Backfill (re-run a range)
• First-run vs. subsequent-run (every run is "process partition P" — no special case)
• "Process only new data" (the partition window IS the filter)

dbt itself has been migrating toward partition-first thinking via microbatch strategy — essentially incremental with mandatory partition key.

What partitioning alone doesn't address

Dedup within a partition by a separate key. Example: partition by created_at daily, but orders_raw is mutable — the same order_id can appear multiple times in one partition (initial create, then amendments). You want orders_daily to hold the latest version per order_id.

DELETE-by-partition + INSERT works only if you reprocess from a source-of-truth source. If you're consuming amendments and need dedup within the slice, you need MERGE on order_id, not DELETE on partition.

This is what dbt's unique_key does. Orthogonal to partitioning: partitioned answers "which slice?"; unique_key answers "how do I dedup inside the slice?"

Pure watermark-based incremental. Mostly subsumed by // partitioned dynamic key=updated_at — a partition becomes "everything since the last seen value of key."

Recommended annotation shape

Don't build "incremental" as a concept. Build:

• // partitioned <kind> — unit of work + state + backfill (already exists).
• // unique_key <col> — opt-in dedup-within-partition. Drives MERGE template vs. DELETE+INSERT template.
• // append — opt-out of dedup entirely (INSERT-only, no DELETE).

This collapses dbt's materialized=incremental + incremental_strategy + unique_key into orthogonal annotations that compose. Partition-first is the better mental model.

Schema drift handling (on_schema_change) is genuinely separate — applies to full-refresh too — and belongs with the schema-contracts work (gap #6).

First implementation slice

Sequencing if we go with the hybrid + partition-first reframing:

1. Partition runtime context — resolve (value, start, end) from scheduler / trigger / backfill, surface as bind vars + env vars. No materialization change yet.
2. Helper library — wmll.partition.window(), wmll.datatable.upsert_partition() for Python/TS, SQL macros for DuckDB/PG.
3. Template updates — when // partitioned X is present, scaffold DELETE+INSERT (or MERGE when // unique_key also present, or INSERT when // append).
4. Materialized-partition state — new table keyed by (workspace, asset_kind, asset_path, partition_value). Asset metadata read API exposes it.
5. Backfill UI — date range picker on the pipeline folder page; fans out runs with concurrency cap.
6. Later, behind a flag: opt-in wrap mode for single-SELECT DuckDB.

Delivers dbt's pragmatic value (incremental, backfill, idempotent reruns) without buying the compile-layer maintenance, and keeps Windmill recognizably Windmill-shaped.