windmill/docs/dbt-runtime.md

Windmill as a dbt runtime

Implementation spec for running an existing dbt project on Windmill with no changes to the project itself. Companion to pipelines-vs-dbt.md, which covers the opposite direction (native pipeline features that replace dbt). The two are complementary: this is the adoption ramp, that is the long game.

Benchmark to beat is Airflow + astronomer-cosmos, the dominant way dbt is orchestrated today.

Scope

• In: run an unmodified dbt project synced into Windmill, one Windmill job per invocation, live per-model observability, dbt models as first-class assets in the existing asset graph.
• Out: one Windmill job per dbt model, slim CI orchestration, dbt docs hosting, semantic layer, dbt platform integration.
• CE: the runtime, the manifest ingest, the asset graph and every piece of UI ship in CE, as do all adapters except two. Only the mssql and oracle adapters are EE, mirroring the native ScriptLang boundary (decision 21).

Decision log

#	Decision	Resolution
1	dbt engine	Three-way toggle (dbt-core-1x | dbt-core-2x | fusion); shipped default dbt-core-1x, instance-configurable. See below
2	Artifact shape	ScriptLang::Dbt
3	Graph in v0	Yes, both runtime and graph
4	Execution granularity	One job per invocation
5	Project storage	The project is the script's module bundle; nothing is cloned. See "Where the dbt project lives"
6	Multiple run configs	Per-run select on one script; N scripts means N projects
7	Run-time select	Descriptor default plus run-arg override
8	Credentials	Workspace warehouses, plus profiles.yml passthrough. A descriptor never names a resource. See below
9	Adapter mappings	postgres, redshift, mysql, snowflake, bigquery, databricks translate from their Windmill resource; every adapter dbt has is reachable from a dbt_profile resource, or the project's own profiles.yml
10	Private repo auth	Not applicable: the project is synced, not fetched
11	Asset kind	dbt://<warehouse>/<schema>/<name> — keyed on the relation, not on dbt's node id. See below
12	Graph refresh	Deploy-time, re-ingested per run only when the descriptor is dynamic, plus an explicit parse of the editor's buffer. See below
13	Manifest storage	Sidecar table for nodes/edges; the whole manifest is kept once per environment, for deferral — see below
14	Metadata depth	Tests, strategy, tags, freshness, column descriptions. Column lineage and real column schemas come from the engine's parquet index, opt-in per project — see below
15	Node rendering	Asset nodes per model plus one runnable node for the script
16	Progress	Live, from the JSON event stream
17	Test failures	Honor dbt's own severity
18	Retry	Automatic node-level retry in-job, plus dbt retry as a run argument. See below
19	Caching	Worker-local global cache, keyed by the project digest and the resolution the deploy pinned
20	Images	Full images only
21	Licensing	CE except the mssql / oracle adapters. See below
22	Naming	Match Cosmos field names; importer deferred
23	Descriptor	wm_dbt.yaml inside the project, OPTIONAL. See below
24	Warehouse	Configured on the workspace by name, main by default. See below
25	Cascade direction	Into a relation, not out of a run: // materialize manual dbt://… declares a write from any language but dbt's own and wakes # on dbt://… subscribers; a finished dbt run still does not dispatch. See "No cascade from dbt"
26	Deferral	A durable state per environment, published by the runs whose relations are the script's; defer is a per-run toggle. See below

Decision 1: engine toggle, and why the shipped default is not Fusion yet

engine: dbt-core-1x | dbt-core-2x | fusion in the descriptor. Omitted, it is dbt-core-1x, which runs today's projects untouched.

No engine is baked into any image. Each is fetched or built on first use and cached, for a different reason in each case.

Engine	Distribution	Cold start	License
dbt-core-1x (default)	A uv venv resolved per adapter on first use, then cached. Cannot be baked: the adapter is a Python package chosen per project	One venv build per (core range, adapter)	Apache 2.0
dbt-core-2x	One adapter-agnostic Rust binary, fetched from GitHub releases on first use, cached	One download	Apache 2.0
fusion	Never bundled. Fetched from dbt Labs on first use, cached	One download (~290MB)	dbt Fusion engine license agreement

2.x is the one that could be baked, and deliberately is not: it is a pre-release (2.0.0-alpha.5) that nothing is defaulted onto, so baking it costs a layer in every image and a version pinned in two places with nothing keeping them in step. An operator who wants an engine pre-staged — an air-gapped instance, or a fleet that should not fetch per worker — populates DBT_BUNDLED_DIR (default /usr/local/dbt) with core2x-<version>/dbt-sa-cli in a derived image; the worker prefers it over its own cache.

Two things to know before choosing 2.x: it is a pre-release, and it does not emit the per-node events the run page animates (see "Live per-model progress"), so a run on it reports its models only at the end.

The 1.x venv resolves dbt-core>=1.8,<2.0.0 together with the adapter rather than pinning a core version, because several adapters cap below the newest core (dbt-oracle and dbt-databricks below 1.12) and an independent pin makes those projects unprovisionable. The lockfile records whichever version the resolver actually chose. Both bounds and each engine version are env-overridable (DBT_CORE_1X_FLOOR, DBT_CORE_1X_CEILING, DBT_CORE_2X_VERSION).

Fusion is the fastest option and the toggle exists so users can choose it. Two things block making it the shipped default, both verifiable rather than matters of taste:

1. Redistribution terms. The Fusion license grants only a "limited, non-exclusive, non-transferable, non-sublicensable" redistribution right, and 4.1 forbids introducing "obstacles or delays that have the effect of hampering or interfering with (a) communication between Provider and End User, (b) User's ability to view, access, or use the Product and/or any Account Features." A sandboxed non-interactive job runner sits squarely in that clause's path, and "may not share, pool, or relay its own login credentials to any End User" reads directly onto putting one dbt platform token in a workspace secret. That needs counsel, not an engineering judgment. Fetch-at-runtime is the mitigation: the user's own instance pulls the binary from dbt Labs directly, so Windmill never redistributes and never interposes. Do not bake Fusion into any image.
2. Fusion is v2 semantics, and v2 drops all deprecated functionality. Every deprecation warning, including historic ones and those added in 1.10, must be resolved before a project runs on it. An arbitrary existing dbt 1.x project therefore may not run unchanged, which is this feature's entire premise. dbt ships an autofix tool and Fusion/Core interoperate side by side, so it is a migration users can do, but not one Windmill should silently require of them.

Consequence: ship with dbt-core-1x, which runs today's projects untouched, and flip the instance default to fusion once counsel clears the runtime-fetch model and a real project is verified end to end on it. Both dbt-core engines are exercised by the e2e suite, so the flip is a config change, not a port.

Decision 21: mirror the native warehouse boundary, do not invent one

Everything structural is CE: the executor, all three engines, the manifest ingest, the dbt:// asset graph, live progress, the editor. The only gate is on two adapters, and it is not a dbt-specific policy — it is the same boundary the native script languages already draw. Since bigquery and snowflake became CE, the only warehouse ScriptLangs still behind a license are mssql and oracledb, so those two dbt adapters are EE and every other one (postgres, mysql, duckdb, snowflake, bigquery, databricks, redshift, clickhouse, salesforce) is CE. Gating any of the others would make reaching a warehouse through dbt stricter than reaching it natively, which is backwards.

Those two are recognized (for the gate and for the pip package the 1.x engine's venv needs), but no Windmill connection resource translates into them: an oracledb resource is {user, password, database} with no host/protocol/service, and dbt-sqlserver needs an ODBC driver the images do not install. They reach their warehouse through a dbt_profile resource or the project's own profiles.yml, which is also how duckdb, clickhouse and salesforce work.

Recognition is what the gate keys on, and it survives the open adapter set: a dbt_profile stating sqlserver, mssql or oracle resolves to the same KnownAdapter a resource type would, so it is gated identically. An adapter Windmill has never heard of is never enterprise — the boundary mirrors the two native warehouse languages, and an adapter with no Windmill runtime behind it is not one of them.

The gate almost never fires in practice: dbt-core-2x supports neither adapter, so it can only apply to dbt-core-1x with one of those two.

The mechanism differs from the native languages. They gate at compile time, so a CE binary simply lacks the executor. That is not available here: there is one dbt executor and the adapter is only known once the profile resolves. So it is a runtime check on the resolved adapter, at both deploy and run, and it must say what is wrong — a silent degradation that surfaces later as a connection error is worse than no gate at all.

One trap: ee_oss::LICENSE_KEY_VALID is initialized to true in the OSS variant, so reading it alone passes on a CE build. The check is cfg!(feature = "enterprise") && LICENSE_KEY_VALID, which rejects both a CE build and an enterprise build whose key did not verify.

Decision 11: dbt://, keyed on the relation and not on the dbt node

dbt://<warehouse>/<schema>/<name>, one AssetKind, where <warehouse> is the workspace warehouse's NAME, so two scripts running against the same warehouse agree on identity.

The SCHEME names the namespace dbt made, not an exclusive producer. dbt is what put warehouse relations in the asset graph and is what derives them from a project; no other language infers one, and calling the kind something generic promised a parity with native Snowflake and BigQuery scripts that does not exist. A script can nonetheless DECLARE that it writes one — // materialize manual dbt://<warehouse>/<schema>/<name>, in any language but dbt's own, whose writes come from its manifest — and that declaration lands on the same node the dbt model reading the relation does, because identity is the relation rather than the tool. See "No cascade from dbt" below.

The PATH is the physical relation, and that is the load-bearing half. dbt-core has no cross-project ref(): two projects meet when one materializes a mart and the next declares it a source. Their dbt identities differ there — model.a_pkg.orders against source.b_pkg.analytics.orders — while the relation does not, so keying on unique_id would make every project an island and turn the handoff into two unconnected nodes. unique_id also embeds the package name from dbt_project.yml, which two unrelated projects may both call analytics, collapsing two different tables onto one node. The relation cannot collide that way. It is also what a DuckDB, Python, TS or Ansible script can name in a // on dbt://… annotation to join the lineage — those four are the languages with a body-asset parser; the native SQL ones cannot declare assets at all.

A dbt run does not trigger those readers. See "no cascade from dbt" below.

An ephemeral model (an inlined CTE, never written), an exposure, or a source that is not separately modelled has no physical relation and therefore no place in this namespace. If those ever prove worth rendering they need a key of their own — unique_id suits them, precisely because nothing else can refer to them.

Two traps, both of which quietly defeat the point if handled wrong.

Identifier canonicalization. manifest.json gives relation_name pre-quoted ("windmill"."Analytics"."Orders"), an annotation is written by hand, and the warehouses disagree on case: Snowflake folds unquoted identifiers up, Postgres folds them down, DuckDB compares case-insensitively. Two spellings of one table produce two nodes, no edge, and nothing looks broken in isolation. So one rule is applied in exactly one place — parse_asset_syntax, the single point where an asset URI becomes a graph key: strip the quote characters (", backtick, [/]) from the schema and name, then ASCII-lowercase them, matching the case-insensitive identifier comparison the DuckDB paths already use. The warehouse-name prefix is spelled as the workspace configures it and stays case-sensitive.

Warehouse identity is the workspace warehouse's name, exactly as ducklake://main.orders keys on the workspace lake's name — never the host, account or database. A descriptor cannot name a resource at all (Decision 24), so there is exactly one spelling per warehouse and the ambiguity a per-project resource would create does not arise. The warehouse names the default database too, so it stays out of the key; a model that overrides its database (Snowflake database, BigQuery project) is genuinely elsewhere and qualifies its schema segment as <database>.<schema>, so two same-named relations in different databases cannot collapse onto one node. A project that brings its own profiles.yml reports its target's database from that file, read with the same keys the renderer writes, so it spells a relation exactly as a workspace-warehouse project does and the two meet on one node. Only where the target leaves its database implicit does every relation qualify, because assuming they share one database is exactly what would collapse them.

Three call sites derive this key: the manifest ingest that creates the node, and the live-progress and end-of-run paths that record status against it. They share one function, because a site that derives it differently records progress against a path no node has — the run still succeeds and the graph simply never moves. The same warehouse is reachable under several hostnames, and credential material has no business in an asset key. Accepted limitation, worth knowing before it is filed as a bug: two workspace warehouses pointing at the same physical warehouse do not unify, so assets under one will not share edges with assets under the other. Point both projects at one warehouse to link them.

Decision 24: the warehouse is a workspace setting, named, and the only one

A descriptor names a warehouse by NAME (profile.warehouse, main when it names none) and cannot name a resource. Admins configure the warehouses under Settings → dbt, where each entry points at a resource, exactly as large_file_storage points at the object-storage resource and a DuckLake names its catalog.

What a warehouse may point at. Either a Windmill connection resource whose type render_profile translates (postgresql, redshift, mysql, snowflake, snowflake_oauth, bigquery, gcp_service_account, databricks), or a dbt_profile resource, whose VALUE IS one entry of that file's outputs map — type included, nothing lifted out or renamed. A block is copied from a working profiles.yml and pasted in, which is the whole point: a type that asked the user to restructure their block first would be doing the translation this exists to avoid. Its schema declares no properties, so the resource form renders one JSON editor over the value (ResourceForm.svelte). The picker is constrained to exactly these (WAREHOUSE_RESOURCE_TYPES); anything else has no way to become a target at all, which is why an unconstrained picker was a trap: it offered slack and github resources for a field that can only be a warehouse.

The two exist for different reasons. A Windmill resource is the ergonomic path and is shared with everything else that connects to that warehouse, but it is not a dbt target: each adapter arm translates the fields Windmill's resource happens to carry into the keys dbt reads, so only what an arm covers can be expressed, and an adapter with no arm cannot be reached from one at all. dbt_profile inverts that — nothing is translated, so any adapter and any key it documents works.

Which of the two a value is cannot be read off the value: both are objects with a type, and Windmill's bigquery resource is a service-account JSON that says type: service_account. So the warehouse carries its resource's TYPE (DbtWarehouseConnection.resource_type), and that is also what finally makes decision 9's "the resource type name is the authority" true at runtime rather than aspirational — the translated path resolved its adapter by sniffing connection fields until it had the name.

dbt_profile is open, deliberately. Its type is not checked against a list: DbtAdapter carries an optional KnownAdapter beside the name, so the eleven adapters Windmill has facts about (a field mapping, a pip package, the license gate) keep them, and every other adapter dbt has — trino, athena, spark, whatever ships next — is carried by name and rendered, licensed and identified without Windmill knowing anything about it. A closed list would have made "whatever dbt supports" mean "whatever this enum lists", and each new adapter a Windmill release. The name is constrained to [a-z0-9_-] starting alphanumeric because it is open: it reaches a pip requirement and a venv path on the host, where a leading - is a flag and a / is a path segment.

Installing one is a separate question from using one. dbt-core-1x fetches dbt-<name> from PyPI, dbt- is not a reserved prefix there, and that install runs through run_tool — outside the nsjail ordinary Python dependency installation uses, with uv executing a source distribution's PEP 517 backend. An unbounded name would therefore let a script author publish dbt-<x> and run code as the worker, on the one dependency path that is not sandboxed. So ensure_adapter_installable gates that install on PUBLISHED_ADAPTERS plus whatever an operator lists in DBT_EXTRA_ADAPTERS: the author chooses which adapter to use, the admin decides which packages this instance trusts. Nothing else is gated — a profile still renders for any adapter, and dbt-core-2x and fusion carry their adapters in the binary, install nothing, and take any type at all.

Two keys are not passed through: type (Windmill writes the adapter's own dbt spelling) and root_certificate_pem, which is a PEM body rather than the path dbt hands the driver — it is written beside profiles.yml and pointed at by sslrootcert, as it is for a translated postgres resource. profile.schema and threads from the descriptor override their block keys rather than joining them.

Three things follow, and they are the reason for the rule rather than consequences to work around.

A dbt project carries no connection. The same project runs locally against a developer's own ~/.dbt/profiles.yml and on Windmill against the workspace warehouse, with no Windmill-specific file in between and nothing to strip before committing it to a repository. This is what makes Decision 23 possible at all: if a project had to name its own resource, the descriptor could never be optional.

Asset identity has exactly one spelling. Keying on a name is only sound because a name is all there is. Had both profile.resource and profile.warehouse existed, one physical warehouse would be reachable under two spellings and two projects on it would silently fail to share nodes — the exact failure Decision 11 exists to prevent.

dbt is unpermissioned, and the blast radius is bounded by construction instead. The warehouse resource is read with NO permission check on the runner, exactly as s3:// reaches the workspace bucket without the caller being granted the storage resource: configuring a warehouse is what makes it available, and anyone who may run a dbt script may build with it and read its models. What is reachable stays bounded because only an admin writes the setting and a descriptor cannot name a resource, only one of the names an admin configured.

Per-relation rules were considered and rejected. s3:// can enforce a path glob because Windmill mediates every object operation through its proxy; dbt has no such chokepoint — Windmill renders profiles.yml and dbt opens its own connection. A rule could only be a pre-run check against the manifest, and a pre-hook, a macro or dbt run-operation issues arbitrary SQL on the same connection, so it would stop the ordinary case while implying a guarantee it cannot keep.

A project that brings its own profiles.yml still connects with it, and then names a warehouse only to say where its assets belong. The name must still match a configured warehouse — a typo is not identity, it strands the project's models on a node nothing else reaches — but it grants nothing, since nothing here is granted. It gets no identity by default, because defaulting to main would key a self-hosted profile's tables onto a workspace warehouse it never connected to.

That label is worth having only because such a project spells its relations the same way: Windmill reads the target's database out of the project's own file (Decision 11), so a mart it builds and a workspace-warehouse project's source on the same relation land on ONE node. Without that the label would name a namespace and still share nothing, which is the failure it exists to prevent.

An agent worker cannot read the database, so it resolves the name through a job-scoped API route. That route returns the resolved connection, which is why it requires a job token: a running job already holds those credentials in its rendered profiles.yml, and a browsable route would hand them to anyone. The same worker posts its per-model outcomes to a second job-scoped route, since the live reporter tails a log straight into the database and cannot run there. An agent's run page therefore fills in when the run ends rather than during it. Both routes are posted with the JOB's token: an agent's own credential authenticates only against the agent surface.

Decision 23: the descriptor is optional, and lives inside the project

<script>__dbt/wm_dbt.yaml. An unmodified dbt project — one cp -r away from a developer's working copy, or a repository cloned as-is — is already a complete Windmill script: it runs the whole project against the workspace's default warehouse. The descriptor appears only when the project wants something Windmill-specific: run arguments, a named warehouse, an engine pin, a test policy.

It lives INSIDE the project rather than beside it so that an author writes nothing outside the directory dbt itself reads. A dbt developer's working copy and a Windmill bundle are then the same directory, which is the whole bargain of Decision 5.

Absent means an empty descriptor, never a missing script. dbt_project.yml is what identifies a project — the descriptor cannot, being optional — and three rules keep "absent" from reading as a change: the export omits an empty descriptor, the sync map gives BOTH sides the empty descriptor an absence means (so neither reads as an addition), and a pull deletes rather than writes one. Without all three a descriptor-less project either diffs forever or grows the very file this decision exists to avoid.

Decision 12: the graph refreshes with the deploy

The project's files are the script's, so a deploy already sees exactly what will run: it parses the bundle and stores the graph. "Refresh" is just "redeploy". No manual button, no webhook, no separate mechanism.

The one case that cannot be settled at deploy is a descriptor that is dynamic by construction: a vars value spelled with a {{ placeholder }}, or an env value spelled $var: (re-resolved every run). dbt vars can steer enabled, aliases, schemas, databases and materializations, so for those the deploy cannot know what will run and the graph is re-ingested from every run's own manifest, under that run's job id. A run that cannot refresh those rows fails rather than showing a stale graph. What the SCRIPT owns stays the deploy's — see "Which run's graph becomes what the script owns" for why the two cannot diverge.

An agent worker reaches the database only through the API, so it POSTs the graph it parsed to /api/agent_workers/dbt_graph/{workspace} instead of writing it — which is why it needs no way to READ the stored relation root: it re-ingests every run, so its own run page shows the profile it actually used. What it publishes is that per-run snapshot alone — the path-keyed ownership rows are written by the deploy and by database-connected workers. Dynamic descriptors and Windmill-resolved profiles therefore both run there. What an agent does not get is LIVE progress — that is a per-model event stream, and a round trip per node is the wrong trade — so its per-model state is settled from run_results.json when the run ends, and its retry state lives only in the worker-local generation. See agent-worker-e2e.md.

The refresh happens before the build, from a dbt parse with this run's own vars and env, so a run in flight is already showing the models it is building.

A dynamic descriptor's graph is a property of the RUN, not of the deployed version, so it is stored per job — see "The graph belongs to a script version" below. Two concurrent runs of one such script therefore keep their own, and each run page shows the models that run built.

Re-ingesting is nearly free: the run parses the project (about a second) before building it and ingests that manifest.

The parse is what makes a newly added model appear in the same run that builds it, rather than one run late: the graph is written before the build, so the run page shows the model while it is being built.

What a share-link viewer sees

A share link is not anonymous access: the token is HMAC'd with the workspace key and scoped to one job and its descendants. It is an extra grant for a logged-in user who lacks access to that job — which is normally why someone was sent a link.

Both halves of a dbt run page go through one gate: /jobs/run_progress/{id} and /jobs/dbt_graph/{id}, each behind require_job_read_access, which validates the token. The graph then has a second, independent filter — RLS on the script row — and a viewer sent a link usually has no grant there. Deciding the graph's SHAPE under that filter is wrong: it would answer for the caller's access to the project rather than for the run they were given, and the Models panel would come back blank beneath working progress rows.

So a pinned run resolves its version from the JOB ROW, not from script: the live CTE takes the path and hash the handler read after authorizing the job. Two things make that safe rather than a widening:

• It leaks nothing new. v2_job_completed.result already carries every node's unique_id and relation_name, and this viewer can read it — the model set and its relations are already visible to them.
• raw_code is gated separately, on an EXISTS against script in the authed transaction. The body of a model is the project's source code and stays behind access to the project, whatever the shape query resolved.

The path and hash coming from the job row rather than the query also means a caller cannot pin one project's version while naming another's run.

What a dbt job returns, and which half of it is a contract

The result is {engine, engine_version, command, totals, nodes, invocation_args}, and each node carries both status and outcome.

invocation_args is the arguments the run used, as SUBMITTED — a $var: stays a reference, so no resolved value is published — and it is omitted when empty. It exists because a dbt retry restores the failed run's arguments inside the worker and never writes them back to the retry job, whose own args are just {"command": {"label": "retry", "dbt_retry_job": "<id>"}}: the row preview, which is a dbt show of the same project, has nowhere else to get them. On a retry it is therefore ANOTHER invocation's arguments, which is why a hidden run saves no state at all (see the retry section).

status is dbt's own word, verbatim — success, error, partial success, no-op. It is what the log says and what dbt's docs describe, so it belongs in the result, but it is dbt's vocabulary and dbt may change it: 1.x and 2.x already differ on casing, and no-op arrived in a minor release.

outcome is the same result in Windmill's terms — passed, failed, warned, skipped, no_op, unknown — and it is the half a downstream script should branch on. A dbt release that renames a status moves status and leaves outcome where it is. Publishing only dbt's word would have made every such release either a break for users or a lie in our mapping.

The graph belongs to a script version

dbt_node / dbt_edge are keyed (workspace_id, script_path, script_hash, job_id, unique_id). Each deployed version keeps its own graph, and a job records the version it ran (v2_job.runnable_id), so a run page asks for that one: /assets/graph?dbt_script_hash=<hex> renders the project as it was — its models, its SQL, its ref() lineage — instead of whatever is deployed today.

job_id is the second half, and it exists for dynamic descriptors only. A {{ }} placeholder in vars can enable a different set of models per run, so those runs re-ingest; keyed by version alone, each re-ingest overwrote the last and reopening an older run showed the newer run's project, with any model only the older run built simply gone. A run of a dynamic descriptor therefore writes its own snapshot under its job id, and its page reads the graph through GET /w/{w_id}/jobs/dbt_graph/{id}, passing the version hash.

A static descriptor writes nothing per run: its graph is the version's, under the zero-UUID DEPLOYED_GRAPH sentinel, and every run of it reads that. The sentinel is a value rather than NULL because job_id is in the primary key and Postgres does not treat two NULLs as one key, so a re-ingest would accumulate row sets instead of replacing one. The route falls back to it whenever the job has no snapshot, which is why a run page can use it unconditionally rather than having to know whether its descriptor was dynamic.

Pinning to a run is job-scoped, so it is a job route and not a parameter on /assets/graph: it needs the whole job-read contract, which is require_job_read_access. That helper lives in windmill-api, which depends on windmill-api-assets, so the read moved to the check rather than the check to the read. The route charges assets:read on top of the jobs:read its URL implies, since the body it returns is asset data.

A snapshot is only written when it DIFFERS from the version's graph, compared by a digest of the nodes, edges and relation root. Marking a descriptor dynamic is conservative — a {{ }} in vars says the arguments reach dbt, not that they change which models exist — so the usual dynamic run (a date var) resolves to exactly the graph the deploy stored, and storing that per run would duplicate an unchanging picture. Those runs write nothing and read the version's graph through the fallback; only a run whose model set really differs pays.

Which run's graph becomes what the script owns

Re-ingesting has several causes and they do not want the same thing, so the reason is carried rather than a bool (GraphRefresh):

Cause	Graph written	Path-keyed asset ownership
Descriptor is dynamic ({{ }} in vars, $var: in env)	under the job id	untouched
The run overrode vars	under the job id	untouched
The run narrowed select/exclude	nothing, unless another cause already made it ingest — then under the job id	untouched
The profile moved since the last publish	the version's graph	republished

Ownership follows the version's graph exactly, which is what the first three rows have in common: the workspace graph takes an asset's relations from the asset rows and its models, SQL, tests and ref() lineage from that version's dbt_node/dbt_edge, so publishing relations the version's graph does not name leaves those assets with no model behind them — a placeholder that moves an alias would empty the current graph of everything dbt contributes to it. A run storing a snapshot of its own therefore publishes nothing, and an override's schemas and aliases do not stand as the script's until the next deploy, which is what a snapshot is for.

The consequence for a dynamic descriptor is that its ownership stays the deploy's, and a profile that moves under one is settled by a redeploy rather than by a run: every run of it already shows its own models and re-parses regardless, so the drift it keeps re-detecting costs it nothing it was not already paying.

The last row is the one that has to publish. The drift check compares the resolved root against relation_root_at_last_ingest, so a run that saw a move and did not republish leaves the next run seeing the same move — forever, with the asset rows still naming the old schema and every run paying a dbt parse for a snapshot nobody reads. It rewrites the VERSION's graph rather than a per-run snapshot for the same reason: once the root is republished no later run detects the move, so a snapshot would leave those runs reading the pre-move rows.

A snapshot wins where they meet: a drifted run that also overrode its arguments, or whose descriptor is dynamic, snapshots under its job id and publishes nothing, and the drift is settled by an ordinary run of a static descriptor or by a redeploy — a wasted parse per overriding run, where the alternative is one caller's subset standing as the script's own, or replacing the version's graph with a picture missing every model that run did not select.

Both halves have a retention story, and they differ because their readers do. A run's snapshot expires on a clock — 30 days — because the run page that reads it is transient. A VERSION's graph cannot: its reader is every finished run of that version, and a run page is as old as its job. So version graphs are bounded by deploy COUNT instead — the newest 50 per path keep theirs — which makes growth versions x models rather than unbounded in time. Without it a CI deploying on every commit adds a full model set per commit and nothing ever reclaims it. The bound is generous on purpose: reaching it empties that version's run pages, so it exists to stop unbounded growth rather than to be hit in normal use.

Both are pruned by every dbt run, so no background sweep has to know about the tables. The prune is deliberately not hung off the progress reporter, which exists only for engines that emit node events: retention that stops working because an instance chose Fusion is not retention. A version's own graph lives as long as the version.

The third provenance: a parse of the editor's buffer

The dbt editor draws a graph of the project as it is in the editor, refreshed on demand by a dbt_command: "parse" job over the buffer — the deploy's own deps → parse → ingest path, with no build. That graph is neither of the two above: the buffer differs from what is deployed, which is the point of refreshing it, and a project being written may have no deployed version at all.

So it is keyed to its own PREVIEW JOB with no version — script_hash IS NULL — and readable only back through that job id (GET /jobs/dbt_graph/{id}), never through the path. That is what keeps the property GraphPublisher::Unversioned exists for: a parse publishes no path-keyed asset usages and no relation root, so a principal who needs only jobs:run still cannot restate what a deployed project's graph says.

Three consequences of the version being absent:

• script_hash is nullable, so the primary keys of dbt_node, dbt_edge and dbt_graph_snapshot became two partial unique indexes each — versioned rows keyed by their version, editor rows by their job alone. The composite foreign key to script is unchanged: MATCH SIMPLE is satisfied by a NULL, so a versioned row still cascades with its version and a version-less one is outside its reach. A partial arbiter also has to be named, so the marker's ON CONFLICT repeats WHERE script_hash IS NOT NULL.
• Being outside that cascade, they need clearing by hand. A route that deletes the script rows outright reclaims the versioned graph through ON DELETE CASCADE and deliberately locks nothing ahead of the script row; a version-less row references nothing, so it would survive its own script. The delete-by-path and bulk-delete routes therefore call clear_dbt_editor_graphs — AFTER the delete, beside the retry state, since every dbt writer takes the script row first and a sidecar taken ahead of it deadlocks one of the pair. Archiving clears neither: it leaves the script row, and both graphs still answer for finished runs.
• No digest suppression. A run's snapshot that matches the version's stores nothing and reads the version's back; an editor parse always stores, because the editor pins to its own job and a suppressed write leaves it nothing to pin to — and its provenance label would then claim a parse that is not on screen.
• Bounded per (path, PRINCIPAL), not by age: the newest DBT_EDITOR_GRAPHS_KEPT parses of one script by one identity keep their graph, dropped as each refresh lands, since the ones before it are dead the moment a newer parse arrives. The principal is load-bearing rather than incidental — a preview's PATH is chosen by a caller who needs only jobs:run, so a count bounded per path alone is a way to retire the graphs of whoever is actually editing that script. permissioned_as is the execution principal the queue derived, which is why dbt_run_state keys on it too. The instance-wide age sweep every dbt run performs still catches one refreshed once and left.

A parse of a job that DOES name a deployed version — the scriptable form, from a flow or the CLI — writes an ordinary per-run snapshot of that version instead, suppressed when it agrees with the deploy. Either way it publishes no ownership: a parse answers for the arguments it was given, so it can no more stand as what the script owns than an overriding run can.

Which graph is on screen is stated rather than left to be inferred — "parsed from the editor at 14:32" against "as of last deploy" — from dbt_graph_ingested_at on the graph response. The two are drawn identically, so without the label the ambiguity the explicit refresh removes would just move into the editor.

A parse renders profiles.yml before dbt runs, so it needs a resolvable warehouse and a misconfigured project fails a refresh the way it would fail a run. That is useful early feedback, and the empty state says so.

Per DEPLOY, not per run: ten thousand runs of one version share one graph. The rows carry a composite foreign key to script (workspace_id, hash) with ON DELETE CASCADE, so a version's graph dies with the version and nothing has to sweep it.

The routes that hard-delete a script rely on exactly that for the VERSIONED rows and clear none of them. Clearing them first would lock the sidecars ahead of the script rows, the reverse of the order a publication takes — script row FOR UPDATE, then the sidecars — and Postgres would abort one of the two for deadlock. So anything the cascade cannot reach is cleared explicitly and AFTER the delete, which keeps that order: dbt_run_state (keyed by path, no script key), and the version-less editor graphs, whose NULL script_hash satisfies the composite key without referencing anything (see "The third provenance" above). dbt_run_progress (keyed by job, no key to either) is reclaimed only by its age sweep.

Two consequences worth knowing:

• Concurrent deploys no longer race for the graph. Two versions write disjoint rows, so neither can lose. claim_graph_publication survives only for what is still keyed by PATH — the asset usage rows, of which there is one set per script — and an older deploy finishing late now records its own graph before declining to touch those.
• A pinned request is scoped differently. Unpinned, the endpoint scopes by the relations in view, using asset. Pinned, asset is the wrong scope: it describes the current deploy, so a model that version had and a later one dropped would be filtered out of its own run's graph. The pinned version's nodes are the scope instead.

No cascade from dbt, and no pipeline membership

A finished dbt run does not trigger anything. Its models are recorded, drawn and tracked; they do not fan out. The opposite direction does: a script that declares // materialize manual dbt://<warehouse>/<schema>/<name> is an ordinary producer of that relation, and its completion wakes # on dbt://<relation> subscribers through the same fan-out every other asset kind uses.

A dbt script is also not a pipeline member (in_pipeline is forced false for ScriptLang::Dbt at deploy). It materializes warehouse tables, so it looks like one, but that membership carries an editor whose premise is that you author the transforms in it — and a dbt project is authored in a local dbt run / dbt test loop, with Windmill as the runner and the viewer. Enrolling it put a dbt project inside the pipeline editor and blurred which of the two a folder holds. Its models are dbt:// assets in the shared graph regardless: that is what puts a native script reading one of them on the same node, and it is independent of pipeline membership.

dbt already orders its own DAG, so a cascade out of a run would only ever add one thing: waking a Windmill script that reads a mart. That edge is real but narrow, and dispatching it correctly is not cheap. A run's select can build any subset of the project, so the deploy-time write set is not what ran; using it wakes consumers of relations the run never touched, and narrowing it needs a per-job record of what was built, which the per-relation state table cannot supply (it keeps one row per relation, stamped with the last writer).

So dbt materializes and reports, and asset_dispatch returns early for ScriptLang::Dbt. Wiring it up later means deciding what a selective run should notify — that decision is the work, not the plumbing.

Declaring the write, and which subscriptions are refused

// materialize manual dbt://<warehouse>/<schema>/<name> is how an ingestion script says it writes a warehouse relation. manual is not a mode but the only mode: nothing generates warehouse DDL, so the script issues its own write and Windmill records the outcome — the same materialized_partition row a DuckLake target lands, so the relation carries a last writer on the run page and the graph. It is language-agnostic (the DuckLake write ENGINE is DuckDB's; this declaration is anyone's but a dbt project's, whose writes are read from its manifest), and the recording happens in the generic job path (record_declared_warehouse_write) rather than in an executor, for the same reason. Identity is unchanged — the physical relation — so the ingestion script and the dbt model reading it are one node, and a source declared on the relation puts the whole thing on one lineage. // data_test is refused beside it: those checks are probes the DuckDB executor splices around a managed write, so on a warehouse relation — which the script writes itself, from any language — nothing would run them, and a declarer would deploy green with its assertions silently skipped. Assert on the relation with a dbt test in the project that reads it. The <warehouse> segment is resolved at deploy for the same reason a descriptor's profile.warehouse is: a name no warehouse answers to is not a namespace, it strands the write on a node nothing else reaches.

Known boundary, shared with every other runtime pipeline annotation: the record is written from the normal execution path, and recording and cascading are decided separately, so the routes off it differ.

• A dedicated worker never enters that path — it bypasses the record exactly as it bypasses // partitioned resolution — while its job is still a top-level Script, so the fan-out (which reads the deploy-time asset rows) runs. It cascades and records nothing, leaving the relation with no last writer.
• A flow runner bypasses the path too, and is routed by flow_step_id, which is_eligible_kind rejects. Neither record nor cascade.
• A flow step running a deployed script enters the path as a Script job, so it records — and carries a flow_step_id, so it never cascades.
• A flow step with an inline body is a FlowScript job, which the recording guard excludes along with previews: neither.

Fixing the recording half is one change for every runtime pipeline annotation, not this one.

A # on dbt://<relation> subscription is held to the same relation a producer is — a whole <warehouse>/<schema>/<name> under a configured warehouse, checked by the validator the // materialize target goes through, since two spellings of that rule would refuse and accept the same string. Beyond that it is refused in exactly one shape: when every script that writes that relation is a dbt one. Nothing produces it yet is NOT that shape — a subscriber may be deployed before its producer, as for every other asset kind, and refusing there would break deploy-order-independent syncs. A dbt script may neither subscribe nor declare a // materialize: its graph ingest republishes that path's trigger and asset rows wholesale, so either annotation would deploy something the dependency job then silently removes — while the declared write would still stamp the relation on every run.

The producer set is read as it stands committed, minus the deploying script's own rows — those describe the version being replaced, so a script dropping its // materialize while adding a subscription would otherwise count itself as the producer that wakes it, which it could not be anyway (the dispatcher skips self-loops).

What that leaves is a subscription accepted while it was live and later orphaned. A dbt project that claims the relation afterwards names those edges in its own log rather than leaving them silently dormant — the same "an edge that can never fire is worse than saying so" the refusal is for, at the other point where it is knowable. Both points that publish ownership warn: the deploy, and a run whose static descriptor found its profile moved. An agent run publishes none — it is forced to per-run models, so it stores a job-pinned snapshot and leaves workspace ownership with the deployed graph — so it cannot orphan a subscription either.

Two orphanings are reported nowhere, and both are accepted rather than overlooked. A native producer that drops its // materialize and leaves dbt alone on the relation: the deploy that causes it does not touch the subscriber. And the interleaving where a dbt ingest commits between a subscriber's producer check and its own commit — the check sees no producer and accepts, the ingest's warning query sees no trigger and says nothing. Closing the second means a per-relation lock shared by the deploy path and the ingest, and the ingest takes script … FOR UPDATE before its own advisory lock, so a deploy holding relation locks first inverts that order into a deadlock across two subsystems — a worse failure than the cosmetic edge it would prevent. Both are bounded the same way: the next deploy of that project warns, and the canvas is where they show meanwhile.

A plain READ still renders the consumer beside the model, which is what makes the lineage one graph — but it is written in the script's own code, not in a comment: the body parsers resolve an asset URI from a string literal (parse_asset_syntax), so "dbt://<resource>/<schema>/<name>" appearing in a Python, TS/Bun/Deno, DuckDB or Ansible script is the read. Those four are the languages with a body-asset parser; the native warehouse ones (snowflake, bigquery, postgresql, mysql, mssql) declare no assets at all today, so a mart they consume joins the graph only once that inference exists — while a relation one of them WRITES joins it now, through the annotation.

Live per-model progress, and why only dbt-core 1.x has it

DbtEngine::emits_node_events() is true for dbt-core-1x alone, so only 1.x moves nodes on the run-page graph while it builds. The other two engines settle every relation at the end instead.

That is a statement about where the engines put their events, not about whether they produce them. Both Rust engines emit exactly the structured node events the tailer parses:

$ dbt-sa-cli build --log-format json   # and likewise the fusion binary
{"info":{"name":"NodeStart"},"data":{"node_info":{
  "node_status":"started","unique_id":"model.probe.m3",
  "node_relation":{"relation_name":"windmill_dbt_runtime.probe_sch.m3", ...}}}}

Measured on 2.0.0-alpha.5 and fusion 2.0.0-preview.202, a three-model project: 15 node events each on the console, 0 in the file log. --log-format-file json is accepted by both — json is a listed value — and ignored: the file is text either way.

The events are therefore only on stdout, which is the human-readable job log. Taking them would mean setting --log-format json and rendering the log ourselves from each event's info.msg, so the run's log stays readable. That buys live progress on two pre-release engines at the price of permanently owning log presentation, to work around something upstream has already declared it intends to support. Not worth it: when either engine honours --log-format-file json, flipping emits_node_events() is the whole change, and the existing tailer starts working untouched.

A finished run is unaffected on every engine — it is coloured from the run's own result, not from these events (decision 11's note on run_progress).

Where the dbt project lives

In Windmill. One dbt project is one Windmill script: the script's content is the descriptor, and the project's files ride with it as its module bundle, a path-keyed map the worker materialises into the job directory before invoking dbt. There is one way to do this. Nothing is cloned, so there is no repository resource, no ref, no commit and no clone cache.

A team whose repository must stay canonical keeps it: git-sync points at that repository and pushes it into the workspace, so the repository still holds the truth and Windmill receives the project. A team with no repository at all pushes straight from a working copy.

On disk, the project is a canonical dbt project

wmill sync pull writes the bundle verbatim, so the tree under the module folder is exactly what dbt expects, with the extensions dbt expects:

f/analytics/
└── analytics__dbt/                 the module bundle: the project, unmodified
    ├── wm_dbt.yaml                 the descriptor (the script's content) — OPTIONAL
    ├── dbt_project.yml
    ├── packages.yml
    ├── models/staging/stg_orders.sql
    ├── models/marts/_marts__models.yml
    ├── macros/cents_to_dollars.sql
    ├── seeds/country_codes.csv
    └── snapshots/orders_snapshot.sql

Import is therefore a copy, never a transformation:

cp -r my-dbt-project/. f/analytics/analytics__dbt/
wmill sync push

The descriptor is optional, and nothing above it is authored: an unmodified dbt project is already a complete Windmill script, running the whole project against the workspace's default warehouse. wm_dbt.yaml appears only when the project needs something Windmill-specific — run arguments, a named warehouse, an engine pin — and it lives inside the project so that a dbt developer's working copy and a Windmill bundle are the same directory.

Locally, dbt runs against the bundle with --project-dir analytics__dbt (or a cd), which is what a monorepo holding several dbt projects already does, and what dbt Cloud exposes as its "project subdirectory" setting.

Why a module bundle rather than one script per model. Models as scripts was considered and rejected on three counts. A Windmill path admits no dots, and the CLI rejects bare .sql as ambiguous (.pg.sql, .duckdb.sql, … are the convention), so a model could only be typed by its location inside the project, which breaks the rule that extension determines language. dbt resolves ref() project-wide and cannot run a model alone, so each model job would reassemble and reparse the whole project anyway. And schema.yml describes many models at once, so splitting models into objects while their tests and docs stay in shared YAML puts a model's contract in a different object. The bundle keeps the project whole, and per-model execution is offered as an action on the graph node (--select <model>+) rather than as a separate object.

What that costs, stated plainly: a model has no permissions or version history of its own. The unit of both is the project.

Consequences

The version is the script version. Deploying the script deploys the project atomically; rollback is redeploying a previous version. The lockfile keeps the resolved engine and adapter versions and the manifest digest.

Windmill holds the files, so the graph can show them. A model's compiled SQL is readable from its node in the asset graph. A dbt project has its own editor — the file tree, the descriptor, the run arguments and the model graph, which is the artifact's actual shape — but it is not where a dbt project is developed: that is a CLI loop against a local warehouse (dbt run --select, dbt test). Windmill is the runner, the viewer and the place a project is corrected.

Two scripts against one project means two copies. Splitting a project across scripts, so an upstream selection and a downstream one compose, assumed a shared repository. With bundles they would duplicate the project and drift. Prefer one script per project with per-run select, and treat two scripts as two projects (decision 6).

Seeds are the only thing that can bloat a version. Measured on real dbt code, .sql files run about 500 bytes median and 1.9 KB at p90, so even a 5000-model project is a few MB before compression. A single committed CSV can exceed all of it, so the CLI drops any file over 5 MB from the bundle and says which, rather than counting models.

Only text is carried. A dbt project's authored files are text; a binary one (an image under docs/, a stray .DS_Store, a parquet seed) is skipped with the reason. Left in, it would be read as mojibake and, if it carried a NUL, rejected by Postgres with an opaque unsupported Unicode escape sequence. Binary is detected the way git does it, by a NUL in the first 8000 bytes, because docs/ and dotfiles do not follow extensions. The push, the staleness hash and the sync diff share one predicate: a file one drops and another keeps is a change no push can resolve.

Secrets are not carried. .env, .env.* and .envrc are skipped with the reason. The import above copies whatever the checkout holds, and what a .gitignore was keeping out of the repo is exactly the file that must not become a script version, readable by anyone who can read the script and handed back on every pull. dbt does not read them either — env_var() takes the process environment, which Windmill fills from the descriptor's env and the script's environment variables.

dbt_project.yml is rendered before it is read. dbt allows env_var() in that file, so a project may name its profile or its packages directory through one. Windmill renders those two settings against the environment the run gives dbt before acting on them: reading the template instead leaves a rendered profiles.yml keyed under a name dbt never looks up, and a package cache watching a directory dbt deps never fills.

The script artifact

New ScriptLang::Dbt. Content is a YAML descriptor whose field names track dbt's and Cosmos's vocabulary so the mental model ports without translation:

engine: dbt-core-1x               # or dbt-core-2x | fusion
profile:
  warehouse: main                 # a warehouse configured on the workspace, by
                                  # name; omitted takes `main`
  target: prod
  # schema: marts                 # target schema; REQUIRED for BigQuery, whose
                                  # resource is a service-account JSON with no
                                  # dataset in it
  # profiles_yml: profiles.yml    # alternative: keep your own file; it then
                                  # names a warehouse only to say where its
                                  # assets belong (see below)
select: ["tag:nightly+"]
exclude: []
test_behavior: build              # build | after_all | none
column_lineage: false             # opt in to the static-analysis pass that
                                  # produces column-level lineage (decision 14)
vars:                             # typed: numbers/bools/lists keep their type,
  run_date: "{{ run_date }}"      # and string leaves take job arguments
  strict: false
threads: 8
full_refresh: false
env:                              # for the project's own `{{ env_var() }}`
  DBT_PASSWORD: $var:u/rf/wh_password

env values spelled $var:<path> are resolved to that Windmill variable, so a project keeping its own profiles.yml never needs a credential written into the descriptor — which is versioned script content. Both this map and the script's own environment variables apply to the deploy-time parse as well as the run, so an env_var() feeding a schema, alias or enabled produces the same relation in the stored graph and in the build either way. Prefer the descriptor's env when the value belongs to the project rather than to one deployment of it: it is versioned with the descriptor, so a redeploy from git carries it.

select/exclude/selector are passed verbatim to dbt. Do not reimplement the selector grammar; Cosmos's manifest path had to, and it is a recurring source of divergence. One thing is decided before dbt sees them: a run that spells out select or exclude drops the descriptor's selector, because dbt resolves --selector instead of --select and passing both would silently build the descriptor's nodes rather than the ones the run asked for. "Spells out" means DIFFERS from the descriptor's own value, not merely "was submitted": the generated run form posts a default back for every field left untouched, and a selector descriptor's select default is [], so reading a submitted [] as an override dropped --selector from every run started from the UI, a schedule or a webhook and built the whole project. A run that wants the whole project despite the selector asks for it with a selection that differs — ["*"]. select and vars are overridable per run via job args. The graph stays the deployed descriptor's: asset rows are written at deploy, like every other language's, so a run-arg override changes what gets built without changing what the graph says the script owns. Split the project into several scripts (decision 6) when the graph itself should differ.

A vars override does re-ingest — vars steer enabled, aliases, schemas and materializations, so the deployed graph would name another run's relations — but under the job id alone, never as what the script owns: publishing an override's relations would leave them recorded for the next default run, which then builds the descriptor's while the graph shows the override's. See "Which run's graph becomes what the script owns" for the whole table, including the profile move that is the one cause a run publishes.

vars interpolates from job args with interpolate_template (common.rs, shared with the Ansible executor). The syntax is {{ arg_name }}.

select/exclude also scope what the script owns in the graph, resolved by asking dbt (dbt ls --output json) rather than by interpreting the selector string. Without that a narrowly-selected script registers as the producer of every model in the project, and two scripts splitting one project would each claim all of it. Running several scripts with different selections only composes because of this.

Deploy path

New ScriptLang::Dbt arm in worker_lockfiles.rs (near the ScriptLang::Ansible arm at :2758), producing:

struct DbtDependencyLocks {
    manifest_digest: String,
    engine: String,
    engine_version: String,
    adapter_version: Option<String>,
    package_lock_digest: Option<String>,
    profile_relation_root: Option<String>,
}

Steps: write the script's modules into the job directory, dbt deps, dbt parse for the manifest, then ingest.

Ingestion writes the rows the native parser writes, via replace_static_asset_usage (windmill-common/src/assets.rs:254) into asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns). The language dispatch point is parse_assets_for_lang (windmill-api-scripts/src/asset_inference.rs:33).

The one architectural wrinkle. Every other language's asset parsing there is a pure function of script content. dbt's needs the bundle on disk and a dbt invocation, so it cannot run inline: it runs as a deploy-time job, persists the manifest, and parse_assets_for_lang reads the persisted result. Prototype this first, it is the assumption most likely to reshape the phasing.

Dependencies resolve at deploy, and are pinned for every run

A project declaring packages.yml ranges or a mutable git revision asks dbt to resolve them, and dbt re-resolves on every dbt deps. Windmill resolves once, at deploy, and pins the result — the same contract every other language's lockfile gets here.

The deploy records the digest of the package-lock.yml dbt produced into DbtDependencyLocks. That digest keys the worker-local package cache and joins the run identity that gates dbt retry. A run restores the tree under that key; a worker that resolves anything else is refused rather than run, because accepting it would let one resolution's run_results.json decide what a retry rebuilds.

Only a run has a resolution to be held to. The deploy establishes one and accepts whatever dbt returns — including a package-lock.yml dbt rewrites itself, which it does whenever the sha1_hash it stored for packages.yml no longer matches. Holding the deploy to a committed lock would refuse the first deploy after a package is added, with no way out, since redeploying resolves the same way again.

Consequences worth knowing before choosing whether to commit a lockfile:

• To pick up a newer version of a ranged dependency, deploy a CHANGE. Only a deploy re-resolves, and an unchanged push is skipped as a no-op — the lock and the schema are both derived, so they are compared as they would be stored rather than as they arrive. Editing packages.yml, or committing the package-lock.yml you want, is what moves a pinned resolution.
• A committed package-lock.yml lets a deploy hit the cache, since it is the digest the lookup is keyed on before dbt deps has run. A project without one, or whose committed lock is not what dbt resolves, pays a real dbt deps per deploy. This is dbt's own recommendation for the same reason.
• The refusal is per worker, not per script. dbt deps writes its lock into the job directory, so a project with a range and no committed lock reproduces its resolution only from a cache hit. Once upstream publishes a new version, that script keeps running on every worker already holding its tree and fails on the first cold one — same commit, same arguments, different outcome by worker. A deploy that changes something re-pins and clears it; committing the lock avoids it entirely.

Nothing evicts these worker-local caches — package trees, engine installs and retry state alike — and cache_clear does not reach them either: it removes $WINDMILL_DIR/cache/, while all three live under $WINDMILL_DIR/cache_nomount/, as bun's cache does. An operator reclaims them by deleting that directory. Engine installs dominate the space by two orders of magnitude (~270–290 MB each, bounded by engine version) and package trees grow one tree per edit of a project that declares packages. Sweeping either on an age or a size bound is follow-up work.

Run path

One dbt build per job, the shape Cosmos arrived at with ExecutionMode.WATCHER after per-model Airflow tasks proved roughly 6x slower (about 5.5 minutes for one dbt run versus about 32 minutes for 184 per-model invocations on google/fhir-dbt-analytics). dbt's own threading provides parallelism; Windmill provides observability.

The run's arguments are one command block

A run takes a single command argument, plus one argument per {{ placeholder }} the descriptor interpolates. command is a oneOf whose variant IS the command, so it carries exactly the overrides that command takes:

{"command": {"label": "build", "select": [], "exclude": [], "vars": {}, "full_refresh": false}}
{"command": {"label": "retry", "dbt_retry_job": "019fb410-8ea9-…"}}
{"command": {"label": "show",  "model": "stg_orders", "vars": {}, "limit": 100}}
{"command": {"label": "parse", "vars": {}}}

show and parse are accepted by the worker but are not run-form variants: each is a thing to do to the project in front of you rather than a job to fill a form in for, and the graph, the assets list and the dbt editor are where they live. Both stay reachable from a flow, the CLI and the API, which is what makes the editor's refresh scriptable and testable rather than a UI-only affordance.

The union is the point: dbt_retry_job is required where it means something and absent everywhere else, show takes the ONE model it previews rather than the select/exclude pair that narrows a build, full_refresh cannot reach a command that ignores it, and the run form renders a toggle over the variants rather than a list of fields that quietly do nothing. The worker spreads the block over the run's arguments to read them — label becomes dbt_command — which is why a {{ placeholder }} may not take one of those names (RESERVED_ARG_NAMES). What a run SUBMITTED keeps the block, since that is what dbt_run_state saves and invocation_args publishes.

1. Materialise the script's modules into the job directory, restore dbt_packages/ from cache.

2. Render profiles.yml from the resource, or use the project's own file with Windmill secrets injected as env vars for {{ env_var() }}.

3. dbt build --log-format json plus select/exclude/vars/threads.

4. Stream events: each NodeFinished updates per-model status live and emits RecordMaterializationRequest (windmill-common/src/materialization.rs:53), which already carries asset_kind, asset_path, partition, status, row_count, job_id, error, schema. run_results.json supplies all of it.

5. Structured job result (per-model status, timing, rows, failed tests), not just an exit code. Partial failure is dbt's normal case and must be legible without reading logs.

6. Node-level retry. retry_failed_nodes: {attempts, delay_seconds} in the descriptor rebuilds only what a failed build left failed or skipped, in the same job, before reporting failure. dbt confines a failure to its own subtree, so a transient warehouse error costs those nodes rather than the project. In-job is what keeps the state question out of it: the previous attempt's run_results.json is still in the job directory, so there is nothing to persist and no worker to land back on. This is the granularity astronomer-cosmos gets from one Airflow task per model, without the ~6x that per-model tasks measured (decision 4).

   A retry's run_results.json names only the nodes it redid, so it overlays the accumulated results rather than replacing them: the job's result must be every node the job touched, or the nodes that succeeded before the retry settle no materializations.

7. dbt retry resumes from the failure point using run_results.json, which is what makes one-job-per-invocation defensible. It is saved twice: to the worker's local cache, and to dbt_run_state in the database, so a retry works from any worker with a database connection. An agent worker reaches the database only through the API, which does not expose this, so it keeps only its own local copy; the automatic node retry is refused there for the same reason, since its wait could not observe a cancellation. Only run_results.json is stored there. dbt retry also needs manifest.json, roughly sixty times larger and growing with the project (732 KB against 12 KB on a six-node fixture), but the manifest is a pure function of the project files, vars and env — all of which the stored identity already pins — so a worker restoring from the database re-derives it with a dbt parse of about a second. It is a run argument (the retry command variant) rather than the automatic behavior of Windmill's generic retry, which has no per-language hook to change the invoked command. Each attempt gets a fresh job dir, so the previous run's target/ is cached per (workspace, script) on the worker and restored for a retry.

   Naming the run. The retry variant requires dbt_retry_job, the id of the run to resume, and the worker refuses one that is not the run it holds — naming both ids, since "that is not the one" is otherwise indistinguishable from "nothing is saved". Only the latest failure of a script is kept, so an unnamed retry would mean "whatever failed last" and would quietly resume a different run than the caller was looking at. The run page's Resume this run and Run again → dbt retry with same args both fill it in; on the run form, choosing retry prefills it with the run that caller's own retry would land on. It is not a selector: naming a run other than the saved one is refused, not resumed.

   Concurrency is the script's, not the retry's. A retry that starts while another run of the same script is in flight rebuilds nodes that run may also be rebuilding — appending an incremental model twice. That is what two concurrent builds of one project do as well: dbt takes no cross-process lock, so a project that must not run twice at once sets the script's concurrency limit, which covers its retries with it. A lock held across a dbt execution instead would outlive worker deaths and cancellations.

   Who may resume it. The state is keyed (workspace, script_path, permissioned_as) — one saved run per script per identity it executes as — so anyone entitled to run the script as that principal may resume its last failure, which is the same capability as re-running that job: running the script requires read access on it, and that access shows them the run and its arguments already. Naming the run neither widens nor narrows that; the id is checked against what the state holds, not against the caller.

   For an on_behalf_of script that means every caller shares one saved run, since they all execute as the owner — deliberately, since the state describes the script's last run under the owner's identity rather than any one caller's.

   What a retry actually adds, and where that crosses a line. Resuming grants no capability a caller lacks: they may already run the script as that principal, and a plain run builds everything the descriptor selects, of which a retry rebuilds a subset. The one thing it adds is information — the result carries invocation_args, the resumed run's arguments as SUBMITTED, because a retry job's own args are just the command and the run it names, and the row preview needs the real ones.

   For almost every shape the caller could already read that run, so nothing crosses: under f/, see_folder_extra_perms_user makes a job readable to everyone with read on the folder, which is also what grants execution; an ordinary script's runs are permissioned_as the caller, and another user's runs are keyed under their own principal and unreachable. It takes all three of a u/<owner> path, extra_perms sharing and on_behalf_of for "may run as this principal" to be broader than "may read this principal's jobs" — and there a grantee learns the literal argument values of another's run. References stay references, so no resolved secret is among them.

   That residual is accepted rather than gated. A gate needs the caller's identity, and a worker has only created_by — display_username(), which a token LABEL supplies. Resolving it as a username denies every labelled token (a CI token becomes label-<name>, which is no workspace member) and still trusts a name; authorizing where the caller is real means the submitting path, not the worker. The exposure did not justify either.

   That equivalence holds only while the run is READABLE, so the one run that breaks it saves nothing: a job pushed invisible_to_owner is hidden from the script's owners, and a retry publishes the arguments it restored, which would make that retry the one way to see them. A hidden run therefore keeps no retry state at all — it cannot be resumed by anyone, including whoever launched it, which is the cheaper half of the trade. Keying by the initiating caller would not have worked instead: created_by is display_username(), which a token LABEL supplies, so two callers can share one value and one can name a third person (GHSA-8x8x-88qc-qp4r, whose fix was to stop trusting that name for authorization). A retry does name the run it resumes, but that name is checked against the saved state rather than authorized as a job read — doing the latter, which is what would let a hidden run be resumed by its own author, needs the submitting path, where the caller is real.

8. Test failures honor dbt's severity: error fails the job, warn surfaces without failing. Overriding this would make the same project behave differently on Windmill than locally, breaking the core promise.

Durable state per environment, and what defers to it

dbt --defer --state <dir> resolves a ref() the run does not build to the relation the manifest in <dir> names, instead of to the schema this run writes into. That is what lets one model be rebuilt into a scratch schema without rebuilding everything above it, and it needs a manifest of the environment the project actually lives in.

Nothing that already existed could supply one. dbt_run_state answers a different question — it holds the LAST run whatever its outcome, keyed by the principal, so dbt retry can resume its failures — and the worker-local generations behind it are a cache: the next run of a project usually lands on a worker holding neither artifact. So the state is its own table, dbt_environment_state, one row per (workspace, script path, environment), holding manifest.json and run_results.json from the last SUCCESSFUL run. Success is half of the contract: a relation a later run defers to has to exist.

The environment is the warehouse, the target and where they resolve to

The workspace warehouse's name, the target dbt actually runs, and the database and schema that target resolves to — the pair relation_root reports to the graph's drift check. Each component is length-prefixed rather than joined on a separator — <warehouse>|<target>|<schema>|<database>, each written <len>:<value>, so main/prod/analytics/dbt_wh_defer is stored as 4:main|4:prod|9:analytics|12:dbt_wh_defer. A target name and a schema are both the user's own strings, so prod|analytics + scratch and prod + analytics|scratch would otherwise be one key, and a profile moving between them would read as the same environment rather than as one nothing has published. What a message names is spelled out instead, never the encoded key.

The target is the EFFECTIVE one, not the descriptor's profile.target: a descriptor naming none inherits the workspace warehouse's, or the default in the project's own profiles.yml, so reading the descriptor's would file every inherited target under one empty name — and a target.name macro decides where a model is built.

The last two are in the key because deferring is resolving a relation NAME. A warehouse repointed at another database, or a profile.schema moved by a redeploy, keeps the first two while putting every relation somewhere else, and a manifest is a list of relation names — there is no other way to notice. Keyed on the first two alone, such a move would hand the next deferring run the names of relations that are no longer there. Keyed on all four, it reads as an environment nothing has published yet, which is what it is.

What the key deliberately does NOT carry is the resolved connection. That is the profile_digest a retry is held to, and it moves when a password is rotated, which moves no relation; a warehouse pointing somewhere else entirely is decision 11's accepted limitation, spelled the same way here as everywhere else.

Today one script has one environment, because a descriptor fixes both the warehouse and the target and a run cannot override either. The key is what makes the later item — fork and preview environments — an addition rather than a migration, and what makes a profile move detectable now.

Which runs publish it

A successful build that did not itself defer, and whose graph becomes what the script owns (GraphRefresh::publishes_ownership) — the same condition as the graph's and the same reason: an invocation that scoped its own model set — a vars or select override, or a descriptor dynamic by construction — describes where the CALLER put those relations, not where this project's models live. Publishing it would point every later deferral at one caller's scratch schema.

A run that deferred never publishes, whatever narrowed it, and that is a separate condition rather than a consequence of the first. A deferring run built some of the relations its manifest names and resolved the rest out of the state it read, so recording that manifest would claim relations nothing built — and a model renamed since would be recorded under a name only a full build creates, breaking every later deferral until one repairs it. publishes_ownership cannot see this: it reads the caller's overrides, and a descriptor that already narrows select needs none.

A retry publishes nothing. Its run_results.json names only the nodes it redid, so the environment would come to claim a run of a handful of models. The environment's state is therefore the last full successful build, exactly as dbt Cloud's "last successful run" is, and a run recovered by a retry leaves it at the previous one.

The AUTOMATIC in-job node retry is the same artifact under a different name: a build it recovers is a successful build, but the run_results.json on disk is the retry's. Such a run publishes the manifest without results, rather than with a set describing some other slice of the build — the manifest is a function of the project rather than of what ran, so deferral is unaffected. A result: selector is the one thing left with nothing to read, and it is refused by name against such a publication rather than passed to dbt (see "Selectors that read the state" below).

Under test_behavior: after_all the stored run_results.json is the test phase's, because that is what the second invocation leaves in the target directory — the same artifact a local dbt run && dbt test leaves behind.

What that condition means for what the artifacts may carry, and why this table is keyed by environment where dbt_run_state is keyed by principal. dbt records the invocation's flags into run_results.json, and Windmill resolves $var: / $res: references before dbt sees them — which is exactly why the retry state is per-principal, so one caller's resolved select and vars are not restorable by the next. Here they cannot be one caller's: a publishing run added nothing of its own, and a descriptor that interpolates a {{ }} placeholder into vars never publishes at all, so what is recorded is the descriptor's own arguments — the script's content, which anyone entitled to run it may already read. Widen the publish condition and that stops being true.

Where the blob goes

run_results.json is small; manifest.json is not, and grows with the project (535 KB on a two-model fixture). Each takes the same two homes: inline in the row under DBT_STATE_INLINE_MAX_BYTES (8 MiB), and the INSTANCE's object storage above it, with the row keeping the key. Inline is what makes the feature work on an instance that has configured no storage at all; the ceiling is what stops one project's manifest from becoming a multi-megabyte row rewritten by every run. A project past the ceiling with no storage configured is told so, in the job log, naming the setting and the variable — the run itself still succeeds, since losing the state costs the next deferral rather than the build that just ran.

The instance store, not the workspace's, which is where every other internal worker artifact already lives (bun bundles, python wheels, job logs, the global cache). The workspace bucket is the one members read and write through job_helpers/* and wmill.write_s3_file with a caller-supplied key, and only volumes/ is reserved there — so a manifest under it is one any member could replace, and the next deferring run would hand dbt an attacker-chosen defer_relation for every unbuilt ref() while holding the script's warehouse credentials. Its compiled SQL would be readable there too, for a project the reader may have no access to. The consequence to know: a project past the ceiling needs the instance store configured, which is an EE feature, so on CE the ceiling is the limit and DBT_STATE_INLINE_MAX_BYTES is how it moves.

Each publication writes its OWN keys (wmill_dbt_state/<workspace>/<digest of path and environment>/<job>.<nonce>/<artifact>) and the row switches to them in one statement, so an upload never overwrites an artifact the committed row still names: a run that fails between its two uploads, or between them and its row, leaves the state pointing at the pair it already had. The objects the commit displaced are dropped afterwards, never before, since a reader that has already read the row is about to fetch them; a reader that loses that race re-reads for as long as the row keeps MOVING, rather than reporting a state that is there. A reader takes no lock, so successive publications can each overtake one; an unmoved row whose objects are gone is the error that means what it says, and a bound on the re-reads is the other, for a project republishing faster than a run can read. What a publication uploaded and then could not commit is dropped on the way out — except after a commit that REPORTED an error, where what was lost may be only the acknowledgement: dropping then would leave a committed row naming objects that are gone, so an orphan is the cheaper side to take.

The path and the environment are only a prefix of that key. The row is what says where an artifact is, which is why state can travel with a renamed script and go on naming objects under the old path's digest. The rest of the key is the job and a per-EXECUTION nonce — zombie recovery re-runs a job under its own id, so keyed on that alone a second attempt would overwrite the objects the first attempt's committed row still names, then read those keys back as displaced and drop them.

Publishers of one environment serialize on pg_advisory_xact_lock, so only one of them settles the row and the objects it displaces at a time — an advisory lock rather than the row's, because the first publish of an environment has no row to lock and is exactly when two runs of a newly deployed script are most likely to race.

Retention

None, deliberately, and this is where it differs from the graph tables next door. Those are pruned by age by the dbt runs themselves because their reader is a transient run page. This one holds a single row per script per environment, replaced in place, so it does not grow with runs — and its reader is every later run of that script, so a project that runs monthly must still find last month's state. It goes with the script instead: a path no live dbt version occupies any more clears it, alongside dbt_run_state (clear_dbt_script_state, clear_dbt_script_state_if_path_retired).

The write carries a guard of its own, and it names the VERSION rather than the path: the live dbt script there must be the one this job ran, or a later version of it (hash = $n OR $n = ANY(parent_hashes)). "Some live dbt script is here" — which is what the retry state settles for — is also satisfied by a script created at a path this one was renamed away from, and this job's manifest would then become that project's deferral state. A preview names no version and so publishes nothing, which is right for a run of content that was never deployed.

The job's KIND is checked beside it, because a preview carries a caller-supplied script_hash into runnable_id (run_preview_script): the version alone would let anyone who may run a job publish arbitrary content as a deployed script's state. A flow or app step naming a deployed dbt script by path is an ordinary script job carrying that script's own hash, so it publishes like any other run; only INLINE flow code is a FlowScript, and that has no deployed version to publish for.

That guard HOLDS the script row (FOR SHARE) for the rest of the publication, so a rename, archive or delete of the path either waits for it or is seen by it. Read unlocked, it leaves a window where the lifecycle clear finds no row to take, finishes, and the publication then commits state at a path a new script goes on to occupy. The script row is taken before the sidecar, which is the order every other dbt writer takes and what keeps the two off a deadlock.

An artifact too large for its row is left in the store when the row is cleared, as a deleted script leaves its bundle: reaching it from the delete would mean an object-store client in windmill-common and a delete that has to land after the caller's transaction commits, for one object per environment of a script that is gone.

Asking for it

defer is a field on the build command block, defaulting to the descriptor's own defer:. A per-run toggle rather than a descriptor-only setting, because the run that publishes an environment's state and the run that defers to it are two invocations of ONE script (decision 6: N scripts means N projects): a project that could only defer by descriptor could never populate the state it reads.

A project whose profile selects its schema or database with a TEMPLATE — either delimiter, since dbt renders {% … %} blocks as well as {{ … }} — is refused a deferral outright, and publishes no state either: dbt renders those and Windmill does not, so two renderings resolve to one relation_root, and a deferral after the value changed would resolve every unbuilt ref() through the previous location's manifest. Both sides, because a published template would sit under a key a literal profile shares, and de-templating later would make that stale manifest readable as the new location's. It covers a project-owned profiles.yml, a dbt_profile resource — one block of the user's own file, copied through unchanged — and a profile.schema written as given. Plainly absent is different: that is the adapter's default, which does not move.

A run that asks to defer with nothing published is refused, naming the environment and the runs that cannot publish one. The alternative — running without deferral — fails deep inside dbt with a relation-not-found the caller has no way to connect back to a missing state. An agent worker is refused the same way and for a reason it can act on: it reaches the database only through the API, which does not expose this table.

A show defers too, and every engine takes the flags on it. It compiles the model it previews, so a model whose upstream this environment built and this run did not is exactly the case a deferral exists for. So does the dbt ls that resolves what a run's selection owns, without which a result: selector — which reads run_results.json out of the state directory, and which select passes to dbt verbatim — would fail before the build that would have honoured it.

The result carries deferred_to, the run whose state was used. Without it what a deferring run built against is unrecoverable, since the next successful run of that environment replaces the state.

Selectors that read the state, and why they are refused rather than passed

--state also feeds dbt's own selector methods, so publishing the state is what makes state:modified+, state:new and result:error+ resolve at all. Only a deferring run is handed the directory, so a state: or result: method in select or exclude without defer is refused before dbt starts.

Refused, rather than left to dbt, because the engines disagree about it and two of the three disagree silently. Given a state selector and no --state, dbt-core 1.x raises (Got a state selector method, but no comparison manifest, exit 2), but dbt-sa-cli 2.x and fusion read a MISSING state as an EMPTY one and exit 0: state:modified then selects nothing and the run reports success having built nothing, while state:new selects everything, because against an empty state every node is new. A scheduled run that quietly stops doing work, or quietly rebuilds the project, is the failure this state exists to prevent.

From the DESCRIPTOR they are refused whether or not the run defers, and the message says so. That selection is also what decides which nodes the script owns, and the deploy resolves it before any run exists, with no state to compare against. "Whatever changed last" is not an ownership answer. They describe one run, so they belong in a run's own select.

source_status: is refused under any setting: it compares sources.json, which dbt source freshness writes and no run publishes here, so there is nothing to compare against even while deferring.

Two more refusals follow from the same argument, that a selector with nothing to read must say so rather than resolve to a silent answer:

• A result: method while deferring to a state that carries no run_results.json. Publishing that is deliberate — a build recovered by automatic node retry stores the manifest alone, its results describing the retried nodes rather than the build ("Which runs publish it") — so defer being on is not enough to know the file is there. Answerable only once the state is loaded, so it is checked right after, naming the run that published.
• Any of them on a parse. A parse resolves a selection to store the graph and never defers, so defer would not hand it a state at any setting, and the remedy the other refusal offers would lead nowhere. It says that instead.

Matching nothing is then an ordinary outcome for these methods, and for no others. state:modified+ selects the empty set exactly when nothing changed since the published state, which is the answer a CI run wants, so a selection naming a state: or result: method may resolve to no nodes. Such a run scoped its own selection, so what it stores is a snapshot of its own and never what the script owns, and nothing is un-wired by the empty set.

The exemption is by METHOD, not by who chose the selection. Exempting every caller-chosen one would take a misspelled model name, which resolves to nothing just as surely, and report it as a build that did its work. An ordinary selection matching nothing stays refused, from a run as from the descriptor — from the descriptor because that one also decides ownership.

Only what select and exclude spell directly. A method reached through a selectors.yml definition is named nowhere the worker reads, and dbt's own behaviour — including the silent one — is what stands there.

--state is also a retry's own argument, and that is a trap

dbt retry reads the run it RESUMES from --state. Handed the deferral's directory it resumes the successful run stored there, finds nothing failed, and reports a green retry having rebuilt nothing — silently, on dbt-core 1.x, which warns and exits 0.

dbt-core 1.x has --defer-state, the deferral-only half of the pair, so a retry there passes that and leaves --state alone. The Rust engines do not have it, and a run that deferred is refused a retry on them, before the build: the alternative is rebuilding the failed nodes with every ref() resolving into the schema this run writes into, which for the narrowed run a deferral exists to serve means writing them somewhere they do not belong. The automatic in-job node retry is dropped for the same reason and says so in the log.

The state directory is passed RELATIVE (wm_dbt_state, beside wm_target in the job directory). dbt records the invocation's flags into run_results.json and a later dbt retry restores them, so an absolute path would name the job directory of the run being resumed, which is gone by then. Relative, it resolves against the project root — whichever job directory the retry landed in.

Three engine facts found while wiring this up, all worth knowing before filing a bug against the feature. dbt retry on dbt-core 2.x restores neither the resumed invocation's --vars nor its deferral: it re-parses with the current (empty) ones, so a retry of a run that overrode vars rebuilds into the descriptor's schema rather than the run's. That is independent of deferral and predates it; the refusal above stops the deferring case from being the way it is discovered. dbt show on either Rust engine prints a bare JSON array where dbt-core frames it as {"node": …, "show": […]}, which run_show is written against — so a preview there fails to parse whether or not it defers, and the deferral itself resolves correctly under it. And neither Rust engine reached dbt's own service-backed State (--manage-state) on any run measured here, so no flag is passed to disable it.

Because select reaches dbt verbatim, a deferring run also has a --state directory for result: selectors, which is why run_results.json is stored beside the manifest rather than the manifest alone.

Two decisions the implementation narrowed

Decision 13 — the manifest is stored once per environment, not per version. The sidecar holds every field the graph renders, so a copy of manifest.json bought the graph nothing: it is reproducible by redeploying, or for a dynamic descriptor by the next run. Deferral is the reader that changed that — it resolves an unbuilt ref() through a manifest, and one on worker-local disk answers for a machine's history rather than for the environment. So exactly one manifest is kept per (script, environment), replaced by each successful run, rather than one per version (see "Durable state per environment" above).

Decision 14 — column lineage comes from the parquet index, not the manifest. manifest.json carries no column-to-column edges, in any engine, and its columns are the ones an author declared in schema.yml. Both halves exist in a different artifact: dbt compile --static-analysis strict --write-index writes target/index/, and two of its tables are dbt.column_lineage.parquet (from_node_unique_id, from_column_name, to_node_unique_id, to_column_name, lineage_kind) and dbt.node_columns.parquet (every column of every node, with its declared type, its inferred type and its description).

Three measured properties decide the shape of the ingest.

Strict analysis is a stricter dialect. select no_such_column from ref(...) is UnresolvedIdentifier (dbt0227) and exit 1 under strict, and compiles fine under baseline (the default). So this is a separate dbt compile with its own --target-path, never a flag on the build, and it is opt-in per project: column_lineage: true in the descriptor. Off, nothing changes. On, a project that cannot be analyzed keeps exactly the graph it had.

The pass is best-effort about everything that is ITS: a wrong engine, a rejected analysis, a missing or unreadable artifact, an over-long output and outrunning its own time budget all degrade to partial lineage or none, plus a line in the job log saying which. It is not best-effort about the JOB: a cancellation or the job's own deadline fail it, because swallowing those would let a run that blew its timeout inside an optional annotation publish a graph and report success. That split is why the two halves have separate error contracts — the compile owns the job's semantics and may Err; nothing the artifact does or fails to do is a reason to fail a job, so an absent, unreadable or partial index is a value. The decode still runs under the job poller, which both heartbeats through it and ends it if the job is cancelled or completed meanwhile: the job reaching in, not the artifact reaching out. The budget is half the job's remaining wall clock, spent on the compile alone, so the build that follows cannot be starved by it.

A failed pass still writes the index, holding every edge of the models that did analyze, so the artifact is read whatever the exit status and partial lineage is a normal outcome. An unreachable source is milder still: RemoteError (dbt1014) downgrades that model to static_analysis: off and the compile succeeds. (Strict analysis queries the warehouse catalog for source schemas; a ref()ed model is inferred statically and needs no built table.)

The flag is not the capability. dbt-core 2.0.0-alpha.5 — the version DBT_CORE_2X_VERSION pins — accepts --write-index and --write-lineage, and its own views.sql declares views over both tables, but it writes neither parquet; only Fusion does today. The ADAPTER decides too: an experimental one (postgres under DBT_ALLOW_EXPERIMENTAL_ADAPTERS) turns static analysis off and says so only in a warning on an otherwise successful compile. The gate is therefore "the engine has the flag" (everything but 1.x, whose Python CLI has no such option) plus "the file appeared", so a later release picking the feature up needs no change here — and the job log carries the engine's own stderr whenever no index appears, since without it "no column lineage" has no explanation.

lineage_kind is stored as TEXT, not an enum. Three values exist — copy (passthrough), mod (transformed) and scan (the column was read to produce the ROW rather than the value: a join key, a where predicate, a group by) — and the engine's own reader maps those three and passes anything else through, so the set is the engine's to extend. All three are stored, and copy/mod are kept first when the bound bites: a scan edge reaches every output column of its model, so it is most of what a project's index holds and would draw as a complete bipartite graph. Keeping it in the table anyway is what lets a later "show indirect" view ask for it without every project being redeployed.

Storage mirrors dbt_edge exactly: dbt_column_edge, keyed by (path, version, job) with the same composite foreign key to script, so a version's column lineage dies with the version and a run's snapshot with the sweep.

A table of its own, not dbt_edge.column_lineage JSONB. Hanging the links on the ref() edge they sit beneath would inherit its clone, prune, clear and cascade paths for free, and it does not work: a model reading {{ this }} gets column lineage from itself to itself, and parent_map has no self-loop, because a model does not ref() itself. Those pairs have no dbt_edge row to attach to. The loss is not hypothetical — an incremental that selects from {{ this }} (coalesce(p.dbl, s.dbl), p.up as prev_up) yields up → prev_up with kind copy, a drawn edge meaning "this column carries the previous run's value". Inventing self-loop dbt_edge rows to hold it is not an option either: that table is ref() lineage. The typed column list lands in dbt_node.column_schema, beside columns rather than merged into it — columns stays what the author declared.

Stored now, served later. This change lands the ingest and the storage; the endpoint that draws a column trace is a follow-up. What is user-visible today is column_schema — every column of a relation, typed and in the order the model emits them — which rides the asset graph the details pane already fetches, and replaces a panel that could only list the columns an author happened to document. The edges sit in dbt_column_edge waiting for their surface.

column_schema is gated on being able to read the producing project, like the model's SQL: a column-level view is the shape of what the author wrote, one level finer than the ref() graph, which is ungated only because it draws relations the caller already sees. A share-link viewer entitled to a dbt run therefore gets its relations and ref() edges, and neither the SQL nor the columns.

The analysis pass takes the build's own --full-refresh. is_incremental() branches on it, so an incremental model reading {{ this }} compiles its self-join — and any ref() inside that branch — only when the flag is absent. A pass that used the descriptor's default while the run overrode it would store lineage for SQL that run never executed. For the same reason an invocation that overrides the flag counts as per_run_models: its graph is its own, keyed to the job, rather than standing as the version's.

That flag is not the whole of it, and the rest is a property rather than a bug to fix. is_incremental() is also false when the target table does not exist, so an incremental model has two shapes and one ingest holds one of them: a deploy before the first build compiles the cold shape, and the same project deployed again once its tables exist compiles the incremental one. A static descriptor re-ingests on neither runs nor time, so what is stored stays whatever the compile in front of it saw. dbt has no mode that emits both, and re-analyzing per run would buy a second dbt compile on every build to keep a graph nobody asked to refresh. The contract is therefore the honest one: a version's graph describes the compile that produced it, and a run that re-ingests describes its own run.

Concept mapping

dbt	Windmill	Mechanism
model relation	dbt:// asset	new AssetKind
ref() graph	lineage edges	replace_static_asset_usage
materialized: table	materialize_strategy: replace	AssetGraphRunnableNode
materialized: incremental	append or merge (by unique_key)	same
{% snapshot %}	scd2	same, incl. <dim>_current handling
unique/not_null/accepted_values/relationships	data_tests	exact 1:1 with the four // data_test kinds
declared column metadata	columns on the asset node	descriptions only, from the manifest
analyzed column schema	column_schema on the asset node	dbt.node_columns.parquet, opt-in
column-to-column lineage	dbt_column_edge rows (no view yet)	dbt.column_lineage.parquet, opt-in
model tags	node badge	tag
source freshness	freshness	last_success_at chip
run_results.json	materialization records	record_materialization
dbt_packages/	worker-local cache	keyed by packages.yml, the project digest and the package-lock.yml the deploy resolved

Phases

Phase 1: run it. ScriptLang::Dbt across the 41 ADD_NEW_LANG sites (mostly one-liners in EditorBar.svelte, scripts.ts, script_helpers.ts, LanguageIcon.svelte, script_common.ts). Engine provisioning for all three options in Dockerfile and docker/DockerfileFull* (bundle 1x and 2x, fetch Fusion at runtime). New backend/windmill-worker/src/dbt_executor.rs: descriptor parse, bundle materialisation, profiles.yml render, dbt build, log passthrough, structured result, retry.

Phase 2: graph. DbtDependencyLocks and the deploy arm. Migration via cargo sqlx migrate add -r dbt_runtime. New dbt:// AssetKind with its canonical_prefix. Manifest ingest. Deploy-time ingest plus the per-run re-ingest for dynamic descriptors. Extend AssetGraphRunnableNode/AssetGraphAssetNode in frontend/src/lib/components/assets/AssetGraph/types.ts with dbt provenance and render through the existing RunnableNode.svelte / AssetNode.svelte / DataTestNode.svelte.

Phase 3: live progress and ergonomics. JSON event stream to per-model status on the canvas mid-run. record_materialization per model. Profile and select pickers in the editor. Per-model failure triage in the run view.

Phase 4 (not in this PR). Slim CI: the fork and preview environments a deferral would name instead of its own. The selectors themselves are here, since state: and result: read the published state like any deferral does; what is missing is a per-branch environment to compare a CI run against. Partition and backfill integration so BackfillRangeDialog.svelte works on dbt models. wmill dbt import <dag.py> reading DbtDag(...) kwargs.

E2E test requirements

Against a real dbt project (jaffle_shop shape) and the local Postgres:

1. Happy path: deploy a dbt script, run it, assert models exist in the warehouse and the job succeeds with a structured per-model result.
2. Engine parity: the same project passes on dbt-core-1x and dbt-core-2x. Fusion covered by a manually-run test, not CI (runtime fetch).
3. Test severity: a failing error-severity test fails the job; a failing warn-severity test does not.
4. Retry: a run failing midway, retried, resumes via dbt retry and does not rebuild already-successful models.
5. Graph ingest: after deploy, model assets and ref() edges exist; a native script reading one of the marts gets an edge to it.
6. Shared node: a native script that READS a mart renders as a reader of the same node the dbt model writes — one node, not two islands. Declared with a plain read (# dbt://<mart>), never # on: a subscription to a relation dbt alone builds is refused at deploy, since a dbt run does not dispatch.
7. Declared write: a native // materialize manual dbt://<relation> script and a dbt project reading that relation as a source render as one node; a run of the script records its materialization and wakes a # on dbt://<relation> subscriber — a subscription only that producer makes wakeable, the dbt project reading the relation being no producer of it (see "no cascade from dbt").
8. Selection: descriptor select/exclude, and a run-arg override, each build only the expected subset.
9. Dynamic descriptors: a {{ }} placeholder in vars re-ingests the graph from the run's own manifest, so a model that placeholder enables appears in the same run that builds it.
10. Both credential paths: resource-rendered profiles.yml, and the project's own profiles.yml with env-var injection.
11. Caching: a second run reuses the cached dbt_packages/ with no network fetch.
12. Deferral: a full run publishes the environment's state; a second run that builds one downstream model into another schema resolves its unbuilt ref() to the relation the state names, where the same run without defer fails with relation-not-found.
13. State selectors: with a state published, state:modified+ selects nothing while the project is unchanged and exactly the changed model and its children after one is edited. Without defer it is refused rather than passed, and a result: selector against a state published by a node-retry-recovered build is refused too, that one carrying no run_results.json.

Keep only tests that pin behavior a future change could break. Per AGENTS.md, delete development scaffolding before marking the PR ready.