Files
windmill/backend/.sqlx
Ruben FiszelandClaude Opus 5 c6e0302d7c feat: let // materialize declare a dbt:// warehouse-relation write (#10978)
* feat: let `// materialize` declare a `dbt://` warehouse-relation write

`// materialize manual dbt://<warehouse>/<schema>/<name>` lets an ingestion
script in any language declare that it writes a warehouse relation, so it and
the dbt model reading that relation land on one asset node instead of two
disconnected pictures. `manual` is the only mode a warehouse target has —
nothing generates warehouse DDL — and the non-`manual` spelling is refused
rather than silently degraded. The `<warehouse>` segment is resolved against
the workspace's configured warehouses, like a descriptor's `profile.warehouse`.

The run records the same `materialized_partition` row a DuckLake target does,
from the generic job path rather than an executor: the DuckLake write engine is
DuckDB's, this declaration is anyone's.

With a non-dbt producer now possible, the blanket deploy-time refusal of
`# on dbt://<relation>` narrows to the shape that still cannot fire — every
writer of the relation being a dbt script, since a dbt run does not dispatch.
"Nothing produces it yet" stays accepted, as for every other asset kind, so
deploy order does not matter. A dbt script may not subscribe at all: its graph
ingest clears its own `dbt://` trigger rows. The one ordering the deploy cannot
catch — a subscription accepted before any producer, then claimed by a dbt
project — is named in that project's deploy log.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rw1WrKeRRzyYHjfkuB83ek

* fix: address review — preview stamping, stale producer set, public doc

Three findings from the local review round:

- Record the warehouse write only for a DEPLOYED script job. The annotation is
  a deploy-time contract (`manual`, three segments, a configured warehouse)
  checked where write access to the path is also required; honouring it in a
  preview, hub or inline-flow body let `jobs:run` alone restamp any relation's
  last writer from a script that never touched it.
- Exclude the deploying script's own rows from the producer set. Read
  committed, they describe the version being replaced, so a script dropping its
  `// materialize` while adding a subscription counted itself as the producer
  that would wake it and committed a dormant edge. It could not be that
  producer anyway — the dispatcher skips self-loops.
- `AssetKind::Dbt`'s doc no longer claims dbt is the exclusive producer of a
  warehouse relation, on both the types and the parser enum.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: review round 1 — dbt-script materialize, set-form rule, doc

- Refuse `// materialize` on a dbt script, the producer half of the rule the
  trigger loop already applies to `// on`: the graph ingest republishes that
  path's asset rows wholesale, so a declared write is wiped by the deploy that
  accepted it while its runs keep stamping the relation.
- `dormant_dbt_subscriptions` now spells the same predicate its singular sibling
  does: the producer set has to be non-empty (nothing produces it yet is deploy
  order, not a dormant edge) and excludes the subscriber's own path (a script
  never wakes itself). Both divergences are pinned by tests.
- The docs no longer claim the dbt deploy log covers a native producer that drops
  its `// materialize`; it does not, and nothing else reports that case.
- An integration test over the deploy contract, since only a real deploy proves
  the handler feeds `sole_dbt_producer` the canonical key `asset.path` holds —
  the spelling that has to agree across the materialize target, the `// on` ref
  and the refusal that joins them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: qualify the any-language claim, and pin the dbt-script refusal

`AssetKind::Dbt`'s contract (both enums), the two runtime guides and the deploy
comment said a script of any language may declare a `dbt://` write, which the
dbt-script refusal added last round contradicts. They now say "any language but
dbt's own", with the reason: a project's writes are read from its manifest.

The deploy-contract integration test covers that refusal for both annotations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: teach the pipeline AI guidance the warehouse-relation target

The pipeline prompt (both sources, plus the regenerated bundle) told the model
`// materialize` is DuckDB-only and rejected on any other target, which now
steers users away from the very thing this PR adds. It distinguishes the managed
DuckLake write, still DuckDB-only, from the warehouse-relation declaration any
language but dbt's own may make.

`dbt_manifest.rs`'s module doc carried the same "the only thing that creates one"
overclaim the other four sites lost last commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: draw an explicit dbt:// subscription on the canvas

The editor suppressed every `// on dbt://…` overlay, which was right while the
deploy refused all of them. It now refuses only a relation dbt alone builds, so
the suppression hid the author's own annotation for exactly the case this PR
adds — a subscription woken by a native `// materialize manual dbt://…`
producer. The deploy stays the gate.

Also the two stale claims round 4 named: the live pipeline prompt dropped the
dbt-script exception the base prompt carries, and the doc's e2e requirements
still said every `dbt://` subscription is refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: refuse `// data_test` beside a `dbt://` materialize target

`// data_test` checks are verifier probes the DuckDB executor splices around a
managed write. A warehouse relation is written by the script itself, in any
language, so nothing would run them — and unlike the DuckLake `manual` case,
which at least fails loudly in that executor, a declarer in another language
deployed green with its data-quality assertions silently skipped.

Covered in the deploy-contract test and documented beside the annotation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: exclude a renamed producer from the sole-dbt producer set

The producer set already excluded the deploying script's own path, because its
committed rows describe the version being replaced. Under a rename the write
sits at the OLD path — still committed, and removed by the same uncommitted
transaction — so a producer renamed while it drops its `// materialize` and adds
`// on dbt://…` still counted as the producer that would wake it, and committed
a dormant edge.

The deploy-contract test covers it: without the exclusion the rename deploys
201 instead of being refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: take the rename test's parent hash from the create response

`format!("{:x}", …)` over the stored i64 drops leading zeros, while
`ScriptHash`'s deserializer hex-decodes and demands 8 bytes — so a hash below
2^60 would 422 the request instead of reaching the refusal it asserts on, on
roughly one in sixteen spellings of that script body. The create response
already carries the zero-padded form, as the rest of the suite uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: state the concurrent-ingest interleaving honestly

`sole_dbt_producer`'s doc claimed the concurrent-deploy race only ever resolves
toward refusing. It does when the uncommitted producer is native; when it is the
dbt ingest, the check sees an empty producer set and accepts, and if that ingest
then commits and runs its warning query before the subscriber's trigger row
lands, neither side reports the dormant edge.

Not serialized: the two would have to share a per-relation lock, and the ingest
takes `script … FOR UPDATE` before its own advisory lock, so a deploy holding
relation locks first inverts that order into a cross-subsystem deadlock — a worse
failure than the cosmetic edge. Recorded beside the other orphaning the deploy
cannot catch, with the bound both share: the next deploy of that project warns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: refuse a `dbt://` subscription that is not a whole relation

`# on dbt://main/analytics` deployed and persisted a trigger row. Every producer
spells `<warehouse>/<schema>/<name>` — the manifest ingest derives it from
`relation_name`, a `// materialize` target is checked against it — so a partial
one is an edge nothing can ever wake, which is what the dbt-only refusal exists
to prevent.

The shape now has one definition (`is_full_relation_path`) that both halves of
the deploy ask, rather than a segment count spelled twice: a subscription and a
write that disagreed would refuse and accept the same string.

Also rewrites the canvas test's comment as a current constraint per AGENTS.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: hold both halves of the deploy to one `dbt://` relation validator

A subscription checked the relation's shape but not its warehouse, so
`# on dbt://<unconfigured>/<schema>/<name>` deployed and persisted a trigger row
for something no producer can ever write: the write side refuses that exact
string, and a dbt project's `profile.warehouse` resolves against the same config,
so no later deploy fixes it and the dormant-edge warning cannot report it either.

The shape rule and the warehouse rule now live in one `validate_dbt_relation`
that both halves call, rather than being spelled per site — the previous two
rounds each closed one half of one rule, which is the drift that invites.

Also moves the parser test out from between a comment and the test it documents,
and names both refusals in the doc's list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: drop the subscription-only clause from the shared refusal message

"so nothing can produce it" reads backwards on the `// materialize` side, which
is the producer. The remaining sentence says what is wrong on both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: bound a `dbt://` relation by the asset-path column in the shared validator

`asset.path` is VARCHAR(255) and the manifest ingest drops a relation that
outgrows it rather than failing the whole graph, so past the column no producer
row can exist on either side. `script_trigger.trigger_ref` is unbounded text, so
an overlong subscription deployed and stayed dormant for good; an overlong write
reached Postgres and failed the deploy on a `value too long` instead of a message.

Both now refuse in the validator the two halves share, against the ingest's own
constant. The integration case computes the ref from that constant so it cannot
drift back under the bound.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: report a warehouse-lookup failure as the failure it is, and correct the boundary

`dbt_warehouse_exists` fails three ways — no such warehouse, the query itself,
and a setting with no `resource_path` — and all three became a 400 blaming the
user's warehouse name. A pool timeout mid-deploy told a retrying sync that a
transient server error was a permanent client one. Only `NotFound` is the
annotation's fault now.

The known-boundary paragraph claimed a flow-runner run still cascades. It does
not: it is routed by `flow_step_id`, which `is_eligible_kind` rejects, as
`asset_trigger_dispatch.rs` pins. Recording and cascading are decided separately,
so the paragraph now names all three routes rather than merging two of them — and
the row it omitted, an ordinary flow step, which records and never cascades.

E2E item 7 said "deployable" where the rule is "wakeable": with only the dbt
project reading the relation the producer set is empty, which deploys fine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: correct two rationales the last commit got wrong

`Error::SqlErr` already maps to 400 in this codebase, so the query case's status
was never the thing at stake. What the `NotFound` match earns is that a query
failure and a malformed setting stop being described as an unconfigured warehouse
name, and that the malformed-setting `InternalErr` reaches its own 500 instead of
being flattened.

And a flow step is two shapes, not one: a step running a deployed script is a
`Script` job that records and never cascades, while a step with an inline body is
`FlowScript`, which the recording guard excludes along with previews.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: warn about dormant subscriptions from the run that publishes ownership too

A run whose static descriptor finds its profile moved re-ingests the version's
graph and republishes path ownership, exactly as a deploy does — so it can be
what leaves a subscription accepted while the relation had no producer with dbt
as its only one. That path discarded `persist_ingest`'s result and emitted no
warning, which also made the doc's enumeration of unreported orphanings wrong.

Both ownership-publishing points warn now. An agent worker still cannot: it
reaches these tables only through the API and its ingest publishes without
reading back, which the doc now says.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: an agent run publishes no ownership, and the warning has two callers

The agent-worker sentence called it an exception that publishes ownership without
warning. It publishes none: `Connection::Http` forces per-run models, and
`publishes_ownership()` is the negation of that, so an agent stores a job-pinned
snapshot and leaves workspace ownership with the deployed graph — it cannot orphan
a subscription at all.

`warn_dormant_subscribers`' own doc still named the deploy log as the only place
the warning shows, one commit after it gained its second caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: stop the managed-write rule from contradicting the dbt:// target

The sentence after the warehouse-relation paragraph says `// materialize` means
the runtime writes the table for you and the body is a bare SELECT. That is the
managed DuckLake rule, written before a `dbt://` target existed, and unqualified
it tells the model the opposite of what the paragraph above it just said — a
model following the more prominent one emits a SELECT for a warehouse relation,
which deploys and then writes nothing.

Both prompt sources now scope it, and both name the `// data_test` refusal beside
a `dbt://` target, which the badge list advertised without the caveat.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-07 17:38:07 +02:00
..