mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-05 16:03:47 +00:00
docs: state the concurrent-ingest interleaving honestly
`sole_dbt_producer`'s doc claimed the concurrent-deploy race only ever resolves toward refusing. It does when the uncommitted producer is native; when it is the dbt ingest, the check sees an empty producer set and accepts, and if that ingest then commits and runs its warning query before the subscriber's trigger row lands, neither side reports the dormant edge. Not serialized: the two would have to share a per-relation lock, and the ingest takes `script … FOR UPDATE` before its own advisory lock, so a deploy holding relation locks first inverts that order into a cross-subsystem deadlock — a worse failure than the cosmetic edge. Recorded beside the other orphaning the deploy cannot catch, with the bound both share: the next deploy of that project warns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f7f79d45cf
commit
dff574d3d9
@@ -267,8 +267,16 @@ pub fn derive_pipeline_asset_trigger_refs(
|
||||
/// them is free of the opposite error, because a script never wakes its own
|
||||
/// subscription — the dispatcher skips that as a self-loop.
|
||||
///
|
||||
/// A producer another deploy is committing concurrently is still invisible, so
|
||||
/// that race resolves toward refusing with a message the user can retry past.
|
||||
/// A producer another deploy is committing concurrently is invisible either way,
|
||||
/// and the outcome depends on which side it is. An uncommitted NATIVE producer
|
||||
/// leaves a dbt-only set and refuses, with a message the user can retry past. An
|
||||
/// uncommitted DBT one leaves an empty set and accepts — and if that ingest then
|
||||
/// commits and runs [`dormant_dbt_subscriptions`] before this deploy's trigger
|
||||
/// row lands, neither side reports the edge it left dormant. Serializing the two
|
||||
/// is not worth it: they would have to share a per-relation lock, and the ingest
|
||||
/// takes `script … FOR UPDATE` before its own advisory lock, so a deploy holding
|
||||
/// relation locks first inverts that order into a deadlock across the two
|
||||
/// subsystems. The next deploy of that project warns (docs/dbt-runtime.md).
|
||||
pub async fn sole_dbt_producer<'e>(
|
||||
executor: impl PgExecutor<'e>,
|
||||
workspace_id: &str,
|
||||
|
||||
+14
-4
@@ -734,10 +734,20 @@ What that leaves is a subscription accepted while it was live and later orphaned
|
||||
A dbt project deployed afterwards that claims the relation names those edges in
|
||||
its own log rather than leaving them silently dormant — the same "an edge that can
|
||||
never fire is worse than saying so" the refusal is for, at the other point where
|
||||
it is knowable. The remaining case, a native producer that drops its
|
||||
`// materialize` and leaves dbt alone on the relation, is reported nowhere: the
|
||||
deploy that causes it does not touch the subscriber, and the canvas is where it
|
||||
shows.
|
||||
it is knowable.
|
||||
|
||||
Two orphanings are reported nowhere, and both are accepted rather than overlooked.
|
||||
A native producer that drops its `// materialize` and leaves dbt alone on the
|
||||
relation: the deploy that causes it does not touch the subscriber. And the
|
||||
interleaving where a dbt ingest commits between a subscriber's producer check and
|
||||
its own commit — the check sees no producer and accepts, the ingest's warning
|
||||
query sees no trigger and says nothing. Closing the second means a per-relation
|
||||
lock shared by the deploy path and the ingest, and the ingest takes
|
||||
`script … FOR UPDATE` before its own advisory lock, so a deploy holding relation
|
||||
locks first inverts that order into a deadlock across two subsystems — a worse
|
||||
failure than the cosmetic edge it would prevent. Both are bounded the same way:
|
||||
the next deploy of that project warns, and the canvas is where they show
|
||||
meanwhile.
|
||||
|
||||
A plain READ still renders the consumer beside the model, which is what makes
|
||||
the lineage one graph — but it is written in the script's own code, not in a
|
||||
|
||||
Reference in New Issue
Block a user