A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.
Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.
Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:
- a Postgres trigger or capture cannot be created on, or connect to, a data
table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
the data table, its own or a fork's through its pointer. The refusal names
each one to disable.
This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three fixes from review.
`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.
The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.
`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.
Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.
The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit 7dd3275a10.
The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.
`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.
That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.
Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.
Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.
Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.
Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.
The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.
A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.
Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.
Refuse in both, so the clone stops before a database exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.
Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".
The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.
Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
Three from the round, all about deciding on state that could already have moved.
A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.
Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.
The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.
Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.
The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.
One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.
Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.
Three ways a save could leave something resolving to nothing:
A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.
Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.
Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.
Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.
`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.
Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.
Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.
Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.
Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.
The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.
Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.
Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.
`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
Three ways the feature could end up in a state nobody could see or undo.
Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.
Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.
`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.
Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.
Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.
Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.
All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.
A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.
Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.
A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.
Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
* feat: column-level lineage for dbt from the engine's parquet index
`manifest.json` carries no column-to-column edges, which is why decision 14
recorded column lineage as unavailable. The edges live in a different artifact:
`dbt compile --static-analysis strict --write-index` writes `target/index/`,
whose `dbt.column_lineage.parquet` holds them and whose
`dbt.node_columns.parquet` holds every column of every node, typed and ordered
rather than only the ones an author documented.
Strict analysis rejects SQL the default accepts, so this is a separate compile
with its own `--target-path`, opt-in per project via `column_lineage: true`, and
best-effort throughout: a project it cannot analyze keeps exactly the graph it
had, with the engine's own diagnostics in the job log.
Storage mirrors `dbt_edge`: `dbt_column_edge` keyed by (path, version, job) with
the same composite FK to `script` and the same sweeps. The typed column list
lands in `dbt_node.column_schema`, beside `columns` rather than merged into it,
so `columns` stays what the author declared.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ
* fix: address review findings on the dbt column-lineage pass
- The workspace fork copied every other dbt sidecar table and not this one, so
a fork lost its column lineage silently and could not recover it: the cloned
digest covers the column edges, so a dynamic run in the fork matched it and
stored nothing.
- The parquet was collected whole before the edge cap applied, which is exactly
the input the cap exists for — a project whose `scan` lineage is quadratic in
its widest model could take the worker process down. Decoded a row at a time
with the bound enforced during the decode.
- The pass swallowed every error from the runner, including the job poller's
cancellation and deadline, so a run that blew its timeout inside an optional
annotation could still publish a graph and report success. `run_captured`
now carries the exit status in its value, so only a failed COMPILE is
downgraded, and the pass may spend at most half the remaining wall clock so
it cannot starve the build that follows it.
- `scan` edges are stored but no longer served: they are most of a project's
lineage, nothing renders them, and the graph endpoint is polled by the run
page. They are also the first thing the storage cap gives up now, rather than
evicting the direct edges the trace draws.
- `column_schema` and the column edges take the same gate as the model's SQL. A
column-level view is the shape of what the author wrote, one level finer than
the `ref()` graph, which is ungated only because it draws relations the
caller already sees.
- `graph_digest` hashes the new section only when it has edges, so a project
that never asked for the pass keeps the digest it has instead of
re-snapshotting on every dynamic run until it is redeployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ
* fix: the editor buffer's column lineage, and three bounds that were wrong
Round-2 review found four defects, all of them introduced by the round-1 fixes.
- The `script_visible` gate on the column edges was copied from the node query
without its `script_hash IS NULL` arm. `= NULL` is never true, so every
version-less row was filtered out and an editor buffer's parse rendered its
typed columns and none of their lineage — the one place the feature is meant
to be used. Pinned by an assertion in `dbt_pinned_graph.rs`, which is where
this class of bug already had a home.
- The phase budget was handed to the poller, whose expiry is an `Err`
indistinguishable from a cancellation or the job's own deadline, so a slow
but valid analysis aborted the build it exists to annotate. The runner gets
the full deadline again — those two must still fail the job — and the budget
is a race around the whole pass, where expiring is this budget and nothing
else.
- The decode cap counted parquet ROWS, so `scan` and out-of-graph rows could
spend it before a single drawn edge was read. It now counts what is kept,
takes direct kinds in a first pass, and is handed the graph's own nodes so
the budget cannot go on rows that could never be stored.
- Hashing the new digest section conditionally did not preserve old digests,
because an absent `column_schema` still serialized as `null` inside the
nodes. It is skipped when absent instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ
* refactor: split the lineage pass by error contract, and read it in one query
Round 3's findings were all consequences of round 1 and 2's fixes, clustered in
the same two files, so this reshapes those two seams rather than patching again.
The worker pass was one function being three things at once — a subprocess
runner with job-lifecycle error semantics, a bounded decoder, and a best-effort
degrader — which is why each fix to one perturbed another. It is now
`compile_index`, which owns the JOB's semantics (only a cancellation or the
job's deadline can `Err`; a non-zero exit, the output ceiling and the phase
budget are outcomes), and `read_index`, which owns the ARTIFACT's and knows
nothing about the job. The budget wraps the compile alone, so a decode can no
longer outlive the timeout that reported the build would get the rest. The
output ceiling likewise becomes a value rather than a job error, for the caller
that can carry on without the tail of a compile's stdout.
The column edges were read by a fourth hand-written copy of the `live`/`chosen`
CTEs and the version/editor-buffer join conditions, and copying them is what
dropped the `script_hash IS NULL` arm and hid every buffer parse's lineage. Both
kinds of edge now come from ONE statement over a `UNION ALL`'d edge source, so
those conditions exist once. The union is at the source rather than a join
because column lineage can name a node pair `dbt_edge` has no row for: a model
reading `{{ this }}` gets edges from itself to itself, and `parent_map` has no
self-loop.
The cap on the column half now sits after the scope filter, the visibility
check and the graph joins — the scope moved into SQL via the existing
`ScopePathFilter` — so a row the caller may not read can no longer spend it and
leave an allowed project's trace short.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PRtsPQ3Ck69Fu9DNr7bMcJ
* refactor: serve dbt column lineage from its own endpoint
The column edges rode on the folder-wide asset graph, which a run page polls,
while the trace is drawn for one selected relation. That needed a cap, and a cap
has to be applied after every filter that can drop a row.
Keyed to the asset there is no cap: `assets/column_lineage` answers for one
relation, and the caller's `scripts:read` scope and the project's visibility are
decided once, for the script that owns it. Pinning to a run's snapshot or the
editor's parse of its buffer costs the job-read gate, so that form is
`jobs/dbt_column_lineage/{id}` — the same shape `jobs/dbt_graph/{id}` has.
The worker's decode now bounds work and memory separately, and a compile stopped
by the output ceiling reports as truncated rather than complete.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: resolve the owning dbt version the way the graph does
The unpinned arm picked the newest live version at the path without narrowing to
dbt, so a path since redeployed in another language answered with no lineage
while the graph beside it still drew that project's stale nodes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: gate pinned column lineage on reading the project, and answer the component
Four things round 5 found, three of them in code this branch rewrote:
- The pinned arm resolved the version from the job and stopped there, so a
share-link viewer entitled to a run got the project's column names and edges
while the graph beside it still redacted `raw_code` and `column_schema`.
Resolving WHICH version answers is not deciding whether the caller may read
it; the version-less editor buffer keeps its exemption, having no `script` row
to ask.
- The answer was the whole owning project's edges. The canvas lays out the
connected component of the selected relation's columns, so the rest was
unrenderable weight; a recursive walk over both directions returns exactly
what is drawn, and the project key travels with it so a `unique_id` two
projects share cannot walk from one graph into the other.
- The decode had no exit but the 4M-row backstop once its buckets were full,
spending wall clock the build below does not get.
- An unreadable index was reported as a missing one, sending the reader to look
at their engine rather than at the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: stitch the two column graphs, and walk the component in Rust
Round 6's two findings, both regressions this branch introduced:
- The decode returned `Continue` on the edge that FILLED the direct-edge
budget, so a `scan`-only tail after it decoded to the 4M-row backstop with
nowhere to put anything. The read now ends on that edge.
- Seam 3 made the pipeline page choose between the dbt graph and the producer
one. They share node ids — `// column total <- dbt://wh/analytics/orders.amount`
mints the same `(dbt, path, column)` node dbt's own lineage does — so choosing
ended a trace at the boundary in both directions. They are merged again, and
a ducklake selection asks about the dbt relation its producers name so the
chain continues past it. The dbt editor gets the same merge.
Also: the component is walked in Rust rather than by a recursive CTE. A CTE has
no index, so the recursive term rescanned the doubled edge set once per level —
1243ms against 59ms for the query alone on a 3000-model project, 11.7M rows in
the plan. Same answers, same tests; end to end 1.48s to 0.73s there and 1.60s to
0.26s on a 1000-deep chain. The client stops re-asking for a component it
already holds, which is most clicks within one project.
The four doc sites that described a whole-project answer are rewritten around
what it now is, rather than edited where they disagreed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: expand every dbt boundary a selection reaches, and only skip what was asked
Round 7's findings, all in the frontend seam this branch added:
- A ducklake selection seeded the dbt fetch from the FIRST boundary relation it
found, so a table derived from two unconnected dbt relations expanded one and
left the other a leaf — the same "stops at the boundary" symptom the round-6
fix removed, one hop further along. Every distinct boundary is fetched now and
the components merged.
- The component cache skipped a relation merely PRESENT in the graph in hand.
A relation two projects describe has an owner row in each, and a component
fetched for one carries it as an endpoint without the other's half, so that
skipped the request that would have resolved the second owner. Only a relation
actually asked about under this pin is skipped.
- A comment still called the producer graph gated to ducklake selections after
it was widened to dbt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: land dbt column lineage as storage and ingest only
The API surface that draws a column trace moves to a follow-up PR, on
`dbt-column-lineage-surface`. It kept generating findings — a client cache
whose premise was wrong for a two-owner relation, then staleness and a lost
retry from tightening it, and a seed walk that stopped at the first boundary —
and the fix for the last of them is a transitive owner expansion, which has to
re-apply the caller's gate to every newly discovered project. That is the same
shape as the leak four reviewers caught in the pinned arm, and it wants its own
review rather than being the fourth fix at the end of this one.
What lands here stands on its own: the analysis pass, `dbt_column_edge`,
`dbt_node.column_schema`, the engine gating and the error-contract split — plus
the one user-visible half, the typed and ordered column list, which rides the
asset graph the details pane already fetches and replaces a panel that could
only show the columns an author had documented.
Also fixes a real bug in the pass, found in review: it compiled without the
build's `--full-refresh`. `is_incremental()` branches on that flag, so an
incremental model reading `{{ this }}` compiles its self-join — and any `ref()`
inside that branch — only when the flag is absent, and the pass was storing
lineage for SQL a full-refresh run never executed. The flag now comes from one
place shared with the build, and a run that overrides it gets its own graph
rather than standing as the version's.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: say why direct kinds get the budget without naming a view
The bucketing comments explained the priority by what a trace draws, which is
a forward reference now that the surface moved out. The reason stands on its
own: `copy`/`mod` say the value travelled, `scan` says the column was read to
produce the row and so reaches every output column of its model.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: round-9 findings on the descoped PR
- The `full_refresh` helper was inserted between `selection_is_overridden` and
its doc comment, so thirteen lines about `select`/`exclude` echoes documented
the wrong function and the one they were written for had none. Moved below it.
- The parse path ran the analysis compile and the parquet decode BEFORE the
guard that returns when there is no warehouse identity, paying for both and
dropping the result. Moved after it.
- Three sites still described a `/column_lineage` endpoint this branch no longer
has, and two user-facing strings promised a column trace it no longer renders:
the panel's hint and the descriptor template now say what the flag actually
buys, which is the typed column schema.
- Dropped test scaffolding the removed suite left behind: a `raw_orders` node
and `dbt_edge` whose only assertion re-tested pre-existing graph behaviour,
and a second editor-buffer node nothing asserts on.
Documented rather than fixed: an incremental model has two shapes, and which one
the index holds depends on whether the target existed when the pass ran.
`is_incremental()` is false with no target as well as under `--full-refresh`, and
dbt has no mode that emits both — so a version's graph describes the compile that
produced it, and only a re-ingesting run describes its own run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep lineage_kind in the edge key, and one answer for --full-refresh
- Both unique indexes omitted `lineage_kind`, so a column that is projected AND
used as a predicate for the same output column — an ordinary shape — had its
`copy` and `scan` edges collapse under `ON CONFLICT DO NOTHING`, while the
digest counted both. The kind is part of the fact, so it is part of the key.
Edited in the migration rather than added as a second one: it has not landed.
- `full_refresh` was shared between the build and the analysis pass without the
`command != "test"` condition that sat at the build's call site, so the two
disagreed for exactly the runs that build nothing. The condition moved inside
the function, which is the point of sharing it, and the command is threaded to
the pass.
- The "what a trace draws" rewrite missed the copy in `dbt_manifest.rs`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: drop the unreachable full_refresh threading, test the uniqueness key
`DBT_COMMANDS` is `["build", "retry", "show", "parse"]` and `default_command`
returns `build` in every arm, so `command == "test"` cannot happen — the guard
the last commit moved into `full_refresh` was already inert where it came from.
Threading the command through five signatures to preserve it bought nothing, so
it is gone; the build and the pass call one function of the descriptor and the
invocation, which is what the sharing was for.
The uniqueness-key fix now has a test: a column projected AND used as a
predicate for the same output column stores both its `copy` and its `scan` row.
Verified against the old key, where it returns 1 instead of 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: restore the dbt test --full-refresh guard I removed on a wrong premise
The previous commit removed it after reading `DBT_COMMANDS` and concluding
`"test"` was unreachable. That is only true of the command a CALLER can name:
`run_dbt` is invoked with `"test"` directly for the `after_all` test phase, so
an `after_all` project with `full_refresh: true` reached it — and dbt rejects
`--full-refresh` on `test`, failing the phase. Both reviewers caught it.
The guard is back inside the shared function, where the build and the pass get
one answer, and its doc now records why reading the allowlist alone is
misleading. The test covering the `test` case is restored with it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: notice a job that ended during the decode, and name truncation as the cause
- The parquet decode runs on a blocking thread with no poller watching it, so a
cancellation or an expired deadline during it was invisible: `dbt_dep` went on
to publish the graph and the job returned success. The job's state is checked
once the decode returns, before the caller publishes anything, and an ended
job `Err`s — which this module may always do for the job's own semantics.
- A compile stopped by the output ceiling could leave no artifact, and the log
then blamed the engine's capability, sending the reader to check their adapter
rather than the ceiling. Truncation now names itself in the missing and
unreadable branches too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: read cancellation from the DB after the decode, not from a poller's field
`ctx.canceled_by` is only ever written by a poller, and no poller runs during
the blocking decode — which is the exact window the check was added for. So the
guard caught only a cancellation already observed before it, and the comment
beside it claimed more than it did. It now queries `v2_job_queue` directly, the
same probe `worker_lockfiles` uses before it overwrites a flow.
A failed probe answers "still running": this decides whether to discard work
already done, so an unreachable database must not be the reason a healthy deploy
loses its graph.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: reuse job_is_canceled rather than a second copy of it
The probe added last round was `job_is_canceled` from the same file, retyped —
same query, same `Connection::Http` behaviour. Reused instead.
Its doc said a non-database connection was "a failed probe", which reads as an
error path. It is not: it is the agent worker, and on one there is no database
to ask, so only the deadline answers and a cancel issued during the decode is
not observable. The retry path avoids that by refusing to run on an agent worker
at all — which an optional annotation has no business doing — so the gap is
recorded at both ends instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: close the agent-worker cancellation gap instead of documenting it
The previous commit said a cancel issued during the decode is not observable on
an agent worker. It is: `ping_job_status` returns `canceled_by` over both
connection kinds, and is how the poller itself notices one there. So the check
asks through the ping rather than querying `v2_job_queue` directly, and holds on
an agent worker, where a direct query reaches no database at all.
`job_is_canceled` goes back to private and its doc to what it said before — the
retry that calls it still refuses to run on an agent worker for its own reasons.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: decode the index under the job poller instead of checking after it
Two findings with one cause: the decode was the only phase of this pass with no
subprocess behind it, so nothing heartbeated while it ran. A large index left
the worker silent for as long as it took, which the zombie sweep reads as a dead
job and restarts — and the cancellation check bolted on afterwards could only
ever report what had already happened, while dropping the ping's
`already_completed`, so a force-cancelled deploy still published its graph.
Running it under `run_future_with_polling_update_job_poller` answers all of it:
the poller pings throughout, and ends the phase with an `Err` on cancellation,
`AlreadyCompleted` or the phase timeout. The bespoke probe is gone with it.
Verified on a live deploy: 32 edges and 4 typed schemas ingested through the
polled decode.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: stop a cancelled decode, and say what the read phase can now do
Putting the decode under the poller heartbeats it and ends the phase when the
job does, but dropping a `JoinHandle` detaches a blocking task rather than
cancelling it — so a cancelled job left a thread decoding up to four million
rows for a job that was over. The row loop reads an abandonment flag that a drop
guard on the awaiting future sets, so the decode stops at its next row.
That same change made the read phase able to `Err`, and three places still said
it could not — decision 14 in as many words. The distinction that holds is
narrower: nothing the ARTIFACT does or fails to do can fail a job, so absent,
unreadable and partial are all values; the JOB can still end the phase the read
runs in. Stated that way in the module doc, the `Artifact` doc, `MAX_INDEX_ROWS`
and the decision.
Verified on a live deploy: 32 edges and 4 typed schemas.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: share AbortOnDrop, and stop citing a hazard that is now handled
`Abandon` was `ansible_executor`'s `AbortOnDrop` retyped — same struct, same
reason, same `spawn_blocking` shape. Moved to `common` and used from both.
The paragraph explaining why the phase budget wraps the compile alone gave as
its reason "a decode still running on a blocking thread", which is exactly what
the abandonment flag now prevents. The reason that survives is the one that was
always the point: the budget exists to leave the build its share of the clock,
and only the compile can spend that share unboundedly. The decode's end is the
job's, through the poller it runs under.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: put both doc comments back on the items they describe
Moving AbortOnDrop orphaned a doc at each end: it landed between
`raw_to_string`'s doc and `raw_to_string`, and the doc of the struct it replaced
stayed behind to prefix `fetch_repo_archive`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: name the binding the row loop actually reads
`Abandoned` was neither the type nor the binding; the flag is `abandoned`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
