starred/windmill
1
0
Fork 0
mirror of https://github.com/windmill-labs/windmill.git synced 2026-09-21 00:02:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: put a data table's connection under Postgres roles (#11020)

Browse Source 
* feat(datatables): put a data table's connection under Postgres roles

A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): gate the paths that reach a whole database as admin

Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping

A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): fail loudly where a role or a pointer can be left half-recorded

Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): unbreak two operator messages and two comments that described other code

The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract

The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): give the role catalog its own row, out of reach of the config machinery

Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced

The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing

The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* refactor(datatables): put the role catalog in its own table, not in global_settings

Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): validate a rename against the save it describes, and re-check under the locks

Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically

The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse to copy a data table that is under roles

pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse the clone's database too, not only its data

A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* nit worker error msg

* fix pg_dump stuck on version 17 on nix

* fix(datatables): refuse a malformed role annotation instead of ignoring it

`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse a malformed role query string instead of ignoring it

`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): carry the role annotation into the row_to_json retry

The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore(datatables): don't mount the roles UI until the ACL editor lands

Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test

Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): clone a fork's pointer instead of failing after the copy

Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse to roll back the catalog while roles exist

The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* refactor(datatables): settle clonability in one place, before anything is created

A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): let a retried clone reclaim its own leftover database

A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "fix(datatables): let a retried clone reclaim its own leftover database"

This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): record the stale clone database as a known limitation

A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): bounce the streams reading a data table when it is deleted

Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): keep the fork schema baseline, and bounce streams on every removal

Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(datatables): keep Postgres triggers and data table roles apart

A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved

Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): disable a data table role before deleting it

Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with a stream starting

Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): wait out live listeners, and resolve stored names containing `?`

Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): warn when a settings sync strands fork pointers

A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse a save that drops a data table's roles through an undeclared rename

A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): no entry without roles may newly reach a database under roles

The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move data table role catalog and resolution to the enterprise edition

Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table permissions endpoints to the enterprise edition

The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table role catalog endpoints to the enterprise edition

The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the roles tests on the enterprise edition, refusals without it

Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): gate the roles UI mount sites on an enterprise license

Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the tenant matcher test on the enterprise edition

The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607

This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.

* fix(datatables): refuse roles while a same-workspace alias reaches the database

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): let CE migrations connect as an explicitly named admin

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with aliases saved from other workspaces

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): note that legacy names with ? cannot be migrated

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): drop a DuckDB data table secret once its ATTACH has used it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(datatables): resolve a workspace's data tables per pointer hop, not per entry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): hold the parent's settings while a fork points at its data tables

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to 7e338e4dabf91689bfd7fb0333c6534040b17b59

This commit updates the EE repository reference after PR #787 was merged in windmill-ee-private.

Previous ee-repo-ref: 38d6fcf2aeb39cfdac21814bbdbbcc02911e566a

New ee-repo-ref: 7e338e4dabf91689bfd7fb0333c6534040b17b59

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Diego Imbert
2026-09-18 13:56:41 +02:00
committed by GitHub
co-authored by Claude Opus 5 windmill-internal-app[bot]
parent e2a91ca2b1
commit 0e807fb1dd
90 changed files with 5980 additions and 324 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/.sqlx/query-06abbf945bee93349ff88f64906b96ea1e853ef202510281427cfa9beeff81b3.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT 1 AS one FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "one",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "06abbf945bee93349ff88f64906b96ea1e853ef202510281427cfa9beeff81b3"
}
backend/.sqlx/query-06ce02cd7ce2f5a57355153edb573c242f9ba758db66e9a5e16f30e3e1494201.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings ws\n SET datatable = (\n SELECT jsonb_set(ws.datatable, '{datatables}', jsonb_object_agg(\n dt.key,\n CASE WHEN dt.value->'reference'->>'workspace_id' = $1\n AND dt.value->'reference'->>'datatable' = $2\n THEN jsonb_set(dt.value, '{reference,datatable}', to_jsonb($3::text))\n ELSE dt.value END\n ))\n FROM jsonb_each(ws.datatable->'datatables') dt\n )\n WHERE EXISTS (\n SELECT 1 FROM jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) d\n WHERE d.value->'reference'->>'workspace_id' = $1\n AND d.value->'reference'->>'datatable' = $2\n )",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "06ce02cd7ce2f5a57355153edb573c242f9ba758db66e9a5e16f30e3e1494201"
}
backend/.sqlx/query-0d10e0fa5cf4033c7d93c9ed56be8209046007917f44da954eccf2188e5bff1f.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT permissioned_as, permissioned_as_email FROM v2_job\n WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "permissioned_as",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "permissioned_as_email",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Uuid",
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "0d10e0fa5cf4033c7d93c9ed56be8209046007917f44da954eccf2188e5bff1f"
}
backend/.sqlx/query-297c7a40dfce729d44aa37bc7c65560517bd25e40c0752a00467829191e2eb98.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings ws\n SET datatable = (\n SELECT jsonb_set(ws.datatable, '{datatables}', jsonb_object_agg(\n dt.key,\n CASE WHEN dt.value->'reference'->>'workspace_id' = $2\n THEN jsonb_set(dt.value, '{reference,workspace_id}', to_jsonb($1::text))\n ELSE dt.value END\n ))\n FROM jsonb_each(ws.datatable->'datatables') dt\n )\n WHERE jsonb_typeof(ws.datatable->'datatables') = 'object'\n AND ws.datatable::text LIKE '%\"reference\"%'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": []
},
"hash": "297c7a40dfce729d44aa37bc7c65560517bd25e40c0752a00467829191e2eb98"
}
backend/.sqlx/query-2a391cc1bfcd2f75b46144a394c01237e09c3060da88170f1f6e06468309d213.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE capture_config SET server_id = NULL, last_server_ping = NULL\n WHERE workspace_id = $1 AND trigger_kind = 'postgres'\n AND (trigger_config->>'postgres_resource_path' = $2\n OR trigger_config->>'postgres_resource_path' LIKE $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "2a391cc1bfcd2f75b46144a394c01237e09c3060da88170f1f6e06468309d213"
}
backend/.sqlx/query-334dbcd48fb59c96c62c2705ab2d1ce716cd52417f487cc1a8dd376017b2db7d.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = $1)",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "exists",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Name"
]
},
"nullable": [
null
]
},
"hash": "334dbcd48fb59c96c62c2705ab2d1ce716cd52417f487cc1a8dd376017b2db7d"
}
backend/.sqlx/query-4692810d2be817bbb5de9b476d68d695941bd4fb5ccef393e4da522ed479d601.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE postgres_trigger SET server_id = NULL, last_server_ping = NULL\n WHERE workspace_id = $1\n AND (postgres_resource_path = $2 OR postgres_resource_path LIKE $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "4692810d2be817bbb5de9b476d68d695941bd4fb5ccef393e4da522ed479d601"
}
backend/.sqlx/query-5048e21546f9710697100100e1255ab103979433bc386d7c89d0e30db12bfd57.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n FROM workspace_settings ws\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE dt.value->'reference'->>'workspace_id' = $1\n AND dt.value->'reference'->>'datatable' = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "5048e21546f9710697100100e1255ab103979433bc386d7c89d0e30db12bfd57"
}
backend/.sqlx/query-538dd1779874e4003932d7f17750239c625f85e25b3364bf2edf566f518c8ee2.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n FROM workspace_settings ws\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE dt.value->'reference'->>'workspace_id' = $1\n AND dt.value->'reference'->>'datatable' = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "538dd1779874e4003932d7f17750239c625f85e25b3364bf2edf566f518c8ee2"
}
backend/.sqlx/query-58e5cfe9eb87bda9f7de87c403861b6e7b9d35a41594681e2a92a87359e6a018.json
Generated
+23
View File
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT username FROM usr WHERE workspace_id = $1 AND email = $2 AND disabled = false",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "username",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false
]
},
"hash": "58e5cfe9eb87bda9f7de87c403861b6e7b9d35a41594681e2a92a87359e6a018"
}
backend/.sqlx/query-6f9fb5d72f486358fa25d6887bd69b93910e028f140c07048f2c1c8d63ee6909.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO global_settings (name, value) VALUES ($1, $2)\n ON CONFLICT (name) DO UPDATE SET value = $2",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb"
]
},
"nullable": []
},
"hash": "6f9fb5d72f486358fa25d6887bd69b93910e028f140c07048f2c1c8d63ee6909"
}
backend/.sqlx/query-71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0.json
Generated
+38
View File
@@ -0,0 +1,38 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, name, enabled, pwd FROM datatable_role",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "enabled",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "pwd",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false,
false,
true
]
},
"hash": "71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0"
}
backend/.sqlx/query-79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629.json
Generated
+29
View File
@@ -0,0 +1,29 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n FROM workspace_settings ws\n JOIN workspace w ON w.id = ws.workspace_id AND w.deleted = false\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE ws.workspace_id <> $1\n AND dt.value->'database'->>'resource_type' = 'instance'\n AND dt.value->'database'->>'resource_path' = $2\n ORDER BY ws.workspace_id, dt.key\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629"
}
backend/.sqlx/query-86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_role (id, name, enabled, pwd) VALUES ($1, $2, $3, $4)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Bool",
"Text"
]
},
"nullable": []
},
"hash": "86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba"
}
backend/.sqlx/query-975e404ac3a6258bb8220e122e3de094c7ab23330fdbc74d6e4ad472ddd3c820.json
Generated
+15
View File
@@ -0,0 +1,15 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO global_settings (name, value) VALUES ($1, $2)\n ON CONFLICT (name) DO UPDATE SET value = $2, updated_at = now()",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Jsonb"
]
},
"nullable": []
},
"hash": "975e404ac3a6258bb8220e122e3de094c7ab23330fdbc74d6e4ad472ddd3c820"
}
backend/.sqlx/query-9f663180166f53d117e794f3f3a5723a0a43db163ecca7d5a63d4e74ab1d3be1.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT pg_advisory_xact_lock(hashtext('datatable_role_catalog'))",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "pg_advisory_xact_lock",
"type_info": "Void"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "9f663180166f53d117e794f3f3a5723a0a43db163ecca7d5a63d4e74ab1d3be1"
}
backend/.sqlx/query-a57de2bb0442a5ee8a607cd63cfcf675de175796184f620cb4b09670c8b0b19f.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE capture_config SET server_id = NULL, last_server_ping = NULL\n WHERE workspace_id = $1 AND trigger_kind = 'postgres'\n AND (trigger_config->>'postgres_resource_path' = $2\n OR trigger_config->>'postgres_resource_path' LIKE $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "a57de2bb0442a5ee8a607cd63cfcf675de175796184f620cb4b09670c8b0b19f"
}
backend/.sqlx/query-b42af37fb474bea4c5419b0a46d9eadfe384013ab970ccf9c5effd1c78321b7c.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.datatable->'datatables' FROM workspace_settings ws\n WHERE ws.workspace_id = $1 FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
null
]
},
"hash": "b42af37fb474bea4c5419b0a46d9eadfe384013ab970ccf9c5effd1c78321b7c"
}
backend/.sqlx/query-b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT jsonb_object_keys(value->'databases') FROM global_settings\n WHERE name = 'custom_instance_pg_databases'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "jsonb_object_keys",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6"
}
backend/.sqlx/query-c1d026c886799dabc39ce73e1fe09ccb175c7271df75d67aa9c72ad6f825a992.json
Generated
+28
View File
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM usr WHERE email = $1 RETURNING username, workspace_id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "username",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "workspace_id",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
false
]
},
"hash": "c1d026c886799dabc39ce73e1fe09ccb175c7271df75d67aa9c72ad6f825a992"
}
backend/.sqlx/query-c5451ea9d9fa5146af242d1ee8c19ebd65b80e7ed9f29b9fb2e03767c2aa94ba.json
Generated
+28
View File
@@ -0,0 +1,28 @@
{
"db_name": "PostgreSQL",
"query": "SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n FROM workspace_settings ws\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE dt.value->'reference'->>'workspace_id' = $1\n ORDER BY ws.workspace_id, dt.key",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "c5451ea9d9fa5146af242d1ee8c19ebd65b80e7ed9f29b9fb2e03767c2aa94ba"
}
backend/.sqlx/query-c85d362fe2e652d4ac01a35bf470e80b993020a2ff5dcb5849dc570d52798587.json
Generated
+14
View File
@@ -0,0 +1,14 @@
{
"db_name": "PostgreSQL",
"query": "DELETE FROM datatable_role WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text"
]
},
"nullable": []
},
"hash": "c85d362fe2e652d4ac01a35bf470e80b993020a2ff5dcb5849dc570d52798587"
}
backend/.sqlx/query-d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40.json
Generated
+24
View File
@@ -0,0 +1,24 @@
{
"db_name": "PostgreSQL",
"query": "SELECT dt.key AS \"datatable!\"\n FROM workspace_settings ws\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE ws.workspace_id = $1\n AND dt.key <> $2\n AND NOT dt.value ? 'permissions'\n AND dt.value->'database'->>'resource_type' = 'instance'\n AND dt.value->'database'->>'resource_path' = $3\n ORDER BY dt.key",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40"
}
backend/.sqlx/query-d5fb5dde6300862f978739a3d9249fc2b3e7697c0da7d3195398933d3d81aadf.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable FROM workspace_settings WHERE workspace_id = $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
true
]
},
"hash": "d5fb5dde6300862f978739a3d9249fc2b3e7697c0da7d3195398933d3d81aadf"
}
backend/.sqlx/query-da5057c86607327bafc2942c218025ca9181a0c396405984d87e422e129521c1.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings ws\n SET datatable = (\n SELECT jsonb_set(ws.datatable, '{datatables}', jsonb_object_agg(\n dt.key,\n CASE WHEN dt.value->'reference'->>'workspace_id' = $1\n AND dt.value->'reference'->>'datatable' = $2\n THEN jsonb_set(dt.value, '{reference,datatable}', to_jsonb($3::text))\n ELSE dt.value END\n ))\n FROM jsonb_each(ws.datatable->'datatables') dt\n )\n WHERE jsonb_typeof(ws.datatable->'datatables') = 'object'\n AND ws.datatable::text LIKE '%\"reference\"%'",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "da5057c86607327bafc2942c218025ca9181a0c396405984d87e422e129521c1"
}
backend/.sqlx/query-dc8dfc37559e9b6713bde48155f48b5a2c7b8199eace1508e102b60d1ff40c04.json
Generated
+20
View File
@@ -0,0 +1,20 @@
{
"db_name": "PostgreSQL",
"query": "SELECT value->'roles' FROM global_settings WHERE name = 'custom_instance_pg_databases'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "?column?",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "dc8dfc37559e9b6713bde48155f48b5a2c7b8199eace1508e102b60d1ff40c04"
}
backend/.sqlx/query-e159b2ff15633f85e839ee4fe1ec2ecd11caf228ea8d0f52ad66def595644250.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE postgres_trigger SET server_id = NULL, last_server_ping = NULL\n WHERE workspace_id = $1\n AND (postgres_resource_path = $2 OR postgres_resource_path LIKE $3)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": []
},
"hash": "e159b2ff15633f85e839ee4fe1ec2ecd11caf228ea8d0f52ad66def595644250"
}
backend/.sqlx/query-e2061df65ffd4a72146c4ca316829265289c8d6f625ac272655c88e1ad0b1745.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT datatable FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable",
"type_info": "Jsonb"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
true
]
},
"hash": "e2061df65ffd4a72146c4ca316829265289c8d6f625ac272655c88e1ad0b1745"
}
backend/.sqlx/query-ebaf3ed3097621da59dd201b5a4b9d1f440692f183c7c378f59e4b73f1c6e241.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET datatable = jsonb_set(\n jsonb_set(\n datatable #- ARRAY['datatables', $2, 'reference'],\n ARRAY['datatables', $2, 'database'], $3::jsonb),\n ARRAY['datatables', $2, 'forked_from'], $4::jsonb\n )\n WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Jsonb",
"Jsonb"
]
},
"nullable": []
},
"hash": "ebaf3ed3097621da59dd201b5a4b9d1f440692f183c7c378f59e4b73f1c6e241"
}
backend/.sqlx/query-f3ee09fb17955ca8d886f446d397063c4094546a7807343b570b823796372cef.json
Generated
+22
View File
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT workspace_id FROM workspace_settings WHERE datatable::text LIKE $1",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id",
"type_info": "Varchar"
}
],
"parameters": {
"Left": [
"Text"
]
},
"nullable": [
false
]
},
"hash": "f3ee09fb17955ca8d886f446d397063c4094546a7807343b570b823796372cef"
}
backend/.sqlx/query-f4adc9e26ebfebce18a29fb2c21bf06394cacb8a9699a608327b097e0ac1363e.json
Generated
+16
View File
@@ -0,0 +1,16 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE workspace_settings\n SET datatable = CASE WHEN $3::jsonb = 'null'::jsonb\n THEN datatable #- ARRAY['datatables', $2, 'permissions']\n ELSE jsonb_set(datatable, ARRAY['datatables', $2, 'permissions'], $3::jsonb)\n END\n WHERE workspace_id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Text",
"Jsonb"
]
},
"nullable": []
},
"hash": "f4adc9e26ebfebce18a29fb2c21bf06394cacb8a9699a608327b097e0ac1363e"
}
backend/.sqlx/query-fcb34e643b888122766e115a01394ab31ac856252aaa76c75ea27a447009c363.json
Generated
+17
View File
@@ -0,0 +1,17 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE datatable_role SET name = $2, enabled = $3, pwd = $4 WHERE id = $1",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Text",
"Varchar",
"Bool",
"Text"
]
},
"nullable": []
},
"hash": "fcb34e643b888122766e115a01394ab31ac856252aaa76c75ea27a447009c363"
}
backend/Cargo.lock
Generated
+1
View File
@@ -15470,6 +15470,7 @@ dependencies = [
"windmill-ai",
"windmill-alerting",
"windmill-api-auth",
"windmill-audit",
"windmill-common",
"windmill-object-store",
]
backend/ee-repo-ref.txt
+1 -1
View File
@@ -1 +1 @@
d252afcc80e77fcc4f9a2a346b80908c8605a6c0
7e338e4dabf91689bfd7fb0333c6534040b17b59
backend/migrations/20260908142148_datatable_role_catalog.down.sql
+18
View File
@@ -0,0 +1,18 @@
-- Refuse while the catalog holds anything. Each row is a live Postgres login with a password
-- only this table carries, so dropping it would leave credentials on the cluster that Windmill can
-- no longer disable, delete or even name — and re-applying could not recreate them, because the
-- role names would already be taken. Cleaning them up here is not an option either: dropping a
-- role means reassigning what it owns in *every* instance database, and a migration runs in one.
--
-- Delete the roles through instance settings first; that path does the cluster work.
LOCK TABLE datatable_role IN ACCESS EXCLUSIVE MODE;
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM datatable_role) THEN
RAISE EXCEPTION 'Cannot roll back: % data table role(s) still exist as Postgres logins. Delete them in instance settings first, which drops them from the cluster.',
(SELECT count(*) FROM datatable_role);
END IF;
END $$;
DROP TABLE IF EXISTS datatable_role;
backend/migrations/20260908142148_datatable_role_catalog.up.sql
+21
View File
@@ -0,0 +1,21 @@
-- The instance's data table role catalog: one row per Postgres login Windmill created for data
-- table access.
--
-- A table rather than a `global_settings` key, because the value is a set of live cluster
-- credentials and that table has generic read, list, write and CLI round-trip paths that know
-- nothing about what they are carrying. Every one of them is a way to leak the passwords or to
-- overwrite the catalog with a copy that has none, and a row nothing generic touches has none of
-- those. One row per role also makes two concurrent creates two inserts rather than a
-- read-modify-write over one document.
CREATE TABLE datatable_role (
id VARCHAR(50) PRIMARY KEY,
-- The Postgres role name, verbatim. Unique because it is the cluster's own key.
name VARCHAR(63) NOT NULL UNIQUE,
enabled BOOLEAN NOT NULL DEFAULT true,
-- Generated by Windmill, never entered by anyone, and never leaves the server.
pwd TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
GRANT ALL ON datatable_role TO windmill_user;
GRANT ALL ON datatable_role TO windmill_admin;
backend/parsers/windmill-parser/src/asset_parser.rs
+33 -1
View File
@@ -730,7 +730,12 @@ pub fn parse_asset_syntax(
s: &str,
enable_default_syntax: bool,
) -> Option<(AssetKind, Cow<'_, str>)> {
if enable_default_syntax && s == "datatable" {
// `datatable` and `datatable?role=analyst` both name the default data table: the role picks
// which Postgres login the connection is made as, not which data table is read.
if enable_default_syntax
&& s.strip_prefix("datatable")
.is_some_and(|rest| rest.is_empty() || rest.starts_with('?'))
{
return Some((AssetKind::DataTable, Cow::Borrowed("main")));
} else if enable_default_syntax && s == "ducklake" {
return Some((AssetKind::Ducklake, Cow::Borrowed("main")));
@@ -741,6 +746,14 @@ pub fn parse_asset_syntax(
if *kind == AssetKind::Dbt {
return Some((*kind, Cow::Owned(canonicalize_table_asset_path(suffix))));
}
// Same reasoning as above, for the explicit form. Specific to data tables: a
// `Resource`'s `?table=` is part of what it names, and stripping it would merge two
// different assets.
if *kind == AssetKind::DataTable {
if let Some((path, _role)) = suffix.split_once('?') {
return Some((*kind, Cow::Borrowed(path)));
}
}
// The suffix is kept verbatim. For S3 the path encodes the storage:
// `s3://<storage>/<key>`, with an EMPTY storage segment for the
// workspace default — so `s3:///key` yields `/key` (leading slash
@@ -1692,6 +1705,25 @@ fn parse_trigger_spec(s: &str) -> Option<TriggerSpec> {
mod pipeline_annotation_tests {
use super::*;
#[test]
fn a_datatable_role_is_not_part_of_the_asset_it_names() {
// The role picks which Postgres login the connection is made as, so two references that
// differ only by role are the same asset and must land on one graph node.
assert_eq!(
parse_asset_syntax("datatable://sales?role=analytics", false),
Some((AssetKind::DataTable, Cow::Borrowed("sales")))
);
assert_eq!(
parse_asset_syntax("datatable?role=analytics", true),
Some((AssetKind::DataTable, Cow::Borrowed("main")))
);
// A resource's `?table=` is part of what it names, so it is kept.
assert_eq!(
parse_asset_syntax("$res:f/db/pg?table=users", false),
Some((AssetKind::Resource, Cow::Borrowed("f/db/pg?table=users")))
);
}
#[test]
fn s3_path_keeps_storage_distinction() {
// An S3 asset path is `<storage>/<key>` with an empty storage segment
backend/windmill-api-groups/src/folders.rs
+8
View File
@@ -800,6 +800,14 @@ async fn delete_folder(
not_found_if_none(get_folderopt(&mut tx, &w_id, &name).await?, "Folder", &name)?;
// See the same call in `delete_group`: a freed name must not stay in a tenant list.
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
&mut tx,
&w_id,
&format!("f/{name}"),
)
.await?;
let del = sqlx::query_scalar!(
"DELETE FROM folder WHERE name = $1 AND workspace_id = $2 RETURNING 1",
name,
backend/windmill-api-groups/src/groups.rs
+9
View File
@@ -797,6 +797,15 @@ async fn delete_group(
}
not_found_if_none(get_group_opt(&mut tx, &w_id, &name).await?, "Group", &name)?;
// A tenant list names a principal, so a freed name must not linger in one: a later group
// reusing it would silently inherit the data table access this one had.
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
&mut tx,
&w_id,
&format!("g/{name}"),
)
.await?;
sqlx::query!(
"DELETE FROM usr_to_group WHERE group_ = $1 AND workspace_id = $2",
name,
backend/windmill-api-integration-tests/tests/datatable_roles.rs
+1247
View File
File diff suppressed because it is too large Load Diff
backend/windmill-api-integration-tests/tests/fixtures/datatable_roles.sql
Vendored
+45
View File
@@ -0,0 +1,45 @@
-- A data table under roles in `test-workspace`, and a fork whose entry points at it rather than
-- carrying a copy. `test-user-2` is a non-admin of the parent and an admin of the fork: the shape
-- the pointer exists for.
-- Empty registry: role provisioning grants CONNECT on every database named here, and the data
-- table's `dt_main` is a name in workspace settings, not a database that exists.
INSERT INTO global_settings (name, value) VALUES
('custom_instance_pg_databases', '{"user_pwd": "pw", "databases": {}}'::jsonb)
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value;
INSERT INTO datatable_role (id, name, enabled, pwd) VALUES ('role1', 'analytics', true, 'pw');
UPDATE workspace_settings SET datatable = '{
"datatables": {
"main": {
"database": {"resource_type": "instance", "resource_path": "dt_main"},
"permissions": {
"default_role": "role1",
"roles": {
"admin": {"tenants": []},
"role1": {"tenants": ["u/test-user-2", "g/analysts", "f/finance"]}
}
}
}
}
}'::jsonb WHERE workspace_id = 'test-workspace';
INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES
('test-workspace', 'analysts', 'Analysts', '{}');
INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms) VALUES
('test-workspace', 'finance', 'finance', '{}', '{}');
INSERT INTO workspace (id, name, owner, parent_workspace_id) VALUES
('wm-fork-dt', 'fork of test-workspace', 'test2@windmill.dev', 'test-workspace');
INSERT INTO workspace_key (workspace_id, kind, key) VALUES ('wm-fork-dt', 'cloud', 'test-key');
INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES
('wm-fork-dt', 'all', 'All users', '{}');
INSERT INTO usr (workspace_id, email, username, is_admin, role) VALUES
('wm-fork-dt', 'test2@windmill.dev', 'test-user-2', true, 'Admin');
INSERT INTO workspace_settings (workspace_id, datatable) VALUES ('wm-fork-dt', '{
"datatables": {
"main": {"reference": {"workspace_id": "test-workspace", "datatable": "main"}}
}
}'::jsonb);
backend/windmill-api-settings/Cargo.toml
+2 -1
View File
@@ -11,7 +11,7 @@ path = "src/lib.rs"
[features]
default = []
enterprise = ["license"]
private = ["windmill-common/private"]
private = ["windmill-common/private", "windmill-audit/private"]
parquet = ["windmill-common/parquet", "windmill-object-store/parquet"]
license = ["dep:rsa"]
@@ -19,6 +19,7 @@ license = ["dep:rsa"]
windmill-ai = { workspace = true, default-features = false }
windmill-alerting.workspace = true
windmill-api-auth.workspace = true
windmill-audit.workspace = true
windmill-common = { workspace = true, default-features = false }
axum.workspace = true
anyhow.workspace = true
backend/windmill-api-settings/src/datatable_roles_oss.rs
+44
View File
@@ -0,0 +1,44 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Where the data table role catalog endpoints come from: the enterprise implementation, or a
//! refusal. Roles are an Enterprise Edition feature; see `windmill_common::datatable_roles_oss`.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_roles_ee::{
create_datatable_role, delete_datatable_role, list_datatable_roles, update_datatable_role,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) use ce::*;
// The routes stay registered so the API has one shape; each answers after authentication, before
// anything is read.
#[cfg(not(all(feature = "private", feature = "enterprise")))]
mod ce {
use windmill_api_auth::ApiAuthed;
use windmill_common::{
datatable_roles_oss::datatable_roles_unavailable as unavailable, error::Result,
};
pub(crate) async fn list_datatable_roles(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) async fn create_datatable_role(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) async fn update_datatable_role(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) async fn delete_datatable_role(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
}
backend/windmill-api-settings/src/lib.rs
+13
View File
@@ -17,6 +17,9 @@ mod audit_logs_s3;
mod audit_logs_s3_backfill;
#[cfg(feature = "parquet")]
mod background_task;
#[cfg(all(feature = "private", feature = "enterprise"))]
mod datatable_roles_ee;
mod datatable_roles_oss;
#[cfg(feature = "private")]
mod ee;
pub mod ee_oss;
@@ -151,6 +154,16 @@ pub fn global_service() -> Router {
"/list_custom_instance_pg_databases",
post(list_custom_instance_pg_databases),
)
.route(
"/datatable_roles",
get(datatable_roles_oss::list_datatable_roles)
.post(datatable_roles_oss::create_datatable_role),
)
.route(
"/datatable_roles/{id}",
post(datatable_roles_oss::update_datatable_role)
.delete(datatable_roles_oss::delete_datatable_role),
)
.route(
"/refresh_custom_instance_user_pwd",
post(refresh_custom_instance_user_pwd),
backend/windmill-api-users/src/users.rs
+29 -3
View File
@@ -1703,14 +1703,25 @@ async fn delete_user(
.await?;
windmill_common::user_drafts::delete_drafts_of_email(&mut *tx, &email_to_delete).await?;
let usernames = sqlx::query_scalar!(
"DELETE FROM usr WHERE email = $1 RETURNING username",
let memberships = sqlx::query!(
"DELETE FROM usr WHERE email = $1 RETURNING username, workspace_id",
&email_to_delete
)
.fetch_all(&mut *tx)
.await?;
for username in usernames {
for row in memberships {
let username = row.username;
// A tenant list names a principal of its workspace, so the name has to be freed in every
// workspace this account belonged to: a later account taking the username would otherwise
// inherit the data table access it had.
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
&mut tx,
&row.workspace_id,
&format!("u/{username}"),
)
.await?;
sqlx::query!("DELETE FROM password WHERE email = $1", &email_to_delete)
.execute(&mut *tx)
.await?;
@@ -2456,6 +2467,15 @@ pub async fn delete_workspace_user_internal(
tx: &mut Transaction<'_, Postgres>,
authed: Option<&ApiAuthed>, // None for system operations
) -> Result<()> {
// Same reasoning as the `extra_perms` sweep below: a freed username must not stay named
// anywhere that grants access, tenant lists included.
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
tx,
w_id,
&format!("u/{username_to_delete}"),
)
.await?;
// ---- Clean up extra_perms referencing this user ----
let extra_perms_tables = [
"script",
@@ -3965,6 +3985,12 @@ async fn leave_workspace(
) -> Result<String> {
forbid_job_token_account_destruction(&authed)?;
let mut tx = db.begin().await?;
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
&mut tx,
&w_id,
&format!("u/{}", authed.username),
)
.await?;
sqlx::query!(
"DELETE FROM usr WHERE workspace_id = $1 AND username = $2",
&w_id,
backend/windmill-api-workspaces/src/datatable_migrations.rs
+119 -6
View File
@@ -30,6 +30,7 @@ use windmill_api_auth::{require_super_admin, ApiAuthed};
use windmill_api_jobs::run_wait_result_internal;
use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
use windmill_common::datatable_roles::ADMIN_DATATABLE_ROLE;
use windmill_common::db::UserDB;
use windmill_common::error::{pg_error_message, Error, JsonResult, Result};
use windmill_common::jobs::{JobPayload, RawCode};
@@ -38,7 +39,11 @@ use windmill_common::runnable_settings::{ConcurrencySettingsWithCustom, Debounci
use windmill_common::scripts::ScriptLang;
use windmill_common::users::username_to_permissioned_as;
use windmill_common::worker::to_raw_value;
use windmill_common::workspaces::get_datatable_resource_from_db_unchecked;
use windmill_common::worker::SqlAnnotations;
use windmill_common::workspaces::{
ensure_can_use_datatable_role, ensure_datatable_admin_access,
get_datatable_resource_from_db_unchecked, resolve_governing_datatable, DatatableAccess,
};
use windmill_common::{PgDatabase, DB};
use windmill_git_sync::{
handle_deployment_metadata, handle_deployment_metadata_batch, DeployedObject,
@@ -86,6 +91,42 @@ pub(crate) fn routes() -> Router {
)
}
/// Refuse a migration whose role this caller may not use, before a job is pushed or a version
/// recorded.
///
/// A migration that declares `-- role <name>` runs as that role, so the caller has to be one of its
/// tenants. One that declares none runs as `admin` and reaches every object in the database
/// whatever the roles grant, so it is for the admins of the workspace that governs the data table
/// — a fork can run a migration under a role it holds, never a migration under `admin`.
///
/// The executor re-checks the role when it resolves the connection, so this is not the boundary. It
/// is what makes the refusal legible: which migration, and which role.
async fn ensure_migration_role_allowed(
db: &DB,
w_id: &str,
datatable_name: &str,
authed: &ApiAuthed,
sql: &str,
timestamp: i64,
name: &str,
) -> Result<()> {
let context = format!("Migration {timestamp} ({name})");
let access = DatatableAccess::Authed(authed.to_authed_ref());
match SqlAnnotations::datatable_role(sql)? {
Some(role) => {
ensure_can_use_datatable_role(db, w_id, datatable_name, Some(&role), &access, &context)
.await
}
None => ensure_datatable_admin_access(db, w_id, datatable_name, &access)
.await
.map_err(|e| {
Error::NotAuthorized(format!(
"{context} declares no role, so it would run as admin. {e}"
))
}),
}
}
#[derive(Serialize)]
struct AppliedMigration {
version: i64,
@@ -128,7 +169,18 @@ async fn datatable_database_arg(
.await?
.ok_or_else(|| Error::internal_err(format!("datatable {datatable_name} not found")))?;
Ok(to_raw_value(&format!("datatable://{datatable_name}")))
// `?role=admin` rather than a bare reference, so a migration that declares no `-- role` runs
// as the connection that owns the schema instead of falling through to the data table's
// default role — which is what `ensure_migration_role_allowed` gated it as, and which is the
// only role a DDL statement can be expected to succeed under. A migration that does declare a
// role overrides this: the annotation wins over the reference.
//
// A legacy name containing `?` cannot be migrated through this reference: the appended query
// makes it neither an exact name nor a parseable one. Accepted on purpose, since such names can
// no longer be created and none are expected to carry migrations.
Ok(to_raw_value(&format!(
"datatable://{datatable_name}?role={ADMIN_DATATABLE_ROLE}"
)))
}
/// Run a migration's SQL as a normal Windmill `postgresql` job, permissioned as
@@ -384,6 +436,11 @@ async fn run_datatable_migrations(
Path((w_id, datatable_name)): Path<(String, String)>,
Query(query): Query<RunDatatableMigrationsQuery>,
) -> JsonResult<RunDatatableMigrationsResult> {
// Before the admin connection is opened at all: the bookkeeping below is created and read
// through it, so a caller no role covers must be refused here rather than after the fact.
crate::datatable_permissions::ensure_reaches_datatable(&db, &w_id, &datatable_name, &authed)
.await?;
audit_log(
&db,
&authed,
@@ -440,6 +497,16 @@ async fn run_datatable_migrations(
if applied_versions.contains(&m.timestamp) {
continue;
}
ensure_migration_role_allowed(
&db,
&w_id,
&datatable_name,
&authed,
&m.code_up,
m.timestamp,
&m.name,
)
.await?;
run_datatable_migration_job(&db, &user_db, &authed, &w_id, &database_arg, &m.code_up)
.await
.map_err(|e| {
@@ -506,6 +573,11 @@ async fn rollback_datatable_migrations(
Path((w_id, datatable_name)): Path<(String, String)>,
Query(query): Query<RollbackDatatableMigrationsQuery>,
) -> JsonResult<RollbackDatatableMigrationsResult> {
// Before the admin connection is opened at all: the bookkeeping below is created and read
// through it, so a caller no role covers must be refused here rather than after the fact.
crate::datatable_permissions::ensure_reaches_datatable(&db, &w_id, &datatable_name, &authed)
.await?;
audit_log(
&db,
&authed,
@@ -588,6 +660,17 @@ async fn rollback_datatable_migrations(
))
})?;
ensure_migration_role_allowed(
&db,
&w_id,
&datatable_name,
&authed,
&code_down,
version,
&definition.name,
)
.await?;
let database_arg = datatable_database_arg(&db, &w_id, &datatable_name).await?;
run_datatable_migration_job(&db, &user_db, &authed, &w_id, &database_arg, &code_down)
.await
@@ -748,10 +831,15 @@ async fn read_applied_datatable_versions(
/// List a data table's migrations annotated with whether each has been applied.
async fn datatable_migrations_status(
_authed: ApiAuthed,
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path((w_id, datatable_name)): Path<(String, String)>,
) -> JsonResult<DatatableMigrationsStatusResult> {
// Reads `_wm_migrations` through the data table's admin connection, so it answers to the same
// question as running one: may you reach this data table at all.
crate::datatable_permissions::ensure_reaches_datatable(&db, &w_id, &datatable_name, &authed)
.await?;
let enabled = datatable_migrations_enabled(&db, &w_id, &datatable_name).await?;
if !enabled {
return Ok(Json(DatatableMigrationsStatusResult {
@@ -1431,6 +1519,15 @@ async fn generate_initial_datatable_migration(
Extension(db): Extension<DB>,
Path((w_id, datatable_name)): Path<(String, String)>,
) -> JsonResult<DatatableMigration> {
// Returns a `pg_dump` of the whole schema and writes into the data table's own bookkeeping, so
// it answers to the workspace that governs it rather than to whoever is asking.
ensure_datatable_admin_access(
&db,
&w_id,
&datatable_name,
&DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
validate_datatable_path_segment(&datatable_name)?;
ensure_datatable_migrations_enabled(&db, &w_id, &datatable_name).await?;
@@ -1601,9 +1698,21 @@ pub(crate) struct DatatableRename {
pub(crate) to: String,
}
async fn resolve_datatable_pg(db: &DB, w_id: &str, datatable: &str) -> Result<PgDatabase> {
/// The database whose `_wm_migrations` a rename or delete of `datatable` in `w_id` should touch —
/// `None` when that is somebody else's.
///
/// A fork's entry points at the workspace that governs the data table, so renaming or removing it
/// changes what the fork calls the data table and nothing more. Following the pointer here would
/// let a fork admin relabel or wipe the *governing* workspace's migration bookkeeping through
/// their own settings form, and the parent would then re-run every migration from zero.
async fn resolve_datatable_pg(db: &DB, w_id: &str, datatable: &str) -> Result<Option<PgDatabase>> {
let governing = resolve_governing_datatable(db, w_id, datatable).await?;
if governing.workspace_id != w_id {
return Ok(None);
}
let db_resource = get_datatable_resource_from_db_unchecked(db, w_id, datatable).await?;
serde_json::from_value(db_resource)
.map(Some)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
}
@@ -1621,7 +1730,9 @@ fn ignore_missing_wm_migrations(e: tokio_postgres::Error) -> Result<()> {
/// Drop a data table's rows from its own database's `_wm_migrations`.
async fn remote_forget_datatable_migrations(db: &DB, w_id: &str, datatable: &str) -> Result<()> {
let pg_db = resolve_datatable_pg(db, w_id, datatable).await?;
let Some(pg_db) = resolve_datatable_pg(db, w_id, datatable).await? else {
return Ok(());
};
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
let _ = connection.await;
@@ -1646,7 +1757,9 @@ async fn remote_rename_datatable_migrations(
from: &str,
to: &str,
) -> Result<()> {
let pg_db = resolve_datatable_pg(db, w_id, resolve_by).await?;
let Some(pg_db) = resolve_datatable_pg(db, w_id, resolve_by).await? else {
return Ok(());
};
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
let _ = connection.await;
backend/windmill-api-workspaces/src/datatable_permissions.rs
+67
View File
@@ -0,0 +1,67 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Who may connect to a data table as which role.
//!
//! The decision lives on the data table entry of the workspace that governs it, which is not
//! necessarily the workspace asking: a fork's entry points at its parent's, and everything here
//! resolves through that pointer first. Nothing in this module runs SQL against the data table —
//! a save is tenant lists and a default, and the Postgres roles themselves are the instance
//! catalog's business.
use axum::{routing::get, Router};
use windmill_api_auth::ApiAuthed;
use windmill_common::error::Result;
use windmill_common::workspaces::GoverningDatatable;
use windmill_common::DB;
use crate::datatable_permissions_oss as roles;
pub(crate) fn routes() -> Router {
Router::new()
.route(
"/datatable_permissions/{datatable_name}",
get(roles::get_datatable_permissions).post(roles::set_datatable_permissions),
)
.route(
"/datatable_usable_roles/{datatable_name}",
get(roles::list_usable_datatable_roles),
)
}
/// Administering a data table — its permissions, its migrations that declare no role, its exports
/// — is for the admins of the workspace that governs it. A fork can use the data table; it never
/// administers it.
// The gate for whatever administers a data table under roles, which the routes of this module alone
// do not always reach.
#[allow(dead_code)]
pub(crate) async fn ensure_governs_datatable(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
governing: &GoverningDatatable,
) -> Result<()> {
roles::ensure_governs_datatable(db, authed, w_id, governing).await
}
/// Refuse a caller that no tenant of this data table covers.
///
/// The bookkeeping endpoints below open the data table's `admin` connection to read or create
/// `_wm_migrations` before they know which migration will run — so without this, someone covered
/// by no role at all can still force admin-backed reads and writes on a database they may not
/// touch. It asks only "may you reach this data table as anything"; which role a given migration
/// runs as is still decided per migration, and by the executor after that.
pub(crate) async fn ensure_reaches_datatable(
db: &DB,
w_id: &str,
datatable_name: &str,
authed: &ApiAuthed,
) -> Result<()> {
roles::ensure_reaches_datatable(db, w_id, datatable_name, authed).await
}
backend/windmill-api-workspaces/src/datatable_permissions_oss.rs
+73
View File
@@ -0,0 +1,73 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Where the data table permissions endpoints and their gates come from: the enterprise
//! implementation, or a refusal. Roles are an Enterprise Edition feature; see
//! `windmill_common::datatable_roles_oss`.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_permissions_ee::{
ensure_governs_datatable, ensure_reaches_datatable, get_datatable_permissions,
list_usable_datatable_roles, set_datatable_permissions,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) use ce::*;
#[cfg(not(all(feature = "private", feature = "enterprise")))]
mod ce {
use windmill_api_auth::ApiAuthed;
use windmill_common::{
datatable_roles_oss::datatable_roles_unavailable as unavailable,
error::Result,
workspaces::{resolve_governing_datatable, GoverningDatatable},
DB,
};
/// Nobody administers a data table's roles without them.
#[allow(dead_code)]
pub(crate) async fn ensure_governs_datatable(
_db: &DB,
_authed: &ApiAuthed,
_w_id: &str,
_governing: &GoverningDatatable,
) -> Result<()> {
Err(unavailable())
}
/// A data table not under roles is reached as it was before roles existed. One under roles is
/// refused: no role of it can be connected as.
pub(crate) async fn ensure_reaches_datatable(
db: &DB,
w_id: &str,
datatable_name: &str,
_authed: &ApiAuthed,
) -> Result<()> {
let governing = resolve_governing_datatable(db, w_id, datatable_name).await?;
if governing.datatable.permissions.is_none() {
Ok(())
} else {
Err(unavailable())
}
}
// The routes stay registered so the API has one shape; each answers after authentication,
// before anything is read.
pub(crate) async fn get_datatable_permissions(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) async fn set_datatable_permissions(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) async fn list_usable_datatable_roles(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
}
backend/windmill-api-workspaces/src/lib.rs
+5
View File
@@ -2,6 +2,8 @@
pub mod ai_session_backups;
pub mod data_metrics;
pub mod datatable_migrations;
pub mod datatable_permissions;
pub mod datatable_permissions_oss;
pub mod deployment_requests;
pub mod workspaces;
pub mod workspaces_extra;
@@ -9,3 +11,6 @@ pub mod workspaces_oss;
#[cfg(feature = "private")]
pub mod workspaces_ee;
#[cfg(all(feature = "private", feature = "enterprise"))]
pub mod datatable_permissions_ee;
backend/windmill-api-workspaces/src/workspaces.rs
+616 -80
View File
@@ -45,10 +45,12 @@ use windmill_common::workspaces::GitRepositorySettings;
#[cfg(feature = "enterprise")]
use windmill_common::workspaces::WorkspaceDeploymentUISettings;
use windmill_common::workspaces::{
check_deploy_rules, check_user_against_rule, get_datatable_resource_from_db_unchecked,
check_deploy_rules, check_user_against_rule, get_datatable_resource_from_db,
get_datatable_resource_from_db_unchecked, parse_datatable_ref_for, resolve_governing_datatable,
validate_dev_workspace_id, validate_fork_workspace_id, validate_workspace_name, DataTable,
DataTableCatalogResourceType, DataTableForkBehavior, ProtectionRuleKind, ProtectionRules,
ProtectionRuleset, RuleCheckResult, WorkspaceGitSyncSettings, DEV_WORKSPACE_LOCK_RULE_NAME,
DataTableCatalogResourceType, DataTableForkBehavior, DatatableAccess, GoverningDatatable,
ProtectionRuleKind, ProtectionRules, ProtectionRuleset, RuleCheckResult,
WorkspaceGitSyncSettings, DEV_WORKSPACE_LOCK_RULE_NAME,
};
use windmill_common::workspaces::{Ducklake, DucklakeCatalogResourceType};
use windmill_common::PgDatabase;
@@ -141,6 +143,7 @@ pub fn workspaced_service() -> Router {
get(test_datatable_connection),
)
.merge(crate::datatable_migrations::routes())
.merge(crate::datatable_permissions::routes())
.route("/git_sync_enabled", get(get_git_sync_enabled))
.route("/git_sync_deploy_mode", get(get_git_sync_deploy_mode))
.route("/edit_git_sync_config", post(edit_git_sync_config))
@@ -1140,6 +1143,8 @@ async fn get_settings(
if let Some(git_sync) = settings.git_sync.as_mut() {
redact_git_sync_webhook_secrets(git_sync);
}
settings.datatable =
windmill_common::workspaces::strip_datatable_permissions(settings.datatable.take());
Ok(Json(settings))
}
@@ -1176,8 +1181,10 @@ async fn get_public_settings(
.await
.map_err(|e| Error::internal_err(format!("getting public settings: {e:#}")))?;
let settings = not_found_if_none(settings, "workspace settings", &w_id)?;
let mut settings = not_found_if_none(settings, "workspace settings", &w_id)?;
tx.commit().await?;
settings.datatable =
windmill_common::workspaces::strip_datatable_permissions(settings.datatable.take());
Ok(Json(settings))
}
@@ -2184,6 +2191,12 @@ struct DataTableListItem {
name: String,
resource_type: String,
resource_path: String,
/// The workspace whose entry governs this one, when it is not this workspace — a fork pointing
/// at its parent. Its permissions apply here, and only its admins may edit them.
#[serde(skip_serializing_if = "Option::is_none")]
governing_workspace_id: Option<String>,
/// Whether the governing entry is under roles.
permissioned: bool,
}
async fn list_datatables(
@@ -2191,26 +2204,29 @@ async fn list_datatables(
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<DataTableListItem>> {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables' FROM workspace_settings WHERE workspace_id = $1",
&w_id
)
.fetch_one(&db)
.await?;
// A pointer entry owns no database, so what it resolves to is the only truthful answer here.
// One that resolves to nothing — a pointer whose workspace was deleted — is dropped rather than
// listed with a database it does not have; what happened is named where it is actionable
// instead: by the delete that stranded it, and by any attempt to use it.
let resolved =
windmill_common::workspaces::resolve_workspace_governing_datatables(&db, &w_id).await?;
let items: Vec<DataTableListItem> = match config {
Some(val) => {
let map: HashMap<String, DataTable> = serde_json::from_value(val).unwrap_or_default();
map.into_iter()
.map(|(name, dt)| DataTableListItem {
name,
resource_type: dt.database.resource_type.as_ref().to_string(),
resource_path: dt.database.resource_path,
})
.collect()
}
None => vec![],
};
let mut items = Vec::with_capacity(resolved.len());
for (name, governing) in resolved {
let database = governing
.datatable
.database
.as_ref()
.expect("a governing entry owns a database");
items.push(DataTableListItem {
name,
resource_type: database.resource_type.as_ref().to_string(),
resource_path: database.resource_path.clone(),
governing_workspace_id: (governing.workspace_id != w_id)
.then(|| governing.workspace_id.clone()),
permissioned: governing.datatable.permissions.is_some(),
});
}
Ok(Json(items))
}
@@ -2298,6 +2314,15 @@ async fn test_datatable_connection(
Path((w_id, datatable_name)): Path<(String, String)>,
) -> JsonResult<DataTableConnectionCheck> {
require_admin(authed.is_admin, &authed.username)?;
// Reports what the admin connection can do, so it answers to the workspace that governs the
// data table rather than to whichever one is asking.
windmill_common::workspaces::ensure_datatable_admin_access(
&db,
&w_id,
&datatable_name,
&DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
let db_resource = get_datatable_resource_from_db_unchecked(&db, &w_id, &datatable_name).await?;
let pg_db: PgDatabase = serde_json::from_value(db_resource)
@@ -2385,7 +2410,7 @@ async fn test_datatable_connection(
}
async fn list_datatable_schemas(
_authed: ApiAuthed,
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<DataTableSchema>> {
@@ -2393,7 +2418,7 @@ async fn list_datatable_schemas(
let mut results = Vec::new();
for datatable_name in datatable_names {
let schema = match get_datatable_schema(&db, &w_id, &datatable_name).await {
let schema = match get_datatable_schema(&db, &authed, &w_id, &datatable_name).await {
Ok(schemas) => DataTableSchema { datatable_name, schemas, error: None },
Err(e) => DataTableSchema {
datatable_name,
@@ -2408,7 +2433,7 @@ async fn list_datatable_schemas(
}
async fn list_datatable_tables(
_authed: ApiAuthed,
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<DataTableTables>> {
@@ -2416,7 +2441,7 @@ async fn list_datatable_tables(
let mut results = Vec::new();
for datatable_name in datatable_names {
let tables = match get_datatable_tables(&db, &w_id, &datatable_name).await {
let tables = match get_datatable_tables(&db, &authed, &w_id, &datatable_name).await {
Ok(schemas) => DataTableTables { datatable_name, schemas, error: None },
Err(e) => DataTableTables {
datatable_name,
@@ -2431,13 +2456,14 @@ async fn list_datatable_tables(
}
async fn get_datatable_table_schema(
_authed: ApiAuthed,
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Query(query): Query<GetDataTableSchemaQuery>,
) -> JsonResult<DataTableTableSchema> {
let columns = get_datatable_table_columns(
&db,
&authed,
&w_id,
&query.datatable_name,
&query.schema_name,
@@ -2469,13 +2495,39 @@ async fn list_datatable_names(db: &DB, w_id: &str) -> Result<Vec<String>> {
.collect())
}
async fn get_datatable_schema(db: &DB, w_id: &str, datatable_name: &str) -> Result<SchemaMap> {
// Get the datatable resource (connection credentials)
let db_resource = get_datatable_resource_from_db_unchecked(db, w_id, datatable_name).await?;
/// Connect to a data table as the caller, not as `admin`: the role they named, or the data table's
/// default. A data table not under roles resolves as `admin`, exactly as it did before roles.
///
/// Every schema-browsing query below then reports what this Postgres role can actually reach,
/// which is why they filter on `has_schema_privilege` — `pg_catalog` is world-readable, so an
/// unfiltered listing would name schemas the connection cannot even enter.
async fn resolve_datatable_pg_as_caller(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
) -> Result<PgDatabase> {
let db_resource = get_datatable_resource_from_db(
db,
w_id,
datatable_name,
// The data table's default role. Browsing has no way to name another one yet; when the
// database manager grows a role picker it passes the pick through here.
None,
DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
}
// Parse the resource as PgDatabase
let pg_db: PgDatabase = serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))?;
async fn get_datatable_schema(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
) -> Result<SchemaMap> {
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
// Connect to the datatable database
let (client, connection) = pg_db.connect(Some(db)).await?;
@@ -2495,6 +2547,7 @@ async fn get_datatable_schema(db: &DB, w_id: &str, datatable_name: &str) -> Resu
FROM pg_namespace
WHERE nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')
AND nspname NOT LIKE 'pg_%'
AND has_schema_privilege(oid, 'USAGE')
ORDER BY nspname
"#,
&[],
@@ -2562,10 +2615,13 @@ async fn get_datatable_schema(db: &DB, w_id: &str, datatable_name: &str) -> Resu
Ok(schema_map)
}
async fn get_datatable_tables(db: &DB, w_id: &str, datatable_name: &str) -> Result<TableListMap> {
let db_resource = get_datatable_resource_from_db_unchecked(db, w_id, datatable_name).await?;
let pg_db: PgDatabase = serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))?;
async fn get_datatable_tables(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
) -> Result<TableListMap> {
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
@@ -2581,6 +2637,7 @@ async fn get_datatable_tables(db: &DB, w_id: &str, datatable_name: &str) -> Resu
FROM pg_namespace
WHERE nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')
AND nspname NOT LIKE 'pg_%'
AND has_schema_privilege(oid, 'USAGE')
ORDER BY nspname
"#,
&[],
@@ -2629,6 +2686,7 @@ async fn get_datatable_tables(db: &DB, w_id: &str, datatable_name: &str) -> Resu
async fn get_datatable_table_columns(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
schema_name: &str,
@@ -2641,9 +2699,7 @@ async fn get_datatable_table_columns(
)));
}
let db_resource = get_datatable_resource_from_db_unchecked(db, w_id, datatable_name).await?;
let pg_db: PgDatabase = serde_json::from_value(db_resource)
.map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))?;
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
@@ -2853,7 +2909,10 @@ mod tests {
}
/// Resolve a source string to PgDatabase credentials with user-scoped permission checks.
/// For `datatable://name`: accessible to everyone (variables are resolved internally).
///
/// For `datatable://name`: the **admin** connection, so it is gated on admin reach. Every caller
/// copies, dumps or drops a whole database, and a dump taken under a restricted role would be a
/// silently truncated copy rather than an error — which is worse than refusing.
/// For `$res:path`: uses UserDB (row-level security) to verify the user can see the resource,
/// then interpolates `$var:` references in the resource value.
pub(crate) async fn resolve_pg_source_checked(
@@ -2864,6 +2923,13 @@ pub(crate) async fn resolve_pg_source_checked(
source: &str,
) -> Result<PgDatabase> {
let db_resource = if let Some(name) = source.strip_prefix("datatable://") {
windmill_common::workspaces::ensure_datatable_admin_access(
db,
w_id,
name,
&DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
get_datatable_resource_from_db_unchecked(db, w_id, name).await?
} else if let Some(path) = source.strip_prefix("$res:") {
let db_with_authed = windmill_common::db::DbWithOptAuthed::from_authed(
@@ -2904,22 +2970,13 @@ pub(crate) async fn resolve_pg_source_checked(
/// Whether the data table `name` is backed by the Windmill instance's own PostgreSQL
/// rather than a user resource.
pub(crate) async fn is_instance_datatable(db: &DB, w_id: &str, name: &str) -> Result<bool> {
let config = sqlx::query_scalar!(
"SELECT datatable->'datatables'->$2 FROM workspace_settings WHERE workspace_id = $1",
w_id,
name
)
.fetch_optional(db)
.await?
.flatten();
Ok(config
.and_then(|v| {
v.get("database")
.and_then(|d| d.get("resource_type"))
.and_then(|r| r.as_str())
.map(|s| s == "instance")
})
.unwrap_or(false))
// Resolved rather than read: a pointer entry owns no database of its own, so only the entry it
// lands on can answer. A name that resolves to nothing keeps the historical `false`.
Ok(resolve_governing_datatable(db, w_id, name)
.await
.ok()
.and_then(|g| g.datatable.database)
.is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance))
}
/// Same, for the `datatable://<name>` / `$res:<path>` form the import endpoints take.
@@ -3250,6 +3307,14 @@ async fn create_pg_database(
) -> Result<String> {
windmill_common::validate_dbname(&req.target_dbname)?;
// The copy this database is for is refused a call later, and nothing collects an instance
// database that no data table entry names. Refuse here too, so the clone stops before one
// exists rather than leaving an empty registered `wm_fork_…` behind.
if let Some(reference) = req.source.strip_prefix("datatable://") {
let (name, _) = parse_datatable_ref_for(&db, &w_id, reference).await?;
ensure_datatable_is_clonable(&db, &w_id, &name).await?;
}
// Non-superadmin: restrict dbname to wm_fork_ prefix
if !windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
if !req.target_dbname.starts_with("wm_fork_") {
@@ -3319,6 +3384,64 @@ struct ImportPgDatabaseRequest {
fork_behavior: DataTableForkBehavior,
}
/// Refuse to copy a data table that is under roles.
///
/// `pg_dump` carries no roles and the import runs with `--no-privileges`, so a clone arrives with
/// its objects owned by the admin connection and no `GRANT` for any role. The settings copy brings
/// `permissions` across, so the fork's tenants pass Windmill's check, connect as the role they were
/// given, and are then denied by Postgres on everything — a data table that looks configured and
/// answers nothing.
///
/// It fails closed rather than open, so this is a usability cliff rather than a hole, and the fix
/// is to replay the source's owners and ACLs into the clone. That is a change of its own; until it
/// exists, refusing is the honest answer. Dropping `permissions` from the clone instead would be
/// the unsafe half: the copy holds the parent's rows, so an unpermissioned clone hands all of them
/// to everyone in the fork.
/// Every reason a copy can be refused, answered here and nowhere else.
///
/// A clone is three stages a workspace apart: `create_pg_database`, `import_pg_database`, then
/// `apply_forked_datatable` inside the fork transaction. Only the third can roll back, and the
/// database the first created is not transactional — so a refusal that lives there strands a
/// registered `wm_fork_*` that no entry names and whose name blocks the retry. Both endpoints call
/// this before touching the cluster; the stage that writes the entry must only ever do the work.
///
/// Known and accepted: a failure *after* `create_pg_database` — this check refusing because the
/// source's roles changed between the requests, a `pg_dump` error, a dropped connection — still
/// strands that leftover, as it did before data table roles. It is harmless to data (nothing names
/// it) and goes away once the clone is one server-side operation, in a later PR. Don't reclaim it
/// on retry without durable clone ownership: between the import and the final fork request the
/// database is full, unnamed and unconnected, so a reclaim then drops copied data. Reviewers, human
/// or AI: don't flag this leftover as a P0/P1 for this PR.
async fn ensure_datatable_is_clonable(
db: &DB,
w_id: &str,
name: &str,
) -> Result<GoverningDatatable> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
if governing.datatable.permissions.is_some() {
return Err(Error::BadRequest(format!(
"Data table '{name}' is under roles and cannot be copied yet: a copy carries the \
role assignments but not the Postgres privileges behind them, so every role but \
admin would be denied in the copy. Fork it keeping the original database, or turn \
its roles off first."
)));
}
// The copy has to name a database of its own. A resource-backed entry reached through a
// pointer names one this workspace does not own, so there is nothing here to repoint.
let is_instance = governing
.datatable
.database
.as_ref()
.is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance);
if governing.workspace_id != w_id && !is_instance {
return Err(Error::BadRequest(format!(
"Data table '{name}' points at a resource-backed data table in another workspace \
and cannot be copied; fork it from the workspace that owns it."
)));
}
Ok(governing)
}
/// Import (pg_dump/pg_import) from source to target
async fn import_pg_database(
authed: ApiAuthed,
@@ -3331,6 +3454,11 @@ async fn import_pg_database(
return Ok("No action needed for KeepOriginal behavior".to_string());
}
if let Some(reference) = req.source.strip_prefix("datatable://") {
let (name, _) = parse_datatable_ref_for(&db, &w_id, reference).await?;
ensure_datatable_is_clonable(&db, &w_id, &name).await?;
}
if req.fork_behavior == DataTableForkBehavior::SchemaAndData {
require_admin(authed.is_admin, &authed.username)?;
if *CLOUD_HOSTED {
@@ -3524,24 +3652,44 @@ async fn edit_ducklake_config(
Ok(format!("Edit ducklake config for workspace {}", &w_id))
}
/// What a save left behind. `stranded_references` names the data tables in other workspaces that
/// were governed by one this save deleted — a field rather than a sentence in a success string,
/// so the UI decides whether to warn on the data rather than on the server's prose.
#[derive(Serialize)]
pub struct EditDataTableConfigResult {
#[serde(skip_serializing_if = "Vec::is_empty")]
stranded_references: Vec<StrandedReference>,
}
#[derive(Serialize)]
pub struct StrandedReference {
workspace_id: String,
datatable: String,
}
async fn edit_datatable_config(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
ApiAuthed { is_admin, username, .. }: ApiAuthed,
Json(mut new_config): Json<EditDataTableConfig>,
) -> Result<String> {
) -> JsonResult<EditDataTableConfigResult> {
require_admin(is_admin, &username)?;
let is_superadmin = require_super_admin(&db, &authed).await.is_ok();
let mut tx = db.begin().await?;
// Read under the row lock this transaction will write with. `permissions`, `reference` and
// `forked_from` are carried across from what this read returns, so a permissions save
// committing between the read and the whole-document write below would be silently rolled back
// by it.
let old_datatables: HashMap<String, DataTable> = serde_json::from_value(
sqlx::query_scalar!(
"SELECT ws.datatable->'datatables' FROM workspace_settings ws WHERE ws.workspace_id = $1",
"SELECT ws.datatable->'datatables' FROM workspace_settings ws
WHERE ws.workspace_id = $1 FOR UPDATE",
&w_id
)
.fetch_one(&db)
.fetch_one(&mut *tx)
.await?
.unwrap_or(serde_json::Value::Null),
)
@@ -3563,6 +3711,58 @@ async fn edit_datatable_config(
crate::datatable_migrations::validate_datatable_path_segment(&r.from)?;
crate::datatable_migrations::validate_new_datatable_name(&r.to)?;
}
// A rename is a claim about what this save is doing, and other workspaces' pointers are
// rewritten from it — so the claim has to match the configuration it describes, or a caller
// can move every fork of one data table onto another by asserting a rename that did not
// happen. The shape below is what "these old keys became those new keys" actually means.
{
let old_keys = &old_datatables;
let new_keys = &new_config.settings.datatables;
let froms: std::collections::HashSet<&str> =
new_config.renames.iter().map(|r| r.from.as_str()).collect();
let tos: std::collections::HashSet<&str> =
new_config.renames.iter().map(|r| r.to.as_str()).collect();
if froms.len() != new_config.renames.len() {
return Err(Error::BadRequest(
"A data table is renamed twice in one save".to_string(),
));
}
if tos.len() != new_config.renames.len() {
return Err(Error::BadRequest(
"Two data tables are renamed to the same name in one save".to_string(),
));
}
for r in &new_config.renames {
if !old_keys.contains_key(&r.from) {
return Err(Error::BadRequest(format!(
"Cannot rename data table '{}': this workspace has no such data table",
r.from
)));
}
if !new_keys.contains_key(&r.to) {
return Err(Error::BadRequest(format!(
"Cannot rename data table '{}' to '{}': the save does not contain '{}'",
r.from, r.to, r.to
)));
}
// The source has to be gone, or gone-and-reoccupied by another rename — which is what
// a swap is. Without this, `main -> decoy` passes against a save that keeps both, and
// every fork of `main` silently follows onto a different data table.
if new_keys.contains_key(&r.from) && !tos.contains(r.from.as_str()) {
return Err(Error::BadRequest(format!(
"Data table '{}' is renamed to '{}' but the save still contains '{}'",
r.from, r.to, r.from
)));
}
// And the target has to be free, or freed by another rename.
if old_keys.contains_key(&r.to) && !froms.contains(r.to.as_str()) {
return Err(Error::BadRequest(format!(
"Cannot rename data table '{}' to '{}': '{}' already exists",
r.from, r.to, r.to
)));
}
}
}
// Map new name -> old name so a renamed data table inherits the previous
// flag instead of being treated as brand new.
@@ -3583,18 +3783,52 @@ async fn edit_datatable_config(
.get(name.as_str())
.copied()
.unwrap_or(name.as_str());
dt.migrations_enabled = match old_datatables.get(lookup) {
let old = old_datatables.get(lookup);
dt.migrations_enabled = match old {
Some(old) => old.migrations_enabled,
None => {
// Keyed by how the substrate is serialized into `workspace_settings`,
// so these line up with the `datatable_configured` adoption counts.
created_substrates.push(match dt.database.resource_type {
DataTableCatalogResourceType::Instance => "instance",
DataTableCatalogResourceType::Postgresql => "postgresql",
created_substrates.push(match dt.database.as_ref().map(|d| d.resource_type) {
Some(DataTableCatalogResourceType::Instance) => "instance",
Some(DataTableCatalogResourceType::Postgresql) => "postgresql",
None => "reference",
});
Some(true)
}
};
// Carried across from the stored entry rather than taken from the request. `permissions`
// is an access decision, edited through its own endpoint; `reference` is what makes a fork
// answer to the workspace that governs its data table, and letting a save clear it would
// hand the fork the database outright. `forked_from` is the clone stamp the fork flow
// writes: whether an entry has one is carried the same way, since it is what marks the
// database droppable, but the schema baseline inside it is the diff view's to advance.
dt.permissions = old.and_then(|old| old.permissions.clone());
dt.reference = old.and_then(|old| old.reference.clone());
dt.forked_from = match old.and_then(|old| old.forked_from.as_ref()) {
Some(stored) => Some(dt.forked_from.take().unwrap_or_else(|| stored.clone())),
None => None,
};
// Carrying the block onto a resource-backed entry would produce a data table the chokepoint
// refuses on every job — a save that succeeds and breaks everything afterwards. Refuse it
// instead: turning roles off first is one step, and it keeps discarding an access decision
// something somebody chose rather than a side effect of moving a database.
if dt.permissions.is_some()
&& dt
.database
.as_ref()
.is_some_and(|d| d.resource_type != DataTableCatalogResourceType::Instance)
{
return Err(Error::BadRequest(format!(
"Data table '{name}' is under roles, which only a data table on the instance \
database can be. Turn its roles off before moving it to a PostgreSQL resource."
)));
}
// A pointer names no database of its own, so the form's empty `database` is correct there.
if dt.reference.is_some() {
dt.database = None;
}
windmill_common::workspaces::validate_datatable_shape(name, dt)?;
}
let args_for_audit = format!("{:?}", new_config.settings);
@@ -3609,16 +3843,23 @@ async fn edit_datatable_config(
)
.await?;
// Check that non-superadmins are not abusing Instance databases
// Check that non-superadmins are not abusing Instance databases, which reach a database this
// workspace does not own. Pointing an entry at another workspace's data table is not checked
// here because it cannot be requested at all: `reference` is overwritten from the stored entry
// above, for every caller.
if !is_superadmin {
for (name, dt) in new_config.settings.datatables.iter() {
if dt.database.resource_type == DataTableCatalogResourceType::Instance {
let old_dt = old_datatables.get(name);
if old_dt.is_none()
|| old_dt.unwrap().database.resource_type
!= DataTableCatalogResourceType::Instance
|| old_dt.unwrap().database.resource_path != dt.database.resource_path
{
let old_dt = old_datatables.get(name);
if dt
.database
.as_ref()
.is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
{
let unchanged = old_dt.and_then(|o| o.database.as_ref()).is_some_and(|o| {
o.resource_type == DataTableCatalogResourceType::Instance
&& Some(&o.resource_path) == dt.database.as_ref().map(|d| &d.resource_path)
});
if !unchanged {
return Err(Error::BadRequest(
"Only superadmins can create or modify data tables with Instance databases"
.to_string(),
@@ -3628,6 +3869,84 @@ async fn edit_datatable_config(
}
}
// Worked out from the locked entries rather than taken from `deleted_datatables`: a settings
// sync sends the whole map without that list, and dropping a governing entry strands every
// fork pointing at it all the same.
let removed: Vec<String> = old_datatables
.keys()
.filter(|name| {
!new_config.settings.datatables.contains_key(*name)
&& !new_config.renames.iter().any(|r| &r.from == *name)
})
.cloned()
.collect();
// A database under roles is reached only through an entry that carries them. Roles follow an
// entry through a declared rename alone, and a settings sync never declares one, so an entry
// without roles that newly points at such a database — a name added, or an existing one
// repointed — would answer everyone there as `admin`. That holds whichever workspace governs it.
let newly_pointed: Vec<(&String, &str)> = new_config
.settings
.datatables
.iter()
.filter(|(_, dt)| dt.permissions.is_none())
.filter_map(|(name, dt)| {
let db = dt
.database
.as_ref()
.filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
let lookup = rename_src
.get(name.as_str())
.copied()
.unwrap_or(name.as_str());
let repointed = old_datatables
.get(lookup)
.and_then(|old| old.database.as_ref())
.is_none_or(|old_db| {
old_db.resource_type != db.resource_type
|| old_db.resource_path != db.resource_path
});
repointed.then_some((name, db.resource_path.as_str()))
})
.collect();
// Another workspace turning roles on for the same database holds only its own settings row, so
// without this the scan below could read past its uncommitted write.
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut *tx,
newly_pointed.iter().map(|(_, dbname)| *dbname),
)
.await?;
let governed_elsewhere: Vec<String> = if newly_pointed.is_empty() {
vec![]
} else {
sqlx::query_scalar(
"SELECT DISTINCT dt.value->'database'->>'resource_path' FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
WHERE ws.workspace_id <> $1 AND dt.value ? 'permissions'
AND dt.value->'database'->>'resource_type' = 'instance'",
)
.bind(&w_id)
.fetch_all(&mut *tx)
.await?
};
for (name, dbname) in newly_pointed {
let governed_here = old_datatables.values().any(|old| {
old.permissions.is_some()
&& old.database.as_ref().is_some_and(|d| {
d.resource_type == DataTableCatalogResourceType::Instance
&& d.resource_path == dbname
})
});
if governed_here || governed_elsewhere.iter().any(|g| g == dbname) {
return Err(Error::BadRequest(format!(
"Data table '{name}' would point at database '{dbname}', which a data table under \
roles uses, without carrying those roles: everyone reaching '{name}' would connect \
there as `admin`. Rename the data table under roles from the data table settings, \
which carries its roles, or turn its roles off first."
)));
}
}
let config: serde_json::Value = serde_json::to_value(new_config.settings)
.map_err(|err| Error::internal_err(err.to_string()))?;
@@ -3649,6 +3968,44 @@ async fn edit_datatable_config(
)
.await?;
// A fork points at a data table by name, so a rename here has to follow or every fork's entry
// resolves to nothing. In two passes through a temporary name, like the migration cascade one
// layer down: applied in order, `sa -> sb` then `sb -> sa` would move what pointed at `sa` all
// the way back to `sa`, and `A -> B`, `B -> C` would carry `A`'s pointers to `C`. Each pointer
// moves once, from what it named before this save. Inside the transaction: the rename and the
// pointers that name it are one change, and half of it is a fork whose jobs stop.
for (i, r) in new_config.renames.iter().enumerate() {
repoint_datatable_references(&mut tx, &w_id, &r.from, &format!("__wm_rename_tmp/{i}"))
.await?;
}
for (i, r) in new_config.renames.iter().enumerate() {
repoint_datatable_references(&mut tx, &w_id, &format!("__wm_rename_tmp/{i}"), &r.to)
.await?;
}
// A deletion cannot be followed the same way — there is nothing to point at any more. Read who
// is left stranded so the caller is told, the way deleting a workspace does.
let mut stranded: Vec<StrandedReference> = Vec::new();
for name in &removed {
let rows = sqlx::query!(
r#"SELECT ws.workspace_id AS "workspace_id!", dt.key AS "datatable!"
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
WHERE dt.value->'reference'->>'workspace_id' = $1
AND dt.value->'reference'->>'datatable' = $2"#,
&w_id,
name,
)
.fetch_all(&mut *tx)
.await?;
stranded.extend(
rows.into_iter().map(|r| StrandedReference {
workspace_id: r.workspace_id,
datatable: r.datatable,
}),
);
}
tx.commit().await?;
for substrate in created_substrates {
@@ -3663,7 +4020,9 @@ async fn edit_datatable_config(
)
.await?;
Ok(format!("Edit datatable config for workspace {}", &w_id))
Ok(Json(EditDataTableConfigResult {
stranded_references: stranded,
}))
}
#[derive(Deserialize)]
@@ -7545,13 +7904,144 @@ async fn snapshot_datatable_schema(
.map_err(|e| Error::internal_err(format!("Failed to serialize schema: {}", e)))
}
/// Turn every data table the fork chose to keep into a pointer at the parent's entry.
///
/// `clone_workspace_data` copies `workspace_settings` wholesale, so a kept data table arrives as a
/// byte-identical copy naming the parent's database — including the parent's `permissions`, which
/// a fork admin could then edit to widen their own access to it. A pointer has nothing local to
/// edit: the parent's entry stays the only place the decision lives.
///
/// The cloned data tables are skipped: they own a fresh database of their own, and they keep the
/// copied `permissions` as their starting point, which they then govern.
async fn point_kept_datatables_at_parent(
tx: &mut Transaction<'_, Postgres>,
parent_w_id: &str,
forked_w_id: &str,
cloned: &[ForkedDatatableInfo],
) -> Result<()> {
let settings: Option<serde_json::Value> = sqlx::query_scalar!(
"SELECT datatable FROM workspace_settings WHERE workspace_id = $1",
forked_w_id
)
.fetch_optional(&mut **tx)
.await?
.flatten();
let Some(mut settings) = settings else {
return Ok(());
};
let Some(datatables) = settings
.get_mut("datatables")
.and_then(|d| d.as_object_mut())
else {
return Ok(());
};
let mut changed = false;
for (name, entry) in datatables.iter_mut() {
if cloned.iter().any(|c| &c.name == name) {
continue;
}
let dt: DataTable = match serde_json::from_value(entry.clone()) {
Ok(dt) => dt,
Err(_) => continue,
};
// Already a pointer: the parent was itself a fork, and its entry names the workspace that
// governs. Following it from here is the same answer, so leave it alone.
if dt.reference.is_some() {
continue;
}
// Only instance databases. A resource-backed data table names a resource, and the settings
// clone gave the fork its own copy of that resource in its own workspace — pointing at the
// parent's entry would silently move the fork onto the parent's resource instead.
if dt
.database
.as_ref()
.is_none_or(|d| d.resource_type != DataTableCatalogResourceType::Instance)
{
continue;
}
*entry = serde_json::to_value(DataTable {
database: None,
reference: Some(windmill_common::workspaces::DataTableReference {
workspace_id: parent_w_id.to_string(),
datatable: name.clone(),
}),
forked_from: None,
migrations_enabled: dt.migrations_enabled,
permissions: None,
})
.map_err(|e| Error::internal_err(format!("serializing data table '{name}': {e}")))?;
changed = true;
}
if changed {
sqlx::query!(
"UPDATE workspace_settings SET datatable = $1 WHERE workspace_id = $2",
settings,
forked_w_id
)
.execute(&mut **tx)
.await?;
}
Ok(())
}
/// Move every pointer in any workspace that names `(w_id, from)` to `(w_id, to)`.
///
/// `EXISTS` rather than a `LIKE` over the whole document: the update rewrites the row, so matching
/// every workspace that holds any pointer would rewrite rows to a byte-identical value and hold an
/// exclusive lock on them until commit.
async fn repoint_datatable_references(
tx: &mut Transaction<'_, Postgres>,
w_id: &str,
from: &str,
to: &str,
) -> Result<()> {
sqlx::query!(
r#"UPDATE workspace_settings ws
SET datatable = (
SELECT jsonb_set(ws.datatable, '{datatables}', jsonb_object_agg(
dt.key,
CASE WHEN dt.value->'reference'->>'workspace_id' = $1
AND dt.value->'reference'->>'datatable' = $2
THEN jsonb_set(dt.value, '{reference,datatable}', to_jsonb($3::text))
ELSE dt.value END
))
FROM jsonb_each(ws.datatable->'datatables') dt
)
WHERE EXISTS (
SELECT 1 FROM jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) d
WHERE d.value->'reference'->>'workspace_id' = $1
AND d.value->'reference'->>'datatable' = $2
)"#,
w_id,
from,
to,
)
.execute(&mut **tx)
.await?;
Ok(())
}
async fn apply_forked_datatable(
db: &DB,
tx: &mut Transaction<'_, Postgres>,
authed: &ApiAuthed,
parent_w_id: &str,
forked_w_id: &str,
fdt: &ForkedDatatableInfo,
) -> Result<()> {
// Cloning reads the parent's whole schema as admin and hands the copy to the fork, so it is
// for the workspace that governs the data table — a fork can use one, never duplicate it.
windmill_common::workspaces::ensure_datatable_admin_access(
db,
parent_w_id,
&fdt.name,
&DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
let governing = ensure_datatable_is_clonable(db, parent_w_id, &fdt.name).await?;
windmill_common::validate_dbname(&fdt.new_dbname)?;
if !fdt.new_dbname.starts_with("wm_fork_") {
return Err(Error::BadRequest(format!(
@@ -7583,25 +8073,46 @@ async fn apply_forked_datatable(
let dt: DataTable = serde_json::from_value(config_val)
.map_err(|e| Error::internal_err(format!("Failed to parse datatable config: {}", e)))?;
if dt.database.resource_type == DataTableCatalogResourceType::Instance {
// Instance: update resource_path to the new dbname
// A clone owns its copy, so the fork's entry has to be terminal. When the parent was itself a
// fork the settings clone hands down a pointer instead, and what it points at is what the copy
// was taken from. `ensure_datatable_is_clonable` already settled that this shape can be
// cloned, so there is nothing left to refuse here — by now the database exists and is filled.
let database = match dt.database.clone() {
Some(database) => database,
None => governing.datatable.database.clone().ok_or_else(|| {
Error::internal_err(format!(
"Data table '{}' resolves to an entry that owns no database",
fdt.name
))
})?,
};
if database.resource_type == DataTableCatalogResourceType::Instance {
// The whole `database` object, not just its `resource_path`: a pointer entry has none to
// patch. `reference` goes with it — exactly one of the two may be set.
let new_database = serde_json::json!({
"resource_type": "instance",
"resource_path": &fdt.new_dbname,
});
sqlx::query!(
r#"UPDATE workspace_settings
SET datatable = jsonb_set(
jsonb_set(datatable, ARRAY['datatables', $2, 'database', 'resource_path'], to_jsonb($3::text)),
jsonb_set(
datatable #- ARRAY['datatables', $2, 'reference'],
ARRAY['datatables', $2, 'database'], $3::jsonb),
ARRAY['datatables', $2, 'forked_from'], $4::jsonb
)
WHERE workspace_id = $1"#,
forked_w_id,
&fdt.name,
&fdt.new_dbname,
new_database,
forked_from,
)
.execute(&mut **tx)
.await?;
} else {
// Resource: update the resource's dbname and mark as ws_specific
let resource_path = &dt.database.resource_path;
let resource_path = &database.resource_path;
sqlx::query!(
r#"UPDATE resource
SET value = jsonb_set(value, '{dbname}', to_jsonb($3::text))
@@ -8038,6 +8549,14 @@ async fn create_workspace_fork(
.execute(&mut *tx)
.await?;
// The pointers this fork writes to the parent's data tables stay invisible until it commits, so
// a rename of one of them cannot carry them. Holding the parent's settings row makes such a
// rename wait for this commit, and makes the copy below read one that committed first.
sqlx::query("SELECT 1 FROM workspace_settings WHERE workspace_id = $1 FOR SHARE")
.bind(&parent_workspace_id)
.execute(&mut *tx)
.await?;
// Clone all data from the parent workspace using Rust implementation
if let Err(e) =
clone_workspace_data(&mut tx, &db, &parent_workspace_id, &forked_id, &authed).await
@@ -8075,9 +8594,18 @@ async fn create_workspace_fork(
// Update forked datatable settings to point to new databases
for fdt in &nw.forked_datatables {
apply_forked_datatable(&db, &mut tx, &parent_workspace_id, &forked_id, fdt).await?;
apply_forked_datatable(&db, &mut tx, &authed, &parent_workspace_id, &forked_id, fdt)
.await?;
}
point_kept_datatables_at_parent(
&mut tx,
&parent_workspace_id,
&forked_id,
&nw.forked_datatables,
)
.await?;
// The settings clone copies the source's ducklake config verbatim — including a parent
// fork's own `fork_behavior` stamps. Sharing is a per-fork-creation choice, never
// inherited: reset any cloned stamps first, then apply this fork's requested list.
@@ -8880,6 +9408,14 @@ async fn leave_workspace(
) -> Result<String> {
windmill_api_auth::forbid_job_token_account_destruction(&authed)?;
let mut tx = db.begin().await?;
// The membership is what made `u/<username>` mean this person. Leaving it behind in a tenant
// list would hand their data table access back on rejoin, or to whoever takes the name next.
windmill_common::workspaces::remove_datatable_tenant_in_workspace(
&mut tx,
&w_id,
&format!("u/{}", authed.username),
)
.await?;
sqlx::query!(
"DELETE FROM usr WHERE workspace_id = $1 AND email = $2",
&w_id,
backend/windmill-api-workspaces/src/workspaces_extra.rs
+66 -5
View File
@@ -492,6 +492,30 @@ pub(crate) async fn change_workspace_id(
.fetch_all(&mut *tx)
.await?;
// A fork's data table entry names the workspace that governs it by id, so the rename has to
// follow there too — anywhere, not just in the reparented children: a detached workspace can
// point at this one without being its fork. Left behind, the pointer resolves to the archived
// shell and every job through it stops.
info!("Re-pointing data table references to the new workspace id");
sqlx::query!(
r#"UPDATE workspace_settings ws
SET datatable = (
SELECT jsonb_set(ws.datatable, '{datatables}', jsonb_object_agg(
dt.key,
CASE WHEN dt.value->'reference'->>'workspace_id' = $2
THEN jsonb_set(dt.value, '{reference,workspace_id}', to_jsonb($1::text))
ELSE dt.value END
))
FROM jsonb_each(ws.datatable->'datatables') dt
)
WHERE jsonb_typeof(ws.datatable->'datatables') = 'object'
AND ws.datatable::text LIKE '%"reference"%'"#,
&rw.new_id,
&old_id,
)
.execute(&mut *tx)
.await?;
info!("Updating workspace_protection_rule table");
sqlx::query!(
"UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2",
@@ -971,6 +995,22 @@ pub(crate) async fn delete_workspace(
// but the destructive cleanup itself runs only after the commit below: a delete that
// fails mid-way must never leave a live workspace with its fork data destroyed and no
// registry row to retry from. Read-only: nothing is dropped here.
// Read before the delete: another workspace's data table entry can point at one of this
// workspace's, and deleting the workspace it names leaves that pointer resolving to nothing.
// Nothing sweeps them — turning them back into copies would hand each fork the database
// outright — so the deleter is told which data tables they just stranded.
let stranded_pointers = sqlx::query!(
r#"SELECT ws.workspace_id AS "workspace_id!", dt.key AS "datatable!"
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
WHERE dt.value->'reference'->>'workspace_id' = $1
ORDER BY ws.workspace_id, dt.key"#,
&w_id,
)
.fetch_all(&db)
.await
.unwrap_or_default();
let fork_ducklake_cleanups = prepare_fork_ducklake_cleanups(&db, &w_id, None)
.await
.unwrap_or_else(|e| {
@@ -1289,7 +1329,23 @@ pub(crate) async fn delete_workspace(
tracing::warn!("failed to broadcast fork lineage change: {e:#}");
}
Ok(format!("Deleted workspace {}", &w_id))
if stranded_pointers.is_empty() {
Ok(format!("Deleted workspace {}", &w_id))
} else {
let stranded = stranded_pointers
.iter()
.map(|r| format!("{}/{}", r.workspace_id, r.datatable))
.collect::<Vec<_>>()
.join(", ");
Ok(format!(
concat!(
"Deleted workspace {}. These data tables were governed by it and no longer ",
"resolve: {}. Their databases still exist; a superadmin can point them at ",
"another workspace's data table."
),
&w_id, stranded
))
}
}
#[derive(Deserialize)]
@@ -1343,15 +1399,20 @@ pub async fn drop_forked_datatable_databases(
let mut errors: Vec<String> = Vec::new();
for dt_name in &req.datatable_names {
let dt = match datatables.get(dt_name) {
Some(dt) if dt.forked_from.is_some() => dt,
// Only a clone is droppable, and a clone is terminal by construction: a kept data table is
// a pointer at the parent's database, which this fork does not own.
let database = match datatables.get(dt_name) {
Some(dt) if dt.forked_from.is_some() => match dt.database.as_ref() {
Some(database) => database,
None => continue,
},
_ => continue,
};
if dt.database.resource_type
if database.resource_type
== windmill_common::workspaces::DataTableCatalogResourceType::Instance
{
let db_to_drop = &dt.database.resource_path;
let db_to_drop = &database.resource_path;
if !db_to_drop.starts_with("wm_fork_") {
errors.push(format!(
"Refusing to drop instance database '{}' for datatable://{}: name does not start with 'wm_fork_'",
backend/windmill-api/openapi.yaml
+265 -3
View File
@@ -1588,6 +1588,89 @@ paths:
additionalProperties:
$ref: "#/components/schemas/CustomInstanceDb"
/settings/datatable_roles:
get:
summary: list the instance's data table roles
operationId: listInstanceDatatableRoles
tags:
- setting
responses:
"200":
description: the instance role catalog
content:
application/json:
schema:
type: array
items:
$ref: "#/components/schemas/InstanceDatatableRole"
post:
summary: create a data table role on the instance's Postgres cluster
operationId: createInstanceDatatableRole
tags:
- setting
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [name]
properties:
name:
type: string
responses:
"200":
description: the created role
content:
application/json:
schema:
$ref: "#/components/schemas/InstanceDatatableRole"
/settings/datatable_roles/{id}:
post:
summary: rename a data table role or turn its login on and off
operationId: updateInstanceDatatableRole
tags:
- setting
parameters:
- in: path
name: id
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
name:
type: string
enabled:
type: boolean
responses:
"200":
description: the updated role
content:
application/json:
schema:
$ref: "#/components/schemas/InstanceDatatableRole"
delete:
summary: drop a data table role from the cluster and from every workspace that named it
operationId: deleteInstanceDatatableRole
tags:
- setting
parameters:
- in: path
name: id
required: true
schema:
type: string
responses:
"200":
description: deleted
/settings/setup_custom_instance_pg_database/{name}:
post:
summary: Runs CREATE DATABASE on the Windmill Postgres and grants access to the custom_instance_user
@@ -5141,7 +5224,7 @@ paths:
type: array
items:
type: object
required: [name, resource_type, resource_path]
required: [name, resource_type, resource_path, permissioned]
properties:
name:
type: string
@@ -5150,6 +5233,97 @@ paths:
enum: [postgres, instance]
resource_path:
type: string
governing_workspace_id:
type: string
permissioned:
type: boolean
/w/{workspace}/workspaces/datatable_permissions/{datatable_name}:
get:
summary: get who may connect to a data table as which role
operationId: getDatatablePermissions
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: the data table's roles and their tenants
content:
application/json:
schema:
$ref: "#/components/schemas/DatatablePermissions"
post:
summary: set who may connect to a data table as which role
operationId: setDatatablePermissions
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [permissioned]
properties:
permissioned:
type: boolean
default_role:
type: string
roles:
type: array
items:
$ref: "#/components/schemas/DatatableRoleTenants"
responses:
"200":
description: status
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/datatable_usable_roles/{datatable_name}:
get:
summary: list the data table roles the caller may connect as
operationId: listUsableDatatableRoles
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
responses:
"200":
description: usable roles
content:
application/json:
schema:
type: object
required: [permissioned, roles, default_role]
properties:
permissioned:
type: boolean
roles:
type: array
items:
type: string
default_role:
type: string
/w/{workspace}/workspaces/list_datatable_schemas:
get:
@@ -5331,7 +5505,22 @@ paths:
description: status
content:
application/json:
schema: {}
schema:
type: object
properties:
stranded_references:
description: >-
Data tables in other workspaces that were governed by one this save deleted
and no longer resolve.
type: array
items:
type: object
required: [workspace_id, datatable]
properties:
workspace_id:
type: string
datatable:
type: string
/w/{workspace}/workspaces/run_datatable_migrations/{datatable_name}:
post:
@@ -33400,6 +33589,66 @@ components:
- ducklake
- datatable
InstanceDatatableRole:
type: object
required: [id, name, enabled]
properties:
id:
type: string
name:
type: string
enabled:
type: boolean
DatatableRoleTenants:
type: object
required: [id, tenants]
properties:
id:
type: string
name:
type: string
tenants:
type: array
items:
type: string
DatatablePermissions:
type: object
required: [supported, permissioned, default_role, roles, editable, available_roles]
properties:
supported:
type: boolean
description: >-
Whether this data table can be put under roles at all. Only one backed by the
instance database can: a role is a login on that cluster.
permissioned:
type: boolean
default_role:
type: string
roles:
type: array
items:
$ref: "#/components/schemas/DatatableRoleTenants"
governing_workspace_id:
type: string
editable:
type: boolean
available_roles:
type: array
items:
$ref: "#/components/schemas/InstanceDatatableRole"
ungoverned_reachers:
type: array
items:
type: object
required: [workspace_id, datatable]
properties:
workspace_id:
type: string
datatable:
type: string
CustomInstanceDb:
type: object
required:
@@ -35448,9 +35697,11 @@ components:
type: object
additionalProperties:
type: object
required: [database]
properties:
database:
description: >-
Set on an entry that owns its database. Absent on a fork's entry, which points at
another workspace's data table instead.
type: object
properties:
resource_type:
@@ -35462,6 +35713,17 @@ components:
type: string
required:
- resource_type
reference:
description: >-
The workspace and data table that govern this one. Server-owned: written by fork
creation, and carried across a settings save whatever the request says.
type: object
required: [workspace_id, datatable]
properties:
workspace_id:
type: string
datatable:
type: string
migrations_enabled:
type: boolean
description: Whether the SQL migrations feature is opted in for this data table
backend/windmill-api/src/capture.rs
+6
View File
@@ -567,6 +567,9 @@ async fn set_config(
};
let mut tx = user_db.begin(&authed).await?;
if matches!(nc.trigger_kind, TriggerKind::Postgres) {
windmill_common::datatable_roles::lock_datatable_streams(&mut *tx, false).await?;
}
sqlx::query!(
r#"
@@ -614,6 +617,9 @@ async fn ping_config(
)>,
) -> Result<()> {
let mut tx = user_db.begin(&authed).await?;
if matches!(trigger_kind, TriggerKind::Postgres) {
windmill_common::datatable_roles::lock_datatable_streams(&mut *tx, false).await?;
}
sqlx::query!(
r#"
backend/windmill-api/src/jobs.rs
+4 -3
View File
@@ -8365,8 +8365,9 @@ pub async fn run_wait_result_flow_by_version(
/// job lives, in particular DuckDB, which runs in-process in the worker.
///
/// What it does permit is any statement against the workspace's data tables, writes and DDL
/// included: the helper's body is an unrestricted SQL template and data tables carry no
/// per-user ACL. Narrowing that is a separate decision from this exemption.
/// included: the helper's body is an unrestricted SQL template. What that reaches is the
/// operator's own data table role — the preview job is permissioned as them, so the executor
/// resolves it under their tenancy like any other job.
///
/// The database argument is only half the target: the executor honors a `-- database`
/// directive in the SQL over it, and `-- s3` redirects the result set, so both are refused.
@@ -8598,7 +8599,7 @@ async fn run_inline_preview_script(
#[cfg(not(feature = "run_inline"))]
async fn run_inline_preview_script() -> error::Result<Response> {
Err(error::Error::InternalErr(
"inline preview requires the worker feature".to_string(),
"inline preview requires the run_inline feature on the worker".to_string(),
))
}
backend/windmill-api/src/users.rs
+11
View File
@@ -352,6 +352,17 @@ async fn update_username_in_workpsace<'c>(
new_username: &str,
w_id: &str,
) -> error::Result<()> {
// ---- data table tenants ----
// Tenants name the user, so the rename has to follow here too; a list left naming the old
// username silently drops the access instead of moving it.
windmill_common::workspaces::rename_datatable_tenant_in_workspace(
tx,
w_id,
&format!("u/{old_username}"),
&format!("u/{new_username}"),
)
.await?;
// ---- instance and workspace users ----
sqlx::query!(
"UPDATE usr SET username = $1 WHERE email = $2",
backend/windmill-api/src/workspaces_export.rs
+2 -2
View File
@@ -1639,7 +1639,7 @@ pub(crate) async fn tarball_workspace(
mute_critical_alerts: row.mute_critical_alerts,
color: row.color.clone(),
operator_settings: row.operator_settings.clone(),
datatable: row.datatable.clone(),
datatable: windmill_common::workspaces::strip_datatable_permissions(row.datatable.clone()),
slack_team_id: row.slack_team_id.clone(),
slack_name: row.slack_name.clone(),
slack_command_script: row.slack_command_script.clone(),
@@ -1703,7 +1703,7 @@ pub(crate) async fn tarball_workspace(
mute_critical_alerts: row.mute_critical_alerts,
color: row.color,
operator_settings: row.operator_settings,
datatable: row.datatable,
datatable: windmill_common::workspaces::strip_datatable_permissions(row.datatable),
slack_team_id: row.slack_team_id,
slack_name: row.slack_name,
slack_command_script: row.slack_command_script,
backend/windmill-common/src/datatable_roles.rs
+362
View File
@@ -0,0 +1,362 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! The instance's data table role catalog.
//!
//! A data table role is a real Postgres login role on the Windmill cluster, named exactly as the
//! user named it, shared by every instance database. Windmill decides who may ask for a role (the
//! per-data-table tenant lists in [`crate::workspaces`]); Postgres decides what the role may then
//! touch. The catalog here is only the first half's vocabulary plus the cluster provisioning.
//!
//! Entries are keyed by a generated id so a rename moves nothing else: tenants name the id.
use std::collections::BTreeMap;
use crate::{
error::{Error, Result},
DB,
};
/// The connection every data table resolved to before roles existed (`custom_instance_user`). It
/// owns every pre-existing object, so it is a reserved name rather than a catalog entry: never
/// created, renamed or dropped.
pub const ADMIN_DATATABLE_ROLE: &str = "admin";
/// The login the admin connection uses, and the role every created role is granted to — that
/// membership is what later lets it `ALTER ... OWNER TO` a role and drop it.
pub const CUSTOM_INSTANCE_USER: &str = "custom_instance_user";
/// One catalog entry, as stored in `datatable_role`. The password is per role and instance-wide;
/// it belongs to the instance, not to any workspace's settings.
/// No `Serialize`/`Deserialize`: the catalog is rows now, and a derived `Serialize` would emit
/// `pwd` — the same way out for a credential that the hand-written `Debug` below closes on the log
/// side.
#[derive(Clone)]
pub struct InstanceDatatableRole {
/// The Postgres role name, verbatim.
pub name: String,
pub enabled: bool,
/// Absent only for a role whose provisioning did not finish; resolving as it then errors
/// rather than falling back to admin.
///
/// A plain string rather than a `StringOrSecretRef` like the instance user's password: that
/// one is a secret ref because an operator supplies it and may want it to come from their own
/// backend, while this one is minted here and never entered by anyone, so there is nothing for
/// a ref to point at. Encrypting generated secrets at rest is a separate change that would
/// take the replication password with it.
pub pwd: Option<String>,
}
/// Hand-written so `{:?}` on a catalog cannot put a live Postgres password in a log line or an
/// audit record. Everything else about the entry is safe to print.
impl std::fmt::Debug for InstanceDatatableRole {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("InstanceDatatableRole")
.field("name", &self.name)
.field("enabled", &self.enabled)
.field("pwd", &self.pwd.as_ref().map(|_| "<redacted>"))
.finish()
}
}
pub type DatatableRoleCatalog = BTreeMap<String, InstanceDatatableRole>;
/// Names Postgres or Windmill already owns. `admin` is excluded because it never reaches the
/// cluster as a role name at all — it resolves to `custom_instance_user`.
fn is_reserved_role_name(name: &str) -> bool {
let lower = name.to_ascii_lowercase();
lower == ADMIN_DATATABLE_ROLE
|| lower == "postgres"
|| lower == "public"
|| lower.starts_with("pg_")
|| lower.starts_with("windmill_")
|| lower.starts_with("custom_instance_")
}
/// The charset is what makes every downstream interpolation safe: the name reaches Postgres as a
/// quoted identifier, a `-- role <name>` annotation, and a `?role=` query parameter.
pub fn validate_role_name(name: &str) -> Result<()> {
if name.is_empty() || name.len() > 63 {
return Err(Error::BadRequest(format!(
"Invalid data table role name '{name}': it must be between 1 and 63 characters"
)));
}
if !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
{
return Err(Error::BadRequest(format!(
"Invalid data table role name '{name}': only letters, digits, '_' and '-' are allowed"
)));
}
if is_reserved_role_name(name) {
return Err(Error::BadRequest(format!(
"'{name}' is reserved and cannot be used as a data table role name"
)));
}
Ok(())
}
/// A double-quoted Postgres identifier. Doubling `"` is Postgres's own escaping inside one, so this
/// quotes any name — schema, table or role. Role names are validated as well
/// ([`validate_role_name`]) because they also travel unquoted, in `-- role <name>` and `?role=`.
pub fn quote_ident(name: &str) -> String {
format!("\"{}\"", name.replace('"', "\"\""))
}
/// Serialize the mutations that are not already serialized by the row itself.
///
/// A create is an insert and a delete is a delete, which Postgres orders for us — the unique index
/// on `name` is what makes two concurrent creates of the same name one winner and one error. What
/// still needs it is the window between the cluster DDL and the row: `CREATE ROLE` is not visible
/// to another transaction's `pg_roles` check until commit, so without this two creates of the same
/// name both pass their existence check and one fails on the index having already made the login.
/// Held for the transaction, so the DDL has to run on that same transaction to be covered.
pub async fn lock_role_catalog(tx: &mut sqlx::Transaction<'_, sqlx::Postgres>) -> Result<()> {
sqlx::query!("SELECT pg_advisory_xact_lock(hashtext('datatable_role_catalog'))")
.execute(&mut **tx)
.await?;
Ok(())
}
/// A replication stream reads every row whatever a data table's roles grant. Turning roles on looks
/// for streams holding this exclusive; whatever can start a Postgres trigger or capture streaming
/// holds it shared on the transaction that commits it. So either the look sees the stream, or the
/// stream's listener connects after roles are committed and refuses. Held for the transaction.
pub async fn lock_datatable_streams(conn: &mut sqlx::PgConnection, exclusive: bool) -> Result<()> {
let lock = if exclusive {
"pg_advisory_xact_lock"
} else {
"pg_advisory_xact_lock_shared"
};
sqlx::query(&format!("SELECT {lock}(hashtext('datatable_streams'))"))
.execute(conn)
.await?;
Ok(())
}
/// Whether an instance database is reached only through entries under roles is decided by two
/// writes that lock different workspaces' settings rows: turning roles on for one entry, and a
/// settings save pointing an entry without roles at the database. Each holds this for every
/// database it decides on, so neither reads past the other's uncommitted write. Held for the
/// transaction; the names are locked in sorted order so two holders cannot deadlock.
pub async fn lock_instance_databases_governance<'a>(
conn: &mut sqlx::PgConnection,
dbnames: impl IntoIterator<Item = &'a str>,
) -> Result<()> {
let dbnames: std::collections::BTreeSet<&str> = dbnames.into_iter().collect();
for dbname in dbnames {
sqlx::query("SELECT pg_advisory_xact_lock(hashtext('datatable_instance_database:' || $1))")
.bind(dbname)
.execute(&mut *conn)
.await?;
}
Ok(())
}
/// Disclosure: returns every role's stored Postgres password in plaintext. Any server path that
/// has to resolve or name a role may call it — including handlers open to a workspace member, who
/// need the names — but callers MUST NOT let `pwd` reach a response, a log line, an audit record
/// or an export. Nothing about who may call it: the credential is the whole risk, and `Debug` is
/// hand-written to redact it for the same reason.
pub async fn read_role_catalog(db: &DB) -> Result<DatatableRoleCatalog> {
crate::datatable_roles_oss::read_role_catalog(db).await
}
/// As [`read_role_catalog`], reading inside the caller's transaction so the value is the one
/// [`lock_role_catalog`] is protecting. Same disclosure contract.
pub async fn read_role_catalog_tx(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
) -> Result<DatatableRoleCatalog> {
crate::datatable_roles_oss::read_role_catalog_tx(tx).await
}
/// Record a role, in the caller's transaction so it commits with the `CREATE ROLE` it describes.
///
/// Authorization: writes a generated Postgres credential. Callers MUST restrict this to superadmin
/// paths and MUST hold [`lock_role_catalog`] on `tx`.
pub async fn insert_role_catalog_entry(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
id: &str,
role: &InstanceDatatableRole,
) -> Result<()> {
crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, role).await
}
/// Update a role's recorded name, login flag and password. Same contract as
/// [`insert_role_catalog_entry`].
pub async fn update_role_catalog_entry(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
id: &str,
role: &InstanceDatatableRole,
) -> Result<()> {
crate::datatable_roles_oss::update_role_catalog_entry(tx, id, role).await
}
/// Forget a role. Same contract as [`insert_role_catalog_entry`]; run it in the transaction that
/// drops the cluster login, so the two cannot disagree.
pub async fn delete_role_catalog_entry(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
id: &str,
) -> Result<()> {
crate::datatable_roles_oss::delete_role_catalog_entry(tx, id).await
}
/// Resolve the role a caller named to its catalog id. A disabled role is an error rather than a
/// silent fallback: the caller asked for something the instance deliberately turned off.
pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Result<&'a str> {
let entry = catalog
.iter()
.find(|(_, role)| role.name == name)
.ok_or_else(|| {
Error::NotFound(format!(
"'{name}' is not a data table role of this instance. Defined roles: {}.",
catalog
.values()
.map(|r| r.name.as_str())
.collect::<Vec<_>>()
.join(", ")
))
})?;
if !entry.1.enabled {
return Err(Error::BadRequest(format!(
"Data table role '{name}' is disabled on this instance"
)));
}
Ok(entry.0.as_str())
}
/// Every instance database the registry knows about. Role provisioning has to reach all of them:
/// a role that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
///
/// Authorization: checks nothing, and names every instance database across all workspaces. Callers
/// MUST be superadmin-gated or keep the names server-side; never return them to a workspace caller.
pub async fn registered_instance_databases(db: &DB) -> Result<Vec<String>> {
crate::datatable_roles_oss::registered_instance_databases(db).await
}
/// `CONNECT` on `dbname` for every enabled role, and none for `PUBLIC`. Run at role creation, at
/// database creation, and lazily whenever an instance data table is administered, so a database
/// provisioned before a role existed is repaired rather than left silently unreachable.
///
/// Authorization: rewrites a database's ACL with the server's own credentials and checks nothing.
/// Callers MUST have authorized administration of `dbname` — superadmin, or an admin of the
/// workspace governing a data table on it.
pub async fn converge_connect_grants(db: &DB, dbname: &str) -> Result<()> {
crate::datatable_roles_oss::converge_connect_grants(db, dbname).await
}
/// As [`converge_connect_grants`], with a catalog the caller already read. Same contract.
pub async fn converge_connect_grants_with(
db: &DB,
dbname: &str,
catalog: &DatatableRoleCatalog,
) -> Result<()> {
crate::datatable_roles_oss::converge_connect_grants_with(db, dbname, catalog).await
}
/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user`, and `CONNECT` on
/// every registered database. No privileges beyond that — an admin grants them through SQL or the
/// ACL editor.
///
/// Authorization: creates a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn create_instance_role(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
name: &str,
password: &str,
) -> Result<()> {
crate::datatable_roles_oss::create_instance_role(tx, name, password).await
}
/// Authorization: alters a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn set_instance_role_login(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
name: &str,
enabled: bool,
) -> Result<()> {
crate::datatable_roles_oss::set_instance_role_login(tx, name, enabled).await
}
/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one.
///
/// Authorization: renames a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn rename_instance_role(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
from: &str,
to: &str,
password: &str,
) -> Result<()> {
crate::datatable_roles_oss::rename_instance_role(tx, from, to, password).await
}
/// A role owning anything in any database blocks its own `DROP ROLE`, and both its objects and the
/// privileges granted to it are only visible from inside each database — hence the pass over the
/// registry. An unreachable database aborts the whole delete: dropping the role while one database
/// still holds objects owned by it leaves those objects owned by a numeric OID nobody can name.
///
/// Each pass runs as the instance's own Postgres user rather than `custom_instance_user`, which
/// owns the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
/// could only undo what it granted itself, so a privilege planted by an operator in psql — the
/// ordinary way privileges reach a role — would survive and block the drop.
///
/// Authorization: drops a cluster-wide Postgres login and reassigns everything it owns. Callers
/// MUST restrict this to superadmin paths, and MUST hold [`lock_role_catalog`] on `tx`.
///
/// The per-database passes open their own connections and cannot join `tx`; the lock is what keeps
/// a concurrent mutation out while they run. Only the final `DROP ROLE` is on `tx`, so it commits
/// or rolls back with the catalog write that forgets the role. Those passes commit as they go, so
/// callers MUST have disabled the role in an earlier committed transaction: a failure part-way
/// then leaves a disabled role to retry, not an enabled one already stripped in some databases.
pub async fn drop_instance_role(
db: &DB,
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
name: &str,
) -> Result<()> {
crate::datatable_roles_oss::drop_instance_role(db, tx, name).await
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn role_names_are_validated() {
assert!(validate_role_name("analytics").is_ok());
assert!(validate_role_name("read-only_2").is_ok());
assert!(validate_role_name("").is_err());
assert!(validate_role_name(&"a".repeat(64)).is_err());
assert!(validate_role_name("has space").is_err());
assert!(validate_role_name("quote\"injection").is_err());
// Reserved, case-insensitively.
assert!(validate_role_name("admin").is_err());
assert!(validate_role_name("Postgres").is_err());
assert!(validate_role_name("pg_read_all_data").is_err());
assert!(validate_role_name("windmill_user").is_err());
assert!(validate_role_name("custom_instance_user").is_err());
}
#[test]
fn a_disabled_role_is_an_error_not_a_fallback() {
let mut catalog = DatatableRoleCatalog::new();
catalog.insert(
"id1".to_string(),
InstanceDatatableRole {
name: "analytics".to_string(),
enabled: false,
pwd: Some("x".to_string()),
},
);
assert!(role_id_by_name(&catalog, "analytics").is_err());
assert!(role_id_by_name(&catalog, "nope").is_err());
catalog.get_mut("id1").unwrap().enabled = true;
assert_eq!(role_id_by_name(&catalog, "analytics").unwrap(), "id1");
}
}
backend/windmill-common/src/datatable_roles_oss.rs
+208
View File
@@ -0,0 +1,208 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Where data table roles come from: the enterprise implementation, or a refusal.
//!
//! Roles are an Enterprise Edition feature. An edition without them creates, grants and connects
//! as none, and a data table saved under roles — by an enterprise build, before a downgrade — is
//! refused rather than resolved as `admin`. A data table not under roles, asked for no role,
//! resolves as it always has. `private` alone is not that edition: community builds carry it.
use crate::error::Error;
/// What every roles path answers without the Enterprise Edition.
pub fn datatable_roles_unavailable() -> Error {
Error::BadRequest("Data table roles are a Windmill Enterprise Edition feature".to_string())
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_roles_ee::{
can_use_datatable_role, can_use_datatable_role_in_governing_workspace, converge_connect_grants,
converge_connect_grants_with, create_instance_role, delete_role_catalog_entry,
drop_instance_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
ensure_instance_db_grant_options_unchecked, forget_datatable_role_everywhere,
insert_role_catalog_entry, read_role_catalog, read_role_catalog_tx,
registered_instance_databases, rename_instance_role, resolve_datatable_role_connection,
set_instance_role_login, update_role_catalog_entry,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) use ce::*;
#[cfg(not(all(feature = "private", feature = "enterprise")))]
mod ce {
use super::datatable_roles_unavailable as unavailable;
use crate::{
datatable_roles::{DatatableRoleCatalog, InstanceDatatableRole},
db::AuthedRef,
error::Result,
workspaces::{
resolve_governing_datatable, DataTableRoleTenants, DatatableAccess, GoverningDatatable,
},
DB,
};
type Tx<'a> = sqlx::Transaction<'a, sqlx::Postgres>;
pub(crate) async fn read_role_catalog(_db: &DB) -> Result<DatatableRoleCatalog> {
Err(unavailable())
}
pub(crate) async fn read_role_catalog_tx(_tx: &mut Tx<'_>) -> Result<DatatableRoleCatalog> {
Err(unavailable())
}
pub(crate) async fn insert_role_catalog_entry(
_tx: &mut Tx<'_>,
_id: &str,
_role: &InstanceDatatableRole,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn update_role_catalog_entry(
_tx: &mut Tx<'_>,
_id: &str,
_role: &InstanceDatatableRole,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn delete_role_catalog_entry(_tx: &mut Tx<'_>, _id: &str) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn registered_instance_databases(_db: &DB) -> Result<Vec<String>> {
Err(unavailable())
}
/// Nothing to converge: with no roles to admit, an instance database keeps the `CONNECT`
/// grants it was created with, `PUBLIC`'s included, as it did before roles existed.
pub(crate) async fn converge_connect_grants(_db: &DB, _dbname: &str) -> Result<()> {
Ok(())
}
/// As [`converge_connect_grants`].
pub(crate) async fn converge_connect_grants_with(
_db: &DB,
_dbname: &str,
_catalog: &DatatableRoleCatalog,
) -> Result<()> {
Ok(())
}
pub(crate) async fn create_instance_role(
_tx: &mut Tx<'_>,
_name: &str,
_password: &str,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn set_instance_role_login(
_tx: &mut Tx<'_>,
_name: &str,
_enabled: bool,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn rename_instance_role(
_tx: &mut Tx<'_>,
_from: &str,
_to: &str,
_password: &str,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn drop_instance_role(_db: &DB, _tx: &mut Tx<'_>, _name: &str) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn ensure_instance_db_grant_options_unchecked(
_db: &DB,
_dbname: &str,
) -> Result<()> {
Err(unavailable())
}
/// No tenant list covers anyone: there is no role to connect as.
pub(crate) fn can_use_datatable_role(
_tenants: &DataTableRoleTenants,
_authed: &AuthedRef<'_>,
) -> bool {
false
}
pub(crate) async fn can_use_datatable_role_in_governing_workspace(
_db: &DB,
_governing_w_id: &str,
_w_id: &str,
_tenants: &DataTableRoleTenants,
_access: &DatatableAccess<'_>,
) -> Result<bool> {
Err(unavailable())
}
/// Reached only for a data table under roles or a caller naming a role: both are refused.
pub(crate) async fn resolve_datatable_role_connection(
_db: &DB,
_w_id: &str,
_name: &str,
_governing: &GoverningDatatable,
_db_resource: serde_json::Value,
_role: Option<&str>,
_access: DatatableAccess<'_>,
) -> Result<serde_json::Value> {
Err(unavailable())
}
/// A data table not under roles, asked for no role or for `admin`, is not a role decision and
/// passes, as it did before roles existed. Anything else is refused.
pub(crate) async fn ensure_can_use_datatable_role(
db: &DB,
w_id: &str,
name: &str,
role: Option<&str>,
_access: &DatatableAccess<'_>,
_context: &str,
) -> Result<()> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
if governing.datatable.permissions.is_none()
&& role.is_none_or(|r| r == crate::datatable_roles::ADMIN_DATATABLE_ROLE)
{
Ok(())
} else {
Err(unavailable())
}
}
/// A data table not under roles is the `admin` connection for anyone who reaches it, as before
/// roles existed. One under roles is refused.
pub(crate) async fn ensure_datatable_admin_access(
db: &DB,
w_id: &str,
name: &str,
_access: &DatatableAccess<'_>,
) -> Result<()> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
if governing.datatable.permissions.is_none() {
Ok(())
} else {
Err(unavailable())
}
}
pub(crate) async fn forget_datatable_role_everywhere(
_tx: &mut Tx<'_>,
_role_id: &str,
) -> Result<()> {
Err(unavailable())
}
}
backend/windmill-common/src/lib.rs
+47 -11
View File
@@ -37,6 +37,10 @@ pub mod bench;
pub mod cache;
pub mod client;
pub mod data_metrics;
pub mod datatable_roles;
#[cfg(all(feature = "private", feature = "enterprise"))]
mod datatable_roles_ee;
pub mod datatable_roles_oss;
pub mod db;
#[cfg(all(feature = "enterprise", feature = "private"))]
mod db_entra_ee;
@@ -1514,6 +1518,41 @@ pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Resu
Ok(())
}
/// What `custom_instance_user` holds on an instance database.
///
/// `WITH GRANT OPTION` throughout: this is the connection every data table resolves to as `admin`,
/// and it is the one that hands privileges to data table roles. Postgres refuses to let a role pass
/// on a privilege it does not itself hold with grant option, so without these an admin could own
/// the database and still be unable to grant `SELECT` on it to `analytics`.
pub(crate) fn instance_db_grants(dbname: &str) -> String {
format!(
"GRANT CONNECT ON DATABASE \"{dbname}\" TO custom_instance_user WITH GRANT OPTION;
GRANT CREATE ON DATABASE \"{dbname}\" TO custom_instance_user WITH GRANT OPTION;
DO $$ BEGIN
IF EXISTS (SELECT 1 FROM pg_namespace WHERE nspname = 'public') THEN
GRANT USAGE ON SCHEMA public TO custom_instance_user WITH GRANT OPTION;
GRANT CREATE ON SCHEMA public TO custom_instance_user WITH GRANT OPTION;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO custom_instance_user;
END IF;
END $$;"
)
}
/// Re-apply [`instance_db_grants`] to an instance database provisioned before data table roles
/// existed, whose grants carry no grant option. Connects as the instance's own Postgres user —
/// the database and `public` schema owner — since only it can hand out an option it holds.
///
/// Authorization: reaches an instance database with the server's own credentials and checks
/// nothing. Callers MUST have authorized administration of `dbname` — superadmin, or an admin of
/// the workspace governing a data table on it.
pub async fn ensure_instance_db_grant_options_unchecked(
db: &DB,
dbname: &str,
) -> error::Result<()> {
crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, dbname).await
}
/// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake").
pub async fn create_custom_instance_database(
@@ -1553,17 +1592,7 @@ pub async fn create_custom_instance_database(
let (client, connection) = new_pg_creds.connect(Some(db)).await?;
let join_handle = tokio::spawn(async move { connection.await });
if let Err(e) = client
.batch_execute(&format!(
"GRANT CONNECT ON DATABASE \"{dbname}\" TO custom_instance_user;
GRANT USAGE ON SCHEMA public TO custom_instance_user;
GRANT CREATE ON SCHEMA public TO custom_instance_user;
GRANT CREATE ON DATABASE \"{dbname}\" TO custom_instance_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO custom_instance_user;"
))
.await
{
if let Err(e) = client.batch_execute(&instance_db_grants(dbname)).await {
tracing::warn!(
"Failed to grant permissions on '{}': {}. Continuing.",
dbname,
@@ -1592,6 +1621,13 @@ pub async fn create_custom_instance_database(
.execute(db)
.await?;
// A data table role can only reach a database it may CONNECT to, and PUBLIC's default CONNECT
// would otherwise let every role in regardless of what this instance defines. Best-effort: a
// failure here leaves the database usable as `admin`, and the next role change repairs it.
if let Err(e) = crate::datatable_roles::converge_connect_grants(db, dbname).await {
tracing::warn!("Could not set CONNECT grants on instance database '{dbname}': {e}");
}
tracing::info!("Created custom instance database '{}'", dbname);
Ok(())
}
backend/windmill-common/src/worker.rs
+127
View File
@@ -1082,6 +1082,83 @@ pub struct SqlAnnotations {
pub raw_output: bool,
}
impl SqlAnnotations {
/// The data table role a query declares as `-- role <name>`, if any. Only meaningful against a
/// `datatable://` database that is under roles; absent means the data table's default role.
///
/// Hand-written rather than derived because the value matters, not just the presence, and
/// because the executor needs it before it knows the connection is a data table at all. Like
/// every annotation it lives in the leading comment block.
///
/// A leading comment whose first word is `role` is an annotation *attempt*, and a malformed
/// one is an error. The alternative — ignoring what does not parse — resolves the query to the
/// data table's default role instead, so a typo silently runs it under a login the author did
/// not choose, which is the opposite of what naming a role is for. Only callers that already
/// know the target is a `datatable://` reference ever run this, so ordinary SQL keeps its
/// comments.
pub fn datatable_role(code: &str) -> error::Result<Option<String>> {
for line in code.lines() {
let line = line.trim();
if line.is_empty() {
continue;
}
if !line.starts_with("--") {
break;
}
// The keyword may be followed by whitespace, `:` or `=` — `role x`, `role: x`,
// `role=x`, `Role = x` all open an attempt, while `rolexyz` does not. Each accepted
// separator is one spelling that would otherwise take the `continue` below and run the
// query as the data table's default role, which is the silence this exists to remove.
let body = line[2..].trim_start();
let Some(after) = body
.get(..4)
.filter(|kw| kw.eq_ignore_ascii_case("role"))
.map(|_| &body[4..])
else {
continue;
};
if !after.is_empty()
&& !after.starts_with(char::is_whitespace)
&& !after.starts_with([':', '='])
{
continue;
}
// Past this point the line is an attempt to name a role, so a malformed one is an
// error rather than a miss. Falling through would run the query as the data table's
// default role — quietly, and under a login the author did not choose.
let after = after.trim_start();
let after = after.strip_prefix([':', '=']).unwrap_or(after);
let mut tokens = after.split_whitespace();
let role = tokens
.next()
.map(|role| role.strip_suffix(';').unwrap_or(role));
let rest = tokens.next();
match (role, rest) {
(Some(role), None)
if !role.is_empty()
&& role.len() <= 63
&& role
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') =>
{
return Ok(Some(role.to_string()));
}
_ => {
return Err(error::Error::BadRequest(format!(
"Malformed data table role annotation: `{line}`. Write it as \
`-- role <name>` on a line of its own, where <name> is letters, digits, \
'_' or '-'. A comment in the leading block that starts with the word \
'role' is read as this annotation; move it below the first statement if \
it is prose."
)));
}
}
}
Ok(None)
}
}
#[annotations("#")]
pub struct BashAnnotations {
pub docker: bool,
@@ -2653,6 +2730,56 @@ mod tests {
use super::*;
use std::collections::HashMap;
#[test]
fn datatable_role_is_read_from_the_leading_comment_block() {
let role = |code| SqlAnnotations::datatable_role(code);
assert_eq!(
role("-- role analytics\nSELECT 1").unwrap(),
Some("analytics".to_string())
);
// Blank lines and other annotations before it are fine.
assert_eq!(
role("\n-- prepare\n-- role read_only\nSELECT 1").unwrap(),
Some("read_only".to_string())
);
// Past the first statement it is an ordinary comment, not an annotation.
assert_eq!(role("SELECT 1;\n-- role analytics").unwrap(), None);
assert_eq!(role("SELECT 1").unwrap(), None);
// Unambiguous intent is honoured: the keyword matches case-insensitively, a trailing
// semicolon is a habit carried over from SQL rather than a different role, and the colon
// spelling is the one most likely to be typed.
for accepted in [
"-- Role operator\nSELECT 1",
"-- role operator;\nSELECT 1",
"-- role: operator\nSELECT 1",
"-- role:operator\nSELECT 1",
"-- role=operator\nSELECT 1",
"-- Role = operator\nSELECT 1",
] {
assert_eq!(
role(accepted).unwrap(),
Some("operator".to_string()),
"not honoured: {accepted}"
);
}
// Anything else opening with the word is refused rather than resolved to the default role:
// the whole point of naming one is to not run as something else.
for near_miss in [
"-- role operator -- why\nSELECT 1",
"-- role an;alytics\nSELECT 1",
"-- role\nSELECT 1",
"-- role:\nSELECT 1",
"-- role based access is handled below\nSELECT 1",
] {
assert!(role(near_miss).is_err(), "silently ignored: {near_miss}");
}
// A word that merely starts with the keyword is not an attempt.
assert_eq!(role("-- rolebased notes\nSELECT 1").unwrap(), None);
}
fn matcher(id: &str) -> WorkspaceMatcher {
WorkspaceMatcher { id: id.to_string(), include_forks: false }
}
backend/windmill-common/src/workspaces.rs
+771 -46
View File
@@ -6,6 +6,7 @@ use serde::{Deserialize, Serialize};
use strum::AsRefStr;
use crate::{
datatable_roles::{ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER},
error::{self, to_anyhow, Error, Result},
get_database_url,
secret_backend::{get_secret_value, is_external_stored_value},
@@ -1292,9 +1293,17 @@ impl Default for DataTableForkBehavior {
}
}
#[derive(Deserialize, Serialize, Debug)]
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct DataTable {
pub database: DataTableDatabase,
/// Set on a *terminal* entry — one that owns its database. Mutually exclusive with
/// [`DataTable::reference`]; [`validate_datatable_shape`] is the one place that enforces it.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub database: Option<DataTableDatabase>,
/// Set on a *pointer* entry — one that names another workspace's entry and owns nothing.
/// A keep-original fork gets one of these instead of a copy of the parent's entry, so there is
/// nothing local for a fork admin to widen.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub reference: Option<DataTableReference>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub forked_from: Option<DataTableForkedFrom>,
/// Whether the SQL-migrations feature is opted in for this data table.
@@ -1302,22 +1311,85 @@ pub struct DataTable {
/// when migrations already exist (see `datatable_migrations_enabled`).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub migrations_enabled: Option<bool>,
/// Who may connect as which role. Absent = unpermissioned: every caller connects as `admin`,
/// which is how data tables behaved before roles existed. Only meaningful on a terminal entry;
/// a pointer is governed by what it points at.
///
/// Never leaves the instance: stripped from the workspace export and ignored on import, since
/// tenants are workspace-scoped names and syncing them would make repo write access a second
/// door onto the access decision.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub permissions: Option<DataTablePermissions>,
}
#[derive(Deserialize, Serialize, Debug)]
/// A pointer at another workspace's data table entry.
#[derive(Deserialize, Serialize, Debug, Clone, PartialEq)]
pub struct DataTableReference {
pub workspace_id: String,
pub datatable: String,
}
/// The access decision for one data table: which role a caller gets, and who may ask for each.
#[derive(Deserialize, Serialize, Debug, Clone, Default)]
pub struct DataTablePermissions {
/// A role id from the instance catalog, or `admin`. Absent = `admin`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub default_role: Option<String>,
/// Keyed by instance role id, plus the reserved `admin` key. A role absent from this map
/// cannot be used on this data table at all, whatever the instance catalog says.
#[serde(default)]
pub roles: std::collections::BTreeMap<String, DataTableRoleTenants>,
}
impl DataTablePermissions {
pub fn default_role(&self) -> &str {
self.default_role.as_deref().unwrap_or(ADMIN_DATATABLE_ROLE)
}
}
#[derive(Deserialize, Serialize, Debug, Clone, Default)]
pub struct DataTableRoleTenants {
/// `u/<user>`, `g/<group>`, `f/<folder>`, or `*` for every member.
#[serde(default)]
pub tenants: Vec<String>,
}
/// Every member of the governing workspace.
pub const DATATABLE_TENANT_WILDCARD: &str = "*";
/// How deep a chain of pointer entries may go before it is called a loop. Data tables are not
/// expected to chain at all — a fork points at its parent — so this only has to be generous
/// enough to survive a fork of a fork.
const DATATABLE_REFERENCE_MAX_DEPTH: usize = 20;
/// Exactly one of `database` and `reference` must be set. Called wherever an entry is persisted,
/// so nothing downstream has to handle an entry that is both or neither.
pub fn validate_datatable_shape(name: &str, dt: &DataTable) -> Result<()> {
match (&dt.database, &dt.reference) {
(Some(_), None) | (None, Some(_)) => Ok(()),
(Some(_), Some(_)) => Err(Error::BadRequest(format!(
"Data table '{name}' both owns a database and points at another one"
))),
(None, None) => Err(Error::BadRequest(format!(
"Data table '{name}' names neither a database nor another data table"
))),
}
}
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct DataTableForkedFrom {
/// Schema snapshot at fork time
#[serde(default, skip_serializing_if = "Option::is_none")]
pub schema: Option<serde_json::Value>,
}
#[derive(Deserialize, Serialize, Debug)]
#[derive(Deserialize, Serialize, Debug, Clone)]
pub struct DataTableDatabase {
pub resource_type: DataTableCatalogResourceType,
pub resource_path: String,
}
#[derive(Deserialize, Serialize, Debug, PartialEq)]
#[derive(Deserialize, Serialize, Debug, PartialEq, Clone, Copy)]
#[serde(rename_all = "lowercase")]
#[derive(AsRefStr)]
#[strum(serialize_all = "lowercase")]
@@ -1353,37 +1425,13 @@ fn datatable_not_found_error(name: &str, datatables: Option<&serde_json::Value>)
))
}
pub async fn get_datatable_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
get_datatable_resource_inner(db, w_id, name, false).await
}
/// Same as [`get_datatable_resource_from_db_unchecked`] but for postgres trigger
/// connections: custom-instance datatables resolve to
/// `custom_instance_replication_user` rather than `custom_instance_user`. BYO-postgres
/// datatables resolve to the user's own resource unchanged; configuring it for
/// replication there is the user's responsibility.
/// Read one workspace's data table entry, without following a pointer.
///
/// Authorization: like its `_unchecked` sibling, returns resolved connection
/// credentials and performs no authorization — callers MUST have already authorized
/// access to the datatable (e.g. the trigger's own create-time check).
pub async fn get_datatable_replication_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
get_datatable_resource_inner(db, w_id, name, true).await
}
async fn get_datatable_resource_inner(
db: &DB,
w_id: &str,
name: &str,
replication: bool,
) -> Result<serde_json::Value> {
/// Disclosure: this is the primitive [`resolve_governing_datatable`] calls on every path, so it is
/// deliberately open to anything that has to resolve a data table, including for a workspace the
/// caller does not belong to. What it returns is not: callers MUST NOT put `permissions` into a
/// response, an export or a log — it names the governing workspace's users, groups and folders.
pub async fn read_datatable_entry(db: &DB, w_id: &str, name: &str) -> Result<DataTable> {
let datatables = sqlx::query_scalar!(
r#"
SELECT ws.datatable->'datatables' AS datatables
@@ -1401,37 +1449,590 @@ async fn get_datatable_resource_inner(
.and_then(|d| d.get(name))
.filter(|v| !v.is_null())
.ok_or_else(|| datatable_not_found_error(name, datatables.as_ref()))?;
let datatable = serde_json::from_value::<DataTable>(datatable.clone())?;
Ok(serde_json::from_value::<DataTable>(datatable.clone())?)
}
let db_resource = if datatable.database.resource_type == DataTableCatalogResourceType::Instance
{
/// The terminal entry a reference chain lands on: the workspace that governs the data table, the
/// entry name there, and the entry itself. A terminal entry resolves to itself.
///
/// Every decision downstream — which database to connect to, whose `permissions` apply, whose
/// members tenants are evaluated against, who may administer it — is taken on this, never on the
/// entry the caller named.
///
/// Authorization: resolving deliberately crosses into the governing workspace, so it answers for a
/// workspace the caller may not belong to and checks nothing itself. It is the input to the
/// checks, not one of them: callers MUST pass what it returns to
/// [`can_use_datatable_role_in_governing_workspace`] or [`ensure_datatable_admin_access`] before
/// acting on it, and MUST NOT return its `permissions` or `workspace_id` to a caller from
/// elsewhere without gating on the answer.
pub struct GoverningDatatable {
pub workspace_id: String,
pub name: String,
pub datatable: DataTable,
}
impl GoverningDatatable {
/// Backed by the Windmill instance's own Postgres, which is the only substrate data table
/// roles apply to.
pub fn is_instance(&self) -> bool {
self.datatable
.database
.as_ref()
.is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
}
}
pub async fn resolve_governing_datatable(
db: &DB,
w_id: &str,
name: &str,
) -> Result<GoverningDatatable> {
let mut workspace_id = w_id.to_string();
let mut name = name.to_string();
let mut hops = 0;
for _ in 0..DATATABLE_REFERENCE_MAX_DEPTH {
let datatable = read_datatable_entry(db, &workspace_id, &name)
.await
.map_err(|e| {
if hops == 0 {
e
} else {
// A pointer outlives the workspace it names: deleting one only nulls the fork
// lineage, it does not sweep the entries that pointed at it. Say which one is
// gone rather than reporting a data table this workspace never had.
Error::NotFound(format!(
"Data table '{name}' of workspace '{workspace_id}' governs this one and no \
longer exists. A superadmin can point this data table somewhere else."
))
}
})?;
hops += 1;
validate_datatable_shape(&name, &datatable)?;
match &datatable.reference {
None => return Ok(GoverningDatatable { workspace_id, name, datatable }),
Some(reference) => {
workspace_id = reference.workspace_id.clone();
name = reference.datatable.clone();
}
}
}
Err(Error::BadRequest(format!(
"Data table '{name}' points at another data table through more than \
{DATATABLE_REFERENCE_MAX_DEPTH} hops; the chain is likely a loop"
)))
}
/// Every entry of a workspace resolved as [`resolve_governing_datatable`] resolves one, in stored
/// order, reading the settings rows one pointer hop at a time rather than once per entry. An entry
/// that does not resolve — malformed, a dangling pointer, a loop — is left out. Same authorization
/// contract as the single resolution: it checks nothing.
pub async fn resolve_workspace_governing_datatables(
db: &DB,
w_id: &str,
) -> Result<Vec<(String, GoverningDatatable)>> {
type Entries =
std::collections::HashMap<String, std::collections::HashMap<String, serde_json::Value>>;
async fn load(db: &DB, workspaces: &[String], entries: &mut Entries) -> Result<Vec<String>> {
let rows: Vec<(String, String, serde_json::Value)> = sqlx::query_as(
"SELECT ws.workspace_id, dt.key, dt.value FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
WHERE ws.workspace_id = ANY($1)",
)
.bind(workspaces)
.fetch_all(db)
.await?;
for ws in workspaces {
entries.entry(ws.clone()).or_default();
}
let mut keys = Vec::with_capacity(rows.len());
for (ws, key, value) in rows {
keys.push(key.clone());
entries.entry(ws).or_default().insert(key, value);
}
Ok(keys)
}
let mut entries = Entries::new();
let listed = load(db, &[w_id.to_string()], &mut entries).await?;
// (index into `listed`, workspace, entry name) still to be followed.
let mut cursors: Vec<(usize, String, String)> = listed
.iter()
.enumerate()
.map(|(i, name)| (i, w_id.to_string(), name.clone()))
.collect();
let mut resolved: Vec<(usize, GoverningDatatable)> = vec![];
for _ in 0..DATATABLE_REFERENCE_MAX_DEPTH {
let mut next = vec![];
for (i, ws, name) in cursors.drain(..) {
let Some(value) = entries
.get(&ws)
.and_then(|m| m.get(&name))
.filter(|v| !v.is_null())
else {
continue;
};
let Ok(datatable) = serde_json::from_value::<DataTable>(value.clone()) else {
continue;
};
if validate_datatable_shape(&name, &datatable).is_err() {
continue;
}
match &datatable.reference {
None => {
resolved.push((i, GoverningDatatable { workspace_id: ws, name, datatable }))
}
Some(reference) => next.push((
i,
reference.workspace_id.clone(),
reference.datatable.clone(),
)),
}
}
if next.is_empty() {
break;
}
let to_load: Vec<String> = next
.iter()
.map(|(_, ws, _)| ws.clone())
.filter(|ws| !entries.contains_key(ws))
.collect::<std::collections::BTreeSet<_>>()
.into_iter()
.collect();
if !to_load.is_empty() {
load(db, &to_load, &mut entries).await?;
}
cursors = next;
}
resolved.sort_by_key(|(i, _)| *i);
Ok(resolved
.into_iter()
.map(|(i, governing)| (listed[i].clone(), governing))
.collect())
}
/// Build the `admin` connection for a governing entry: `custom_instance_user` for an instance
/// database, the user's own resource for a BYO-postgres one.
async fn resolve_datatable_connection_unchecked(
db: &DB,
governing: &GoverningDatatable,
replication: bool,
) -> Result<serde_json::Value> {
let database = governing
.datatable
.database
.as_ref()
.expect("a governing entry owns a database");
if database.resource_type == DataTableCatalogResourceType::Instance {
let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
pg_creds.dbname = datatable.database.resource_path.clone();
pg_creds.dbname = database.resource_path.clone();
if replication {
pg_creds.user = Some("custom_instance_replication_user".to_string());
pg_creds.password = Some(get_custom_pg_instance_replication_password(&db).await?);
} else {
pg_creds.user = Some("custom_instance_user".to_string());
pg_creds.user = Some(CUSTOM_INSTANCE_USER.to_string());
pg_creds.password = Some(get_custom_pg_instance_password(&db).await?);
}
serde_json::to_value(&pg_creds)
.map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))?
.map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))
} else {
// Name the data table too: the caller asked for one by name, and a bare
// "resource f/x/y does not exist" leaves them to work out which one points at it.
transform_json_unchecked(
&serde_json::Value::String(format!("$res:{}", datatable.database.resource_path)),
w_id,
&serde_json::Value::String(format!("$res:{}", database.resource_path)),
&governing.workspace_id,
db,
)
.await
.map_err(|e| match e {
Error::NotFound(m) => Error::NotFound(format!("data table {name}: {m}")),
Error::NotFound(m) => Error::NotFound(format!("data table {}: {m}", governing.name)),
e => e,
})?
})
}
}
/// Resolve a data table to connection credentials **without authorizing anything**: always the
/// `admin` connection.
///
/// Authorization: callers MUST have authorized access already. Anything that acts for a user or a
/// job wants [`get_datatable_resource_from_db`] instead.
pub async fn get_datatable_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
resolve_datatable_connection_unchecked(db, &governing, false).await
}
/// Same as [`get_datatable_resource_from_db_unchecked`] but for postgres trigger
/// connections: custom-instance datatables resolve to
/// `custom_instance_replication_user` rather than `custom_instance_user`. BYO-postgres
/// datatables resolve to the user's own resource unchanged; configuring it for
/// replication there is the user's responsibility.
///
/// Authorization: a replication connection reads every row whatever the roles grant, so no role or
/// admin check makes it safe. Callers MUST refuse a data table under roles outright — the Postgres
/// trigger crate's `ensure_not_under_roles` — and turning roles on is refused while one streams.
pub async fn get_datatable_replication_resource_from_db_unchecked(
db: &DB,
w_id: &str,
name: &str,
) -> Result<serde_json::Value> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
resolve_datatable_connection_unchecked(db, &governing, true).await
}
/// The identity a resolution is made for. `Unchecked` is for callers that authorized already;
/// everything else is checked against the governing entry's tenants.
pub enum DatatableAccess<'a> {
/// Reaches every role. For callers that already authorized, or that have no user at all.
Unchecked,
Authed(crate::db::AuthedRef<'a>),
/// A job's owner, without reading the job row — only fetched if the data table turns out to
/// be permissioned.
PermissionedAs {
permissioned_as: &'a str,
email: &'a str,
},
/// A job identified by id; its owner is read from `v2_job`. For agent workers and anything
/// else that authenticates as infrastructure rather than as the job's user.
Job(uuid::Uuid),
/// No identity established. Unpermissioned data tables resolve as before; permissioned ones
/// are refused, so a caller predating this feature fails closed.
NoIdentity,
}
/// Does one tenant list cover this identity? Admins of the governing workspace pass everything —
/// they can edit the tenant lists anyway, so refusing them would only be theatre.
pub fn can_use_datatable_role(
tenants: &DataTableRoleTenants,
authed: &crate::db::AuthedRef<'_>,
) -> bool {
crate::datatable_roles_oss::can_use_datatable_role(tenants, authed)
}
/// Evaluate a tenant list **as a member of the governing workspace**, whoever is calling.
///
/// A caller reaching a data table through a pointer is a member of some other workspace, and being
/// its admin means nothing here — that is the whole point of the pointer. They are looked up in
/// the governing workspace by email and evaluated there, or refused when they are not a member.
/// A `g/` or `f/` permissioned-as from a foreign workspace is refused outright: those names are
/// defined per workspace and mean nothing outside the one that defined them.
pub async fn can_use_datatable_role_in_governing_workspace(
db: &DB,
governing_w_id: &str,
w_id: &str,
tenants: &DataTableRoleTenants,
access: &DatatableAccess<'_>,
) -> Result<bool> {
crate::datatable_roles_oss::can_use_datatable_role_in_governing_workspace(
db,
governing_w_id,
w_id,
tenants,
access,
)
.await
}
/// Resolve a data table to connection credentials for one identity.
///
/// This is the chokepoint: everything that opens a connection to a data table on someone's behalf
/// goes through it. `role` is the role name the caller asked for — the `-- role` annotation, the
/// `?role=` on a `datatable://` reference, or `None` for the data table's default.
///
/// The resolved role **logs in as itself**. Never `SET ROLE`: a script could `RESET ROLE` its way
/// back to admin.
pub async fn get_datatable_resource_from_db(
db: &DB,
w_id: &str,
name: &str,
role: Option<&str>,
access: DatatableAccess<'_>,
) -> Result<serde_json::Value> {
let governing = resolve_governing_datatable(db, w_id, name).await?;
let db_resource = resolve_datatable_connection_unchecked(db, &governing, false).await?;
// Not under roles and asked for none, or for `admin` by name: the `admin` connection, as before
// roles existed, in every edition. Anything else is a role decision. Every migration names
// `admin` explicitly, so an edition without roles must not treat that as one.
if governing.datatable.permissions.is_none() && role.is_none_or(|r| r == ADMIN_DATATABLE_ROLE) {
return Ok(db_resource);
}
crate::datatable_roles_oss::resolve_datatable_role_connection(
db,
w_id,
name,
&governing,
db_resource,
role,
access,
)
.await
}
/// Would the chokepoint accept this identity connecting as this role? Answers without resolving
/// credentials, for callers that want to refuse early and say which thing was refused.
///
/// Not the security boundary — [`get_datatable_resource_from_db`] re-checks when it actually opens
/// the connection. This is what turns "permission denied for table x" into a message naming the
/// migration and the role.
pub async fn ensure_can_use_datatable_role(
db: &DB,
w_id: &str,
name: &str,
role: Option<&str>,
access: &DatatableAccess<'_>,
context: &str,
) -> Result<()> {
crate::datatable_roles_oss::ensure_can_use_datatable_role(db, w_id, name, role, access, context)
.await
}
/// Gate the operations that see the whole database whatever the roles grant: a migration that
/// declares no role, exports, and editing the permissions themselves. Not replication, which a
/// data table under roles refuses whoever asks (see `ensure_not_under_roles`). Passing
/// means the caller could have connected as `admin` anyway.
pub async fn ensure_datatable_admin_access(
db: &DB,
w_id: &str,
name: &str,
access: &DatatableAccess<'_>,
) -> Result<()> {
crate::datatable_roles_oss::ensure_datatable_admin_access(db, w_id, name, access).await
}
/// Rewrite the `permissions` of every data table entry of one workspace, in the caller's
/// transaction. `change` reports whether it touched anything; the row is only written when
/// something did.
///
/// Authorization: writes an access decision for any workspace named, with an arbitrary mutation,
/// and checks nothing. It exists for the cascades below — the transaction that frees or renames a
/// principal — so callers MUST be the operation that made the principal change, and MUST run in
/// its transaction. Anything editing a decision on purpose belongs in the permissions endpoint,
/// which is gated on the workspace that governs the data table.
///
/// The tenant lists name principals of this workspace, so anything that frees or renames one has
/// to come through here in the same transaction that frees it — otherwise a `u/alice` reused by a
/// later account silently inherits her access.
pub async fn update_datatable_permissions_in_workspace<F>(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
w_id: &str,
change: F,
) -> Result<()>
where
F: Fn(&mut DataTablePermissions) -> bool,
{
let Some(mut settings) = sqlx::query_scalar!(
"SELECT datatable FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE",
w_id
)
.fetch_optional(&mut **tx)
.await?
.flatten() else {
return Ok(());
};
Ok(db_resource)
let Some(datatables) = settings
.get_mut("datatables")
.and_then(|d| d.as_object_mut())
else {
return Ok(());
};
let mut touched = false;
for entry in datatables.values_mut() {
let Some(permissions) = entry.get("permissions").filter(|p| !p.is_null()) else {
continue;
};
let Ok(mut permissions) =
serde_json::from_value::<DataTablePermissions>(permissions.clone())
else {
continue;
};
if change(&mut permissions) {
entry["permissions"] = serde_json::to_value(&permissions)
.map_err(|e| Error::internal_err(format!("serializing permissions: {e}")))?;
touched = true;
}
}
if touched {
sqlx::query!(
"UPDATE workspace_settings SET datatable = $1 WHERE workspace_id = $2",
settings,
w_id
)
.execute(&mut **tx)
.await?;
}
Ok(())
}
/// Drop a freed principal (`u/alice`, `g/analysts`, `f/finance`) from every tenant list of one
/// workspace. Same contract as [`update_datatable_permissions_in_workspace`]: for the transaction
/// that frees the principal, not for editing a decision.
pub async fn remove_datatable_tenant_in_workspace(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
w_id: &str,
tenant: &str,
) -> Result<()> {
update_datatable_permissions_in_workspace(tx, w_id, |permissions| {
let mut touched = false;
for role in permissions.roles.values_mut() {
let before = role.tenants.len();
role.tenants.retain(|t| t != tenant);
touched |= role.tenants.len() != before;
}
touched
})
.await
}
/// Follow a renamed principal through every tenant list of one workspace. Same contract as
/// [`update_datatable_permissions_in_workspace`].
pub async fn rename_datatable_tenant_in_workspace(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
w_id: &str,
old: &str,
new: &str,
) -> Result<()> {
update_datatable_permissions_in_workspace(tx, w_id, |permissions| {
let mut touched = false;
for role in permissions.roles.values_mut() {
let mut role_touched = false;
for tenant in role.tenants.iter_mut() {
if tenant == old {
*tenant = new.to_string();
role_touched = true;
}
}
if role_touched {
// The rename can collide with a name already in the list, and the two need not be
// adjacent — `Vec::dedup` only collapses neighbours, so it would leave the pair.
let mut seen = std::collections::HashSet::new();
role.tenants.retain(|t| seen.insert(t.clone()));
touched = true;
}
}
touched
})
.await
}
/// Strip a deleted instance role from every workspace that had tenanted it, so nothing is left
/// naming a role that no longer exists.
///
/// Authorization: reaches every workspace on the instance. Callers MUST be the superadmin path
/// dropping the role from the cluster — it exists to follow that, not to edit tenants.
///
/// Takes that path's transaction rather than opening its own: run afterwards, a failure part-way
/// leaves the catalog row already gone, so the retry answers `NotFound` while some workspaces
/// still name a role nothing can connect as. In the transaction, the cluster drop, the catalog row
/// and every tenant list commit together or not at all. A data table whose default role was the deleted one falls
/// back to `admin` — the one role that is always present.
pub async fn forget_datatable_role_everywhere(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
role_id: &str,
) -> Result<()> {
crate::datatable_roles_oss::forget_datatable_role_everywhere(tx, role_id).await
}
/// Drop the `permissions` block from a `workspace_settings.datatable` value before it leaves the
/// server.
///
/// Who may connect as which role is an access decision, not configuration, and its tenants name
/// principals of one workspace — `g/analysts` in dev is a different group from `g/analysts` in
/// prod. Shipping it would both mean nothing at the far end and turn a settings push into a way to
/// widen access, so the decision stays where it was made. [`DataTable`] deserializes fine without
/// it, and the settings-editing endpoint carries the stored block across untouched.
pub fn strip_datatable_permissions(
datatable: Option<serde_json::Value>,
) -> Option<serde_json::Value> {
let mut datatable = datatable?;
if let Some(entries) = datatable
.get_mut("datatables")
.and_then(|d| d.as_object_mut())
{
for entry in entries.values_mut() {
if let Some(entry) = entry.as_object_mut() {
entry.remove("permissions");
}
}
}
Some(datatable)
}
/// As [`parse_datatable_ref`], except that an entry whose stored name itself contains `?` — which
/// names could before they were restricted — resolves by that exact name, without a role. It is
/// looked up first, so `sales?role=x` never reaches a different entry than the one stored so.
///
/// Authorization: checks nothing, and its answer reveals whether `w_id` stores that exact name.
/// Callers MUST already act for `w_id` — a job of it, or a caller authenticated into it — and
/// MUST still pass the name to [`get_datatable_resource_from_db`] or an admin-access check.
pub async fn parse_datatable_ref_for(
db: &DB,
w_id: &str,
reference: &str,
) -> Result<(String, Option<String>)> {
if reference.contains('?') {
let exists = sqlx::query_scalar::<_, Option<bool>>(
"SELECT (datatable->'datatables') ? $2 FROM workspace_settings WHERE workspace_id = $1",
)
.bind(w_id)
.bind(reference)
.fetch_optional(db)
.await?
.flatten()
.unwrap_or(false);
if exists {
return Ok((reference.to_string(), None));
}
}
let (name, role) = parse_datatable_ref(reference)?;
Ok((name.to_string(), role.map(str::to_string)))
}
/// Split a `datatable://` reference into its name and the role its query string names.
///
/// A query string that does not parse is an error rather than an absent role. Falling back would
/// resolve the reference to the data table's default role, so `?Role=analytics` or a mistyped
/// `?role=` would quietly connect as something the caller did not ask for — the same trap as a
/// malformed `-- role` annotation, and `role` is the only parameter a reference takes.
pub fn parse_datatable_ref(reference: &str) -> Result<(&str, Option<&str>)> {
let (name, query) = reference.split_once('?').unwrap_or((reference, ""));
let mut role = None;
for param in query.split('&').filter(|p| !p.is_empty()) {
let (key, value) = param.split_once('=').unwrap_or((param, ""));
if !key.eq_ignore_ascii_case("role") {
return Err(Error::BadRequest(format!(
"Data table reference '{name}' carries an unknown parameter '{key}'. \
The only one it takes is `?role=<name>`."
)));
}
if role.is_some() {
return Err(Error::BadRequest(format!(
"Data table reference '{name}' names a role more than once."
)));
}
if value.is_empty() || !is_datatable_role_name(value) {
return Err(Error::BadRequest(format!(
"Data table reference '{name}' has a malformed role '{value}'. Write it as \
`?role=<name>`, where <name> is letters, digits, '_' or '-'."
)));
}
role = Some(value);
}
Ok((name, role))
}
fn is_datatable_role_name(role: &str) -> bool {
!role.is_empty()
&& role.len() <= 63
&& role
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
}
#[derive(Deserialize, Serialize, Debug)]
@@ -2642,6 +3243,130 @@ async fn transform_json_unchecked(
mod tests {
use super::*;
fn tenants(list: &[&str]) -> DataTableRoleTenants {
DataTableRoleTenants { tenants: list.iter().map(|t| t.to_string()).collect() }
}
#[cfg(all(feature = "private", feature = "enterprise"))]
#[test]
fn a_tenant_list_covers_users_groups_folders_and_the_wildcard() {
let groups = vec!["analysts".to_string()];
let folders = vec![("finance".to_string(), true, false)];
let scopes = None;
let token_prefix = None;
let is_admin = false;
let is_operator = false;
let authed = crate::db::AuthedRef {
email: "alice@windmill.dev",
username: "alice",
is_admin: &is_admin,
is_operator: &is_operator,
groups: &groups,
folders: &folders,
scopes: &scopes,
token_prefix: &token_prefix,
};
assert!(can_use_datatable_role(&tenants(&["u/alice"]), &authed));
assert!(can_use_datatable_role(&tenants(&["g/analysts"]), &authed));
assert!(can_use_datatable_role(&tenants(&["f/finance"]), &authed));
assert!(can_use_datatable_role(&tenants(&["*"]), &authed));
assert!(!can_use_datatable_role(&tenants(&[]), &authed));
assert!(!can_use_datatable_role(
&tenants(&["u/bob", "g/ops"]),
&authed
));
// A bare name is not a principal: only the three prefixes and the wildcard match.
assert!(!can_use_datatable_role(&tenants(&["alice"]), &authed));
// An admin of the governing workspace reaches every role: they can edit the lists anyway.
let is_admin = true;
let admin = crate::db::AuthedRef { is_admin: &is_admin, ..authed };
assert!(can_use_datatable_role(&tenants(&[]), &admin));
}
#[cfg(not(all(feature = "private", feature = "enterprise")))]
#[test]
fn without_the_enterprise_edition_no_tenant_list_covers_anyone() {
let groups = vec![];
let folders = vec![];
let scopes = None;
let token_prefix = None;
let is_admin = true;
let is_operator = false;
let admin = crate::db::AuthedRef {
email: "alice@windmill.dev",
username: "alice",
is_admin: &is_admin,
is_operator: &is_operator,
groups: &groups,
folders: &folders,
scopes: &scopes,
token_prefix: &token_prefix,
};
assert!(!can_use_datatable_role(&tenants(&["*"]), &admin));
assert!(!can_use_datatable_role(&tenants(&["u/alice"]), &admin));
}
#[test]
fn a_datatable_ref_splits_off_its_role() {
assert_eq!(parse_datatable_ref("sales").unwrap(), ("sales", None));
assert_eq!(
parse_datatable_ref("sales?role=analytics").unwrap(),
("sales", Some("analytics"))
);
// The key matches case-insensitively, the way the `-- role` annotation does.
assert_eq!(
parse_datatable_ref("sales?Role=analytics").unwrap(),
("sales", Some("analytics"))
);
// A query string that does not parse is refused rather than read as "no role": resolving
// it to the data table's default would connect as a login the caller never asked for.
for malformed in [
"sales?role=",
"sales?role=an;alytics",
"sales?x=1&role=analytics",
"sales?role=a&role=b",
] {
assert!(
parse_datatable_ref(malformed).is_err(),
"silently ignored: {malformed}"
);
}
}
#[test]
fn an_entry_owns_a_database_or_points_at_one_but_never_both() {
let terminal = DataTable {
database: Some(DataTableDatabase {
resource_type: DataTableCatalogResourceType::Instance,
resource_path: "dt_main".to_string(),
}),
reference: None,
forked_from: None,
migrations_enabled: None,
permissions: None,
};
assert!(validate_datatable_shape("main", &terminal).is_ok());
let pointer = DataTable {
database: None,
reference: Some(DataTableReference {
workspace_id: "prod".to_string(),
datatable: "main".to_string(),
}),
..terminal.clone()
};
assert!(validate_datatable_shape("main", &pointer).is_ok());
let both = DataTable { database: terminal.database.clone(), ..pointer.clone() };
assert!(validate_datatable_shape("main", &both).is_err());
let neither = DataTable { database: None, reference: None, ..terminal.clone() };
assert!(validate_datatable_shape("main", &neither).is_err());
}
#[test]
fn test_parse_fork_branch() {
// Generated fork (`wm-fork-abc`) and dev workspace (`staging`) forms.
backend/windmill-trigger-postgres/src/handler.rs
+28 -2
View File
@@ -21,11 +21,12 @@ use windmill_common::{
use windmill_git_sync::DeployedObject;
use windmill_api_auth::{check_scopes, ApiAuthed};
use windmill_trigger::{Trigger, TriggerCrud, TriggerData};
use windmill_trigger::{Trigger, TriggerCrud, TriggerData, TriggerMode};
use super::{
check_if_valid_publication_for_postgres_version, create_logical_replication_slot,
create_pg_publication, drop_publication, generate_random_string, get_default_pg_connection,
create_pg_publication, drop_publication, ensure_not_under_roles, generate_random_string,
get_default_pg_connection,
mapper::{Mapper, MappingInfo},
PostgresConfig, PostgresConfigRequest, PostgresPublicationReplication, PostgresTrigger,
PublicationData, Relations, Slot, SlotList, TableToTrack, TemplateScript, TestPostgresConfig,
@@ -64,6 +65,29 @@ impl TriggerCrud for PostgresTrigger {
DeployedObject::PostgresTrigger { path, parent_path }
}
async fn validate_config(
&self,
db: &DB,
config: &Self::TriggerConfigRequest,
workspace_id: &str,
) -> Result<()> {
ensure_not_under_roles(db, workspace_id, &config.postgres_resource_path).await
}
async fn authorize_set_trigger_mode(
&self,
_authed: &ApiAuthed,
tx: &mut PgConnection,
_workspace_id: &str,
_path: &str,
mode: &TriggerMode,
) -> Result<()> {
if *mode != TriggerMode::Disabled {
windmill_common::datatable_roles::lock_datatable_streams(tx, false).await?;
}
Ok(())
}
async fn create_trigger(
&self,
db: &DB,
@@ -72,6 +96,7 @@ impl TriggerCrud for PostgresTrigger {
w_id: &str,
trigger: TriggerData<Self::TriggerConfigRequest>,
) -> Result<()> {
windmill_common::datatable_roles::lock_datatable_streams(&mut *tx, false).await?;
let resolved_edited_by = trigger.base.resolve_edited_by(authed);
let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed);
let Self::TriggerConfigRequest {
@@ -161,6 +186,7 @@ impl TriggerCrud for PostgresTrigger {
path: &str,
trigger: TriggerData<Self::TriggerConfigRequest>,
) -> Result<()> {
windmill_common::datatable_roles::lock_datatable_streams(&mut *tx, false).await?;
let resolved_edited_by = trigger.base.resolve_edited_by(authed);
let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed);
let Self::TriggerConfigRequest {
backend/windmill-trigger-postgres/src/lib.rs
+30
View File
@@ -374,6 +374,35 @@ pub async fn get_raw_postgres_connection(
Ok(client)
}
/// A replication stream reads every row of every table whatever the data table's roles grant, so
/// the two don't mix: a data table under roles takes no triggers or captures, and roles cannot be
/// turned on while one is enabled on it.
///
/// Authorization: checks nothing, and its refusal says whether `w_id`'s data table is under roles.
/// Callers MUST have established that the caller may manage triggers in `w_id` first.
pub(crate) async fn ensure_not_under_roles(
db: &DB,
w_id: &str,
postgres_resource_path: &str,
) -> Result<()> {
let Some(datatable_name) = postgres_resource_path.strip_prefix("datatable://") else {
return Ok(());
};
if windmill_common::workspaces::resolve_governing_datatable(db, w_id, datatable_name)
.await?
.datatable
.permissions
.is_some()
{
return Err(Error::BadRequest(format!(
"Data table '{datatable_name}' is under roles, and a Postgres trigger or capture \
cannot read one: a replication stream sees every row whatever the roles grant. \
Turn its roles off to stream it."
)));
}
Ok(())
}
pub async fn resolve_postgres_resource(
authed: &ApiAuthed,
user_db: Option<UserDB>,
@@ -382,6 +411,7 @@ pub async fn resolve_postgres_resource(
w_id: &str,
) -> Result<Postgres> {
if let Some(datatable_name) = postgres_resource_path.strip_prefix("datatable://") {
ensure_not_under_roles(db, w_id, postgres_resource_path).await?;
// Trigger connections (publication/slot management + logical replication) run
// as the dedicated replication user on custom-instance databases.
let resource_value =
backend/windmill-trigger-postgres/src/listener.rs
+12 -3
View File
@@ -20,7 +20,8 @@ use windmill_common::{
use windmill_trigger::{listener::ListeningTrigger, trigger_helpers::TriggerJobArgs, Listener};
use super::{
drop_publication, get_default_pg_connection, get_raw_postgres_connection,
drop_publication, ensure_not_under_roles, get_default_pg_connection,
get_raw_postgres_connection,
handler::drop_logical_replication_slot,
relation::RelationConverter,
replication_message::{
@@ -135,8 +136,8 @@ impl PostgresSimpleClient {
/// Resolves the Postgres resource, validates that the configured publication and
/// replication slot still exist, and opens a fresh logical replication stream.
///
/// Returns `Error::BadConfig` when the publication or slot is missing (an
/// unrecoverable misconfiguration). Any other error is treated as transient
/// Returns `Error::BadConfig` when the publication or slot is missing, or the
/// data table is under roles (unrecoverable misconfigurations). Any other error is treated as transient
/// (connection refused, network interruption, ...) and is retried by the caller.
/// The resource is re-resolved on every call so credential rotations are picked
/// up across reconnections.
@@ -149,6 +150,14 @@ async fn connect_logical_replication_stream(
let PostgresConfig { postgres_resource_path, publication_name, replication_slot_name, .. } =
trigger_config;
// Retrying cannot lift roles, so this disables the trigger like a missing slot does.
ensure_not_under_roles(db, workspace_id, postgres_resource_path)
.await
.map_err(|e| match e {
Error::BadRequest(msg) => Error::BadConfig(msg),
e => e,
})?;
let database = resolve_postgres_resource(
authed,
Some(UserDB::new(db.clone())),
backend/windmill-worker/src/agent_workers.rs
+11 -2
View File
@@ -64,16 +64,25 @@ pub async fn get_ducklake_from_agent_http(
.await
}
/// An agent worker authenticates as infrastructure, not as the job's user, so the job id travels
/// with the request: the server reads the job's owner from it and evaluates the data table's
/// tenants against them. A worker predating this sends neither, and the server fails it closed on
/// a data table under roles.
#[allow(dead_code)]
pub async fn get_datatable_resource_from_agent_http(
client: &HttpClient,
name: &str,
w_id: &str,
role: Option<&str>,
job_id: &uuid::Uuid,
) -> anyhow::Result<serde_json::Value> {
let role_query = role
.map(|r| format!("&role={}", urlencoding::encode(r)))
.unwrap_or_default();
client
.get(&format!(
"/api/w/{}/agent_workers/get_datatable_resource/{}",
w_id, &name
"/api/w/{}/agent_workers/get_datatable_resource/{}?job_id={}{}",
w_id, &name, job_id, role_query
))
.await
}
backend/windmill-worker/src/duckdb_executor.rs
+110 -26
View File
@@ -13,8 +13,8 @@ use windmill_common::error::{to_anyhow, Error, Result};
use windmill_common::utils::sanitize_string_from_password;
use windmill_common::worker::{get_memory, to_raw_value, Connection, SqlResultCollectionStrategy};
use windmill_common::workspaces::{
get_datatable_resource_from_db_unchecked, get_ducklake_from_db_unchecked,
strip_fork_reserved_attach_args, DucklakeCatalogResourceType,
get_datatable_resource_from_db, get_ducklake_from_db_unchecked,
strip_fork_reserved_attach_args, DatatableAccess, DucklakeCatalogResourceType,
};
use windmill_common::PgDatabase;
use windmill_object_store::S3_PROXY_LAST_ERRORS_CACHE;
@@ -1494,13 +1494,9 @@ pub async fn do_duckdb(
.await?
{
probe_blocks.extend(q);
} else if let Some(q) = transform_attach_datatable(
&query_block,
conn,
&mut hidden_passwords,
&job.workspace_id,
)
.await?
} else if let Some(q) =
transform_attach_datatable(&query_block, conn, &mut hidden_passwords, job)
.await?
{
probe_blocks.extend(q);
} else {
@@ -1575,13 +1571,9 @@ pub async fn do_duckdb(
.await?
{
v.extend(ducklake_query);
} else if let Some(datatable_query) = transform_attach_datatable(
&query_block,
conn,
&mut hidden_passwords,
&job.workspace_id,
)
.await?
} else if let Some(datatable_query) =
transform_attach_datatable(&query_block, conn, &mut hidden_passwords, job)
.await?
{
v.extend(datatable_query);
} else {
@@ -2609,33 +2601,79 @@ fn fork_defer_statements(
Ok(stmts)
}
struct AttachedDatatable<'a> {
/// The data table reference, query string included; a bare `datatable` is `main`.
reference: String,
alias: &'a str,
}
/// `ATTACH 'datatable[://<name>][?role=<role>]' AS <alias>`. A bare `datatable` names the default
/// data table, so the role query string has to be accepted with and without an explicit name. The
/// reference is split only once the workspace can be read, because a stored name may contain `?`.
fn parse_attach_datatable(query: &str) -> Option<AttachedDatatable<'_>> {
lazy_static::lazy_static! {
static ref RE: regex::Regex = regex::Regex::new(
r"(?i)ATTACH\s*'datatable(://[^':]+|\?[^':]*)?'\s*AS\s+([^ ;]+)"
).unwrap();
}
let cap = RE.captures(query)?;
let reference = match cap.get(1).map(|m| m.as_str()) {
Some(named) if named.starts_with("://") => named[3..].to_string(),
Some(query) => format!("main{query}"),
None => "main".to_string(),
};
let alias = cap.get(2).map(|m| m.as_str()).unwrap_or("");
Some(AttachedDatatable { reference, alias })
}
async fn transform_attach_datatable(
query: &str,
conn: &Connection,
hidden_passwords: &mut Arc<Mutex<Vec<String>>>,
w_id: &str,
job: &MiniPulledJob,
) -> Result<Option<Vec<String>>> {
lazy_static::lazy_static! {
static ref RE: regex::Regex = regex::Regex::new(r"(?i)ATTACH\s*'datatable(://[^':]+)?'\s*AS\s+([^ ;]+)").unwrap();
}
let Some(cap) = RE.captures(query) else {
let Some(attached) = parse_attach_datatable(query) else {
return Ok(None);
};
let name = cap.get(1).map(|m| &m.as_str()[3..]).unwrap_or("main");
let alias_name = cap.get(2).map(|m| m.as_str()).unwrap_or("");
// A query string that does not parse is refused rather than dropped: attaching under the
// default role when the statement asked for another one is the failure this guards.
let db_resource = match conn {
Connection::Http(client) => {
get_datatable_resource_from_agent_http(client, name, w_id).await?
let (name, role) =
windmill_common::workspaces::parse_datatable_ref(&attached.reference)?;
get_datatable_resource_from_agent_http(client, name, &job.workspace_id, role, &job.id)
.await?
}
Connection::Sql(db) => {
let (name, role) = windmill_common::workspaces::parse_datatable_ref_for(
db,
&job.workspace_id,
&attached.reference,
)
.await?;
get_datatable_resource_from_db(
db,
&job.workspace_id,
&name,
role.as_deref(),
DatatableAccess::PermissionedAs {
permissioned_as: &job.permissioned_as,
email: &job.permissioned_as_email,
},
)
.await?
}
Connection::Sql(db) => get_datatable_resource_from_db_unchecked(db, w_id, name).await?,
};
if let Some(pwd) = db_resource.get("password").and_then(|p| p.as_str()) {
hidden_passwords.lock().unwrap().push(pwd.to_string());
}
Ok(Some(pg_secret_attach_statements(db_resource, alias_name)?))
Ok(Some(pg_secret_attach_statements(
db_resource,
attached.alias,
)?))
}
// Secret names must be plain identifiers; the hash keeps two aliases distinct even
@@ -2680,6 +2718,11 @@ fn pg_secret_attach_statements(db_resource: Value, alias_name: &str) -> Result<V
esc(res.password.as_deref().unwrap_or("")),
),
format!("ATTACH 'sslmode={sslmode}' AS {alias_name} (TYPE postgres, SECRET {secret_name});"),
// The attachment keeps its own resolved connection string, so the secret is dead weight
// once attached — and a live one is a credential the script's own statements can name: an
// `ATTACH 'dbname=<other>' (TYPE postgres, SECRET …)` would reach a database nobody
// authorized this job for, as this role.
format!("DROP TEMPORARY SECRET {secret_name};"),
])
}
@@ -2753,6 +2796,45 @@ pub struct Arg {
mod tests {
use super::*;
#[test]
fn attach_datatable_parses_name_and_role() {
let reference_of = |q: &str| parse_attach_datatable(q).unwrap().reference;
let named =
parse_attach_datatable("ATTACH 'datatable://sales?role=analytics' AS dt").unwrap();
assert_eq!(
(named.reference.as_str(), named.alias),
("sales?role=analytics", "dt")
);
// A bare `datatable` is the default one, and still takes a role.
assert_eq!(
reference_of("ATTACH 'datatable?role=analytics' AS dt"),
"main?role=analytics"
);
assert_eq!(reference_of("ATTACH 'datatable://sales' AS dt"), "sales");
assert_eq!(reference_of("ATTACH 'datatable' AS dt"), "main");
assert!(parse_attach_datatable("SELECT 1").is_none());
// A stored name can contain `?`, so that is left to the workspace lookup to split.
assert_eq!(reference_of("ATTACH 'datatable://a?b' AS dt"), "a?b");
// The key matches case-insensitively, as the `-- role` annotation does, and a query string
// that does not parse is refused rather than attached under the default role.
let parse = |q: &str| {
windmill_common::workspaces::parse_datatable_ref(&reference_of(q))
.map(|(name, role)| (name.to_string(), role.map(str::to_string)))
};
assert_eq!(
parse("ATTACH 'datatable://sales?Role=analytics' AS dt").unwrap(),
("sales".to_string(), Some("analytics".to_string()))
);
for malformed in [
"ATTACH 'datatable://sales?role=' AS dt",
"ATTACH 'datatable://sales?role=an;alytics' AS dt",
"ATTACH 'datatable://sales?x=1&role=analytics' AS dt",
] {
assert!(parse(malformed).is_err(), "silently ignored: {malformed}");
}
}
#[test]
fn decode_ffi_error_unescapes_multiline_and_strips_quotes() {
// Mirror the FFI: JSON-encode the raw DuckDB message, prefix "ERROR ".
@@ -3868,6 +3950,8 @@ mod tests {
stmts[3],
format!("ATTACH 'sslmode=require' AS dt (TYPE postgres, SECRET {secret_name});")
);
assert_eq!(stmts[4], format!("DROP TEMPORARY SECRET {secret_name};"));
assert_eq!(stmts.len(), 5);
}
#[test]
backend/windmill-worker/src/pg_executor.rs
+30 -7
View File
@@ -26,9 +26,11 @@ use windmill_common::azure_workload_identity::WORKLOAD_IDENTITY_PASSWORD;
use windmill_common::error::to_anyhow;
use windmill_common::error::{self, Error};
use windmill_common::worker::{
to_raw_value, Connection, SqlResultCollectionStrategy, CLOUD_HOSTED,
to_raw_value, Connection, SqlAnnotations, SqlResultCollectionStrategy, CLOUD_HOSTED,
};
use windmill_common::workspaces::{
get_datatable_resource_from_db, parse_datatable_ref, parse_datatable_ref_for, DatatableAccess,
};
use windmill_common::workspaces::get_datatable_resource_from_db_unchecked;
use windmill_common::{PgDatabase, PrepareQueryColumnInfo, PrepareQueryResult, DB};
use windmill_parser::{Arg, Typ};
use windmill_parser_sql::{
@@ -680,15 +682,36 @@ pub async fn do_postgresql(
} else {
match pg_args.get("database").cloned() {
Some(Value::String(db_str)) if db_str.starts_with("datatable://") => {
let db_str = db_str.trim_start_matches("datatable://");
let reference = db_str.trim_start_matches("datatable://");
// The annotation wins: a generated query can carry a `?role=` in the reference it
// was handed, but only the script's author writes the leading comment block.
let annotated = SqlAnnotations::datatable_role(&query)?;
Some(match conn {
Connection::Http(client) => {
get_datatable_resource_from_agent_http(client, &db_str, &job.workspace_id)
.await?
let (name, uri_role) = parse_datatable_ref(reference)?;
get_datatable_resource_from_agent_http(
client,
name,
&job.workspace_id,
annotated.as_deref().or(uri_role),
&job.id,
)
.await?
}
Connection::Sql(db) => {
get_datatable_resource_from_db_unchecked(db, &job.workspace_id, &db_str)
.await?
let (name, uri_role) =
parse_datatable_ref_for(db, &job.workspace_id, reference).await?;
get_datatable_resource_from_db(
db,
&job.workspace_id,
&name,
annotated.as_deref().or(uri_role.as_deref()),
DatatableAccess::PermissionedAs {
permissioned_as: &job.permissioned_as,
email: &job.permissioned_as_email,
},
)
.await?
}
})
}
cli/src/core/settings.ts
+8 -1
View File
@@ -393,10 +393,17 @@ export async function pushWorkspaceSettings(
if (!deepEqual(localSettings.datatable, settings.datatable)) {
log.debug(`Updating datatable config...`);
await wmill.editDataTableConfig({
const { stranded_references } = await wmill.editDataTableConfig({
workspace,
requestBody: { settings: localSettings.datatable ?? { datatables: {} } },
});
if (stranded_references?.length) {
log.warn(
`Removed data tables governed data tables in other workspaces, which no longer resolve: ${stranded_references
.map((r) => `${r.workspace_id}/${r.datatable}`)
.join(", ")}. A superadmin can point them somewhere else.`,
);
}
}
if (localSettings.slack_command_script != settings.slack_command_script) {
cli/src/guidance/skills.gen.ts
Generated
+26 -8
View File
@@ -1110,6 +1110,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -1119,8 +1121,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = \${name} AND age = \${age}::int
* \`.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
@@ -1901,6 +1906,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -1910,8 +1917,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = \${name} AND age = \${age}::int
* \`.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
@@ -2786,6 +2796,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -2795,8 +2807,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = \${name} AND age = \${age}::int
* \`.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
@@ -4405,10 +4420,13 @@ def send_teams_message(conversation_id: str, text: str, success: bool = True, ca
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
# Only meaningful on a data table under roles, and only for a role you are a
# tenant of.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main')
def datatable(name: str = 'main', *, role: Optional[str] = None)
# Get a DuckLake client for DuckDB queries.
#
@@ -4626,7 +4644,7 @@ def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
#
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
# async def call_api(payload: dict): ...
def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill script.
#
@@ -4639,7 +4657,7 @@ def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, ti
# @workflow
# async def main():
# data = await extract(url="https://...")
def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill flow.
#
@@ -4652,7 +4670,7 @@ def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = N
# @workflow
# async def main():
# result = await pipeline(input=data)
def task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Decorator marking an async function as a workflow-as-code entry point.
#
@@ -4717,7 +4735,7 @@ async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_
# ...
#
# results = await parallel(items, process, concurrency=5)
async def parallel(items, fn, concurrency: Optional[int] = None)
async def parallel(items, fn, *, concurrency: Optional[int] = None)
# Commit Kafka offsets for a trigger with auto_commit disabled.
#
flake.nix
+4 -1
View File
@@ -65,7 +65,10 @@
# Misc
libtool
postgresql
# Must not trail the server the dev database runs (postgres:18): pg_dump refuses a
# server newer than itself by a major version, which takes out every data table
# export, clone and fork-with-data.
postgresql_18
# Build tooling
pkg-config
frontend/src/lib/components/InstanceSettings.svelte
+4 -2
View File
@@ -1093,7 +1093,8 @@
migrations are set up and used, how often an empty workspace home is seen, how often
the home page’s create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, and whether a pre-approved trial offer was opened, last 30 days)</li
import got, whether a pre-approved trial offer was opened, and whether data tables are
put under roles and whether callers name a role or take the default, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
@@ -1159,7 +1160,8 @@
migrations are set up and used, how often an empty workspace home is seen, how often
the home page’s create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, and whether a pre-approved trial offer was opened, last 30 days)</li
import got, whether a pre-approved trial offer was opened, and whether data tables are
put under roles and whether callers name a role or take the default, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
frontend/src/lib/components/SqlRepl.svelte
+15 -8
View File
@@ -103,14 +103,21 @@
// We don't always put the fix by default for row ordering concerns
let transformedCode = code
if (doPostgresRowToJsonFix) {
transformedCode = statements
.map((statement) => {
if (READ_OPS.some((op) => statement.trim().toUpperCase().startsWith(op))) {
return `SELECT row_to_json(__t__) FROM (${statement}) __t__`
}
return statement
})
.join(';')
// Rebuilt from the pruned statements, which drops the leading comment block — and
// with it the `-- role <name>` annotation that decides which login the query runs
// as. Carry it over, or the retry connects as the data table's default role and a
// query the first attempt was denied succeeds on the second.
const leadingAnnotations = code.match(/^(?:[^\S\n]*\n|[^\S\n]*--[^\n]*\n)*/)?.[0] ?? ''
transformedCode =
leadingAnnotations +
statements
.map((statement) => {
if (READ_OPS.some((op) => statement.trim().toUpperCase().startsWith(op))) {
return `SELECT row_to_json(__t__) FROM (${statement}) __t__`
}
return statement
})
.join(';')
}
const dbArg = getDatabaseArg(input)
frontend/src/lib/components/sidebar/DeleteForkedWorkspaceModal.svelte
+11 -4
View File
@@ -45,11 +45,13 @@
const settings = await WorkspaceService.getPublicSettings({ workspace: $workspaceStore })
const datatables = settings.datatable?.datatables ?? {}
forkedDatatables = Object.entries(datatables)
.filter(([_, dt]) => dt.forked_from != null)
// A clone owns its database and is droppable; an entry pointing at the parent's is
// not this workspace's to drop, and never carries a clone stamp anyway.
.filter(([_, dt]) => dt.forked_from != null && dt.database != null)
.map(([name, dt]) => ({
name,
resourceType: dt.database.resource_type ?? 'instance',
resourcePath: dt.database.resource_path ?? '',
resourceType: dt.database?.resource_type ?? 'instance',
resourcePath: dt.database?.resource_path ?? '',
dropOnDelete: true
}))
} catch {
@@ -120,7 +122,12 @@
}
}
await WorkspaceService.deleteWorkspace({ workspace })
const result = await WorkspaceService.deleteWorkspace({ workspace })
// The server names any data table in another workspace that this delete left governed by
// nothing. Only surfaced when there is something to say.
if (typeof result === 'string' && result.includes('no longer resolve')) {
sendUserToast(result, 'warning', [], undefined, 20000)
}
await deleteSessionsForWorkspace(workspace).catch((e) =>
console.error('Session cleanup after workspace delete failed', e)
)
frontend/src/lib/components/workspaceSettings/DataTablePermissionsButton.svelte
+277
View File
@@ -0,0 +1,277 @@
<script lang="ts">
import { Alert, Badge, Button, Drawer, DrawerContent } from '../common'
import Select from '../select/Select.svelte'
import MultiSelect from '../select/MultiSelect.svelte'
import Toggle from '../Toggle.svelte'
import Tooltip from '../Tooltip.svelte'
import CloseButton from '../common/CloseButton.svelte'
import { KeyRound, Plus } from 'lucide-svelte'
import {
FolderService,
GroupService,
UserService,
WorkspaceService,
type DatatablePermissions,
type InstanceDatatableRole
} from '$lib/gen'
import { sendUserToast } from '$lib/toast'
const ADMIN_ROLE = 'admin'
let {
workspace,
datatable,
disabled = false
}: {
workspace: string
datatable: string
disabled?: boolean
} = $props()
let drawer: Drawer | undefined = $state(undefined)
let loading = $state(false)
let saving = $state(false)
let loadError = $state<string | undefined>(undefined)
let info = $state<DatatablePermissions | undefined>(undefined)
// Edited copy. `id` is the instance role's catalog id (or the reserved `admin`), which is what
// the tenant lists are keyed by — so renaming a role instance-side moves nothing here.
let permissioned = $state(false)
let defaultRole = $state(ADMIN_ROLE)
let rows = $state<{ id: string; name: string | undefined; tenants: string[] }[]>([])
// Tenants name principals of the workspace that governs the data table, which is not
// necessarily the one we are browsing from.
let tenantOptions = $state<{ value: string; label: string }[]>([])
const editable = $derived(!!info?.editable)
const governing = $derived(info?.governing_workspace_id)
const availableRoles: InstanceDatatableRole[] = $derived(info?.available_roles ?? [])
const unusedRoles = $derived(availableRoles.filter((r) => !rows.some((row) => row.id === r.id)))
async function load() {
loading = true
loadError = undefined
try {
const res = await WorkspaceService.getDatatablePermissions({ workspace, datatableName: datatable })
info = res
permissioned = res.permissioned
defaultRole = res.default_role
rows = (res.roles ?? [])
.map((r) => ({ id: r.id, name: r.name, tenants: r.tenants ?? [] }))
.sort((a, b) => (a.id === ADMIN_ROLE ? -1 : b.id === ADMIN_ROLE ? 1 : 0))
if (rows.length === 0) {
rows = [{ id: ADMIN_ROLE, name: ADMIN_ROLE, tenants: [] }]
}
await loadTenantOptions(res.governing_workspace_id ?? workspace)
} catch (e) {
loadError = e?.body ?? e?.message ?? String(e)
} finally {
loading = false
}
}
async function loadTenantOptions(ws: string) {
try {
const [users, groups, folders] = await Promise.all([
UserService.listUsernames({ workspace: ws }),
GroupService.listGroupNames({ workspace: ws }),
FolderService.listFolderNames({ workspace: ws })
])
tenantOptions = [
{ value: '*', label: 'Everyone in the workspace' },
...users.map((u) => ({ value: `u/${u}`, label: `u/${u}` })),
...groups.map((g) => ({ value: `g/${g}`, label: `g/${g}` })),
...folders.map((f) => ({ value: `f/${f}`, label: `f/${f}` }))
]
} catch {
// A fork member may not be able to list the governing workspace's principals. The
// tenants they cannot name are still shown, they just cannot pick new ones.
tenantOptions = []
}
}
function addRole(id: string) {
const role = availableRoles.find((r) => r.id === id)
if (!role) return
rows = [...rows, { id: role.id, name: role.name, tenants: [] }]
}
function removeRole(id: string) {
rows = rows.filter((r) => r.id !== id)
if (defaultRole === id) defaultRole = ADMIN_ROLE
}
async function save() {
saving = true
try {
const msg = await WorkspaceService.setDatatablePermissions({
workspace,
datatableName: datatable,
requestBody: {
permissioned,
default_role: defaultRole,
roles: rows.map((r) => ({ id: r.id, tenants: r.tenants }))
}
})
sendUserToast(msg)
await load()
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
} finally {
saving = false
}
}
export function open() {
drawer?.openDrawer()
load()
}
</script>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: KeyRound }}
iconOnly
{disabled}
title={disabled ? 'Save settings first' : 'Roles: who may connect as which Postgres role'}
on:click={open}
/>
<Drawer bind:this={drawer} size="700px">
<DrawerContent
title="Roles for {datatable}"
on:close={() => drawer?.closeDrawer()}
tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant privileges with SQL. Roles are defined for the whole instance; here you say who may use each one on this data table."
>
{#if loading}
<p class="text-sm text-secondary">Loading…</p>
{:else if loadError}
<Alert type="error" title="Could not load roles" size="xs">{loadError}</Alert>
{:else}
<div class="flex flex-col gap-4">
{#if !info?.supported}
<Alert type="info" title="Not available on this data table" size="xs">
A data table role is a Postgres login on the Windmill instance's own database, so only a
data table backed by that database can use one. This one is backed by a PostgreSQL
resource — grant access on that server directly.
</Alert>
{:else if governing}
<Alert type="info" title="Governed by {governing}" size="xs">
This data table points at the one in workspace <span class="font-mono">{governing}</span
>, so its roles are decided there. You are evaluated as a member of that workspace.
</Alert>
{:else if !editable}
<Alert type="info" title="Read only" size="xs">
Only admins of this workspace can change who may use which role.
</Alert>
{/if}
{#if info?.ungoverned_reachers?.length}
<Alert type="warning" title="Other workspaces reach this database" size="xs">
These data tables point at the same database with their own entry, so what you set here
does not reach them:
<ul class="mt-1 list-disc list-inside font-mono">
{#each info.ungoverned_reachers as reacher}
<li>{reacher.workspace_id} / {reacher.datatable}</li>
{/each}
</ul>
</Alert>
{/if}
<Toggle
bind:checked={permissioned}
disabled={!editable || !info?.supported}
options={{
right: 'Put this data table under roles',
rightTooltip:
'Off, every job connects as admin — the connection that owns every table. On, every job resolves to a role, and a caller no tenant covers is refused.'
}}
/>
{#if permissioned}
{#if availableRoles.length === 0}
<Alert type="warning" title="No role defined on this instance" size="xs">
Only <span class="font-mono">admin</span> can be used until a superadmin adds a data table
role in the data table settings page.
</Alert>
{/if}
<div class="flex flex-col gap-3">
{#each rows as row (row.id)}
<div class="flex flex-col gap-1 border rounded-md p-3 bg-surface-secondary">
<div class="flex items-center gap-2">
<Badge color={row.id === ADMIN_ROLE ? 'blue' : 'gray'}>
{row.name ?? row.id}
</Badge>
{#if row.id === ADMIN_ROLE}
<Tooltip>
The connection every data table resolved to before roles. It owns every
existing object, so it is always available and cannot be removed.
</Tooltip>
{:else if !row.name}
<span class="text-xs text-secondary italic">
no longer defined on this instance
</span>
{/if}
{#if defaultRole === row.id}
<Badge color="green">Default</Badge>
{:else if editable}
<Button
unifiedSize="2xs"
variant="subtle"
on:click={() => (defaultRole = row.id)}
>
Make default
</Button>
{/if}
<div class="grow"></div>
{#if editable && row.id !== ADMIN_ROLE}
<CloseButton small on:close={() => removeRole(row.id)} />
{/if}
</div>
<MultiSelect
items={tenantOptions}
bind:value={row.tenants}
disabled={!editable}
placeholder="Nobody yet — add a user, group or folder"
size="sm"
/>
</div>
{/each}
</div>
{#if editable && unusedRoles.length > 0}
<div class="flex items-center gap-2">
<Plus size={14} class="text-secondary" />
<Select
items={unusedRoles.map((r) => ({
value: r.id,
label: r.enabled ? r.name : `${r.name} (disabled)`
}))}
placeholder="Add a role"
bind:value={
() => undefined,
(id) => {
if (id) addRole(id)
}
}
class="w-64"
size="sm"
/>
</div>
{/if}
{/if}
{#if editable}
<div class="flex justify-end">
<Button unifiedSize="sm" variant="accent" loading={saving} on:click={save}>
Save roles
</Button>
</div>
{/if}
</div>
{/if}
</DrawerContent>
</Drawer>
frontend/src/lib/components/workspaceSettings/DataTableRolesSection.svelte
+210
View File
@@ -0,0 +1,210 @@
<script lang="ts">
import { Alert, Badge, Button } from '../common'
import CloseButton from '../common/CloseButton.svelte'
import TextInput from '../text_input/TextInput.svelte'
import Toggle from '../Toggle.svelte'
import Tooltip from '../Tooltip.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import Cell from '../table/Cell.svelte'
import DataTable from '../table/DataTable.svelte'
import Head from '../table/Head.svelte'
import Row from '../table/Row.svelte'
import { Pencil, Plus } from 'lucide-svelte'
import { SettingService, type InstanceDatatableRole } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
let roles = $state<InstanceDatatableRole[]>([])
let loading = $state(true)
let loadError = $state<string | undefined>(undefined)
let busy = $state(false)
let newName = $state('')
/** Which role's name is being edited, and to what. */
let renaming = $state<{ id: string; name: string } | undefined>(undefined)
const confirmationModal = createAsyncConfirmationModal()
async function load() {
loading = true
loadError = undefined
try {
roles = await SettingService.listInstanceDatatableRoles()
} catch (e) {
loadError = e?.body ?? e?.message ?? String(e)
} finally {
loading = false
}
}
load()
async function run(fn: () => Promise<unknown>, success: string) {
busy = true
try {
await fn()
sendUserToast(success)
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
} finally {
// Reloaded whether or not it worked: the login toggle is driven by what the server
// holds, so a failed flip has to snap back rather than sit there claiming it landed.
await load()
busy = false
}
}
async function create() {
const name = newName.trim()
if (!name) return
await run(
() =>
SettingService.createInstanceDatatableRole({
requestBody: { name }
}),
`Created the data table role ${name}`
)
newName = ''
}
async function remove(role: InstanceDatatableRole) {
const confirmed = await confirmationModal.ask({
title: `Delete the role ${role.name}?`,
children:
'Everything it owns in every instance database is handed back to the admin connection, its grants are dropped, and it is removed from every data table that named it. This cannot be undone.',
confirmationText: 'Delete role'
})
if (!confirmed) return
await run(
() => SettingService.deleteInstanceDatatableRole({ id: role.id }),
`Deleted the data table role ${role.name}`
)
}
</script>
<ConfirmationModal {...confirmationModal.props} />
<div class="flex flex-col gap-2">
<div class="flex items-baseline gap-1">
<h3 class="font-semibold text-sm">Instance roles</h3>
<Tooltip>
A data table role is a real Postgres login on this instance, shared by every instance
database. A job that names one connects as it, and Postgres decides what it may touch — grant
it privileges with SQL. Which people may use a role on a given data table is set per data
table, in its roles drawer.
</Tooltip>
</div>
{#if loadError}
<Alert type="error" title="Could not load the instance roles" size="xs">{loadError}</Alert>
{:else}
<DataTable>
<Head>
<tr>
<Cell head first>Name</Cell>
<Cell head>Login</Cell>
<Cell head last></Cell>
</tr>
</Head>
<tbody class="divide-y bg-surface-tertiary">
{#if loading}
<Row>
<Cell colspan={3} class="text-center py-4 text-secondary text-xs">Loading…</Cell>
</Row>
{:else if roles.length === 0}
<Row>
<Cell colspan={3} class="text-center py-4 text-secondary text-xs">
No data table role yet. Every job connects as
<span class="font-mono">admin</span>.
</Cell>
</Row>
{/if}
{#each roles as role (role.id)}
<Row>
<Cell first class="w-64">
{#if renaming?.id === role.id}
<div class="flex gap-1 items-center">
<TextInput bind:value={renaming.name} inputProps={{ placeholder: 'Name' }} />
<Button
unifiedSize="xs"
variant="accent"
disabled={busy}
on:click={async () => {
const name = renaming?.name?.trim()
renaming = undefined
if (name && name !== role.name) {
await run(
() =>
SettingService.updateInstanceDatatableRole({
id: role.id,
requestBody: { name }
}),
`Renamed the data table role to ${name}`
)
}
}}
>
Rename
</Button>
<CloseButton small on:close={() => (renaming = undefined)} />
</div>
{:else}
<div class="flex items-center gap-1">
<span class="font-mono text-sm">{role.name}</span>
<Button
unifiedSize="2xs"
variant="subtle"
startIcon={{ icon: Pencil }}
iconOnly
title="Rename this role"
on:click={() => (renaming = { id: role.id, name: role.name })}
/>
</div>
{/if}
</Cell>
<Cell>
<div class="flex items-center gap-2">
<Toggle
checked={role.enabled}
disabled={busy}
on:change={(e) =>
run(
() =>
SettingService.updateInstanceDatatableRole({
id: role.id,
requestBody: { enabled: e.detail }
}),
e.detail ? `Enabled ${role.name}` : `Disabled ${role.name}`
)}
/>
{#if !role.enabled}
<Badge color="gray">Cannot log in</Badge>
{/if}
</div>
</Cell>
<Cell last class="w-12">
<CloseButton small on:close={() => remove(role)} />
</Cell>
</Row>
{/each}
<Row class="!border-0">
<Cell colspan={3} class="pt-0 pb-2">
<div class="flex gap-2 items-center">
<TextInput
bind:value={newName}
inputProps={{ placeholder: 'analytics', id: 'new-datatable-role' }}
/>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: Plus }}
disabled={busy || !newName.trim()}
on:click={create}
>
Add role
</Button>
</div>
</Cell>
</Row>
</tbody>
</DataTable>
{/if}
</div>
frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte
+144 -54
View File
@@ -12,6 +12,10 @@
resource_type: 'postgresql' | 'instance'
resource_path?: string | undefined
}
/** Set on a fork's entry: it names the workspace whose data table governs this one, and
* owns no database of its own. Read-only here — only forking writes it, and the server
* carries it across a save rather than taking it from this form. */
reference?: { workspace_id: string; datatable: string }
}[]
}
@@ -24,7 +28,10 @@
s.dataTables.push({
id: randomUUID(),
name,
...rest
...rest,
// A pointer entry owns no database. The row renders read-only in that case, so this
// placeholder is never shown or sent.
database: rest.database ?? { resource_type: 'instance' }
})
}
}
@@ -38,6 +45,12 @@
const database = dataTable.database
if (dataTable.name in s.datatables)
throw 'Settings contain duplicate dataTable name: ' + dataTable.name
// A pointer owns no database, so it has nothing to validate and nothing to send: the
// server keeps the stored reference whatever this payload says.
if (dataTable.reference) {
s.datatables[dataTable.name] = {}
continue
}
if (!database.resource_path) throw 'No resource selected for ' + dataTable.name
if (database.resource_type === 'instance' && database.resource_path === 'windmill')
throw dataTable.name + ' database cannot be called "windmill"'
@@ -79,6 +92,7 @@
type GetSettingsResponse,
type TestDataTableConnectionResponse
} from '$lib/gen'
// `superadmin` gates the commented-out roles section at the bottom; restore it there.
import { workspaceStore } from '$lib/stores'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
@@ -87,6 +101,10 @@
import { Popover } from '../meltComponents'
import ExploreAssetButton from '../ExploreAssetButton.svelte'
import DataTableMigrationsButton from './DataTableMigrationsButton.svelte'
// Both components are complete and reviewed; their call sites in this file are commented
// out until the ACL editor lands. Uncomment these with them.
// import DataTablePermissionsButton from './DataTablePermissionsButton.svelte'
// import DataTableRolesSection from './DataTableRolesSection.svelte'
import { deepEqual } from 'fast-equals'
import { clone } from '$lib/utils'
import SettingsFooter from './SettingsFooter.svelte'
@@ -208,12 +226,28 @@
const deleted_datatables = dataTableSettings.dataTables
.filter((d) => !tempIds.has(d.id))
.map((d) => d.name)
await WorkspaceService.editDataTableConfig({
const result = await WorkspaceService.editDataTableConfig({
workspace: $workspaceStore!,
requestBody: { settings, renames, deleted_datatables }
})
dataTableSettings = clone(tempSettings)
sendUserToast('Data table settings saved successfully')
// A delete can leave another workspace's data table governed by nothing. Swallowing
// that is what made it silent for the person who caused it.
const stranded = result?.stranded_references ?? []
if (stranded.length > 0) {
sendUserToast(
`These data tables were governed by one you deleted and no longer resolve: ${stranded
.map((s) => `${s.workspace_id}/${s.datatable}`)
.join(', ')}. Their databases still exist; a superadmin can point them at another ` +
`workspace's data table.`,
'warning',
[],
undefined,
20000
)
} else {
sendUserToast('Data table settings saved successfully')
}
} catch (e) {
sendUserToast(e, true)
console.error('Error saving data table settings', e)
@@ -350,62 +384,85 @@
{#each tempSettings.dataTables as dataTable, dataTableIndex (dataTable.id)}
<Row>
<Cell first class="w-48 relative">
<TextInput bind:value={dataTable.name} inputProps={{ placeholder: 'Name', id: 'name' }} />
{#if dataTable.reference}
<span class="font-mono text-sm">{dataTable.name}</span>
{:else}
<TextInput
bind:value={dataTable.name}
inputProps={{ placeholder: 'Name', id: 'name' }}
/>
{/if}
</Cell>
<Cell>
<div class="flex gap-2">
<div class="relative">
{#if dataTable.database.resource_type === 'instance'}
<Tooltip wrapperClass="absolute mt-[0.6rem] right-2 z-20" placement="bottom-start">
Use Windmill's PostgreSQL instance
</Tooltip>
{/if}
<Select
items={[
{ value: 'postgresql', label: 'PostgreSQL' },
{
value: 'instance',
label: 'Instance',
disabled: isCloudHosted(),
subtitle: $isCustomInstanceDbEnabled
? undefined
: isCloudHosted()
? 'Not available on cloud'
: 'Superadmin only'
}
]}
bind:value={
() => dataTable.database.resource_type,
(resource_type) => {
dataTable.database = {
resource_type,
resource_path:
resource_type === 'instance' ? defaultInstanceDbName() : undefined
{#if dataTable.reference}
<div class="flex items-center gap-1 text-sm text-secondary">
<span>Governed by</span>
<span class="font-mono">{dataTable.reference.workspace_id}</span>
<span>/</span>
<span class="font-mono">{dataTable.reference.datatable}</span>
<Tooltip>
This fork uses its parent's data table rather than a copy of it, so the database and
its roles are decided in that workspace.
</Tooltip>
</div>
{:else}
<div class="flex gap-2">
<div class="relative">
{#if dataTable.database.resource_type === 'instance'}
<Tooltip
wrapperClass="absolute mt-[0.6rem] right-2 z-20"
placement="bottom-start"
>
Use Windmill's PostgreSQL instance
</Tooltip>
{/if}
<Select
items={[
{ value: 'postgresql', label: 'PostgreSQL' },
{
value: 'instance',
label: 'Instance',
disabled: isCloudHosted(),
subtitle: $isCustomInstanceDbEnabled
? undefined
: isCloudHosted()
? 'Not available on cloud'
: 'Superadmin only'
}
]}
bind:value={
() => dataTable.database.resource_type,
(resource_type) => {
dataTable.database = {
resource_type,
resource_path:
resource_type === 'instance' ? defaultInstanceDbName() : undefined
}
}
}
}
id="database-type-select"
class="w-28"
/>
</div>
<div class="flex items-center gap-1 w-80 relative">
{#if dataTable.database.resource_type !== 'instance'}
<ResourcePicker
class="flex-1"
bind:value={dataTable.database.resource_path}
resourceType={dataTable.database.resource_type}
id="database-type-select"
class="w-28"
/>
{:else}
<CustomInstanceDbSelect
class="flex-1"
{confirmationModal}
{customInstanceDbs}
bind:value={dataTable.database.resource_path}
tag="datatable"
/>
{/if}
</div>
<div class="flex items-center gap-1 w-80 relative">
{#if dataTable.database.resource_type !== 'instance'}
<ResourcePicker
class="flex-1"
bind:value={dataTable.database.resource_path}
resourceType={dataTable.database.resource_type}
/>
{:else}
<CustomInstanceDbSelect
class="flex-1"
{confirmationModal}
{customInstanceDbs}
bind:value={dataTable.database.resource_path}
tag="datatable"
/>
{/if}
</div>
</div>
</div>
{/if}
</Cell>
<Cell class="whitespace-nowrap">
@@ -415,6 +472,23 @@
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
<!-- Data table roles: not mounted yet. The enforcement ships first and this
drawer is what turns it on, so leaving it reachable would expose a half of the
feature whose other half (the ACL editor, which grants the privileges a role
actually needs) does not exist yet.
DataTablePermissionsButton.svelte is complete and reviewed — reuse it rather
than rewriting it, and uncomment this together with the roles section at the
bottom of this file and the two imports at the top.
{#if $enterpriseLicense}
<DataTablePermissionsButton
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
{/if}
-->
<Button
size="xs"
color="light"
@@ -448,7 +522,9 @@
</div>
</Cell>
<Cell class="w-12">
<CloseButton small on:close={() => removeDataTable(dataTableIndex)} />
{#if !dataTable.reference}
<CloseButton small on:close={() => removeDataTable(dataTableIndex)} />
{/if}
</Cell>
</Row>
{/each}
@@ -539,6 +615,20 @@
{/if}
{/if}
<!-- The instance role catalog, superadmin-only. Not mounted for the same reason as the
permissions drawer above: creating roles is only useful once there is a way to grant them
privileges, which arrives with the ACL editor.
DataTableRolesSection.svelte is complete and reviewed — reuse it rather than rewriting it,
and uncomment this together with the permissions button above and the two imports at the top.
{#if $superadmin && $enterpriseLicense && !isCloudHosted()}
<div class="mt-8">
<DataTableRolesSection />
</div>
{/if}
-->
<SettingsFooter
class="mt-8"
{hasUnsavedChanges}
python-client/wmill/wmill/client.py
+23 -5
View File
@@ -1430,16 +1430,19 @@ class Windmill:
},
)
def datatable(self, name: str = "main"):
def datatable(self, name: str = "main", *, role: Optional[str] = None):
"""Get a DataTable client for SQL queries.
Args:
name: Database name (default: "main")
role: Connect as this data table role instead of the data table's default one.
Only meaningful on a data table under roles, and only for a role you are a
tenant of.
Returns:
DataTableClient instance
"""
return DataTableClient(self, name)
return DataTableClient(self, name, role=role)
def ducklake(self, name: str = "main"):
"""Get a DuckLake client for DuckDB queries.
@@ -2278,16 +2281,17 @@ def username_to_email(username: str) -> str:
@init_global_client
def datatable(name: str = "main") -> DataTableClient:
def datatable(name: str = "main", *, role: Optional[str] = None) -> DataTableClient:
"""Get a DataTable client for SQL queries.
Args:
name: Database name (default: "main")
role: Connect as this data table role instead of the data table's default one.
Returns:
DataTableClient instance
"""
return _client.datatable(name)
return _client.datatable(name, role=role)
@init_global_client
def ducklake(name: str = "main") -> DucklakeClient:
@@ -2362,17 +2366,28 @@ def stream_result(stream) -> None:
for text in stream:
append_to_result_stream(text)
# Interpolated into a `-- role <name>` line, so a value carrying a newline could append
# statements of its own. Mirrors the server's own role-name rule.
_ROLE_NAME_RE = re.compile(r"^[A-Za-z0-9_-]{1,63}$")
class DataTableClient:
"""Client for executing SQL queries against Windmill DataTables."""
def __init__(self, client: Windmill, name: str):
def __init__(self, client: Windmill, name: str, role: Optional[str] = None):
"""Initialize DataTableClient.
Args:
client: Windmill client instance
name: DataTable name
role: Data table role to connect as, or None for the data table's default
"""
if role is not None and not _ROLE_NAME_RE.match(role):
raise ValueError(
f"Invalid data table role '{role}': only letters, digits, '_' and '-' are allowed"
)
self.client = client
self.role = role
self.name, self.schema = parse_sql_client_name(name)
def query(self, sql: str, *args) -> SqlQuery:
"""Execute a SQL query against the DataTable.
@@ -2393,6 +2408,9 @@ class DataTableClient:
args_dict[f"arg{i+1}"] = arg
args_def += f"-- ${i+1} arg{i+1} ({infer_sql_type(arg)})\n"
sql = args_def + sql
# Must lead: the executor's annotation parser stops at the first non-comment line.
if self.role is not None:
sql = f"-- role {self.role}\n" + sql
return SqlQuery(
sql,
lambda sql: self.client.run_inline_script_preview(
system_prompts/auto-generated/prompts.ts
+25 -9
View File
@@ -1911,6 +1911,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -1920,8 +1922,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = \${name} AND age = \${age}::int
* \`.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
@@ -2436,10 +2441,13 @@ def send_teams_message(conversation_id: str, text: str, success: bool = True, ca
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
# Only meaningful on a data table under roles, and only for a role you are a
# tenant of.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main')
def datatable(name: str = 'main', *, role: Optional[str] = None)
# Get a DuckLake client for DuckDB queries.
#
@@ -2657,7 +2665,7 @@ def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
#
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
# async def call_api(payload: dict): ...
def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill script.
#
@@ -2670,7 +2678,7 @@ def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, ti
# @workflow
# async def main():
# data = await extract(url="https://...")
def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill flow.
#
@@ -2683,7 +2691,7 @@ def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = N
# @workflow
# async def main():
# result = await pipeline(input=data)
def task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Decorator marking an async function as a workflow-as-code entry point.
#
@@ -2748,7 +2756,7 @@ async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_
# ...
#
# results = await parallel(items, process, concurrency=5)
async def parallel(items, fn, concurrency: Optional[int] = None)
async def parallel(items, fn, *, concurrency: Optional[int] = None)
# Commit Kafka offsets for a trigger with auto_commit disabled.
#
@@ -3195,6 +3203,8 @@ interface DatatableSqlTemplateFunction {
Create a SQL template function for PostgreSQL/datatable queries
@param name - Database/datatable name (default: "main")
@param opts.role - Connect as this data table role instead of the data table's default one.
Only meaningful on a data table under roles, and only for a role you are a tenant of.
@returns SQL template function for building parameterized queries
@example
let sql = wmill.datatable()
@@ -3204,8 +3214,12 @@ await sql\`
SELECT * FROM friends
WHERE name = \${name} AND age = \${age}::int
\`.fetch()
@example
// Read through a restricted role
let sql = wmill.datatable("main", { role: "analytics" })
\`\`\`typescript
function datatable(name: string = "main"): DatatableSqlTemplateFunction
function datatable(name: string = "main",
opts?: DatatableOptions): DatatableSqlTemplateFunction
\`\`\`
`;
@@ -3217,10 +3231,11 @@ Import: \`import wmill\`
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main') -> DataTableClient
def datatable(name: str = 'main', *, role: Optional[str] = None) -> DataTableClient
# Client for executing SQL queries against Windmill DataTables.
class DataTableClient:
@@ -3229,7 +3244,8 @@ class DataTableClient:
# Args:
# client: Windmill client instance
# name: DataTable name
def __init__(client: Windmill, name: str)
# role: Data table role to connect as, or None for the data table's default
def __init__(client: Windmill, name: str, role: Optional[str] = None)
# Execute a SQL query against the DataTable.
#
system_prompts/auto-generated/script.md
+14 -6
View File
@@ -2000,6 +2000,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -2009,8 +2011,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
@@ -2525,10 +2530,13 @@ def send_teams_message(conversation_id: str, text: str, success: bool = True, ca
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
# Only meaningful on a data table under roles, and only for a role you are a
# tenant of.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main')
def datatable(name: str = 'main', *, role: Optional[str] = None)
# Get a DuckLake client for DuckDB queries.
#
@@ -2746,7 +2754,7 @@ def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
#
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
# async def call_api(payload: dict): ...
def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill script.
#
@@ -2759,7 +2767,7 @@ def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, ti
# @workflow
# async def main():
# data = await extract(url="https://...")
def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill flow.
#
@@ -2772,7 +2780,7 @@ def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = N
# @workflow
# async def main():
# result = await pipeline(input=data)
def task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Decorator marking an async function as a workflow-as-code entry point.
#
@@ -2837,7 +2845,7 @@ async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_
# ...
#
# results = await parallel(items, process, concurrency=5)
async def parallel(items, fn, concurrency: Optional[int] = None)
async def parallel(items, fn, *, concurrency: Optional[int] = None)
# Commit Kafka offsets for a trigger with auto_commit disabled.
#
system_prompts/auto-generated/sdks/datatable-python.md
+4 -2
View File
@@ -6,10 +6,11 @@ Import: `import wmill`
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main') -> DataTableClient
def datatable(name: str = 'main', *, role: Optional[str] = None) -> DataTableClient
# Client for executing SQL queries against Windmill DataTables.
class DataTableClient:
@@ -18,7 +19,8 @@ class DataTableClient:
# Args:
# client: Windmill client instance
# name: DataTable name
def __init__(client: Windmill, name: str)
# role: Data table role to connect as, or None for the data table's default
def __init__(client: Windmill, name: str, role: Optional[str] = None)
# Execute a SQL query against the DataTable.
#
system_prompts/auto-generated/sdks/datatable-typescript.md
+7 -1
View File
@@ -62,6 +62,8 @@ interface DatatableSqlTemplateFunction {
Create a SQL template function for PostgreSQL/datatable queries
@param name - Database/datatable name (default: "main")
@param opts.role - Connect as this data table role instead of the data table's default one.
Only meaningful on a data table under roles, and only for a role you are a tenant of.
@returns SQL template function for building parameterized queries
@example
let sql = wmill.datatable()
@@ -71,6 +73,10 @@ await sql`
SELECT * FROM friends
WHERE name = ${name} AND age = ${age}::int
`.fetch()
@example
// Read through a restricted role
let sql = wmill.datatable("main", { role: "analytics" })
```typescript
function datatable(name: string = "main"): DatatableSqlTemplateFunction
function datatable(name: string = "main",
opts?: DatatableOptions): DatatableSqlTemplateFunction
```
system_prompts/auto-generated/sdks/python.md
+8 -5
View File
@@ -469,10 +469,13 @@ def send_teams_message(conversation_id: str, text: str, success: bool = True, ca
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
# Only meaningful on a data table under roles, and only for a role you are a
# tenant of.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main')
def datatable(name: str = 'main', *, role: Optional[str] = None)
# Get a DuckLake client for DuckDB queries.
#
@@ -690,7 +693,7 @@ def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
#
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
# async def call_api(payload: dict): ...
def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill script.
#
@@ -703,7 +706,7 @@ def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, ti
# @workflow
# async def main():
# data = await extract(url="https://...")
def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill flow.
#
@@ -716,7 +719,7 @@ def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = N
# @workflow
# async def main():
# result = await pipeline(input=data)
def task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Decorator marking an async function as a workflow-as-code entry point.
#
@@ -781,7 +784,7 @@ async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_
# ...
#
# results = await parallel(items, process, concurrency=5)
async def parallel(items, fn, concurrency: Optional[int] = None)
async def parallel(items, fn, *, concurrency: Optional[int] = None)
# Commit Kafka offsets for a trigger with auto_commit disabled.
#
system_prompts/auto-generated/sdks/typescript.md
+6 -1
View File
@@ -565,6 +565,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -574,8 +576,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
system_prompts/auto-generated/skills/write-script-bun/SKILL.md
+6 -1
View File
@@ -736,6 +736,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -745,8 +747,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md
+6 -1
View File
@@ -736,6 +736,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -745,8 +747,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
system_prompts/auto-generated/skills/write-script-deno/SKILL.md
+6 -1
View File
@@ -738,6 +738,8 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -747,8 +749,11 @@ parseS3Object(s3Object: S3Object): S3ObjectRecord
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
datatable(name: string = "main"): DatatableSqlTemplateFunction
datatable(name: string = "main", opts?: DatatableOptions): DatatableSqlTemplateFunction
/**
* Create a SQL template function for DuckDB/ducklake queries
system_prompts/auto-generated/skills/write-script-python3/SKILL.md
+8 -5
View File
@@ -654,10 +654,13 @@ def send_teams_message(conversation_id: str, text: str, success: bool = True, ca
#
# Args:
# name: Database name (default: "main")
# role: Connect as this data table role instead of the data table's default one.
# Only meaningful on a data table under roles, and only for a role you are a
# tenant of.
#
# Returns:
# DataTableClient instance
def datatable(name: str = 'main')
def datatable(name: str = 'main', *, role: Optional[str] = None)
# Get a DuckLake client for DuckDB queries.
#
@@ -875,7 +878,7 @@ def parse_sql_client_name(name: str) -> tuple[str, Optional[str]]
#
# @task(retry={"attempts": 3, "delay": 30, "multiplier": 2})
# async def call_api(payload: dict): ...
def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill script.
#
@@ -888,7 +891,7 @@ def task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, ti
# @workflow
# async def main():
# data = await extract(url="https://...")
def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Create a task that dispatches to a separate Windmill flow.
#
@@ -901,7 +904,7 @@ def task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = N
# @workflow
# async def main():
# result = await pipeline(input=data)
def task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
def task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None, retry: Optional[dict] = None)
# Decorator marking an async function as a workflow-as-code entry point.
#
@@ -966,7 +969,7 @@ async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_
# ...
#
# results = await parallel(items, process, concurrency=5)
async def parallel(items, fn, concurrency: Optional[int] = None)
async def parallel(items, fn, *, concurrency: Optional[int] = None)
# Commit Kafka offsets for a trigger with auto_commit disabled.
#
system_prompts/generate.py
+12 -1
View File
@@ -223,7 +223,12 @@ def extract_py_functions(content: str) -> list[dict]:
if args.vararg:
params.append(f"*{args.vararg.arg}")
# Handle keyword-only args
# Handle keyword-only args. Same bare `*` as `_format_py_params`: without it the rendered
# signature reads as all-positional, and code written against this reference passes a
# keyword-only argument positionally and gets a TypeError.
if args.kwonlyargs and not args.vararg:
params.append('*')
for i, arg in enumerate(args.kwonlyargs):
param_str = arg.arg
if arg.annotation:
@@ -1266,6 +1271,12 @@ def _format_py_params(node: ast.FunctionDef, skip_self: bool = False) -> str:
vararg_str += f": {ast.unparse(args.vararg.annotation)}"
params.append(vararg_str)
# A bare `*` before the keyword-only args, when nothing else already introduced them.
# Without it the rendered signature reads as all-positional, and a caller written against
# these docs passes a keyword-only argument positionally and gets a TypeError.
if args.kwonlyargs and not args.vararg:
params.append('*')
for i, arg in enumerate(args.kwonlyargs):
param_str = arg.arg
if arg.annotation:
typescript-client/sqlUtils.d.ts
Vendored
+4 -1
View File
@@ -85,7 +85,10 @@ export interface DatatableSqlTemplateFunction extends SqlTemplateFunction {
query<T = any>(sql: string, ...params: any[]): SqlStatement<T>;
}
export declare function datatable(name: string): DatatableSqlTemplateFunction;
export interface DatatableOptions {
role?: string;
}
export declare function datatable(name?: string, opts?: DatatableOptions): DatatableSqlTemplateFunction;
export declare function ducklake(name: string): SqlTemplateFunction;
export interface DucklakeMaterializeOptions {
typescript-client/sqlUtils.ts
+40 -5
View File
@@ -108,11 +108,19 @@ export interface DatatableSqlTemplateFunction extends SqlTemplateFunction {
query<T = any>(sql: string, ...params: any[]): SqlStatement<T>;
}
export interface DatatableOptions {
/** The data table role to connect as. Omit to get the data table's default role. */
role?: string;
}
// ---------------------------------------------------------------------------
// Provider interface — captures what differs between datatable and ducklake
// ---------------------------------------------------------------------------
interface SqlProvider {
/** Leading annotation lines. Emitted before everything else: the executor's annotation parser
* stops at the first non-comment line. */
annotations(): string;
formatArgDecl(argNum: number, argType: string): string;
formatArgUsage(
argNum: number,
@@ -125,11 +133,25 @@ interface SqlProvider {
providerName: string;
}
function datatableProvider(name: string, schema?: string): SqlProvider {
// Interpolated into a `-- role <name>` line, so a value carrying a newline could append
// statements of its own. Mirrors the server's own role-name rule.
const ROLE_NAME_RE = /^[A-Za-z0-9_-]{1,63}$/;
function datatableProvider(
name: string,
schema?: string,
role?: string
): SqlProvider {
if (role !== undefined && !ROLE_NAME_RE.test(role)) {
throw new Error(
`Invalid data table role '${role}': only letters, digits, '_' and '-' are allowed`
);
}
return {
providerName: "datatable",
language: "postgresql",
extraArgs: { database: `datatable://${name}` },
annotations: () => (role !== undefined ? `-- role ${role}\n` : ""),
formatArgDecl: (argNum) => `-- $${argNum} arg${argNum}`,
formatArgUsage: (argNum, explicitType, inferredType) =>
explicitType !== undefined
@@ -144,6 +166,7 @@ function ducklakeProvider(name: string, schema?: string): SqlProvider {
providerName: "ducklake",
language: "duckdb",
extraArgs: {},
annotations: () => "",
formatArgDecl: (argNum, argType) => `-- $arg${argNum} (${argType})`,
formatArgUsage: (argNum) => `$arg${argNum}`,
// `USE dl."schema"` sets the active schema so unqualified tables resolve there.
@@ -272,7 +295,8 @@ function buildSqlTemplateFunction(provider: SqlProvider): SqlTemplateFunction {
return provider.formatArgDecl(info.argNum, argType);
});
let content = argDecls.length ? argDecls.join("\n") + "\n" : "";
let content = provider.annotations();
content += argDecls.length ? argDecls.join("\n") + "\n" : "";
content += provider.preamble();
// SQL body — inline raw values, reference params via provider syntax
@@ -322,6 +346,8 @@ function buildSqlTemplateFunction(provider: SqlProvider): SqlTemplateFunction {
/**
* Create a SQL template function for PostgreSQL/datatable queries
* @param name - Database/datatable name (default: "main")
* @param opts.role - Connect as this data table role instead of the data table's default one.
* Only meaningful on a data table under roles, and only for a role you are a tenant of.
* @returns SQL template function for building parameterized queries
* @example
* let sql = wmill.datatable()
@@ -331,10 +357,16 @@ function buildSqlTemplateFunction(provider: SqlProvider): SqlTemplateFunction {
* SELECT * FROM friends
* WHERE name = ${name} AND age = ${age}::int
* `.fetch()
* @example
* // Read through a restricted role
* let sql = wmill.datatable("main", { role: "analytics" })
*/
export function datatable(name: string = "main"): DatatableSqlTemplateFunction {
export function datatable(
name: string = "main",
opts?: DatatableOptions
): DatatableSqlTemplateFunction {
let { name: n, schema } = parseName(name);
let provider = datatableProvider(n, schema);
let provider = datatableProvider(n, schema, opts?.role);
let sqlFn = buildSqlTemplateFunction(provider) as DatatableSqlTemplateFunction;
// `.query(sql, ...params)` is for SQL strings that already contain
// positional placeholders ($1, $2, ...). We DON'T go through the template
@@ -353,7 +385,10 @@ export function datatable(name: string = "main"): DatatableSqlTemplateFunction {
.join("\n");
let contentBody = sqlString;
let content =
(argDecls ? argDecls + "\n" : "") + provider.preamble() + sqlString;
provider.annotations() +
(argDecls ? argDecls + "\n" : "") +
provider.preamble() +
sqlString;
let args = {
...Object.fromEntries(
params.map((v, i) => [`arg${i + 1}`, serializeArgValue(v)])