Compare commits

...
Author SHA1 Message Date
Diego ImbertandClaude Opus 5 c96ddc0333 fix(datatables): refuse moving the external cluster to another host or port while it holds databases
Every settings writer goes through the same check as removal: the login, TLS and maintenance
database may still change, the cluster may not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:32:59 +02:00
Diego Imbert 1d76c06426 Merge branch 'fork-database-authorization' into datatable-external-instance 2026-09-18 16:07:21 +02:00
Diego ImbertandClaude Opus 5 a853733bba fix(datatables): hold the fork lock across a rename's settings copy
Fork cleanup of the old id could otherwise drop a copy the renamed workspace goes on using. Also
states the authorization contract of the instance database drop helpers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 16:07:16 +02:00
Diego Imbert 64c1f28ee2 Merge branch 'fork-database-authorization' into datatable-external-instance 2026-09-18 15:48:37 +02:00
Diego ImbertandClaude Opus 5 d3ae20be67 fix(datatables): lock the fork copy at finalization, and order cleanup's registry write after its settings row
- Fork finalization holds the copy's database lock from its availability check to the commit.
- Cleanup removes the registry entry in its own transaction, in a task of its own, instead of on a
  second connection; a rename migrates reservations after its settings rewrites, so both take the
  settings rows before the registry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:48:37 +02:00
Diego Imbert 64f49226aa Merge branch 'fork-database-authorization' into datatable-external-instance 2026-09-18 15:28:03 +02:00
Diego ImbertandClaude Opus 5 a621bd9c42 fix(datatables): state the authorization contract of the external cluster status and write check
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:28:03 +02:00
Diego ImbertandClaude Opus 5 e29dacb938 fix(datatables): serialize fork database reservations, cleanup, setup and Ducklake saves on the database
- A fork copy is created and registered under its workspace's fork lock, and refused once the
  workspace is archived; a rename re-migrates reservations under that lock after archiving.
- Ducklake saves lock every instance database they newly name, as data table saves do.
- Instance database setup holds the database's lock until its entry is written.
- Fork import also holds the database's lock across the restore.
- Cleanup re-reads the entry under its locks before dropping anything.
- Non-superadmins no longer see fork copies reserved for workspaces they are not in.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:27:35 +02:00
Diego ImbertandClaude Opus 5 5aa09bc871 chore: drop the unused json import from the settings crate
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 15:01:17 +02:00
Diego Imbert b337ade93b Merge remote-tracking branch 'origin/datatable-roles-redesign-part-4' into datatable-external-instance
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-18 15:00:57 +02:00
Diego Imbert 89bee36774 Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-integration-tests/tests/datatable_roles.rs
#	backend/windmill-api-workspaces/src/datatable_permissions_oss.rs
#	backend/windmill-api-workspaces/src/workspaces.rs
#	backend/windmill-common/src/datatable_roles_oss.rs
#	backend/windmill-common/src/workspaces.rs
#	frontend/src/lib/components/datatableAcl/PgAclEditor.svelte
#	frontend/src/lib/components/workspaceSettings/DataTablePermissionsButton.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableRolesSection.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte
#	frontend/src/lib/components/workspaceSettings/InstanceRolesButton.svelte
2026-09-18 14:11:39 +02:00
Diego Imbert 45e24e5098 Merge remote-tracking branch 'origin/fork-database-authorization' into HEAD 2026-09-18 14:04:12 +02:00
windmill-internal-app[bot] 64f95bcb7b chore: update ee-repo-ref to 7e338e4dabf91689bfd7fb0333c6534040b17b59
This commit updates the EE repository reference after PR #787 was merged in windmill-ee-private.

Previous ee-repo-ref: 0edd40979cf36bfba59323f3f6a0811ae1369cf5

New ee-repo-ref: 7e338e4dabf91689bfd7fb0333c6534040b17b59

Automated by sync-ee-ref workflow.
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 908b98ef78 fix: check ACL read reach against the entry it connects from
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 de7be97124 fix: build the ACL connection from the authorized data table entry
Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 0f99cce43c fix: hold the ACL connection to the database that was authorized
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 d985f5d0ff fix: run one data table ACL apply at a time per server before it connects
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 9b9d164524 fix: add only missing grant options before an ACL apply, never default privileges
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 45dd4cb271 fix: refresh grant options only after the ACL apply validates its plan
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 f0f1f7c186 fix: take every pooled connection before the ACL apply locks
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 76b796bc62 feat(datatables): add an ACL editor for data table roles
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 14:00:05 +02:00
Diego ImbertandClaude Opus 5 92c044af6a fix(datatables): hold the fork lock across a fork import, and carry the reservation inside the setup write
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 13:57:18 +02:00
Diego ImbertandClaude Opus 5 0f7cadb19c fix(datatables): keep fork reservations private, drop a cleaned-up entry with its database, and serialize cleanup with settings saves
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 01:11:43 +02:00
Diego Imbert 154bade187 Merge remote-tracking branch 'origin/fork-database-authorization' into HEAD 2026-09-18 00:58:33 +02:00
Diego ImbertandClaude Opus 5 e550374f06 fix(datatables): migrate fork reservations on workspace rename, and lock the parent's data tables for the whole fork
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 00:58:19 +02:00
Diego Imbert eab084b36f Merge remote-tracking branch 'origin/fork-database-authorization' into HEAD 2026-09-17 18:41:27 +02:00
Diego ImbertandClaude Opus 5 4cba79b7f0 fix(datatables): authenticate instance database setup before writing its status, and keep a fork reservation across it
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:41:11 +02:00
Diego Imbert c9e2246e1c Merge remote-tracking branch 'origin/fork-database-authorization' into HEAD 2026-09-17 18:17:06 +02:00
Diego ImbertandClaude Opus 5 b68768084e fix(datatables): bind fork database copies to their workspace, and count every use before dropping one
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:16:48 +02:00
Diego Imbert a928253af0 Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/windmill-common/src/workspaces.rs
2026-09-17 15:22:45 +02:00
Diego Imbert 5d8f031cf6 Merge branch 'datatable-roles-redesign' into datatable-roles-redesign-part-2 2026-09-17 15:22:26 +02:00
Diego ImbertandClaude Opus 5 8a7f364cfe fix(datatables): hold the parent's settings while a fork points at its data tables
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 13:29:35 +02:00
Diego ImbertandClaude Opus 5 ee176e24d5 perf(datatables): resolve a workspace's data tables per pointer hop, not per entry
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 13:04:26 +02:00
Diego ImbertandClaude Opus 5 4bdc5a9e20 fix(datatables): drop a DuckDB data table secret once its ATTACH has used it
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 12:17:30 +02:00
Diego ImbertandClaude Opus 5 dbe0230f48 fix(datatables): never grant CREATEROLE to custom_instance_user on the external cluster
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 11:49:20 +02:00
Diego Imbert b3a3f1cf5a Merge remote-tracking branch 'origin/datatable-roles-redesign-part-4' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 11:34:28 +02:00
Diego Imbert 92e51f19aa Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-integration-tests/tests/datatable_roles.rs
#	backend/windmill-api-workspaces/src/datatable_permissions_oss.rs
#	backend/windmill-api-workspaces/src/workspaces.rs
#	backend/windmill-common/src/datatable_roles_oss.rs
#	backend/windmill-common/src/workspaces.rs
#	frontend/src/lib/components/datatableAcl/PgAclEditor.svelte
#	frontend/src/lib/components/workspaceSettings/DataTablePermissionsButton.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableRolesSection.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte
#	frontend/src/lib/components/workspaceSettings/InstanceRolesButton.svelte
2026-09-17 10:03:05 +02:00
Diego ImbertandClaude Opus 5 36532c35d8 fix: check ACL read reach against the entry it connects from
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 79481a2f02 fix: build the ACL connection from the authorized data table entry
Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 c884597e56 fix: hold the ACL connection to the database that was authorized
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 115448d3d1 fix: run one data table ACL apply at a time per server before it connects
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 1ece278fd2 fix: add only missing grant options before an ACL apply, never default privileges
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 7819d48df9 fix: refresh grant options only after the ACL apply validates its plan
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 ce1f720d85 fix: take every pooled connection before the ACL apply locks
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 93d346e54b feat(datatables): add an ACL editor for data table roles
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 9378828472 docs(datatables): note that legacy names with ? cannot be migrated
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 79bb761c5d fix(datatables): serialize roles going on with aliases saved from other workspaces
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 d191cfe4eb fix(datatables): let CE migrations connect as an explicitly named admin
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 31b5b0a062 fix(datatables): refuse roles while a same-workspace alias reaches the database
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
windmill-internal-app[bot] d1c2f801f4 chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607
This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 d3ee67c9bc test(datatables): run the tenant matcher test on the enterprise edition
The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 eafc551651 feat(datatables): gate the roles UI mount sites on an enterprise license
Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 db05c8b7c9 test(datatables): run the roles tests on the enterprise edition, refusals without it
Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 11ba3cd477 feat(datatables): move the data table role catalog endpoints to the enterprise edition
The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 7ae3335f03 feat(datatables): move the data table permissions endpoints to the enterprise edition
The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 c7269e975f feat(datatables): move data table role catalog and resolution to the enterprise edition
Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 8a0babd7a6 fix(datatables): no entry without roles may newly reach a database under roles
The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 6d9ef5f6b7 fix(datatables): refuse a save that drops a data table's roles through an undeclared rename
A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 1dbc1e7759 fix(datatables): warn when a settings sync strands fork pointers
A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 df471e37d8 fix(datatables): wait out live listeners, and resolve stored names containing ?
Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 122a3bd164 fix(datatables): serialize roles going on with a stream starting
Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 2f95c22242 fix(datatables): disable a data table role before deleting it
Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 7279ee15e2 fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved
Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 fa1e8bd3aa refactor(datatables): keep Postgres triggers and data table roles apart
A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 dc3ebfbe74 fix(datatables): keep the fork schema baseline, and bounce streams on every removal
Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 8255f2ad2b fix(datatables): bounce the streams reading a data table when it is deleted
Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 ebe076cbe6 docs(datatables): record the stale clone database as a known limitation
A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 643bc1b51a Revert "fix(datatables): let a retried clone reclaim its own leftover database"
This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 29fccb0978 fix(datatables): let a retried clone reclaim its own leftover database
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 432ccc0fbd refactor(datatables): settle clonability in one place, before anything is created
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 e6794984d2 fix(datatables): refuse to roll back the catalog while roles exist
The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 2abcd122a8 fix(datatables): clone a fork's pointer instead of failing after the copy
Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 30f039d292 fix(datatables): honour -- role: x, and fix the DuckDB attach test
Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 5fd9ab24b8 chore(datatables): don't mount the roles UI until the ACL editor lands
Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 f7812187cc fix(datatables): carry the role annotation into the row_to_json retry
The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 b5785e8e7a fix(datatables): refuse a malformed role query string instead of ignoring it
`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 ac8587e452 fix(datatables): refuse a malformed role annotation instead of ignoring it
`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego Imbert 3ae1688c44 fix pg_dump stuck on version 17 on nix 2026-09-17 10:01:15 +02:00
Diego Imbert d69da0bbd6 nit worker error msg 2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 eeed332591 fix(datatables): refuse the clone's database too, not only its data
A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 78a12fdfcc fix(datatables): refuse to copy a data table that is under roles
pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 189bc3297b fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically
The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 0efddb94cf fix(datatables): validate a rename against the save it describes, and re-check under the locks
Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 1db77dda87 refactor(datatables): put the role catalog in its own table, not in global_settings
Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 d0ea1ce598 fix(datatables): close the last ways a role or a pointer can be left pointing at nothing
The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 c400469e67 fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced
The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 a74f73ccb1 fix(datatables): give the role catalog its own row, out of reach of the config machinery
Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 a7bff8de97 fix(datatables): serialize role catalog mutations, and state each helper's authorization contract
The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 0d5106f618 fix(datatables): unbreak two operator messages and two comments that described other code
The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 b8d6b3c953 fix(datatables): fail loudly where a role or a pointer can be left half-recorded
Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 c2c24598a2 fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping
A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 21d97f3a4c fix(datatables): gate the paths that reach a whole database as admin
Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 34ed0c4230 feat(datatables): put a data table's connection under Postgres roles
A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-17 10:01:15 +02:00
Diego Imbert 13162656bf Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-workspaces/src/datatable_permissions_oss.rs
2026-09-17 09:50:52 +02:00
Diego ImbertandClaude Opus 5 e79bc14903 fix: check ACL read reach against the entry it connects from
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 09:00:04 +02:00
Diego ImbertandClaude Opus 5 b7623f12eb fix: build the ACL connection from the authorized data table entry
Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 00:09:29 +02:00
Diego ImbertandClaude Opus 5 631c01d20d fix(datatables): compare the external cluster settings under a row lock before storing setup
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 00:07:50 +02:00
Diego ImbertandClaude Opus 5 0a2ebfb3db fix: hold the ACL connection to the database that was authorized
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 23:55:43 +02:00
Diego Imbert 0489cb1630 Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-integration-tests/tests/datatable_roles.rs
2026-09-16 23:45:48 +02:00
Diego ImbertandClaude Opus 5 d7417f5cc2 Merge branch 'datatable-roles-redesign' into datatable-roles-redesign-part-2
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 23:40:04 +02:00
Diego ImbertandClaude Opus 5 5ef5f58eeb fix(datatables): scope external cluster credential readers to the crate
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:38:06 +02:00
Diego ImbertandClaude Opus 5 4533ec460e fix(datatables): send external cluster passwords as SCRAM verifiers
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 22:44:11 +02:00
Diego ImbertandClaude Opus 5 b3cda2f548 feat(datatables): set up an external instance cluster for data tables
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 22:35:00 +02:00
Diego ImbertandClaude Opus 5 8d0d51e99d fix: run one data table ACL apply at a time per server before it connects
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 22:25:51 +02:00
Diego ImbertandClaude Opus 5 110876d0c9 docs(datatables): note that legacy names with ? cannot be migrated
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 22:25:27 +02:00
Diego ImbertandClaude Opus 5 e5ec5217d4 fix(datatables): serialize roles going on with aliases saved from other workspaces
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 18:09:07 +02:00
Diego ImbertandClaude Opus 5 6a89ba0179 fix: add only missing grant options before an ACL apply, never default privileges
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 17:31:06 +02:00
Diego ImbertandClaude Opus 5 707b0dc8ee fix(datatables): let CE migrations connect as an explicitly named admin
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:25:42 +02:00
Diego ImbertandClaude Opus 5 dcc37bf6f2 fix: declare the default role in migrations written for a data table whose name contains '?'
Such a data table connects as its default role without naming it, so the
migrations the manager wrote for it declared no role and ran as admin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:20:19 +02:00
Diego ImbertandClaude Opus 5 10861bb529 fix(datatables): refuse a reference naming both a legacy data table and a role
When a workspace stores both `sales` and a legacy `sales?role=analytics`, the
reference resolved to the legacy entry without a role, so browsing `sales` as
`analytics` reached another data table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 17:03:32 +02:00
Diego ImbertandClaude Opus 5 d459b371d8 fix: refresh grant options only after the ACL apply validates its plan
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 16:53:13 +02:00
Diego ImbertandClaude Opus 5 41e1101a2d fix: take every pooled connection before the ACL apply locks
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-16 16:39:14 +02:00
Diego ImbertandClaude Opus 5 4416283b17 fix: read the roles of a data table whose name contains '?'
The generated client leaves a '?' in a path param unencoded, so the lookup
404'd and the raw-app picker blocked Start on such a data table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:32:36 +02:00
Diego ImbertandClaude Opus 5 e0360bce5c fix: never add a role to the reference of a data table whose name contains '?'
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:07:00 +02:00
Diego ImbertandClaude Opus 5 f36aa69fc3 feat(datatables): data table roles in the DB manager and raw apps
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:23:20 +02:00
Diego ImbertandClaude Opus 5 a1b91690fd feat(datatables): add an ACL editor for data table roles
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:22:05 +02:00
Diego ImbertandClaude Opus 5 0676f55856 fix(datatables): refuse roles while a same-workspace alias reaches the database
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
windmill-internal-app[bot] aa6a52271b chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607
This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 64d5f762dc test(datatables): run the tenant matcher test on the enterprise edition
The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 4c2f1a7b60 feat(datatables): gate the roles UI mount sites on an enterprise license
Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 16fdba4d82 test(datatables): run the roles tests on the enterprise edition, refusals without it
Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 1986ec2535 feat(datatables): move the data table role catalog endpoints to the enterprise edition
The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 a7bac22c13 feat(datatables): move the data table permissions endpoints to the enterprise edition
The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 1d6da0084e feat(datatables): move data table role catalog and resolution to the enterprise edition
Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 3d4a2133ed fix(datatables): no entry without roles may newly reach a database under roles
The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 f8a6347052 fix(datatables): refuse a save that drops a data table's roles through an undeclared rename
A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 46dfcc5c49 fix(datatables): warn when a settings sync strands fork pointers
A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 e9c27d63db fix(datatables): wait out live listeners, and resolve stored names containing ?
Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 65a6cc3eb5 fix(datatables): serialize roles going on with a stream starting
Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 dd2ca5513e fix(datatables): disable a data table role before deleting it
Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 6a3a2ed13c fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved
Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 791bb8d843 refactor(datatables): keep Postgres triggers and data table roles apart
A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 b68b97f5ea fix(datatables): keep the fork schema baseline, and bounce streams on every removal
Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 92c723f735 fix(datatables): bounce the streams reading a data table when it is deleted
Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 c7f28ba8ed docs(datatables): record the stale clone database as a known limitation
A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 bea9667516 Revert "fix(datatables): let a retried clone reclaim its own leftover database"
This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 bb8c8ed735 fix(datatables): let a retried clone reclaim its own leftover database
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 4610f8a35c refactor(datatables): settle clonability in one place, before anything is created
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 284d1ecb43 fix(datatables): refuse to roll back the catalog while roles exist
The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:29 +02:00
Diego ImbertandClaude Opus 5 306a35e3b0 fix(datatables): clone a fork's pointer instead of failing after the copy
Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 5a98030fb7 fix(datatables): honour -- role: x, and fix the DuckDB attach test
Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 d558b6508a chore(datatables): don't mount the roles UI until the ACL editor lands
Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 fce20faeae fix(datatables): carry the role annotation into the row_to_json retry
The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 da3babccc7 fix(datatables): refuse a malformed role query string instead of ignoring it
`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 69742c0b48 fix(datatables): refuse a malformed role annotation instead of ignoring it
`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego Imbert 3dfbf64d9d fix pg_dump stuck on version 17 on nix 2026-09-16 15:14:28 +02:00
Diego Imbert d32ff92eac nit worker error msg 2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 01eba26582 fix(datatables): refuse the clone's database too, not only its data
A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 cafcf3afc3 fix(datatables): refuse to copy a data table that is under roles
pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 030254af47 fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically
The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 9872bff69f fix(datatables): validate a rename against the save it describes, and re-check under the locks
Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 e400daabb8 refactor(datatables): put the role catalog in its own table, not in global_settings
Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 7f3c7a19af fix(datatables): close the last ways a role or a pointer can be left pointing at nothing
The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 1d9ee09b31 fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced
The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 715d8a0e6d fix(datatables): give the role catalog its own row, out of reach of the config machinery
Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 ea14c01a3c fix(datatables): serialize role catalog mutations, and state each helper's authorization contract
The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 54f4467027 fix(datatables): unbreak two operator messages and two comments that described other code
The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 8276ae09fa fix(datatables): fail loudly where a role or a pointer can be left half-recorded
Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 6f9457fc8e fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping
A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 6ea682d741 fix(datatables): gate the paths that reach a whole database as admin
Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
Diego ImbertandClaude Opus 5 22b3e69c07 feat(datatables): put a data table's connection under Postgres roles
A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR
2026-09-16 15:14:28 +02:00
74 changed files with 7728 additions and 1046 deletions
+1
View File
@@ -15653,6 +15653,7 @@ dependencies = [
"pin-project-lite",
"pkcs1",
"postgres-native-tls 0.5.3",
"postgres-protocol",
"prometheus",
"quick_cache",
"rand 0.9.0",
+1
View File
@@ -624,6 +624,7 @@ wasm-bindgen-test = "^0"
convert_case = "0.6.0"
getrandom = "0.2"
tokio-postgres = {version = "^0.7", features = ["array-impls", "with-serde_json-1", "with-chrono-0_4", "with-uuid-1", "with-bit-vec-0_6"]}
postgres-protocol = "0.6"
rust-postgres = { package = "tokio-postgres", git = "https://github.com/MaterializeInc/rust-postgres", rev = "78c1222577bb091d69bc22b1bc7ad01c14675abe"}
rust-postgres-native-tls = { package = "postgres-native-tls", git = "https://github.com/MaterializeInc/rust-postgres", features = ["runtime"], rev = "78c1222577bb091d69bc22b1bc7ad01c14675abe" }
bit-vec = "=0.6.3"
+1 -1
View File
@@ -1 +1 @@
7e338e4dabf91689bfd7fb0333c6534040b17b59
23d12f73e44a24bb91fa54d79dfc4ae1436e0227
+46
View File
@@ -1583,3 +1583,49 @@ async fn declarative_sync_rejects_an_unusable_default_allowed_origins(db: Pool<P
.await
.expect("a valid origin list must sync");
}
/// While Windmill has databases on the external cluster, a sync may change its login but not
/// point it at another cluster: the databases would be stranded there, and their registry and
/// role passwords applied to a cluster that has neither.
#[cfg(all(feature = "enterprise", feature = "private"))]
#[sqlx::test(fixtures("base"))]
async fn declarative_sync_keeps_the_external_cluster_while_it_holds_databases(db: Pool<Postgres>) {
clear_settings_and_configs(&db).await;
let cluster = |host: &str, password: &str| serde_json::json!({ "host": host, "port": 5432, "user": "wm_admin", "password": password });
sqlx::query(
"INSERT INTO global_settings (name, value) VALUES
('external_instance_pg', $1),
('external_instance_pg_state', '{\"databases\": {\"dt_a\": {\"success\": true}}}')",
)
.bind(cluster("pg-a.internal", "one"))
.execute(&db)
.await
.unwrap();
let current = BTreeMap::from([(
"external_instance_pg".to_string(),
cluster("pg-a.internal", "one"),
)]);
let sync = |value: serde_json::Value| {
let desired = BTreeMap::from([("external_instance_pg".to_string(), value)]);
let (db, current) = (db.clone(), current.clone());
async move {
windmill_common::instance_config::sync_global_settings_declarative(
&db, &current, &desired,
)
.await
}
};
let err = sync(cluster("pg-b.internal", "one"))
.await
.expect_err("another host must be refused while dt_a is registered");
assert!(err.to_string().contains("dt_a"), "got: {err}");
assert_eq!(
get_global_setting(&db, "external_instance_pg").await,
Some(cluster("pg-a.internal", "one"))
);
sync(cluster("PG-A.internal ", "two"))
.await
.expect("a new login on the same cluster must sync");
}
@@ -0,0 +1,108 @@
//! Who may read and change a data table's grants and owners. On the Enterprise Edition: its
//! administrators, from the workspace that governs it. Without it: nobody. Each refusal is decided
//! before anything connects to the data table, so the fixture's database never has to exist.
use serde_json::{json, Value};
use sqlx::{Pool, Postgres};
use windmill_test_utils::*;
fn grant_select_on_public() -> Value {
json!({
"target": {"kind": "schema", "schema": "public"},
"change": {"type": "grant", "role": "analytics", "privileges": ["SELECT"],
"scope": "all_tables"},
"statements": [r#"GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO "analytics""#]
})
}
async fn post_acl(
port: u16,
w_id: &str,
action: &str,
token: &str,
) -> anyhow::Result<reqwest::Response> {
Ok(reqwest::Client::new()
.post(format!(
"http://localhost:{port}/api/w/{w_id}/workspaces/datatable_acl/main/{action}"
))
.header("Authorization", format!("Bearer {token}"))
.json(&grant_select_on_public())
.send()
.await?)
}
/// A fork reaches the data table through a pointer: it may use it, never change what each role may
/// touch on it — not even as an admin of the fork.
#[cfg(all(feature = "private", feature = "enterprise"))]
#[sqlx::test(migrations = "../migrations", fixtures("base", "datatable_roles"))]
async fn a_fork_cannot_change_access_on_the_data_table_it_points_at(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
for action in ["plan", "apply"] {
let resp = post_acl(port, "wm-fork-dt", action, "SECRET_TOKEN_2").await?;
assert_eq!(resp.status(), 401, "{action}: {}", resp.text().await?);
}
Ok(())
}
#[cfg(all(feature = "private", feature = "enterprise"))]
#[sqlx::test(migrations = "../migrations", fixtures("base", "datatable_roles"))]
async fn a_member_who_is_not_an_admin_cannot_change_access(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
for action in ["plan", "apply"] {
let resp = post_acl(port, "test-workspace", action, "SECRET_TOKEN_2").await?;
assert_eq!(resp.status(), 401, "{action}: {}", resp.text().await?);
}
Ok(())
}
/// Not even reading, and not even on a data table that is not under roles — which any member
/// reaches, so only the edition stands between them and the instance's credentials.
#[cfg(not(all(feature = "private", feature = "enterprise")))]
#[sqlx::test(migrations = "../migrations", fixtures("base", "datatable_roles"))]
async fn only_the_enterprise_edition_has_the_access_editor(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
sqlx::query(
"UPDATE workspace_settings
SET datatable = datatable #- '{datatables,main,permissions}'
WHERE workspace_id = 'test-workspace'",
)
.execute(&db)
.await?;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let read = reqwest::Client::new()
.get(format!(
"http://localhost:{port}/api/w/test-workspace/workspaces/datatable_acl/main?kind=database"
))
.header("Authorization", "Bearer SECRET_TOKEN")
.send()
.await?;
let mut responses = vec![("read", read)];
for action in ["plan", "apply"] {
responses.push((
action,
post_acl(port, "test-workspace", action, "SECRET_TOKEN").await?,
));
}
for (action, resp) in responses {
assert_eq!(resp.status(), 400, "{action}");
let body = resp.text().await?;
assert!(
body.contains("Data table roles are a Windmill Enterprise Edition feature"),
"{action}: {body}"
);
}
Ok(())
}
@@ -912,6 +912,18 @@ async fn a_stored_name_containing_a_question_mark_resolves_as_itself(
resolve("main?dt").await.is_err(),
"an unknown parameter was ignored"
);
sqlx::query(
"UPDATE workspace_settings
SET datatable = jsonb_set(datatable, '{datatables,main?role=analytics}', datatable->'datatables'->'main')
WHERE workspace_id = 'test-workspace'",
)
.execute(&db)
.await?;
assert!(
resolve("main?role=analytics").await.is_err(),
"a reference naming both a stored data table and a role on another resolved to one of them"
);
Ok(())
}
@@ -1009,6 +1021,70 @@ async fn an_entry_without_roles_cannot_newly_reach_a_database_under_roles(
Ok(())
}
/// Browsing names the role it connects as, and a role the caller may not use is refused rather
/// than quietly listed as the default. The refusal is decided before connecting, so the fixture's
/// database never has to exist.
#[cfg(all(feature = "private", feature = "enterprise"))]
#[sqlx::test(migrations = "../migrations", fixtures("base", "datatable_roles"))]
async fn browsing_as_a_role_the_caller_may_not_use_is_refused(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let base = format!("http://localhost:{port}/api/w/test-workspace/workspaces");
// `test-user-2` is a tenant of `analytics` only.
let resp = authed(
client().get(format!(
"{base}/list_datatable_tables?role_for=main&role=admin"
)),
"SECRET_TOKEN_2",
)
.send()
.await?;
assert_eq!(resp.status(), 200);
let body: Value = resp.json().await?;
let entry = body
.as_array()
.and_then(|a| a.iter().find(|e| e["datatable_name"] == "main"))
.expect("main is listed");
assert_eq!(entry["usable_roles"], json!(["analytics"]), "{entry}");
assert_eq!(entry["default_role"], "analytics", "{entry}");
assert_eq!(entry["permissioned"], true, "{entry}");
assert_eq!(entry["instance"], true, "{entry}");
let error = entry["error"].as_str().unwrap_or_default();
assert!(
error.contains("Not allowed to use role 'admin'"),
"listed as another role than the one asked for: {entry}"
);
let resp = authed(
client().get(format!(
"{base}/get_datatable_table_schema?datatable_name=main&schema_name=public&table_name=t&role=admin"
)),
"SECRET_TOKEN_2",
)
.send()
.await?;
let status = resp.status();
let text = resp.text().await?;
assert!(
text.contains("Not allowed to use role 'admin'"),
"{status}: {text}"
);
// A role means nothing without the data table it belongs to.
let resp = authed(
client().get(format!("{base}/list_datatable_tables?role=analytics")),
"SECRET_TOKEN_2",
)
.send()
.await?;
assert_eq!(resp.status(), 400, "{}", resp.text().await?);
Ok(())
}
#[sqlx::test(migrations = "../migrations", fixtures("base", "datatable_roles"))]
async fn an_alias_saved_elsewhere_waits_for_roles_going_on_for_its_database(
db: Pool<Postgres>,
+145 -10
View File
@@ -42,7 +42,6 @@ use axum::{
routing::{get, post},
Json, Router,
};
use serde_json::json;
use serde::{Deserialize, Serialize};
use windmill_ai::ai_cache::bump_instance_ai_config_revision;
@@ -60,7 +59,7 @@ use windmill_common::{
global_settings::{
AI_CONFIG_SETTING, APP_WORKSPACED_ROUTE_SETTING, AUTOMATE_USERNAME_CREATION_SETTING,
CRITICAL_ALERT_MUTE_UI_SETTING, CUSTOM_TAGS_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING,
DISABLE_HUB_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS,
DISABLE_HUB_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS, EXTERNAL_INSTANCE_PG_SETTING,
GITHUB_APP_WEBHOOK_BASE_URL_SETTING, HTTP_ROUTE_DEFAULT_ALLOWED_ORIGINS_SETTING,
HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING,
INSTANCE_BANNER_SETTING, MAX_RETENTION_OVERRIDE_WORKSPACES,
@@ -168,6 +167,14 @@ pub fn global_service() -> Router {
"/refresh_custom_instance_user_pwd",
post(refresh_custom_instance_user_pwd),
)
.route(
"/external_instance_pg/status",
get(get_external_instance_pg_status),
)
.route(
"/external_instance_pg/setup",
post(setup_external_instance_pg),
)
.route(
"/setup_custom_instance_pg_database/{name}",
post(setup_custom_instance_pg_database),
@@ -938,6 +945,13 @@ async fn run_setting_pre_write_hook(
value: &serde_json::Value,
) -> error::Result<()> {
match key {
EXTERNAL_INSTANCE_PG_SETTING => {
windmill_common::external_instance_pg::check_external_instance_pg_write(
db,
Some(value),
)
.await?;
}
// The instance AI config is written as an untyped blob through this generic
// endpoint, so it never passes the typed check the workspace handler applies.
// Rates that reach a cost total unbounded would make it negative or infinite.
@@ -1289,6 +1303,14 @@ async fn set_instance_config(
for (key, value) in &settings_diff.upserts {
run_setting_pre_write_hook(&db, key, value).await?;
}
if settings_diff
.deletes
.iter()
.any(|k| k == EXTERNAL_INSTANCE_PG_SETTING)
{
windmill_common::external_instance_pg::check_external_instance_pg_write(&db, None)
.await?;
}
instance_config::apply_settings_diff(&db, &settings_diff)
.await
@@ -1647,6 +1669,8 @@ struct CustomInstanceDb {
tag: Option<String>,
#[serde(default, skip_serializing_if = "Vec::is_empty")]
used_by_workspaces: Vec<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
workspace_id: Option<String>,
}
#[derive(Deserialize, Debug, Serialize, Default)]
@@ -1687,7 +1711,40 @@ async fn list_custom_instance_pg_databases(
))
})?;
if windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
if !windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
// A fork copy's name gives away the workspace it was reserved for, so every pending fork on
// the instance would be listed. Kept for members of that workspace, and wherever the
// caller's workspaces use it, e.g. the fork it was finalized into.
let reserved_visible: BTreeSet<String> = sqlx::query_scalar(
r#"SELECT e.k FROM global_settings gs
CROSS JOIN LATERAL jsonb_each(gs.value->'databases') AS e(k, v)
WHERE gs.name = 'custom_instance_pg_databases' AND e.v->>'workspace_id' IS NOT NULL
AND (EXISTS (SELECT 1 FROM usr WHERE usr.email = $1
AND usr.workspace_id = e.v->>'workspace_id')
OR EXISTS (SELECT 1 FROM usr JOIN workspace_settings ws
ON ws.workspace_id = usr.workspace_id
CROSS JOIN LATERAL jsonb_each(
CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
WHERE usr.email = $1
AND dt.value->'database'->>'resource_type' = 'instance'
AND dt.value->'database'->>'resource_path' = e.k))"#,
)
.bind(&authed.email)
.fetch_all(&db)
.await?
.into_iter()
.collect();
result.retain(|dbname, entry| {
entry.workspace_id.is_none() || reserved_visible.contains(dbname)
});
// Which workspace reserved a copy is still only for superadmins.
for entry in result.values_mut() {
entry.workspace_id = None;
}
return Ok(Json(result));
}
{
// Enrich each database with the list of workspaces referencing it through
// either a ducklake catalog or a datatable database whose resource_type is
// 'instance'. Not stored in DB to avoid drift.
@@ -1743,6 +1800,54 @@ async fn refresh_custom_instance_user_pwd(
Ok(Json(()))
}
async fn get_external_instance_pg_status(
authed: ApiAuthed,
Extension(db): Extension<DB>,
) -> JsonResult<windmill_common::external_instance_pg::ExternalInstancePgStatus> {
require_super_admin(&db, &authed).await?;
Ok(Json(
windmill_common::external_instance_pg::external_instance_pg_status(&db).await?,
))
}
#[derive(Deserialize)]
struct SetupExternalInstancePgBody {
#[serde(default)]
rotate_passwords: bool,
}
async fn setup_external_instance_pg(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Json(body): Json<SetupExternalInstancePgBody>,
) -> JsonResult<windmill_common::external_instance_pg::ExternalInstancePgSetupReport> {
require_super_admin(&db, &authed).await?;
let report = windmill_common::external_instance_pg::setup_external_instance_pg_unchecked(
&db,
body.rotate_passwords,
)
.await?;
let rotated = body.rotate_passwords.to_string();
let success = report.success.to_string();
windmill_audit::audit_oss::audit_log(
&db,
&authed,
"settings.setup_external_instance_pg",
windmill_audit::ActionKind::Update,
"global",
Some(&authed.email),
Some(
[
("rotate_passwords", rotated.as_str()),
("success", success.as_str()),
]
.into(),
),
)
.await?;
Ok(Json(report))
}
#[derive(Deserialize)]
struct SetupCustomInstanceDbBody {
tag: Option<String>,
@@ -1754,18 +1859,48 @@ async fn setup_custom_instance_pg_database(
Path(dbname): Path<String>,
Json(body): Json<SetupCustomInstanceDbBody>,
) -> JsonResult<CustomInstanceDb> {
// Before anything is recorded: the status written below replaces the registry entry, and with it
// the workspace a fork copy is reserved for.
require_super_admin(&db, &authed).await?;
// Fork cleanup checks and drops the database and its entry under this lock. Held from before
// the setup creates the database to after its entry is written, neither lands on the other's
// half-done state: a dropped database with its entry written back, or the reverse.
let mut tx = db.begin().await?;
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut tx,
[dbname.trim()],
)
.await?;
let mut logs = CustomInstanceDbLogs::default();
let result = setup_custom_instance_pg_database_inner(authed, &db, &dbname, &mut logs).await;
let success = result.is_ok();
let error = result.err().map(|e| e.to_string());
let status =
CustomInstanceDb { logs, success, error, tag: body.tag, used_by_workspaces: vec![] };
let status = CustomInstanceDb {
logs,
success,
error,
tag: body.tag,
used_by_workspaces: vec![],
workspace_id: None,
};
let status_json = serde_json::to_value(&status).map_err(to_anyhow)?;
// Save that the database was setup successfully
sqlx::query!(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
json!({ dbname: status_json })
).execute(&db).await?;
// The fork reservation is carried over inside the write, from whatever the row holds then: a
// rename migrating it while the setup above ran would otherwise be overwritten with the value
// this request started from, stranding the copy under the archived workspace.
let saved = sqlx::query_scalar::<_, serde_json::Value>(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}',
COALESCE(value->'databases', '{}'::jsonb)
|| jsonb_build_object($1::text, $2::jsonb || jsonb_build_object(
'workspace_id', value->'databases'->$1::text->'workspace_id')))
WHERE name = 'custom_instance_pg_databases'
RETURNING value->'databases'->$1::text"#,
)
.bind(&dbname)
.bind(&status_json)
.fetch_one(&mut *tx)
.await?;
tx.commit().await?;
let status: CustomInstanceDb = serde_json::from_value(saved).map_err(to_anyhow)?;
Ok(Json(status))
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,44 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Where the ACL planner comes from: the enterprise one, or a refusal.
//!
//! Data table roles are an Enterprise Edition feature, and so is everything here — reading who
//! owns what included. `private` alone is not that edition — community builds carry it — so the
//! planner is behind `enterprise` as well.
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_acl_ee::plan_statements;
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) fn ensure_datatable_acl_available() -> windmill_common::error::Result<()> {
Ok(())
}
#[cfg(not(all(feature = "private", feature = "enterprise")))]
use {
crate::datatable_acl::{AclChange, AclPlan, AclTarget, CatalogFacts},
windmill_common::{datatable_roles_oss::datatable_roles_unavailable, error::Result},
};
/// Checked first by every ACL route, before anything is read or connected to.
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) fn ensure_datatable_acl_available() -> Result<()> {
Err(datatable_roles_unavailable())
}
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) fn plan_statements(
_target: &AclTarget,
_change: &AclChange,
_dbname: &str,
_pg_role: &str,
_facts: &CatalogFacts,
) -> Result<AclPlan> {
Err(datatable_roles_unavailable())
}
@@ -65,3 +65,15 @@ pub(crate) async fn ensure_reaches_datatable(
) -> Result<()> {
roles::ensure_reaches_datatable(db, w_id, datatable_name, authed).await
}
/// [`ensure_reaches_datatable`] against an entry already resolved, for a caller that goes on to
/// connect from that same entry.
pub(crate) async fn ensure_reaches_governing_datatable(
db: &DB,
w_id: &str,
datatable_name: &str,
governing: &GoverningDatatable,
authed: &ApiAuthed,
) -> Result<()> {
roles::ensure_reaches_governing_datatable(db, w_id, datatable_name, governing, authed).await
}
@@ -12,8 +12,9 @@
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_permissions_ee::{
ensure_governs_datatable, ensure_reaches_datatable, get_datatable_permissions,
list_usable_datatable_roles, set_datatable_permissions,
ensure_governs_datatable, ensure_reaches_datatable, ensure_reaches_governing_datatable,
get_datatable_permissions, list_usable_datatable_roles, set_datatable_permissions,
usable_datatable_roles,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
@@ -56,6 +57,20 @@ mod ce {
}
}
pub(crate) async fn ensure_reaches_governing_datatable(
_db: &DB,
_w_id: &str,
_datatable_name: &str,
governing: &GoverningDatatable,
_authed: &ApiAuthed,
) -> Result<()> {
if governing.datatable.permissions.is_none() {
Ok(())
} else {
Err(unavailable())
}
}
// The routes stay registered so the API has one shape; each answers after authentication,
// before anything is read.
@@ -70,4 +85,28 @@ mod ce {
pub(crate) async fn list_usable_datatable_roles(_authed: ApiAuthed) -> Result<String> {
Err(unavailable())
}
pub(crate) struct UsableDatatableRoles {
pub(crate) permissioned: bool,
pub(crate) roles: Vec<String>,
pub(crate) default_role: String,
}
/// A data table not under roles is used as `admin`, as before roles existed. One under roles
/// is refused: no role of it can be connected as.
pub(crate) async fn usable_datatable_roles(
_db: &DB,
_authed: &ApiAuthed,
_w_id: &str,
governing: &GoverningDatatable,
) -> Result<UsableDatatableRoles> {
if governing.datatable.permissions.is_some() {
return Err(unavailable());
}
Ok(UsableDatatableRoles {
permissioned: false,
roles: vec![],
default_role: windmill_common::datatable_roles::ADMIN_DATATABLE_ROLE.to_string(),
})
}
}
@@ -1,6 +1,8 @@
#[cfg(feature = "parquet")]
pub mod ai_session_backups;
pub mod data_metrics;
pub mod datatable_acl;
pub mod datatable_acl_oss;
pub mod datatable_migrations;
pub mod datatable_permissions;
pub mod datatable_permissions_oss;
@@ -12,5 +14,8 @@ pub mod workspaces_oss;
#[cfg(feature = "private")]
pub mod workspaces_ee;
#[cfg(all(feature = "private", feature = "enterprise"))]
pub mod datatable_acl_ee;
#[cfg(all(feature = "private", feature = "enterprise"))]
pub mod datatable_permissions_ee;
+245 -30
View File
@@ -144,6 +144,7 @@ pub fn workspaced_service() -> Router {
)
.merge(crate::datatable_migrations::routes())
.merge(crate::datatable_permissions::routes())
.merge(crate::datatable_acl::routes())
.route("/git_sync_enabled", get(get_git_sync_enabled))
.route("/git_sync_deploy_mode", get(get_git_sync_deploy_mode))
.route("/edit_git_sync_config", post(edit_git_sync_config))
@@ -2262,6 +2263,25 @@ struct DataTableTables {
schemas: TableListMap,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<String>,
/// On the instance database: the only kind that can be under roles or have its access edited.
instance: bool,
permissioned: bool,
/// The roles this caller may connect as, by name; empty when not under roles.
usable_roles: Vec<String>,
default_role: String,
/// What the role the listing connected as may create.
can_create_schema: bool,
creatable_schemas: Vec<String>,
}
#[derive(Deserialize)]
struct ListDataTableTablesQuery {
/// List only this data table: each entry opens a connection to its database.
datatable_name: Option<String>,
/// The data table `role` applies to. Every other one is listed as its default role, since a
/// role name means nothing outside the data table it belongs to.
role_for: Option<String>,
role: Option<String>,
}
#[derive(Deserialize)]
@@ -2269,6 +2289,7 @@ struct GetDataTableSchemaQuery {
datatable_name: String,
schema_name: String,
table_name: String,
role: Option<String>,
}
#[derive(Serialize, Debug)]
@@ -2436,25 +2457,89 @@ async fn list_datatable_tables(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Query(query): Query<ListDataTableTablesQuery>,
) -> JsonResult<Vec<DataTableTables>> {
let datatable_names = list_datatable_names(&db, &w_id).await?;
if query.role.is_some() && query.role_for.is_none() {
return Err(Error::BadRequest(
"`role` needs `role_for`, the data table it is a role of".to_string(),
));
}
if let (Some(only), Some(role_for)) =
(query.datatable_name.as_deref(), query.role_for.as_deref())
{
if only != role_for {
return Err(Error::BadRequest(format!(
"`role_for` names '{role_for}', which `datatable_name` leaves out of the listing"
)));
}
}
let mut datatable_names = list_datatable_names(&db, &w_id).await?;
for named in [query.role_for.as_deref(), query.datatable_name.as_deref()]
.into_iter()
.flatten()
{
if !datatable_names.iter().any(|n| n == named) {
return Err(Error::NotFound(format!(
"No data table named '{named}' in this workspace"
)));
}
}
if let Some(only) = query.datatable_name.as_deref() {
datatable_names.retain(|n| n == only);
}
let mut results = Vec::new();
for datatable_name in datatable_names {
let tables = match get_datatable_tables(&db, &authed, &w_id, &datatable_name).await {
Ok(schemas) => DataTableTables { datatable_name, schemas, error: None },
Err(e) => DataTableTables {
datatable_name,
schemas: HashMap::new(),
error: Some(e.to_string()),
},
};
results.push(tables);
let role = query
.role
.as_deref()
.filter(|_| query.role_for.as_deref() == Some(datatable_name.as_str()));
results.push(list_one_datatable_tables(&db, &authed, &w_id, datatable_name, role).await);
}
Ok(Json(results))
}
async fn list_one_datatable_tables(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: String,
role: Option<&str>,
) -> DataTableTables {
let mut entry = DataTableTables {
datatable_name,
schemas: HashMap::new(),
error: None,
instance: false,
permissioned: false,
usable_roles: vec![],
default_role: windmill_common::datatable_roles::ADMIN_DATATABLE_ROLE.to_string(),
can_create_schema: false,
creatable_schemas: vec![],
};
let result: Result<()> = async {
let governing = resolve_governing_datatable(db, w_id, &entry.datatable_name).await?;
entry.instance = governing.is_instance();
let usable =
crate::datatable_permissions_oss::usable_datatable_roles(db, authed, w_id, &governing)
.await?;
entry.permissioned = usable.permissioned;
entry.usable_roles = usable.roles;
entry.default_role = usable.default_role;
let listing = get_datatable_tables(db, authed, w_id, &entry.datatable_name, role).await?;
entry.schemas = listing.schemas;
entry.can_create_schema = listing.can_create_schema;
entry.creatable_schemas = listing.creatable_schemas;
Ok(())
}
.await;
if let Err(e) = result {
entry.error = Some(e.to_string());
}
entry
}
async fn get_datatable_table_schema(
authed: ApiAuthed,
Extension(db): Extension<DB>,
@@ -2468,6 +2553,7 @@ async fn get_datatable_table_schema(
&query.datatable_name,
&query.schema_name,
&query.table_name,
query.role.as_deref(),
)
.await?;
@@ -2506,14 +2592,13 @@ async fn resolve_datatable_pg_as_caller(
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
role: Option<&str>,
) -> Result<PgDatabase> {
let db_resource = get_datatable_resource_from_db(
db,
w_id,
datatable_name,
// The data table's default role. Browsing has no way to name another one yet; when the
// database manager grows a role picker it passes the pick through here.
None,
role,
DatatableAccess::Authed(authed.to_authed_ref()),
)
.await?;
@@ -2527,7 +2612,7 @@ async fn get_datatable_schema(
w_id: &str,
datatable_name: &str,
) -> Result<SchemaMap> {
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name, None).await?;
// Connect to the datatable database
let (client, connection) = pg_db.connect(Some(db)).await?;
@@ -2615,13 +2700,20 @@ async fn get_datatable_schema(
Ok(schema_map)
}
struct DatatableTableListing {
schemas: TableListMap,
can_create_schema: bool,
creatable_schemas: Vec<String>,
}
async fn get_datatable_tables(
db: &DB,
authed: &ApiAuthed,
w_id: &str,
datatable_name: &str,
) -> Result<TableListMap> {
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
role: Option<&str>,
) -> Result<DatatableTableListing> {
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name, role).await?;
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
@@ -2633,7 +2725,7 @@ async fn get_datatable_tables(
let schema_rows = client
.query(
r#"
SELECT nspname::text AS schema_name
SELECT nspname::text AS schema_name, has_schema_privilege(oid, 'CREATE') AS can_create
FROM pg_namespace
WHERE nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')
AND nspname NOT LIKE 'pg_%'
@@ -2647,11 +2739,29 @@ async fn get_datatable_tables(
Error::internal_err(format!("Failed to query schemas: {}", pg_error_message(&e)))
})?;
let can_create_schema: bool = client
.query_one(
"SELECT has_database_privilege(current_database(), 'CREATE')",
&[],
)
.await
.map_err(|e| {
Error::internal_err(format!(
"Failed to read database privileges: {}",
pg_error_message(&e)
))
})?
.get(0);
let mut table_map: TableListMap = HashMap::new();
let mut creatable_schemas = Vec::new();
let schema_names: Vec<String> = schema_rows
.iter()
.map(|row| {
let name: String = row.get(0);
if row.get::<_, bool>(1) {
creatable_schemas.push(name.clone());
}
table_map.entry(name.clone()).or_default();
name
})
@@ -2681,7 +2791,7 @@ async fn get_datatable_tables(
table_map.entry(table_schema).or_default().push(table_name);
}
Ok(table_map)
Ok(DatatableTableListing { schemas: table_map, can_create_schema, creatable_schemas })
}
async fn get_datatable_table_columns(
@@ -2691,6 +2801,7 @@ async fn get_datatable_table_columns(
datatable_name: &str,
schema_name: &str,
table_name: &str,
role: Option<&str>,
) -> Result<ColumnMap> {
if is_system_pg_schema(schema_name) {
return Err(Error::BadRequest(format!(
@@ -2699,7 +2810,7 @@ async fn get_datatable_table_columns(
)));
}
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name).await?;
let pg_db = resolve_datatable_pg_as_caller(db, authed, w_id, datatable_name, role).await?;
let (client, connection) = pg_db.connect(Some(db)).await?;
tokio::spawn(async move {
@@ -3326,8 +3437,27 @@ async fn create_pg_database(
}
if is_instance_datatable_source(&db, &w_id, &req.source).await? {
windmill_common::create_custom_instance_database(&db, &req.target_dbname, "datatable")
.await?;
// Held until the copy is registered, as a rename migrates reservations to the new id under
// it once the old one is archived: a copy registered after that would be reserved for an
// id nothing answers on.
let mut tx = db.begin().await?;
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
let live = sqlx::query_scalar::<_, bool>("SELECT NOT deleted FROM workspace WHERE id = $1")
.bind(&w_id)
.fetch_optional(&mut *tx)
.await?
.unwrap_or(false);
if !live {
return Err(Error::BadRequest(format!("Workspace '{w_id}' is archived")));
}
windmill_common::create_custom_instance_database(
&db,
&req.target_dbname,
"datatable",
Some(&w_id),
)
.await?;
tx.commit().await?;
} else {
let source_pg =
resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
@@ -3469,6 +3599,7 @@ async fn import_pg_database(
}
let schema_only = req.fork_behavior == DataTableForkBehavior::SchemaOnly;
let mut fork_lock: Option<Transaction<'_, Postgres>> = None;
let source_pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let mut target_pg =
resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.target).await?;
@@ -3481,6 +3612,21 @@ async fn import_pg_database(
.to_string(),
));
}
if is_instance_datatable_source(&db, &w_id, &req.target).await? {
// Held until the restore is done: fork finalization takes the first, and every
// save newly naming a database, in any workspace, the second. Nothing may start
// using this database while `psql` is still filling it.
let mut tx = db.begin().await?;
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut tx,
[override_dbname.as_str()],
)
.await?;
windmill_common::ensure_fork_database_available_to(&db, override_dbname, &w_id)
.await?;
fork_lock = Some(tx);
}
}
target_pg.dbname = override_dbname.clone();
}
@@ -3499,6 +3645,9 @@ async fn import_pg_database(
)
.await?;
pg_import_dump(&target_pg, &dump_file).await?;
if let Some(tx) = fork_lock {
tx.commit().await?;
}
Ok(format!(
"Imported from '{}' into '{}'",
@@ -3590,20 +3739,38 @@ async fn edit_ducklake_config(
)
.await?;
let old_ducklakes = sqlx::query_scalar!(
r#"
SELECT ws.ducklake->'ducklakes' AS ducklake_name
FROM workspace_settings ws
WHERE ws.workspace_id = $1
"#,
&w_id
// Under the row lock the save writes with, taken before the database locks below as fork
// cleanup takes the two.
let old_ducklakes = sqlx::query_scalar::<_, Option<serde_json::Value>>(
"SELECT ws.ducklake->'ducklakes' FROM workspace_settings ws
WHERE ws.workspace_id = $1 FOR UPDATE",
)
.bind(&w_id)
.fetch_one(&mut *tx)
.await?
.unwrap_or(serde_json::Value::Null);
let old_ducklakes: HashMap<String, Ducklake> =
serde_json::from_value(old_ducklakes).unwrap_or_default();
// Fork cleanup decides nothing uses an instance database under this lock, so a catalog newly
// put on one must not commit between its check and its drop.
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut *tx,
new_config
.settings
.ducklakes
.iter()
.filter(|(name, dl)| {
dl.catalog.resource_type == DucklakeCatalogResourceType::Instance
&& old_ducklakes.get(name.as_str()).is_none_or(|old| {
old.catalog.resource_type != DucklakeCatalogResourceType::Instance
|| old.catalog.resource_path != dl.catalog.resource_path
})
})
.map(|(_, dl)| dl.catalog.resource_path.as_str()),
)
.await?;
// Check that non-superadmins are not abusing Instance databases
if !is_superadmin {
for (name, dl) in new_config.settings.ducklakes.iter() {
@@ -3678,6 +3845,8 @@ async fn edit_datatable_config(
let is_superadmin = require_super_admin(&db, &authed).await.is_ok();
let mut tx = db.begin().await?;
// Ahead of the settings row, as fork cleanup of this workspace takes the two.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
// Read under the row lock this transaction will write with. `permissions`, `reference` and
// `forked_from` are carried across from what this read returns, so a permissions save
@@ -3910,10 +4079,38 @@ async fn edit_datatable_config(
})
.collect();
// Another workspace turning roles on for the same database holds only its own settings row, so
// without this the scan below could read past its uncommitted write.
// without this the scan below could read past its uncommitted write. Every managed database
// this save newly names is locked, not just the ones the scan is about: fork cleanup takes the
// same lock to decide nothing uses the database it is dropping.
let newly_named: std::collections::BTreeSet<&str> = new_config
.settings
.datatables
.iter()
.filter_map(|(name, dt)| {
let db = dt
.database
.as_ref()
.filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
let lookup = rename_src
.get(name.as_str())
.copied()
.unwrap_or(name.as_str());
old_datatables
.get(lookup)
.and_then(|old| old.database.as_ref())
.is_none_or(|old_db| {
old_db.resource_type != db.resource_type
|| old_db.resource_path != db.resource_path
})
.then_some(db.resource_path.as_str())
})
.collect();
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut *tx,
newly_pointed.iter().map(|(_, dbname)| *dbname),
newly_pointed
.iter()
.map(|(_, dbname)| *dbname)
.chain(newly_named.iter().copied()),
)
.await?;
let governed_elsewhere: Vec<String> = if newly_pointed.is_empty() {
@@ -8087,6 +8284,21 @@ async fn apply_forked_datatable(
})?,
};
if database.resource_type == DataTableCatalogResourceType::Instance {
// Held until the fork commits, as every save newly naming a database takes it: none may
// claim the copy between the check below and this fork's entry landing on it.
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut **tx,
[fdt.new_dbname.as_str()],
)
.await?;
}
if database.resource_type == DataTableCatalogResourceType::Instance
&& !windmill_api_auth::is_super_admin_authed(db, authed).await?
{
windmill_common::ensure_fork_database_available_to(db, &fdt.new_dbname, parent_w_id)
.await?;
}
if database.resource_type == DataTableCatalogResourceType::Instance {
// The whole `database` object, not just its `resource_path`: a pointer entry has none to
// patch. `reference` goes with it — exactly one of the two may be set.
@@ -8480,6 +8692,9 @@ async fn create_workspace_fork(
}
let mut tx: Transaction<'_, Postgres> = db.begin().await?;
// Before the settings clone reads the parent's data tables: a pointer this fork ends up with
// must not be written after cleanup of the parent decided that nothing points at its copies.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &parent_workspace_id).await?;
if nw.is_dev_workspace {
// The checks above ran outside a transaction, so the parent's eligibility and the chain's
@@ -56,6 +56,11 @@ pub(crate) async fn change_workspace_id(
let mut tx = db.begin().await?;
// The settings copy below carries every data table entry to the new id, which fork cleanup of
// the old id cannot see until this commits: without the lock it could drop a copy the renamed
// workspace goes on using. Before the pairing lock, as forking takes the two in that order.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
// A rename rewrites the workspace's dev flag and reparents its children, so it decides on the
// same state the pairing handlers do: without this lock a concurrent create/attach could commit
// an active dev workspace under the shell this rename is about to archive. Both ids, since the
@@ -850,6 +855,10 @@ pub(crate) async fn change_workspace_id(
}
}
// After every workspace_settings write above: fork cleanup locks a settings row before the
// registry, so taking the registry first here would deadlock with it.
migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
// Audit log in the same transaction as the workspace changes
audit_log(
&mut *tx,
@@ -918,6 +927,14 @@ pub(crate) async fn change_workspace_id(
let (_schedules_count, canceled_count, _deleted_tokens_count) =
archive_workspace_impl(&db, &old_id, &authed.username, None).await?;
// The old id stays live between the commit above and the archive, and a fork copy created for
// it in that window registers under it. Creation checks the workspace is live under the fork
// lock, so once this has run under it, no copy can be reserved for the old id any more.
let mut tx = db.begin().await?;
windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
tx.commit().await?;
info!(
"Workspace id change completed: moved {} to {}, archived old workspace",
old_id, rw.new_id
@@ -929,6 +946,28 @@ pub(crate) async fn change_workspace_id(
))
}
/// A fork copy reserved for the old id would otherwise be unreachable: its creator cannot import
/// into it or finish its fork under the new id, and nothing else would ever drop it.
async fn migrate_fork_reservations(
tx: &mut Transaction<'_, Postgres>,
old_id: &str,
new_id: &str,
) -> Result<()> {
sqlx::query(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (
SELECT COALESCE(jsonb_object_agg(k, CASE WHEN v->>'workspace_id' = $1
THEN jsonb_set(v, '{workspace_id}', to_jsonb($2::text)) ELSE v END), '{}'::jsonb)
FROM jsonb_each(COALESCE(value->'databases', '{}'::jsonb)) AS e(k, v)
))
WHERE name = 'custom_instance_pg_databases'"#,
)
.bind(old_id)
.bind(new_id)
.execute(&mut **tx)
.await?;
Ok(())
}
#[derive(Deserialize)]
pub(crate) struct DeleteWorkspaceQuery {
pub(crate) only_delete_forks: Option<bool>,
@@ -1420,7 +1459,91 @@ pub async fn drop_forked_datatable_databases(
));
continue;
}
if let Err(e) = windmill_common::drop_custom_instance_database(&db, db_to_drop).await {
// The fork's own entry is what is going away; anything else still reaching the copy,
// a child fork's pointer at this entry included, keeps it. The lock keeps a child fork
// from gaining such a pointer before the drop.
// A task of its own, so a client going away cannot stop it between dropping the
// database and committing the entry's removal.
let dropped = tokio::spawn({
let (db, w_id, dt_name, db_to_drop) = (
db.clone(),
w_id.clone(),
dt_name.clone(),
db_to_drop.clone(),
);
let resource_type = database.resource_type;
async move {
let mut tx = db.begin().await?;
// The three locks a settings save takes, in its order: this workspace's data
// tables, its settings row, and the database itself. Without them a save could
// rename this entry, or point another one here, either side of the check below.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
// The snapshot above was read unlocked: a save committing since could have
// repointed this entry, and the entry is removed below whatever it names by then.
let current = sqlx::query_scalar::<_, Option<serde_json::Value>>(
"SELECT datatable->'datatables'->$2 FROM workspace_settings
WHERE workspace_id = $1 FOR UPDATE",
)
.bind(&w_id)
.bind(&dt_name)
.fetch_optional(&mut *tx)
.await?
.flatten()
.and_then(|v| serde_json::from_value::<DataTable>(v).ok());
if !current.is_some_and(|dt| {
dt.forked_from.is_some()
&& dt.database.is_some_and(|d| {
d.resource_type == resource_type && d.resource_path == db_to_drop
})
}) {
return Err(Error::BadRequest(
"the data table changed while it was being cleaned up".to_string(),
));
}
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut tx,
[db_to_drop.as_str()],
)
.await?;
let uses = windmill_common::workspaces::managed_database_uses(
&mut tx,
windmill_common::workspaces::DataTableCatalogResourceType::Instance,
&db_to_drop,
Some((w_id.as_str(), dt_name.as_str())),
)
.await?;
if !uses.is_empty() {
return Err(Error::BadRequest(format!(
"it is still used by {}",
uses.join(", ")
)));
}
// The entry goes with the database: a fork this one is cloned into afterwards must
// not inherit a pointer at a data table whose database is gone.
sqlx::query(
"UPDATE workspace_settings SET datatable = datatable #- ARRAY['datatables', $2]
WHERE workspace_id = $1",
)
.bind(&w_id)
.bind(&dt_name)
.execute(&mut *tx)
.await?;
windmill_common::drop_custom_instance_database_keep_entry(&db, &db_to_drop)
.await?;
sqlx::query(
"UPDATE global_settings SET value = value #- ARRAY['databases', $1]
WHERE name = 'custom_instance_pg_databases'",
)
.bind(&db_to_drop)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok::<_, Error>(())
}
})
.await
.unwrap_or_else(|e| Err(Error::internal_err(format!("cleanup task failed: {e}"))));
if let Err(e) = dropped {
errors.push(format!(
"Could not drop instance database '{}' for datatable://{}: {}",
db_to_drop, dt_name, e
+469 -1
View File
@@ -1572,6 +1572,43 @@ paths:
schema:
type: object
/settings/external_instance_pg/status:
get:
summary: Returns whether the external instance cluster is configured and how its last setup went
operationId: getExternalInstancePgStatus
tags:
- setting
responses:
"200":
description: external instance cluster status
content:
application/json:
schema:
$ref: "#/components/schemas/ExternalInstancePgStatus"
/settings/external_instance_pg/setup:
post:
summary: Sets up the external instance cluster with its saved admin login, optionally rotating the passwords Windmill manages on it (enterprise edition only)
operationId: setupExternalInstancePg
tags:
- setting
requestBody:
required: true
content:
application/json:
schema:
type: object
properties:
rotate_passwords:
type: boolean
responses:
"200":
description: the setup report, also stored as the last setup
content:
application/json:
schema:
$ref: "#/components/schemas/ExternalInstancePgSetupReport"
/settings/list_custom_instance_pg_databases:
post:
summary: Returns the set-up statuses of custom instance pg databases
@@ -5294,6 +5331,97 @@ paths:
schema:
type: string
/w/{workspace}/workspaces/datatable_acl/{datatable_name}:
get:
summary: read the owner and grants of an instance data table's database, schema or table
operationId: getDatatableAcl
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
- name: kind
in: query
required: true
schema:
type: string
enum: [database, schema, table]
- name: schema
in: query
required: false
schema:
type: string
- name: table
in: query
required: false
schema:
type: string
responses:
"200":
description: owner and grants
content:
application/json:
schema:
$ref: "#/components/schemas/DatatableAclInfo"
/w/{workspace}/workspaces/datatable_acl/{datatable_name}/plan:
post:
summary: preview the SQL an ownership or grant change would run (data table administrators only)
operationId: planDatatableAcl
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AclChangeRequest"
responses:
"200":
description: statements that would run, in a single transaction
content:
application/json:
schema:
$ref: "#/components/schemas/AclPlan"
/w/{workspace}/workspaces/datatable_acl/{datatable_name}/apply:
post:
summary: run an ownership or grant change exactly as planned (data table administrators only)
operationId: applyDatatableAcl
tags:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: path
required: true
schema:
type: string
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/AclChangeRequest"
responses:
"200":
description: change applied
content:
text/plain:
schema:
type: string
/w/{workspace}/workspaces/datatable_usable_roles/{datatable_name}:
get:
summary: list the data table roles the caller may connect as
@@ -5397,6 +5525,21 @@ paths:
- workspace
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: datatable_name
in: query
description: list only this data table; each listed data table opens a connection to its database
schema:
type: string
- name: role_for
in: query
description: the data table `role` applies to; every other one is listed as its default role
schema:
type: string
- name: role
in: query
description: the role to list `role_for` as; refused, in that entry's `error`, if the caller may not use it
schema:
type: string
responses:
"200":
description: table metadata of all datatables
@@ -5430,6 +5573,11 @@ paths:
required: true
schema:
type: string
- name: role
in: query
description: the data table role to read the table as; defaults to the data table's default role
schema:
type: string
responses:
"200":
description: schema of one datatable table
@@ -33589,6 +33737,44 @@ components:
- ducklake
- datatable
ExternalInstancePgSetupStep:
type: object
required: [name, status, message]
properties:
name:
type: string
status:
type: string
enum: [ok, warning, error]
message:
type: string
ExternalInstancePgSetupReport:
type: object
required: [success, finished_at, steps]
properties:
success:
type: boolean
description: no step failed; warnings leave it true
finished_at:
type: string
format: date-time
steps:
type: array
items:
$ref: "#/components/schemas/ExternalInstancePgSetupStep"
ExternalInstancePgStatus:
type: object
required: [configured, database_count]
properties:
configured:
type: boolean
database_count:
type: integer
last_setup:
$ref: "#/components/schemas/ExternalInstancePgSetupReport"
InstanceDatatableRole:
type: object
required: [id, name, enabled]
@@ -33649,6 +33835,255 @@ components:
datatable:
type: string
AclTarget:
description: what access is read or changed on
oneOf:
- $ref: "#/components/schemas/AclTargetDatabase"
- $ref: "#/components/schemas/AclTargetSchema"
- $ref: "#/components/schemas/AclTargetTable"
discriminator:
propertyName: kind
mapping:
database: "#/components/schemas/AclTargetDatabase"
schema: "#/components/schemas/AclTargetSchema"
table: "#/components/schemas/AclTargetTable"
AclTargetDatabase:
type: object
required: [kind]
properties:
kind:
type: string
enum: [database]
AclTargetSchema:
type: object
required: [kind, schema]
properties:
kind:
type: string
enum: [schema]
schema:
type: string
AclTargetTable:
type: object
required: [kind, schema, table]
properties:
kind:
type: string
enum: [table]
schema:
type: string
table:
type: string
AclChange:
description: one change to plan or apply
oneOf:
- $ref: "#/components/schemas/AclChangeSetOwner"
- $ref: "#/components/schemas/AclChangeGrant"
- $ref: "#/components/schemas/AclChangeRevoke"
discriminator:
propertyName: type
mapping:
set_owner: "#/components/schemas/AclChangeSetOwner"
grant: "#/components/schemas/AclChangeGrant"
revoke: "#/components/schemas/AclChangeRevoke"
AclChangeSetOwner:
type: object
description: >-
hands the target to role — for a schema, with everything already in it but an extension's
members, which stay with the extension
required: [type, role]
properties:
type:
type: string
enum: [set_owner]
role:
type: string
description: a data table role of the instance, or admin
AclChangeGrant:
type: object
required: [type, role, privileges, scope]
properties:
type:
type: string
enum: [grant]
role:
type: string
description: a data table role of the instance, or admin
privileges:
type: array
items:
type: string
scope:
$ref: "#/components/schemas/AclGrantScope"
AclChangeRevoke:
type: object
required: [type, role, privileges, scope]
properties:
type:
type: string
enum: [revoke]
role:
type: string
description: a data table role of the instance, other than admin
privileges:
type: array
items:
type: string
scope:
$ref: "#/components/schemas/AclGrantScope"
objects:
type: array
description: >-
objects inside the target the revoke covers, empty for the target itself. Only with the
target scope; a revoke on all objects of a kind is refused, since it cannot say which
grants it takes back.
items:
$ref: "#/components/schemas/AclObject"
AclGrantScope:
type: string
enum:
[
target,
all_tables,
all_sequences,
all_functions,
future_tables,
future_sequences,
future_functions,
]
AclChangeRequest:
type: object
required: [target, change]
properties:
target:
$ref: "#/components/schemas/AclTarget"
change:
$ref: "#/components/schemas/AclChange"
statements:
type: array
description: >-
The statements the plan showed. Required to apply, which plans again and refuses if
the result differs.
items:
type: string
AclPlan:
type: object
required: [statements, warnings]
properties:
statements:
type: array
items:
type: string
warnings:
type: array
items:
type: string
AclObject:
type: object
required: [name, kind]
properties:
name:
type: string
kind:
type: string
description: >-
TABLE, SEQUENCE, FUNCTION, PROCEDURE or TYPE — what the object is. A revoke turns it
into the keyword it takes, ROUTINE for both routine kinds; a type's grants are read
only.
args:
type: string
description: identity arguments of a routine, which is what tells two of the same name apart
AclGrant:
type: object
required: [grantee, privileges, sources]
properties:
grantee:
type: string
privileges:
type: array
items:
type: string
object:
$ref: "#/components/schemas/AclObject"
future:
type: string
description: >-
set for a default privilege, naming the kind of object it covers (TABLES, SEQUENCES,
FUNCTIONS, TYPES, or SCHEMAS). On a schema, the defaults set in that schema; on the
database, the ones set database-wide, which apply in every schema and which no schema's
own defaults take back.
sources:
type: array
description: >-
the roles the grant comes from, each once — who granted it, or for a default privilege
the role whose future objects it covers. A revoke of some of the grant's privileges
takes them back from every source that gave them.
items:
$ref: "#/components/schemas/AclSource"
AclSource:
type: object
required: [role, privileges, reachable]
properties:
role:
type: string
privileges:
type: array
description: >-
what role gave of the grant's privileges. A revoke is held back only by a source out of
reach that gave some of what it takes back.
items:
type: string
reachable:
type: boolean
description: >-
whether the data table's connection can take back what role gave. On an object that is
the owner, when the connection acts for the owner, and otherwise the connection itself;
for a default privilege, a creating role the connection acts for. What a source out of
reach gave is not revocable from here; privileges only other sources gave still are.
DatatableAclInfo:
type: object
required: [owner, roles, editable, supports_maintain, dbname, grants, children]
properties:
owner:
type: string
roles:
type: array
description: the roles a change may name; empty unless the caller may change anything
items:
type: string
editable:
type: boolean
description: whether the caller may plan and apply changes
supports_maintain:
type: boolean
description: whether the server is Postgres 17+, which added the MAINTAIN table privilege
dbname:
type: string
description: the database the target lives in
grants:
type: array
items:
$ref: "#/components/schemas/AclGrant"
children:
type: array
description: a database's schemas, or a schema's tables
items:
type: string
CustomInstanceDb:
type: object
required:
@@ -33673,6 +34108,9 @@ components:
items:
type: string
description: Workspaces that reference this database via a ducklake catalog or datatable database with resource_type 'instance'. Computed at request time, not persisted.
workspace_id:
type: string
description: The workspace a member created this database for as a fork copy. Only that workspace can import into it or point a fork at it.
NewSqsTrigger:
type: object
@@ -35792,7 +36230,17 @@ components:
DataTableTables:
type: object
required: [datatable_name, schemas]
required:
[
datatable_name,
schemas,
instance,
permissioned,
usable_roles,
default_role,
can_create_schema,
creatable_schemas,
]
properties:
datatable_name:
type: string
@@ -35805,6 +36253,26 @@ components:
type: string
error:
type: string
instance:
type: boolean
description: on the instance database, the only kind that can be under roles or have its access edited
permissioned:
type: boolean
usable_roles:
type: array
description: the roles the caller may connect as, by name; empty when not under roles
items:
type: string
default_role:
type: string
can_create_schema:
type: boolean
description: whether the role the listing connected as may create schemas
creatable_schemas:
type: array
description: the schemas the role the listing connected as may create in
items:
type: string
DataTableTableSchema:
type: object
+6 -1
View File
@@ -8723,7 +8723,12 @@ fn register_potential_assets_on_inline_execution(
.as_ref()
.and_then(|args| args.get("database"))
.map(|v| v.get().trim_matches('"'))
.and_then(|dt| dt.strip_prefix("datatable://"));
.and_then(|dt| dt.strip_prefix("datatable://"))
// `?role=` picks the connection, not the data table. Anything else after a `?` may be
// part of a name stored before names were restricted, so it stays.
.map(|dt| {
windmill_common::workspaces::parse_datatable_ref(dt).map_or(dt, |(name, _)| name)
});
if let Some(datatable) = datatable {
let re = regex::Regex::new(r#"SET search_path TO "([^"]+)";"#).unwrap();
let (schema, content) = if let Some(captures) = re.captures(&preview.content) {
+1
View File
@@ -76,6 +76,7 @@ bitflags.workspace = true
once_cell.workspace = true
phf.workspace = true
tokio-postgres.workspace = true
postgres-protocol.workspace = true
postgres-native-tls.workspace = true
native-tls.workspace = true
@@ -15,7 +15,8 @@
use crate::error::Error;
/// What every roles path answers without the Enterprise Edition.
/// What every roles path answers without the Enterprise Edition. The frontend matches this exact
/// sentence (`datatableUsableRoles.ts`) to read the refusal as "not under roles": reword both.
pub fn datatable_roles_unavailable() -> Error {
Error::BadRequest("Data table roles are a Windmill Enterprise Edition feature".to_string())
}
@@ -0,0 +1,210 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! The external Postgres cluster behind `external_instance` data tables and Ducklake catalogs.
//!
//! Windmill administers that cluster itself, logged in as the user in
//! [`EXTERNAL_INSTANCE_PG_SETTING`]. It creates `custom_instance_user` and
//! `custom_instance_replication_user` there, with passwords it generates and keeps in
//! [`EXTERNAL_INSTANCE_PG_STATE_SETTING`]. They share their names with the roles on Windmill's own
//! cluster, but they are different roles with different passwords.
//!
//! The cluster may hold data Windmill did not create. Two Windmill instances sharing one is not
//! supported: each would keep resetting the passwords the other depends on.
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use crate::{
error::{Error, Result},
global_settings::{EXTERNAL_INSTANCE_PG_SETTING, EXTERNAL_INSTANCE_PG_STATE_SETTING},
instance_config::{CustomInstanceDb, ExternalInstancePg},
DB,
};
/// What Windmill keeps about the external cluster. Server-managed and hidden: never part of the
/// instance config, never readable by an agent worker. No `Debug`: it carries live passwords.
#[derive(Serialize, Deserialize, Clone, Default)]
pub struct ExternalInstancePgState {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub user_pwd: Option<String>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub replication_pwd: Option<String>,
/// The databases Windmill created on the cluster. It only ever drops one of these.
#[serde(default)]
pub databases: BTreeMap<String, CustomInstanceDb>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub last_setup: Option<ExternalInstancePgSetupReport>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ExternalInstancePgSetupReport {
/// No step failed. Warnings leave it true.
pub success: bool,
pub finished_at: chrono::DateTime<chrono::Utc>,
pub steps: Vec<ExternalInstancePgSetupStep>,
}
#[derive(Serialize, Deserialize, Clone, Debug)]
pub struct ExternalInstancePgSetupStep {
pub name: String,
pub status: SetupStepStatus,
pub message: String,
}
#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum SetupStepStatus {
Ok,
Warning,
Error,
}
/// The status the settings page shows without running anything.
#[derive(Serialize, Debug)]
pub struct ExternalInstancePgStatus {
pub configured: bool,
pub database_count: usize,
#[serde(skip_serializing_if = "Option::is_none")]
pub last_setup: Option<ExternalInstancePgSetupReport>,
}
/// Authorization: returns the cluster's admin password and checks nothing. Callers MUST be
/// superadmin or an internal server path.
pub(crate) async fn read_external_instance_pg_config<'c>(
executor: impl sqlx::PgExecutor<'c>,
) -> Result<Option<ExternalInstancePg>> {
let value = sqlx::query_scalar!(
"SELECT value FROM global_settings WHERE name = $1",
EXTERNAL_INSTANCE_PG_SETTING
)
.fetch_optional(executor)
.await?;
value
.map(|v| {
serde_json::from_value(v).map_err(|e| {
Error::internal_err(format!("reading {EXTERNAL_INSTANCE_PG_SETTING}: {e}"))
})
})
.transpose()
}
/// Authorization: returns the passwords Windmill generated on the cluster and checks nothing.
/// Callers MUST be superadmin or an internal server path.
pub(crate) async fn read_external_instance_pg_state<'c>(
executor: impl sqlx::PgExecutor<'c>,
) -> Result<ExternalInstancePgState> {
let value = sqlx::query_scalar!(
"SELECT value FROM global_settings WHERE name = $1",
EXTERNAL_INSTANCE_PG_STATE_SETTING
)
.fetch_optional(executor)
.await?;
match value {
None => Ok(ExternalInstancePgState::default()),
Some(v) => serde_json::from_value(v).map_err(|e| {
Error::internal_err(format!("reading {EXTERNAL_INSTANCE_PG_STATE_SETTING}: {e}"))
}),
}
}
/// Authorization: reads the hidden cluster state and checks nothing. Callers MUST be superadmin.
pub async fn external_instance_pg_status(db: &DB) -> Result<ExternalInstancePgStatus> {
let configured = read_external_instance_pg_config(db).await?.is_some();
let state = read_external_instance_pg_state(db).await?;
Ok(ExternalInstancePgStatus {
configured,
database_count: state.databases.len(),
last_setup: state.last_setup,
})
}
/// Refuse to unset the cluster while Windmill still has databases on it: every data table and
/// Ducklake catalog there would stop resolving. Allowed on every edition, so a downgraded
/// instance can still clear a setting it no longer uses.
async fn ensure_external_instance_pg_removable(db: &DB) -> Result<()> {
let state = read_external_instance_pg_state(db).await?;
if state.databases.is_empty() {
return Ok(());
}
let names = state
.databases
.keys()
.cloned()
.collect::<Vec<_>>()
.join(", ");
Err(Error::BadRequest(format!(
"The external instance cluster still holds databases Windmill created ({names}). Drop \
them before removing {EXTERNAL_INSTANCE_PG_SETTING}."
)))
}
/// Check a write to [`EXTERNAL_INSTANCE_PG_SETTING`] before it happens: `None`, null or an empty
/// string unsets it. Every writer of global settings calls this, the per-key and bulk endpoints
/// as well as the declarative sync.
///
/// Authorization: reads the hidden cluster state and names the databases on the cluster in its
/// refusal. Callers MUST be superadmin or an internal server path writing global settings.
pub async fn check_external_instance_pg_write(
db: &DB,
value: Option<&serde_json::Value>,
) -> Result<()> {
match value {
None | Some(serde_json::Value::Null) => ensure_external_instance_pg_removable(db).await,
Some(serde_json::Value::String(s)) if s.trim().is_empty() => {
ensure_external_instance_pg_removable(db).await
}
Some(value) => {
crate::external_instance_pg_oss::validate_external_instance_pg_setting(value)?;
ensure_same_cluster_while_in_use(db, value).await
}
}
}
/// Pointing the setting at another cluster strands the databases Windmill created on this one as
/// surely as unsetting it, and would hand their registry and role passwords to a cluster that has
/// neither. The login, TLS and maintenance database may change; the host and port may not.
async fn ensure_same_cluster_while_in_use(db: &DB, value: &serde_json::Value) -> Result<()> {
let state = read_external_instance_pg_state(db).await?;
if state.databases.is_empty() {
return Ok(());
}
let identity = |c: &ExternalInstancePg| (c.host.trim().to_lowercase(), c.port.unwrap_or(5432));
let new: ExternalInstancePg = serde_json::from_value(value.clone())
.map_err(|e| Error::BadRequest(format!("{EXTERNAL_INSTANCE_PG_SETTING}: {e}")))?;
let current = read_external_instance_pg_config(db).await?;
if current.is_some_and(|current| identity(&current) == identity(&new)) {
return Ok(());
}
Err(Error::BadRequest(format!(
"The external instance cluster still holds databases Windmill created ({}). Drop them \
before pointing {EXTERNAL_INSTANCE_PG_SETTING} at another host or port.",
state
.databases
.keys()
.cloned()
.collect::<Vec<_>>()
.join(", ")
)))
}
/// Converge the external cluster on the configured login: check what it can do, create or update
/// Windmill's two roles with the stored passwords, and report anything that would get in the way.
/// With `rotate_passwords`, generate new passwords first. Safe to run again; running it again is
/// how a failed rotation is repaired.
///
/// Authorization: administers the external cluster with its admin credentials and checks nothing.
/// Callers MUST be superadmin.
pub async fn setup_external_instance_pg_unchecked(
db: &DB,
rotate_passwords: bool,
) -> Result<ExternalInstancePgSetupReport> {
crate::external_instance_pg_oss::setup_external_instance_pg_unchecked(db, rotate_passwords)
.await
}
@@ -0,0 +1,43 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! Where the external instance cluster comes from: the enterprise implementation, or a refusal.
//! `private` alone is not that edition: community builds carry it.
use crate::error::Error;
pub fn external_instance_pg_unavailable() -> Error {
Error::BadRequest(
"External instance databases are a Windmill Enterprise Edition feature".to_string(),
)
}
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::external_instance_pg_ee::{
setup_external_instance_pg_unchecked, validate_external_instance_pg_setting,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
pub(crate) use ce::*;
#[cfg(not(all(feature = "private", feature = "enterprise")))]
mod ce {
use super::external_instance_pg_unavailable as unavailable;
use crate::{error::Result, external_instance_pg::ExternalInstancePgSetupReport, DB};
pub(crate) fn validate_external_instance_pg_setting(_value: &serde_json::Value) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn setup_external_instance_pg_unchecked(
_db: &DB,
_rotate_passwords: bool,
) -> Result<ExternalInstancePgSetupReport> {
Err(unavailable())
}
}
@@ -57,6 +57,8 @@ pub const SAML_METADATA_SETTING: &str = "saml_metadata";
pub const SMTP_SETTING: &str = "smtp_settings";
pub const TEAMS_SETTING: &str = "teams";
pub const INDEXER_SETTING: &str = "indexer_settings";
pub const EXTERNAL_INSTANCE_PG_SETTING: &str = "external_instance_pg";
pub const EXTERNAL_INSTANCE_PG_STATE_SETTING: &str = "external_instance_pg_state";
pub const TIMEOUT_WAIT_RESULT_SETTING: &str = "timeout_wait_result";
pub const UNIQUE_ID_SETTING: &str = "uid";
@@ -355,6 +357,9 @@ pub const AGENT_WORKER_BLOCKED_SETTINGS: &[&str] = &[
// resolve datatable connections through the dedicated datatable endpoints, never these.
"custom_instance_pg_databases",
"custom_instance_replication_pwd",
// The external cluster's admin login, and the passwords Windmill generated on it.
EXTERNAL_INSTANCE_PG_SETTING,
EXTERNAL_INSTANCE_PG_STATE_SETTING,
];
/// Whether an agent worker may read the given global setting over HTTP.
+76 -31
View File
@@ -350,6 +350,8 @@ pub struct GlobalSettings {
pub ducklake_settings: Option<DucklakeSettings>,
#[serde(skip_serializing_if = "Option::is_none")]
pub custom_instance_pg_databases: Option<CustomInstancePgDatabases>,
#[serde(skip_serializing_if = "Option::is_none")]
pub external_instance_pg: Option<ExternalInstancePg>,
// Opaque settings (EE-private structs or no clear schema)
#[serde(skip_serializing_if = "Option::is_none")]
@@ -809,6 +811,9 @@ pub struct CustomInstanceDb {
pub error: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub tag: Option<String>,
/// The workspace a member created this fork copy for. Absent when a superadmin created it.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub workspace_id: Option<String>,
}
/// Setup log entries for a custom instance database.
@@ -833,6 +838,36 @@ pub struct CustomInstanceDbLogs {
pub replication_user_error: Option<String>,
}
// ---------------------------------------------------------------------------
// External instance PG cluster
// ---------------------------------------------------------------------------
/// The external Postgres cluster Windmill manages for `external_instance` data tables and Ducklake
/// catalogs. `user` logs in as the cluster's administrator: it needs `CREATEDB` and `CREATEROLE`.
/// `dbname` is only where that login connects to run cluster-wide statements.
///
/// Every field defaults rather than being required: this deserializes as part of the whole
/// instance config, and one malformed row must not make every other setting unreadable. The
/// write path and every use reject an incomplete value instead.
#[derive(Deserialize, Serialize, Clone, Debug, Default)]
#[cfg_attr(feature = "instance_config_schema", derive(schemars::JsonSchema))]
pub struct ExternalInstancePg {
#[serde(default)]
pub host: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub port: Option<u16>,
#[serde(default)]
pub user: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub password: Option<StringOrSecretRef>,
#[serde(skip_serializing_if = "Option::is_none")]
pub dbname: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub sslmode: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub root_certificate_pem: Option<String>,
}
// ---------------------------------------------------------------------------
// Autoscaling (worker config)
// ---------------------------------------------------------------------------
@@ -969,6 +1004,7 @@ pub const PROTECTED_SETTINGS: &[&str] = &[
"ducklake_settings",
"custom_instance_pg_databases",
"custom_instance_replication_pwd",
"external_instance_pg_state",
"uid",
"rsa_keys",
"jwt_secret",
@@ -994,6 +1030,8 @@ pub const HIDDEN_SETTINGS: &[&str] = &[
// Server-only (written by setup/refresh via direct SQL), never operator-authored —
// hidden so the config machinery can't read, rewrite, or drop it.
"custom_instance_replication_pwd",
// Same for the passwords and database registry Windmill keeps for the external cluster.
"external_instance_pg_state",
];
/// Top-level settings whose entire value is sensitive and must be fully redacted in logs.
@@ -1005,6 +1043,7 @@ const SENSITIVE_SETTINGS: &[&str] = &[
"license_key",
"ducklake_user_pg_pwd",
"custom_instance_replication_pwd",
"external_instance_pg_state",
"pip_index_url",
"pip_extra_index_url",
"npm_config_registry",
@@ -1030,6 +1069,7 @@ const NESTED_SENSITIVE_FIELDS: &[(&str, &[&str])] = &[
&["secret_key", "serviceAccountKey"],
),
("custom_instance_pg_databases", &["user_pwd"]),
("external_instance_pg", &["password"]),
];
fn redact_json_value(value: &serde_json::Value) -> serde_json::Value {
@@ -1359,6 +1399,13 @@ pub async fn sync_global_settings_declarative(
.map_err(|e| anyhow::anyhow!("{origins_key}: {e}"))?;
let diff = diff_global_settings(current, desired, ApplyMode::Replace);
let external_pg_key = crate::global_settings::EXTERNAL_INSTANCE_PG_SETTING;
if diff.deletes.iter().any(|k| k == external_pg_key) {
crate::external_instance_pg::check_external_instance_pg_write(db, None).await?;
}
if let Some(value) = diff.upserts.get(external_pg_key) {
crate::external_instance_pg::check_external_instance_pg_write(db, Some(value)).await?;
}
apply_settings_diff(db, &diff).await?;
Ok(())
@@ -1491,6 +1538,10 @@ pub fn resolve_env_refs(settings: &mut GlobalSettings) -> Result<(), String> {
resolve_env_option(&mut pg.user_pwd)?;
}
if let Some(pg) = &mut settings.external_instance_pg {
resolve_env_option(&mut pg.password)?;
}
Ok(())
}
@@ -2460,39 +2511,33 @@ mod tests {
}
#[test]
fn custom_instance_replication_pwd_is_isolated_from_config() {
// The replication-role password is server-only: written by setup/refresh via direct
// SQL, never operator-authored. It must stay out of the declarative config surface
// (hidden on read) and be undeletable, so config sync can't read, rewrite, or drop it.
assert!(HIDDEN_SETTINGS.contains(&"custom_instance_replication_pwd"));
assert!(PROTECTED_SETTINGS.contains(&"custom_instance_replication_pwd"));
assert!(SENSITIVE_SETTINGS.contains(&"custom_instance_replication_pwd"));
fn server_generated_db_passwords_are_isolated_from_config() {
// These hold passwords the server generates: written by setup/refresh via direct SQL,
// never operator-authored. They must stay out of the declarative config surface
// (hidden on read) and be undeletable, so config sync can't read, rewrite, or drop them.
for key in [
"custom_instance_replication_pwd",
"external_instance_pg_state",
] {
assert!(HIDDEN_SETTINGS.contains(&key), "{key}");
assert!(PROTECTED_SETTINGS.contains(&key), "{key}");
assert!(SENSITIVE_SETTINGS.contains(&key), "{key}");
// A stray desired value (e.g. flattened into `extra`) is ignored, not upserted.
let mut desired = BTreeMap::new();
desired.insert(
"custom_instance_replication_pwd".to_string(),
serde_json::json!("attacker-set"),
);
let diff = diff_global_settings(&BTreeMap::new(), &desired, ApplyMode::Merge);
assert!(
diff.upserts.is_empty(),
"hidden setting must not be upserted"
);
// A stray desired value (e.g. flattened into `extra`) is ignored, not upserted.
let mut desired = BTreeMap::new();
desired.insert(key.to_string(), serde_json::json!("attacker-set"));
let diff = diff_global_settings(&BTreeMap::new(), &desired, ApplyMode::Merge);
assert!(diff.upserts.is_empty(), "{key} must not be upserted");
// A current value is never deleted by a Replace that omits it.
let mut current = BTreeMap::new();
current.insert(
"custom_instance_replication_pwd".to_string(),
serde_json::json!("live"),
);
let diff = diff_global_settings(&current, &BTreeMap::new(), ApplyMode::Replace);
assert!(
!diff
.deletes
.contains(&"custom_instance_replication_pwd".to_string()),
"hidden setting must not be deleted"
);
// A current value is never deleted by a Replace that omits it.
let mut current = BTreeMap::new();
current.insert(key.to_string(), serde_json::json!("live"));
let diff = diff_global_settings(&current, &BTreeMap::new(), ApplyMode::Replace);
assert!(
!diff.deletes.contains(&key.to_string()),
"{key} must not be deleted"
);
}
}
#[test]
+84 -10
View File
@@ -58,6 +58,10 @@ pub mod ee_oss;
pub mod email_ee;
pub mod email_oss;
pub mod error;
pub mod external_instance_pg;
#[cfg(all(feature = "private", feature = "enterprise"))]
mod external_instance_pg_ee;
pub mod external_instance_pg_oss;
pub mod external_ip;
#[cfg(feature = "private")]
pub mod feature_usage_ee;
@@ -1024,6 +1028,20 @@ impl Future for TokioPgConnection {
}
}
impl TokioPgConnection {
/// Drive the connection and hand back what the server sends outside of a query's response —
/// notices above all, which driving it as a future silently discards.
pub fn poll_message(
&mut self,
cx: &mut core::task::Context<'_>,
) -> core::task::Poll<Option<Result<tokio_postgres::AsyncMessage, tokio_postgres::Error>>> {
match self {
TokioPgConnection::Tls(conn) => conn.poll_message(cx),
TokioPgConnection::NoTls(conn) => conn.poll_message(cx),
}
}
}
impl PgDatabase {
/// The role the connection logs in as, whichever way it authenticates.
pub fn login_name(&self) -> &str {
@@ -1461,7 +1479,26 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> {
}
/// Drop a custom instance database: validate, terminate connections, DROP DATABASE, remove from global_settings.
///
/// Authorization: drops any instance database but Windmill's own and checks nothing. Callers MUST
/// be superadmin, or have established the caller may drop this one — a fork's owner cleaning up
/// its own copy that nothing else uses.
pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Result<()> {
drop_custom_instance_database_keep_entry(db, dbname).await?;
sqlx::query!(
r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
dbname.trim()
)
.execute(db)
.await?;
Ok(())
}
/// [`drop_custom_instance_database`] leaving its registry entry, for a caller holding row locks in
/// a transaction: the registry write has to go through that transaction, as waiting on another
/// connection for a lock the transaction's own peers hold is a deadlock Postgres cannot see. Same
/// authorization contract.
pub async fn drop_custom_instance_database_keep_entry(db: &DB, dbname: &str) -> error::Result<()> {
let dbname = dbname.trim();
validate_dbname(dbname)?;
@@ -1507,14 +1544,6 @@ pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Resu
tracing::info!("Database '{}' does not exist, skipping drop", dbname);
}
// Always remove from global_settings
sqlx::query!(
r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
dbname
)
.execute(db)
.await?;
Ok(())
}
@@ -1554,11 +1583,13 @@ pub async fn ensure_instance_db_grant_options_unchecked(
}
/// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake").
/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake"). `for_workspace`
/// is the workspace a member creates a fork copy for; see [`ensure_fork_database_available_to`].
pub async fn create_custom_instance_database(
db: &DB,
dbname: &str,
tag: &str,
for_workspace: Option<&str>,
) -> error::Result<()> {
let dbname = dbname.trim();
validate_dbname(dbname)?;
@@ -1612,7 +1643,8 @@ pub async fn create_custom_instance_database(
},
"success": true,
"error": null,
"tag": tag
"tag": tag,
"workspace_id": for_workspace,
});
sqlx::query!(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
@@ -1632,6 +1664,48 @@ pub async fn create_custom_instance_database(
Ok(())
}
/// Refuse a workspace member writing a fork copy into, or pointing a fork at, the instance database
/// `dbname`, unless `w_id` created it for that ([`create_custom_instance_database`]) and nothing uses
/// it yet. The `wm_fork_` prefix is no authorization: every instance database answers to the same
/// `custom_instance_user`, so a name is all it takes to reach another workspace's copy.
///
/// Authorization: reads the global registry and every workspace's settings, and names other
/// workspaces in its refusal. Callers MUST have authorized `w_id` for the caller first — a member
/// of it forking or importing there — and MUST NOT call it on a workspace the caller is not in.
pub async fn ensure_fork_database_available_to(
db: &DB,
dbname: &str,
w_id: &str,
) -> error::Result<()> {
let created_for = sqlx::query_scalar::<_, Option<String>>(
"SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
WHERE name = 'custom_instance_pg_databases'",
)
.bind(dbname)
.fetch_optional(db)
.await?
.flatten();
if created_for.as_deref() != Some(w_id) {
return Err(Error::BadRequest(format!(
"Database '{dbname}' was not created for a fork of workspace '{w_id}'"
)));
}
let uses = workspaces::managed_database_uses(
&mut *db.acquire().await?,
workspaces::DataTableCatalogResourceType::Instance,
dbname,
None,
)
.await?;
if !uses.is_empty() {
return Err(Error::BadRequest(format!(
"Database '{dbname}' is already in use: {}",
uses.join(", ")
)));
}
Ok(())
}
/// Connection options parsed from a database URL.
///
/// The only place a database URL becomes `PgConnectOptions`. Providers that mint the password
+36 -1
View File
@@ -329,6 +329,7 @@ pub fn try_expand_internal_db_query(
"ALTER_TABLE" => expand_alter_table(json_str, db_type).map(ExpandedQuery::sql),
"CREATE_SCHEMA" => expand_create_schema(json_str, db_type).map(ExpandedQuery::sql),
"DROP_SCHEMA" => expand_drop_schema(json_str, db_type).map(ExpandedQuery::sql),
"RENAME_SCHEMA" => expand_rename_schema(json_str, db_type).map(ExpandedQuery::sql),
// Metadata queries
"LOAD_TABLE_METADATA" => expand_load_table_metadata(json_str, db_type),
"FOREIGN_KEYS" => expand_foreign_keys(json_str, db_type).map(ExpandedQuery::sql),
@@ -1716,6 +1717,13 @@ struct DropSchemaPayload {
ducklake: Option<String>,
}
#[derive(Deserialize)]
struct RenameSchemaPayload {
schema: String,
new_schema: String,
ducklake: Option<String>,
}
#[derive(Debug, Clone, Deserialize)]
struct TableEditorColumn {
name: String,
@@ -2004,6 +2012,23 @@ fn expand_drop_schema(json_str: &str, db_type: DbType) -> Result<String, String>
Ok(maybe_wrap_ducklake(query, p.ducklake.as_deref()))
}
fn expand_rename_schema(json_str: &str, db_type: DbType) -> Result<String, String> {
let p: RenameSchemaPayload = serde_json::from_str(json_str)
.map_err(|e| format!("Invalid RENAME_SCHEMA payload: {}", e))?;
if !matches!(db_type, DbType::Postgresql | DbType::Snowflake) || p.ducklake.is_some() {
return Err(format!(
"Renaming a schema is not supported on {:?}",
db_type
));
}
let query = format!(
"ALTER SCHEMA {} RENAME TO {};",
qi(&p.schema, db_type),
qi(&p.new_schema, db_type)
);
Ok(query)
}
fn expand_create_table(json_str: &str, db_type: DbType) -> Result<String, String> {
let p: CreateTablePayload = serde_json::from_str(json_str)
.map_err(|e| format!("Invalid CREATE_TABLE payload: {}", e))?;
@@ -2598,7 +2623,9 @@ WHERE table_catalog = current_database()",
)
} else {
(
"\nWHERE c.relkind = 'r' AND a.attnum > 0 AND NOT a.attisdropped\n AND ns.nspname != 'pg_catalog' AND ns.nspname != 'information_schema'".to_string(),
// pg_catalog is readable by everyone: without the privilege check this lists
// tables of schemas the connection's role cannot even enter.
"\nWHERE c.relkind = 'r' AND a.attnum > 0 AND NOT a.attisdropped\n AND ns.nspname != 'pg_catalog' AND ns.nspname != 'information_schema'\n AND has_schema_privilege(ns.oid, 'USAGE')".to_string(),
",\n ns.nspname AS schema_name,\n c.relname AS table_name".to_string(),
"\nJOIN pg_catalog.pg_class c ON a.attrelid = c.oid\nJOIN pg_catalog.pg_namespace ns ON c.relnamespace = ns.oid".to_string(),
"ns.nspname, c.relname, a.attnum".to_string(),
@@ -4101,6 +4128,13 @@ mod tests {
assert_eq!(sql, "DROP SCHEMA \"old_schema\" CASCADE;");
}
#[test]
fn test_expand_rename_schema() {
let marker = r#"-- WM_INTERNAL_DB_RENAME_SCHEMA {"schema":"old","new_schema":"new"}"#;
let sql = expand_code(marker, &ScriptLang::Postgresql);
assert_eq!(sql, "ALTER SCHEMA \"old\" RENAME TO \"new\";");
}
#[test]
fn test_expand_create_schema_with_ducklake() {
let marker = r#"-- WM_INTERNAL_DB_CREATE_SCHEMA {"schema":"s","ducklake":"lake"}"#;
@@ -4468,6 +4502,7 @@ mod tests {
assert!(sql.contains("schema_name"));
assert!(sql.contains("table_name"));
assert!(sql.contains("c.relkind = 'r'"));
assert!(sql.contains("has_schema_privilege(ns.oid, 'USAGE')"));
}
#[test]
+84 -5
View File
@@ -1471,6 +1471,73 @@ pub struct GoverningDatatable {
pub datatable: DataTable,
}
/// Everything still using the Windmill-managed database `dbname`, one description per use: data
/// table entries naming it, fork entries pointing at those, Ducklake catalogs on it, and fork
/// Ducklake metadata schemas there that cleanup has not dropped yet. `exempt` is the one data table
/// entry, `(workspace_id, name)`, the caller is about to stop using it through; pointers at that
/// entry still count, since dropping the database would leave them resolving to nothing.
///
/// Authorization: reads every workspace's settings and checks nothing. Callers MUST only turn the
/// answer into a refusal for someone allowed to administer `dbname`.
pub async fn managed_database_uses(
conn: &mut sqlx::PgConnection,
kind: DataTableCatalogResourceType,
dbname: &str,
exempt: Option<(&str, &str)>,
) -> Result<Vec<String>> {
let (exempt_workspace, exempt_name) = exempt.unzip();
Ok(sqlx::query_scalar::<_, String>(
"WITH entries AS (
SELECT ws.workspace_id::text AS workspace_id, dt.key AS name, dt.value
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(
CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
), naming AS (
SELECT workspace_id, name FROM entries
WHERE value->'database'->>'resource_type' = $1
AND value->'database'->>'resource_path' = $2
)
SELECT format('data table ''%s'' in workspace ''%s''', name, workspace_id) FROM naming
WHERE $3::text IS NULL OR NOT (workspace_id = $3 AND name = $4)
UNION ALL
SELECT format('data table ''%s'' in workspace ''%s'', which points at the one in ''%s''',
e.name, e.workspace_id, n.workspace_id)
FROM entries e JOIN naming n
ON e.value->'reference'->>'workspace_id' = n.workspace_id
AND e.value->'reference'->>'datatable' = n.name
UNION ALL
SELECT format('Ducklake ''%s'' in workspace ''%s''', dl.key, ws.workspace_id)
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(
CASE WHEN jsonb_typeof(ws.ducklake->'ducklakes') = 'object'
THEN ws.ducklake->'ducklakes' ELSE '{}'::jsonb END) dl
WHERE dl.value->'catalog'->>'resource_type' = $1
AND dl.value->'catalog'->>'resource_path' = $2
UNION ALL
SELECT format('the Ducklake namespace of fork ''%s'', not cleaned up yet', workspace_id)
FROM fork_ducklake_namespace
WHERE catalog = $1 || ':' || $2 AND NOT schema_dropped
ORDER BY 1",
)
.bind(kind.as_ref())
.bind(dbname)
.bind(exempt_workspace)
.bind(exempt_name)
.fetch_all(&mut *conn)
.await?)
}
/// Held by fork cleanup of `w_id`'s data tables and by forking `w_id`, which can hand the new fork
/// pointers at them, so a pointer cannot appear between cleanup's check and its drop.
pub async fn lock_fork_datatables(conn: &mut sqlx::PgConnection, w_id: &str) -> Result<()> {
sqlx::query("SELECT pg_advisory_xact_lock(hashtext('fork_datatables:' || $1))")
.bind(w_id)
.execute(&mut *conn)
.await?;
Ok(())
}
impl GoverningDatatable {
/// Backed by the Windmill instance's own Postgres, which is the only substrate data table
/// roles apply to.
@@ -1967,6 +2034,8 @@ pub fn strip_datatable_permissions(
/// As [`parse_datatable_ref`], except that an entry whose stored name itself contains `?` — which
/// names could before they were restricted — resolves by that exact name, without a role. It is
/// looked up first, so `sales?role=x` never reaches a different entry than the one stored so.
/// When `sales` is stored too, the reference means either one, and is refused rather than
/// resolved to whichever is looked up first.
///
/// Authorization: checks nothing, and its answer reveals whether `w_id` stores that exact name.
/// Callers MUST already act for `w_id` — a job of it, or a caller authenticated into it — and
@@ -1977,16 +2046,26 @@ pub async fn parse_datatable_ref_for(
reference: &str,
) -> Result<(String, Option<String>)> {
if reference.contains('?') {
let exists = sqlx::query_scalar::<_, Option<bool>>(
"SELECT (datatable->'datatables') ? $2 FROM workspace_settings WHERE workspace_id = $1",
let role_target = parse_datatable_ref(reference)
.ok()
.and_then(|(name, role)| role.map(|_| name));
let (exists, target_exists) = sqlx::query_as::<_, (Option<bool>, Option<bool>)>(
"SELECT (datatable->'datatables') ? $2, (datatable->'datatables') ? $3
FROM workspace_settings WHERE workspace_id = $1",
)
.bind(w_id)
.bind(reference)
.bind(role_target)
.fetch_optional(db)
.await?
.flatten()
.unwrap_or(false);
if exists {
.unwrap_or((None, None));
if exists.unwrap_or(false) {
if let (Some(name), Some(true)) = (role_target, target_exists) {
return Err(Error::BadRequest(format!(
"Data table reference '{reference}' names both the data table '{reference}' \
and a role on the data table '{name}'. Rename '{reference}' to use either."
)));
}
return Ok((reference.to_string(), None));
}
}
+2
View File
@@ -55,6 +55,8 @@ export interface AppFile {
tables?: string[];
datatable?: string;
schema?: string;
/** The role the app uses each data table through, by data table name. */
roles?: Record<string, string>;
};
// Mirrors granular ACLs on the raw_app path. Synced via /acls/* by
// applyExtraPermsDiff — never through update_app_raw — so a perm-only
+4
View File
@@ -5826,6 +5826,8 @@ data:
tables:
- main/users # Table in public schema
- main/app_schema:items # Table in specific schema
roles: # Optional: the role the app uses each datatable through
main: analyst
\`\`\`
**Table reference formats:**
@@ -5833,6 +5835,8 @@ data:
- \`<datatable>/<table>\` — Specific table in public schema
- \`<datatable>/<schema>:<table>\` — Table in specific schema
**Roles:** when a datatable is under roles, its queries run as a role, which only reaches what it was granted. \`roles\` records the role the app uses each datatable through; the app's code must pass the same role: \`wmill.datatable('main', { role: 'analyst' })\` in TypeScript, \`wmill.datatable('main', role='analyst')\` in Python. A datatable without an entry is used as its default role.
## SQL Migrations (sql_to_apply/)
The \`sql_to_apply/\` folder is for creating/modifying database tables during development.
File diff suppressed because it is too large Load Diff
@@ -1,16 +1,21 @@
<script lang="ts">
import { dbSchemas, workspaceStore, type DBSchema } from '$lib/stores'
import type { DataTableTables } from '$lib/gen'
import { sortArray } from '$lib/utils'
import { Loader2, RefreshCcw } from 'lucide-svelte'
import Alert from './common/alert/Alert.svelte'
import Button from './common/button/Button.svelte'
import { dbSupportsSchemas } from './apps/components/display/dbtable/utils'
import {
dbSupportsSchemas,
getLanguageByResourceType
} from './apps/components/display/dbtable/utils'
import DbManager from './DBManager.svelte'
import DbWorkerTagPicker from './DbWorkerTagPicker.svelte'
import MissingWorkerTagAlert from './jobs/MissingWorkerTagAlert.svelte'
import {
dbSchemaOpsWithPreviewScripts,
dbTableOpsWithPreviewScripts,
getDatabaseArg,
getDbType,
getDefaultDbTag,
getDucklakeSchema
@@ -18,11 +23,11 @@
import { Pane, Splitpanes } from 'svelte-splitpanes'
import SqlRepl from './SqlRepl.svelte'
import SimpleAgTable from './SimpleAgTable.svelte'
import { type Snippet } from 'svelte'
import type { DbInput } from './dbTypes'
import type { DatatableRowAction, DbInput } from './dbTypes'
import { schemaCacheKey } from './dbSchemaCache'
import { getDbSchemas, loadAllTablesMetaData } from './apps/components/display/dbtable/metadata'
import type { SelectedTable } from './DBManager.svelte'
import type { PendingRowAction, SelectedTable } from './DBManager.svelte'
import { getDbFeatures } from './apps/components/display/dbtable/dbFeatures'
import { resource } from 'runed'
import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte'
@@ -36,7 +41,15 @@
hasReplResult?: boolean
selectedSchemaKey?: string | undefined
selectedTableKey?: string | undefined
dbSelector?: Snippet<[]>
/** Every data table with its schemas and tables, for the left-pane tree. Undefined when
* the manager is not on a data table, which drops the tree's top level. */
datatableTree?: DataTableTables[]
datatableTreeLoading?: boolean
onSelectDatatable?: (datatable: string) => void
onSelectRole?: (datatable: string, role: string) => void
pendingAction?: PendingRowAction | undefined
onDatatableAction?: (datatable: string, action: DatatableRowAction) => void
canManageDatatable?: boolean
/** Enable multi-select mode with checkboxes in sidebar */
multiSelectMode?: boolean
/** Selected tables in multi-select mode */
@@ -59,7 +72,13 @@
hasReplResult = $bindable(false),
selectedSchemaKey = $bindable(undefined),
selectedTableKey = $bindable(undefined),
dbSelector,
datatableTree,
datatableTreeLoading,
onSelectDatatable,
onSelectRole,
pendingAction = $bindable(),
onDatatableAction,
canManageDatatable,
multiSelectMode = false,
selectedTables = $bindable([]),
disabledTables = [],
@@ -70,33 +89,25 @@
let ws = $derived(workspace ?? $workspaceStore)
let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[schemaCacheKey(input)])
let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[schemaCacheKey(ws, input)])
const outOfOrderModal = createAsyncConfirmationModal()
function getDbSchemasPath(input: DbInput): string {
switch (input.type) {
case 'database':
return input.resourcePath
case 'ducklake':
return 'ducklake://' + input.ducklake
}
}
// Scope the shared `dbSchemas` cache by the acting workspace: a datatable of
// the same name can exist in both the nav and the acting workspace, so the
// bare resource path alone would let one workspace's schema be reused for the
// other while DB operations target the acting one.
function schemaCacheKey(input: DbInput): string {
return `${ws}:${getDbSchemasPath(input)}`
}
// Reported in place of the loading spinner: both queries run as jobs, so
// anything from a bad connection to a tag no worker serves surfaces here
// instead of leaving the manager spinning with no explanation. Each query
// owns its slot so neither can clear the other's error on a refetch.
let schemaError = $state<string | undefined>(undefined)
let colDefsError = $state<string | undefined>(undefined)
function emptySchemaFor(db: DbInput): DBSchema {
return {
lang: db.type === 'ducklake' ? 'ducklake' : getLanguageByResourceType(db.resourceType),
schema: {},
publicOnly: undefined,
stringified: ''
} as DBSchema
}
let loadError = $derived(
schemaError
? { title: 'Could not load the database schema', message: schemaError }
@@ -136,14 +147,23 @@
const run = ++schemaRun
schemaError = undefined
if (!input) return
const dbSchemasPath = schemaCacheKey(input)
const dbSchemasPath = schemaCacheKey(ws, input)
if (input.type == 'database') {
let connection = input.resourcePath
try {
// The role'd reference, validated: an invalid role fails here rather than
// reading the schema as the data table's default role.
if (connection.startsWith('datatable://')) connection = getDatabaseArg(input).database!
} catch (e) {
schemaError = (e as Error)?.message ?? String(e)
return
}
// Reported through a local, not `schemaError` directly, so a superseded
// run's callback can't fail a load that already succeeded.
let queryError: string | undefined
const schema = await getDbSchemas(
input.resourceType,
input.resourcePath,
connection,
ws,
(message: string) => (queryError = message),
{ customTag: workerTag }
@@ -223,17 +243,23 @@
}}
/>
<!-- The error branch comes first on purpose: `dbSchema` is read from a cache that
survives a failed refetch, so ordering it first would hide the failure behind
stale content. -->
{#if loadError}
<!-- A load error replaces only the data pane: the tree, its role badge and menus, and the REPL
stay usable, so another data table or role can be picked and the connection tried by hand.
The tree then gets an empty schema: the cached one survives a failed refetch and would pass
stale content off as what this connection reaches. -->
{#snippet errorPane()}
<div class="h-full w-full flex flex-col items-center justify-center gap-3 p-8">
<div class="max-w-2xl w-full flex flex-col gap-3">
<Alert type="error" title={loadError.title} size="xs">
{loadError.message}
<Alert type="error" title={loadError?.title ?? ''} size="xs">
{loadError?.message}
</Alert>
<div class="self-start">
<Button size="xs" color="light" startIcon={{ icon: RefreshCcw }} on:click={() => refresh()}>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: RefreshCcw }}
on:click={() => refresh()}
>
Retry
</Button>
</div>
@@ -247,9 +273,12 @@
/>
</div>
</div>
{:else if dbSchema && ws && input}
{/snippet}
{#if (loadError || dbSchema) && ws && input}
{@const _input = input}
{@const dbType = getDbType(_input)}
{@const shownSchema = loadError || !dbSchema ? emptySchemaFor(_input) : dbSchema}
<Splitpanes horizontal>
<Pane class="relative">
<!-- svelte-ignore a11y_click_events_have_key_events -->
@@ -274,9 +303,11 @@
</div>
<DbManager
dbSupportsSchemas={dbSupportsSchemas(dbType)}
databaseIsEmpty={!Object.values(dbSchema.schema).flatMap((s) => Object.values(s)).length}
{dbSchema}
colDefs={colDefs.current}
databaseIsEmpty={!loadError &&
!Object.values(shownSchema.schema).flatMap((s) => Object.values(s)).length}
dbSchema={shownSchema}
mainPane={loadError ? errorPane : undefined}
colDefs={loadError ? undefined : colDefs.current}
dbTableOpsFactory={({ colDefs, tableKey, whereClause }) =>
dbTableOpsWithPreviewScripts({
colDefs,
@@ -306,7 +337,15 @@
: undefined}
{dbType}
refresh={() => refresh()}
{dbSelector}
{datatableTree}
{datatableTreeLoading}
{onSelectDatatable}
{onSelectRole}
workspace={ws}
currentRole={input.type === 'database' ? input.role : undefined}
bind:pendingAction
{onDatatableAction}
{canManageDatatable}
{onImport}
bind:selectedSchemaKey
bind:selectedTableKey
@@ -329,12 +368,12 @@
onSchemaChange={() => refresh()}
placeholderTableName={sortArray(
Object.keys(
dbSchema?.schema[
'public' in dbSchema?.schema
shownSchema.schema[
'public' in shownSchema.schema
? 'public'
: 'dbo' in dbSchema?.schema
: 'dbo' in shownSchema.schema
? 'dbo'
: Object.keys(dbSchema?.schema ?? {})?.[0]
: Object.keys(shownSchema.schema ?? {})?.[0]
] ?? {}
)
)?.[0]}
@@ -1,25 +1,25 @@
<script lang="ts">
import { superadmin, userStore, workspaceStore } from '$lib/stores'
import { WorkspaceService } from '$lib/gen'
import { enterpriseLicense, superadmin, userStore, workspaceStore } from '$lib/stores'
import { WorkspaceService, type DataTableTables } from '$lib/gen'
import { listUsableDatatableRoles } from './datatableUsableRoles'
import Button from './common/button/Button.svelte'
import Drawer from './common/drawer/Drawer.svelte'
import DrawerContent from './common/drawer/DrawerContent.svelte'
import Select from './select/Select.svelte'
import {
ArrowLeft,
Copy,
Download,
Expand,
LoaderCircle,
Minimize,
RefreshCcw,
Upload
} from 'lucide-svelte'
import { ArrowLeft, Copy, Download, Expand, Minimize, RefreshCcw, Upload } from 'lucide-svelte'
import DBManagerContent from './DBManagerContent.svelte'
import type { PendingRowAction } from './DBManager.svelte'
import DataTableMigrationsButton from './workspaceSettings/DataTableMigrationsButton.svelte'
import DataTablePermissionsButton from './workspaceSettings/DataTablePermissionsButton.svelte'
import { resource } from 'runed'
import { untrack } from 'svelte'
import { tick, untrack } from 'svelte'
import type { DbManagerUriState } from './dbManagerDrawerModel.svelte'
import {
ADMIN_DATATABLE_ROLE,
datatableNameTakesRole,
defaultMigrationRole,
type DatatableRowAction
} from './dbTypes'
import ResourcePicker from './ResourcePicker.svelte'
import Alert from './common/alert/Alert.svelte'
import { sendUserToast } from '$lib/toast'
@@ -41,31 +41,113 @@
// the editor that opened it (set via openDrawer), else the nav workspace.
let ws = $derived(uriState.workspace ?? $workspaceStore)
// Load available datatables when drawer opens with datatable input
const datatables = resource<string[]>([], async () => {
if (!ws) return []
try {
return (await WorkspaceService.listDataTables({ workspace: ws })).map((d) => d.name)
} catch (e) {
console.error('Failed to load datatables:', e)
return []
}
})
// A create started on a data table other than the current one: survives the
// re-mount the switch causes.
let pendingAction = $state<PendingRowAction | undefined>(undefined)
const datatableItems = $derived(
datatables.current.map((dt) => ({
value: dt,
label: dt
}))
// Read once through primitives: the getters return values of a freshly parsed URL, which
// changes on every table click, and the listings below must not refetch for that.
const selectedDatatable = $derived(uriState.selectedDatatable)
const selectedRole = $derived(uriState.selectedRole)
// Roles the caller may use, to settle the role before anything connects. Offering only
// these is a convenience: the server refuses any other.
const usableRoles = resource(
() => [ws, selectedDatatable] as const,
async ([workspace, datatable]) => {
if (!workspace || !datatable) return undefined
try {
return {
datatable,
...(await listUsableDatatableRoles(workspace, datatable))
}
} catch (e) {
// Never leave the drawer waiting on this: fall back to the
// unpermissioned shape so it opens and the server picks the role.
console.error('Failed to load datatable roles:', e)
return { datatable, permissioned: false, roles: [], default_role: ADMIN_DATATABLE_ROLE }
}
}
)
// Refetch datatables when switching to a datatable input
// A resource keeps its previous value while refetching, and roles are per data
// table: settling from the last one's answer would connect to the new data
// table as a role it may not even have.
const rolesOfCurrent = $derived(
usableRoles.current?.datatable === selectedDatatable ? usableRoles.current : undefined
)
// Nothing that connects runs until the role is settled: a first round sent without a role
// would run — and cache — as whatever the server defaults to.
const roleSettled = $derived(
!uriState.isDatatableInput ||
(rolesOfCurrent !== undefined &&
(!rolesOfCurrent.permissioned ||
rolesOfCurrent.roles.length === 0 ||
selectedRole !== undefined ||
// Its reference cannot name a role, so it connects as the default one.
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))))
)
// Make the role explicit before anything queries the data table, so the URL, the
// cache and every migration the manager writes name it. A role already in the URL
// is kept even when it is not usable: the server refuses it, visibly.
$effect(() => {
if (uriState.isDatatableInput) {
untrack(() => datatables.refetch())
}
const roles = rolesOfCurrent
if (
!roles?.permissioned ||
selectedRole !== undefined ||
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))
)
return
const effective = roles.roles.includes(roles.default_role) ? roles.default_role : roles.roles[0]
if (effective) untrack(() => (uriState.selectedRole = effective))
})
const contentInput = $derived.by(() => {
const input = uriState.effectiveInput
if (input?.type !== 'database' || selectedDatatable === undefined) return input
const migrationRole = defaultMigrationRole(
selectedDatatable,
rolesOfCurrent?.permissioned,
rolesOfCurrent?.default_role
)
return migrationRole === undefined ? input : { ...input, migrationRole }
})
// Every data table with its schemas and tables, in one call: this is what the
// left pane's tree navigates, so it has to cover the data tables the user is
// not currently on, not just the selected one. The privileges it reports are
// the connected role's, so the role picked on the open data table is part of
// what is being asked. Gated on the drawer being open on a data table: this
// reaches every data table's database in turn, and the component is mounted on
// every logged-in page.
let datatablesRun = 0
const datatables = resource(
() =>
[
open && uriState.isDatatableInput,
ws,
selectedDatatable,
selectedRole,
roleSettled
] as const,
async ([active, workspace, roleFor, role, settled]): Promise<DataTableTables[]> => {
if (!active || !workspace) return []
if (!settled) return untrack(() => datatables.current)
const run = ++datatablesRun
try {
const result = await WorkspaceService.listDataTableTables({ workspace, roleFor, role })
// An answer for a selection that has since changed describes another role.
return run === datatablesRun ? result : untrack(() => datatables.current)
} catch (e) {
console.error('Failed to load datatables:', e)
return run === datatablesRun ? [] : untrack(() => datatables.current)
}
},
{ initialValue: [] }
)
function handleClose() {
uriState.closeDrawer()
dbManagerContent?.clearReplResult()
@@ -78,6 +160,10 @@
if (!open) {
expand = false
uriState.closeDrawer()
// An action asked for on one data table must not be waiting when the
// drawer is next opened on another database — or on no data table at
// all, where nothing would recognise it as foreign.
pendingAction = undefined
}
})
@@ -97,6 +183,8 @@
let importDrawerOpen = $state(false)
let importLoading = $state(false)
let importSource = $state<string | undefined>(undefined)
/** Which database an import writes into; set when driven from a tree row. */
let importTarget = $state<string | undefined>(undefined)
let importBehavior = $state<'schema_only' | 'schema_and_data'>('schema_only')
let isPostgresqlInput = $derived(
@@ -116,13 +204,49 @@
return toSourceIdentifier(input.resourcePath)
}
// The tree's row menus act on the data table of the row that was clicked, which
// is not necessarily the one currently open — so the target is set first and the
// headless modals are keyed on it.
let actionDatatable = $state<string | undefined>(undefined)
let migrationsModal = $state<DataTableMigrationsButton | undefined>()
let permissionsDrawer = $state<DataTablePermissionsButton | undefined>()
async function runDatatableAction(datatable: string, action: DatatableRowAction) {
actionDatatable = datatable
// Let the keyed block above mount against the new target before driving it.
await tick()
switch (action) {
case 'migrations':
migrationsModal?.open()
break
case 'roles':
permissionsDrawer?.open()
break
case 'export':
await handleExportSchema(`datatable://${datatable}`)
break
case 'import':
importTarget = `datatable://${datatable}`
importDrawerOpen = true
break
}
}
function refreshManager() {
dbManagerContent?.refresh()
dbManagerContent?.dbManager()?.dbTable()?.refresh()
refreshRoles()
}
async function handleExportSchema() {
const source = currentSourceIdentifier()
/** Re-read what the tree and the role picker show: both are answers about the
* data table's roles, which the permissions drawer can have just changed. */
function refreshRoles() {
datatables.refetch()
usableRoles.refetch()
}
async function handleExportSchema(explicitSource?: string) {
const source = explicitSource ?? currentSourceIdentifier()
if (!source || !ws) return
try {
exportResult = await WorkspaceService.exportPgSchema({
@@ -137,7 +261,7 @@
async function handleImportDatabase() {
if (!importSource || !ws) return
const target = currentSourceIdentifier()
const target = importTarget ?? currentSourceIdentifier()
if (!target) return
importLoading = true
try {
@@ -183,52 +307,48 @@
noPadding
id="db-manager-drawer"
>
{#if uriState.effectiveInput && ws}
{#key uriState.selectedDatatable}
{#if contentInput && ws && roleSettled}
{#key `${selectedDatatable}~${selectedRole ?? ''}`}
<DBManagerContent
bind:this={dbManagerContent}
input={uriState.effectiveInput}
input={contentInput}
workspace={uriState.workspace}
datatableTree={uriState.isDatatableInput ? datatables.current : undefined}
datatableTreeLoading={datatables.loading}
onSelectDatatable={(dt) => (uriState.selectedDatatable = dt)}
onSelectRole={(dt, role) => {
// Setting the data table clears the role, so the order matters.
uriState.selectedDatatable = dt
uriState.selectedRole = role
}}
bind:pendingAction
canManageDatatable={!!($superadmin || $userStore?.is_admin) &&
!!$enterpriseLicense &&
!isCloudHosted()}
onDatatableAction={runDatatableAction}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey={uriState.selectedSchema}
bind:selectedTableKey={uriState.selectedTable}
onImport={enableImportExport
? (mode) => ((importDrawerOpen = true), (importBehavior = mode))
? (mode) => (
(importTarget = undefined),
(importDrawerOpen = true),
(importBehavior = mode)
)
: undefined}
>
{#snippet dbSelector()}
{#if uriState.isDatatableInput}
{#if datatables.loading}
<div class="flex items-center gap-2 text-tertiary ml-2">
<LoaderCircle size={14} class="animate-spin" />
<span class="text-sm">Loading...</span>
</div>
{:else if datatables.current.length >= 1}
<Select
transformInputSelectedText={(s) => `Datatable: ${s}`}
items={datatableItems}
bind:value={uriState.selectedDatatable}
placeholder="Select data table"
size="md"
/>
{/if}
{/if}
{/snippet}
</DBManagerContent>
></DBManagerContent>
{/key}
{/if}
{#snippet actions()}
{#if uriState.isDatatableInput && uriState.selectedDatatable && ws}
<DataTableMigrationsButton
workspace={ws}
datatable={uriState.selectedDatatable}
onSchemaChanged={refreshManager}
/>
{/if}
{#if enableImportExport}
<Button startIcon={{ icon: Download }} onClick={handleExportSchema}>Export</Button>
<Button startIcon={{ icon: Upload }} onClick={() => (importDrawerOpen = true)}>
<!-- A data table exports and imports from its row menu in the tree; a plain
database has no tree row to hold them. -->
{#if enableImportExport && !uriState.isDatatableInput}
<Button startIcon={{ icon: Download }} onClick={() => handleExportSchema()}>Export</Button>
<Button
startIcon={{ icon: Upload }}
onClick={() => ((importTarget = undefined), (importDrawerOpen = true))}
>
Import
</Button>
{/if}
@@ -260,6 +380,27 @@
</DrawerContent>
</Drawer>
{#if actionDatatable && ws}
{#key actionDatatable}
<DataTableMigrationsButton
bind:this={migrationsModal}
hideTrigger
workspace={ws}
datatable={actionDatatable}
onSchemaChanged={refreshManager}
/>
{#if $enterpriseLicense && !isCloudHosted()}
<DataTablePermissionsButton
bind:this={permissionsDrawer}
hideTrigger
workspace={ws}
datatable={actionDatatable}
onSaved={refreshRoles}
/>
{/if}
{/key}
{/if}
<Drawer bind:open={exportDrawerOpen} size="800px" offset={offset + 1}>
<DrawerContent title="Export Schemas" on:close={() => (exportDrawerOpen = false)}>
{#if exportResult}
@@ -503,7 +503,7 @@
{/if}
{#if askingForConfirmation?.codeContent}
<div
class="bg-surface-secondary border border-surface-selected rounded-md p-2 relative group"
class="bg-surface-secondary border border-surface-selected rounded-md p-2 relative group min-w-0"
>
<button
class="absolute top-2 right-2 opacity-0 group-hover:opacity-100 transition-opacity p-1 rounded hover:bg-surface-hover"
@@ -512,9 +512,7 @@
>
<ClipboardCopy size={14} />
</button>
<pre class="whitespace-pre-wrap text-sm"
><code>{askingForConfirmation.codeContent}</code></pre
>
<pre class="overflow-x-auto text-sm"><code>{askingForConfirmation.codeContent}</code></pre>
</div>
{/if}
</ConfirmationModal>
@@ -0,0 +1,66 @@
<script lang="ts">
import { ChevronDown } from 'lucide-svelte'
import SelectDropdown from './select/SelectDropdown.svelte'
import Badge from './common/badge/Badge.svelte'
import { clickOutside } from '$lib/utils'
let {
role,
roles,
onSelect
}: {
/** The role in effect, shown on the badge. */
role: string
/** The roles the caller may switch to. */
roles: string[]
onSelect: (role: string) => void
} = $props()
let open = $state(false)
let anchorEl: HTMLSpanElement | undefined = $state()
const items = $derived(roles.map((r) => ({ label: r, value: r })))
// The table picker's drawer opens at `disposables + 10000`, which the
// dropdown's own z-index would sit under.
const dropdownClass = 'z-[20000]'
</script>
<span
bind:this={anchorEl}
class="relative flex min-w-0"
use:clickOutside={{ onClickOutside: () => (open = false) }}
>
<Badge
clickable
color="gray"
wrapperClass="min-w-0"
class="min-w-0 gap-0.5 pl-2 pr-1 bg-surface-sunken hover:bg-surface-sunken text-primary
transition-[filter,transform] hover:brightness-95 active:brightness-90 active:scale-[0.97]
{open ? 'brightness-95' : ''}"
onclick={(e) => {
// The row underneath folds on click, and picking a role is not that.
e.stopPropagation()
open = !open
}}
>
<!-- A long role name gives way rather than pushing the row's own actions
past its right edge. -->
<span class="truncate">{role}</span>
<ChevronDown
size={11}
class="shrink-0 text-secondary transition-transform {open ? 'rotate-180' : ''}"
/>
</Badge>
<SelectDropdown
processedItems={items}
value={role}
{open}
listAutoWidth={false}
class={dropdownClass}
getInputRect={anchorEl && (() => anchorEl!.getBoundingClientRect())}
onSelectValue={(item) => {
open = false
if (item.value !== role) onSelect(item.value)
}}
/>
</span>
@@ -6,8 +6,18 @@
import { joinSqlStatements, splitSqlRuns } from './sqlDdl'
import { logDdlGuardChoice } from './workspaceSettings/datatableTelemetry'
import { CornerDownLeft } from 'lucide-svelte'
import { withMigrationRole } from './datatableMigrationRole'
let { workspace, datatable }: { workspace: string; datatable: string } = $props()
let {
workspace,
datatable,
role
}: {
workspace: string
datatable: string
/** The role the editor runs as. The migration declares it, or it would run as admin. */
role?: string
} = $props()
type Choice = 'run' | 'migrate' | 'cancel'
@@ -73,7 +83,7 @@
function openMigrationModal(sql: string): Promise<boolean> {
return new Promise((resolve) => {
resolveMigrationClosed = (created: boolean) => resolve(created)
newMigrationModal?.open({ codeUp: sql })
newMigrationModal?.open({ codeUp: withMigrationRole(sql, role) })
})
}
@@ -145,6 +155,11 @@
migrations rather than run ad-hoc. Create a migration for it instead?
{/if}
</p>
{#if role}
<p class="text-sm text-secondary">
It will run as role <span class="font-mono">{role}</span>.
</p>
{/if}
<pre
class="text-xs whitespace-pre-wrap font-mono bg-surface-secondary rounded p-3 max-h-48 overflow-auto"
>{promptSql}</pre
@@ -1093,8 +1093,10 @@
migrations are set up and used, how often an empty workspace home is seen, how often
the home pages create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, whether a pre-approved trial offer was opened, and whether data tables are
put under roles and whether callers name a role or take the default, last 30 days)</li
import got, whether a pre-approved trial offer was opened, whether data tables are put
under roles and whether callers name a role or take the default, and which kinds of
access change (grant, revoke, ownership, default privileges) are applied to data
tables, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
@@ -1160,8 +1162,10 @@
migrations are set up and used, how often an empty workspace home is seen, how often
the home pages create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, whether a pre-approved trial offer was opened, and whether data tables are
put under roles and whether callers name a role or take the default, last 30 days)</li
import got, whether a pre-approved trial offer was opened, whether data tables are put
under roles and whether callers name a role or take the default, and which kinds of
access change (grant, revoke, ownership, default privileges) are applied to data
tables, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
+6 -1
View File
@@ -223,5 +223,10 @@
</Splitpanes>
{#if datatableName && ws}
<DdlMigrationGuard bind:this={ddlGuard} workspace={ws} datatable={datatableName} />
<DdlMigrationGuard
bind:this={ddlGuard}
workspace={ws}
datatable={datatableName}
role={input.type === 'database' ? (input.role ?? input.migrationRole) : undefined}
/>
{/if}
+5 -4
View File
@@ -9,9 +9,10 @@
kind: FavoriteKind
summary?: string
workspaceId?: string
size?: number
}
let { path, kind, workspaceId, summary }: Props = $props()
let { path, kind, workspaceId, summary, size = 16 }: Props = $props()
let buttonHover = $state(false)
let starred = $derived(favoriteManager.isStarred(path, kind))
@@ -31,14 +32,14 @@
>
{#if starred}
{#if buttonHover}
<StarOff size={16} fill="currentcolor" />
<StarOff {size} fill="currentcolor" />
{:else}
<Star size={16} fill="currentcolor" />
<Star {size} fill="currentcolor" />
{/if}
{:else}
<Star
class={!buttonHover ? 'opacity-60' : ''}
size={16}
{size}
fill={buttonHover ? 'currentcolor' : 'none'}
/>
{/if}
@@ -282,7 +282,8 @@ const scriptsV2: typeof legacyScripts = {
...legacyScripts.postgresql,
code: `
SELECT table_name, column_name, udt_name, column_default, is_nullable, nsp.nspname AS table_schema FROM information_schema.columns
RIGHT JOIN pg_namespace nsp ON table_schema = nsp.nspname WHERE nsp.nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')`
RIGHT JOIN pg_namespace nsp ON table_schema = nsp.nspname WHERE nsp.nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')
AND NOT starts_with(nsp.nspname, 'pg_') AND has_schema_privilege(nsp.oid, 'USAGE')`
}
}
@@ -186,7 +186,7 @@
<Icon class={theme[type].classes.icon} />
</div>
{/if}
<div class={twMerge('ml-0 text-left flex-1 ', showIcon ? 'ml-4' : '')}>
<div class={twMerge('ml-0 text-left flex-1 min-w-0', showIcon ? 'ml-4' : '')}>
<h3 class="text-lg font-medium text-primary">
{title}
</h3>
@@ -732,12 +732,14 @@ export class AIChatManager implements ChatViewHost {
/** Every mounted flow editor. */
#flowEditors = new Set<FlowAIChatHelpers>()
appAiChatHelpers = $state<AppAIChatHelpers | undefined>(undefined)
/** Datatable creation policy: enabled flag, datatable name, and optional schema */
/** Datatable creation policy: enabled flag, datatable name, optional schema, and the role the
* app uses each data table through */
datatableCreationPolicy = $state<{
enabled: boolean
datatable: string | undefined
schema: string | undefined
}>({ enabled: false, datatable: undefined, schema: undefined })
roles?: Record<string, string>
}>({ enabled: false, datatable: undefined, schema: undefined, roles: undefined })
pendingNewCode = $state<string | undefined>(undefined)
apiTools = $state<Tool<any>[]>([])
aiChatInput = $state<AIChatInput | null>(null)
@@ -67,6 +67,7 @@
<DefaultDatabaseSelector
datatable={aiChatManager.datatableCreationPolicy.datatable}
schema={aiChatManager.datatableCreationPolicy.schema}
roles={aiChatManager.datatableCreationPolicy.roles}
onChange={handleDefaultChange}
description="Set the default datatable and schema for new tables. When table creation is enabled, AI can create tables here if needed."
/>
@@ -18,6 +18,7 @@ import {
type AppCodeSelectionElement,
type AppDatatableElement
} from '../context'
import { appDatatableRole, sdkDatatableCall } from '$lib/components/raw_apps/dataTableRefUtils'
// Backend runnable types
export type BackendRunnableType = 'script' | 'flow' | 'hubscript' | 'inline'
@@ -921,9 +922,20 @@ export function prepareAppSystemMessage(customPrompt?: string): ChatCompletionSy
const policy = aiChatManager.datatableCreationPolicy
const datatableName = policy.datatable ?? 'main'
const schemaPrefix = policy.schema ? `${policy.schema}.` : ''
// Use wmill.datatable() for 'main' (default), otherwise wmill.datatable('name')
const datatableCall =
datatableName === 'main' ? 'wmill.datatable()' : `wmill.datatable('${datatableName}')`
// A role names the privileges the app's queries run with, so it has to be in the code the
// model writes.
const datatableRole = appDatatableRole(policy.roles, datatableName)
const tsDatatableCall = sdkDatatableCall(datatableName, datatableRole, 'typescript')
const pyDatatableCall = sdkDatatableCall(datatableName, datatableRole, 'python')
const roleEntries = Object.entries(policy.roles ?? {})
const rolesNote =
roleEntries.length > 0
? `\n\nThis app uses these data tables through a role: ${roleEntries
.map(([dt, role]) => `\`${dt}\` as \`${role}\``)
.join(
', '
)}. Always pass that role when calling \`wmill.datatable\` on them, as in the examples. The role only reaches what it was granted, so a query on a table it lacks privileges on fails with \`permission denied\`.`
: ''
let content = `You are a helpful assistant that creates and edits apps on the Windmill platform. Apps are defined as a collection of files that contains both the frontend and the backend.
@@ -1024,7 +1036,7 @@ Backend runnables should only perform **data operations** (SELECT, INSERT, UPDAT
import * as wmill from 'windmill-client';
export async function main(user_id: string) {
const sql = ${datatableCall};
const sql = ${tsDatatableCall};
const user = await sql\`SELECT * FROM ${schemaPrefix}users WHERE id = \${user_id}\`.fetchOne();
return user;
}
@@ -1035,12 +1047,12 @@ export async function main(user_id: string) {
import wmill
def main(user_id: str):
db = ${datatableCall}
db = ${pyDatatableCall}
user = db.query('SELECT * FROM ${schemaPrefix}users WHERE id = $1', user_id).fetch_one()
return user
\`\`\`
Use these examples for normal datatable access.
Use these examples for normal datatable access.${rolesNote}
### Schema Modifications (DDL) - Use exec_datatable_sql tool ONLY
@@ -2,6 +2,7 @@ import { z } from 'zod'
import { WorkspaceService, type CompletedJob } from '$lib/gen'
import type { DataTableTables } from '$lib/gen/types.gen'
import { runScript } from '$lib/components/jobs/utils'
import { datatableReference } from '$lib/components/dbTypes'
import {
createToolDef,
executeTestRun,
@@ -15,9 +16,9 @@ import {
*
* Datatables are workspace-level managed PostgreSQL databases. The backend
* endpoints used here (`list_datatable_tables`, `get_datatable_table_schema`)
* and SQL execution (`datatable://<name>`) are gated only by workspace
* membership, so these tools need no app context and operate directly on the
* workspace. This is the unrestricted counterpart to the app-mode datatable
* and SQL execution (`datatable://<name>`) need no app context: the server
* decides what the caller reaches, as the datatable role they name or its
* default. This is the unrestricted counterpart to the app-mode datatable
* tools in `app/core.ts`, which additionally filter by the app's whitelist.
*/
@@ -31,9 +32,19 @@ const memo = <T>(factory: () => T): (() => T) => {
// ============= Pure workspace-scoped operations =============
/** List all datatables configured in the workspace, with their schema/table names. */
export async function listDatatables(workspace: string): Promise<DataTableTables[]> {
return await WorkspaceService.listDataTableTables({ workspace })
/** List the datatables configured in the workspace, with their schema/table names: all of them as
* their default role, or only `datatableName`, as `role` when one is given. */
export async function listDatatables(
workspace: string,
datatableName?: string,
role?: string
): Promise<DataTableTables[]> {
if (datatableName === undefined) return await WorkspaceService.listDataTableTables({ workspace })
return await WorkspaceService.listDataTableTables({
workspace,
datatableName,
...(role !== undefined && { roleFor: datatableName, role })
})
}
/** Get the columns (column_name -> compact_type) of one datatable table. */
@@ -41,13 +52,15 @@ export async function getDatatableColumns(
workspace: string,
datatableName: string,
schemaName: string,
tableName: string
tableName: string,
role?: string
): Promise<Record<string, string>> {
const schema = await WorkspaceService.getDataTableTableSchema({
workspace,
datatableName,
schemaName,
tableName
tableName,
role
})
return schema.columns
}
@@ -81,7 +94,26 @@ const NO_DATATABLES_CONFIGURED_MESSAGE =
// ============= Tool definitions =============
const getListDatatablesSchema = memo(() => z.object({}))
// The same rule the server applies to `-- role <name>`; a name it would refuse fails here instead.
const getRoleSchema = memo(() =>
z
.string()
.regex(/^[A-Za-z0-9_-]{1,63}$/)
.optional()
.describe(
"The datatable role to connect as, when the code you are working on uses one (an app's `data.roles` entry, or the `role` it passes to wmill.datatable). Omit for the datatable's default role."
)
)
const getListDatatablesSchema = memo(() =>
z.object({
datatable_name: z
.string()
.optional()
.describe('List only this datatable. Required with `role`.'),
role: getRoleSchema()
})
)
const getListDatatablesToolDef = memo(() =>
createToolDef(
getListDatatablesSchema(),
@@ -94,7 +126,8 @@ const getGetDatatableTableSchemaSchema = memo(() =>
z.object({
datatable_name: z.string().describe('The datatable name to inspect, e.g. "main".'),
schema_name: z.string().describe('The schema name, e.g. "public".'),
table_name: z.string().describe('The table name to inspect.')
table_name: z.string().describe('The table name to inspect.'),
role: getRoleSchema()
})
)
const getGetDatatableTableSchemaToolDef = memo(() =>
@@ -117,6 +150,7 @@ const getExecDatatableSqlSchema = memo(() =>
.describe(
'The SQL query to execute. Supports SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, ALTER TABLE, DROP TABLE, etc. For SELECT queries, results are returned as an array of objects. A newly created table will appear in list_datatables automatically.'
),
role: getRoleSchema(),
background: z
.boolean()
.optional()
@@ -217,10 +251,18 @@ export function getDatatableTools(): Tool<{}>[] {
{
def: getListDatatablesToolDef(),
planModeSafe: true,
fn: async ({ workspace, toolId, toolCallbacks }) => {
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, { content: 'Listing datatables...' })
try {
const metadata = await listDatatables(workspace)
const parsedArgs = getListDatatablesSchema().parse(args ?? {})
if (parsedArgs.role !== undefined && parsedArgs.datatable_name === undefined) {
throw new Error('`role` needs `datatable_name`, the datatable it is a role of')
}
const metadata = await listDatatables(
workspace,
parsedArgs.datatable_name,
parsedArgs.role
)
if (metadata.length === 0) {
toolCallbacks.setToolStatus(toolId, {
content: 'No datatables configured — set one up in workspace settings'
@@ -236,7 +278,18 @@ export function getDatatableTools(): Tool<{}>[] {
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${metadata.length} datatable(s) with ${totalTables} table(s)`
})
return JSON.stringify(metadata, null, 2)
// Only what the model acts on: the roles it may pass, not the creation privileges
// the manager's UI gates on.
return JSON.stringify(
metadata.map((d) => ({
datatable_name: d.datatable_name,
schemas: d.schemas,
...(d.error && { error: d.error }),
...(d.permissioned && { usable_roles: d.usable_roles, default_role: d.default_role })
})),
null,
2
)
} catch (e) {
const errorMsg = `Error listing datatables: ${e instanceof Error ? e.message : String(e)}`
toolCallbacks.setToolStatus(toolId, { content: errorMsg, error: errorMsg })
@@ -257,7 +310,8 @@ export function getDatatableTools(): Tool<{}>[] {
workspace,
parsedArgs.datatable_name,
parsedArgs.schema_name,
parsedArgs.table_name
parsedArgs.table_name,
parsedArgs.role
)
toolCallbacks.setToolStatus(toolId, {
content: `Retrieved schema for ${parsedArgs.schema_name}.${parsedArgs.table_name}`
@@ -300,7 +354,7 @@ export function getDatatableTools(): Tool<{}>[] {
requestBody: {
language: 'postgresql',
content: parsedArgs.sql,
args: { database: `datatable://${name}` }
args: { database: datatableReference(name, parsedArgs.role) }
}
}),
workspace,
@@ -1431,7 +1431,12 @@ Data Tables:
- Datatables are workspace-scoped managed PostgreSQL databases, shared across the workspace (not owned by any single app). They must be configured by the user in their workspace settings (Workspace settings Data Tables); they cannot be created via SQL.
- Use list_datatables to discover the available datatables and their tables. Reuse an existing table rather than creating a duplicate. If list_datatables reports none, this is a blocking prerequisite tell the user to set up a datatable in their workspace settings and stop; do not assume a "main" datatable exists or call exec_datatable_sql.
- Use get_datatable_table_schema only when you need a table's column names/types; list_datatables is enough for table-list or availability summaries.
- Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement it appears in list_datatables afterward, with no registration step.
- Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement it appears in list_datatables afterward, with no registration step.${
isCloudHosted()
? ''
: `
- A raw app may use a datatable through a role (\`data.roles\` in its raw_app.yaml). When working on such an app, pass that role to the datatable tools, and to wmill.datatable in its runnables, so you see and change only what the app itself can.`
}
- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) it returns only that language so you get just what you need.${
skills.length > 0
? `
@@ -0,0 +1,283 @@
<script lang="ts">
import { WorkspaceService, type AclChange, type AclTarget, type DatatableAclInfo } from '$lib/gen'
import { resource } from 'runed'
import { Trash2 } from 'lucide-svelte'
import { sendUserToast } from '$lib/toast'
import { Alert, Button } from '../common'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import Select from '../select/Select.svelte'
import DataTable from '../table/DataTable.svelte'
import Head from '../table/Head.svelte'
import Row from '../table/Row.svelte'
import Cell from '../table/Cell.svelte'
import PgGrantBuilder from './PgGrantBuilder.svelte'
import {
ADMIN_ROLE,
blockingSources,
grantKey,
grantScopeLabel,
groupGrants,
revocablePrivileges,
revokeScopeOf,
uncoveredCreators
} from './aclScopes'
let {
workspace,
datatable,
target
}: {
workspace: string
datatable: string
/** What owner and grants are read and written for. */
target: AclTarget
} = $props()
const acl = resource(
() => [workspace, datatable, target] as const,
async ([ws, dt, t]) =>
await WorkspaceService.getDatatableAcl({
workspace: ws,
datatableName: dt,
kind: t.kind,
schema: t.kind === 'database' ? undefined : t.schema,
table: t.kind === 'table' ? t.table : undefined
})
)
// Nothing is written before its SQL has been shown, and the apply runs exactly that SQL: the
// server plans again and refuses if the result differs.
let pending = $state<
{ change: AclChange; statements: string[]; warnings: string[]; title: string } | undefined
>(undefined)
let planning = $state(false)
let applying = $state(false)
const info: DatatableAclInfo | undefined = $derived(acl.current)
const grantRows = $derived(groupGrants(info?.grants ?? []))
const ownerItems = $derived(
info
? (info.roles.includes(info.owner) ? info.roles : [info.owner, ...info.roles]).map((r) => ({
value: r,
label: r
}))
: []
)
/** A revoke listed per object takes them all: say that it does. */
const pendingCoversObjects = $derived(
pending?.change.type === 'revoke' && (pending.change.objects?.length ?? 0) > 1
)
function errorText(e: any): string {
return e?.body ?? e?.message ?? String(e)
}
async function confirm(change: AclChange, title: string) {
planning = true
try {
const plan = await WorkspaceService.planDatatableAcl({
workspace,
datatableName: datatable,
requestBody: { target, change }
})
pending = { change, statements: plan.statements, warnings: plan.warnings, title }
} catch (e) {
sendUserToast(errorText(e), true)
} finally {
planning = false
}
}
async function apply() {
if (!pending) return
applying = true
try {
await WorkspaceService.applyDatatableAcl({
workspace,
datatableName: datatable,
requestBody: { target, change: pending.change, statements: pending.statements }
})
sendUserToast(pending.title)
pending = undefined
await acl.refetch()
} catch (e) {
sendUserToast(errorText(e), true)
} finally {
applying = false
}
}
</script>
{#if acl.error}
<Alert type="error" title="Could not read access" size="xs">{errorText(acl.error)}</Alert>
{:else if !info}
<span class="text-xs text-secondary">Loading…</span>
{:else}
<div class="flex flex-col gap-4">
{#if !info.editable}
<span class="text-xs text-secondary">
Read only: access is changed by the admins of the workspace that governs this data table, on
Windmill Enterprise Edition.
</span>
{/if}
{#if target.kind !== 'database'}
<section class="flex flex-col gap-2">
<div class="flex flex-col gap-0.5">
<span class="text-xs font-semibold text-emphasis">Owner</span>
<span class="text-xs text-secondary">
{target.kind === 'schema'
? 'The role that owns the schema and everything already in it, except what belongs to an extension, which stays with the extension. Changing it also keeps the new owner in reach of what the current roles create here from then on; a role added afterwards is not covered.'
: 'The role that owns the table. Its owner may always read and write it, and is who ALTER and DROP answer to.'}
</span>
</div>
{#if info.editable}
<Select
items={ownerItems}
disabled={planning || applying}
size="sm"
class="w-64"
bind:value={
() => info.owner,
(role) => {
// The select shows what the database says; a pick is a request, and only the
// applied change moves it.
if (role && role !== info.owner) {
confirm({ type: 'set_owner', role }, `Ownership transferred to ${role}`)
}
}
}
/>
{:else}
<span class="font-mono text-xs">{info.owner}</span>
{/if}
</section>
{/if}
<section class="flex flex-col gap-2">
<div class="flex flex-col gap-0.5">
<span class="text-xs font-semibold text-emphasis">Grants</span>
<span class="text-xs text-secondary">
{target.kind === 'database'
? 'What each role may do on the database itself — CREATE is the right to create schemas in it — and what default privileges set database-wide give it on what is created later, in every schema. No schema can take those back.'
: 'What each role may do here, beyond what it owns.'}
</span>
</div>
{#if info.editable}
<PgGrantBuilder
{target}
roles={info.roles}
disabled={planning || applying}
supportsMaintain={info.supports_maintain}
dbname={info.dbname}
onAdd={({ role, privileges, scope }) =>
confirm(
{ type: 'grant', role, privileges, scope },
`Granted ${privileges.join(', ')} to ${role}`
)}
/>
{/if}
{#if grantRows.length === 0}
<span class="text-xs text-secondary">No grants yet.</span>
{:else}
<DataTable size="xs">
<Head>
<tr>
<Cell head first>Role</Cell>
<Cell head>Privileges</Cell>
<Cell head>On</Cell>
<Cell head last></Cell>
</tr>
</Head>
<tbody class="divide-y">
{#each grantRows as grant (grantKey(grant))}
{@const revokeScope = revokeScopeOf(grant)}
{@const revocable = revocablePrivileges(grant, target)}
{@const blocked = blockingSources(grant, revocable)}
{@const uncovered = uncoveredCreators(grant, info.roles)}
<Row>
<Cell first>{grant.grantee}</Cell>
<Cell wrap
><span class="font-mono text-2xs">{grant.privileges.join(', ')}</span></Cell
>
<Cell wrap>
{grantScopeLabel(grant)}
{#if blocked.length > 0}
<span
class="text-2xs text-secondary"
title="Only this role can take the grant back: Postgres revokes a grant through the role that made it"
>
from {blocked.join(', ')}
</span>
{/if}
{#if uncovered.length > 0}
<span
class="text-2xs text-secondary"
title="A default privilege covers only the roles it was granted for: grant it again to cover these"
>
· not for what {uncovered.join(', ')}
{uncovered.length === 1 ? 'creates' : 'create'}
</span>
{/if}
</Cell>
<Cell last>
<!-- What `admin` holds is what every role here connects through, so it is not
this editor's to take away. -->
{#if info.editable && revokeScope && revocable.length > 0 && blocked.length === 0 && info.roles.includes(grant.grantee) && grant.grantee !== ADMIN_ROLE}
<Button
unifiedSize="xs"
variant="subtle"
iconOnly
startIcon={{ icon: Trash2 }}
title="Revoke {revocable.join(', ')}"
disabled={planning || applying}
onClick={() =>
confirm(
{
type: 'revoke',
role: grant.grantee,
privileges: revocable,
scope: revokeScope,
objects: grant.objects
},
`Revoked ${revocable.join(', ')} from ${grant.grantee}`
)}
/>
{/if}
</Cell>
</Row>
{/each}
</tbody>
</DataTable>
{/if}
</section>
</div>
{/if}
<ConfirmationModal
open={!!pending}
title="Run the following?"
confirmationText="Run"
type="info"
alwaysPortal
loading={applying}
onConfirmed={apply}
onCanceled={() => (pending = undefined)}
>
<div class="flex flex-col gap-3 min-w-0">
{#if pendingCoversObjects}
<Alert type="info" title="This covers every listed object" size="xs">
The same privileges on several objects read as one row, and are revoked together.
</Alert>
{/if}
{#each pending?.warnings ?? [] as warning (warning)}
<Alert type="warning" title="Warning" size="xs">{warning}</Alert>
{/each}
<span class="text-sm text-secondary">
Runs against <span class="font-mono">{datatable}</span> in a single transaction:
</span>
<pre class="overflow-auto text-xs bg-surface-secondary p-3 rounded select-all max-h-80"
>{(pending?.statements ?? []).join(';\n')};</pre
>
</div>
</ConfirmationModal>
@@ -0,0 +1,95 @@
<script lang="ts">
import type { AclTarget } from '$lib/gen'
import { Plus } from 'lucide-svelte'
import { Button } from '../common'
import Select from '../select/Select.svelte'
import MultiSelect from '../select/MultiSelect.svelte'
import { privilegesOf, scopeSql, scopesOf, type AclScope } from './aclScopes'
let {
target,
roles,
supportsMaintain = false,
dbname,
disabled = false,
onAdd
}: {
target: AclTarget
/** Roles the grant can be handed to. */
roles: string[]
/** Postgres 17+, which has one more table privilege to offer. */
supportsMaintain?: boolean
/** Names the database in the statement a database target builds. */
dbname?: string
disabled?: boolean
onAdd: (grant: { role: string; privileges: string[]; scope: AclScope }) => void
} = $props()
let role = $state<string | undefined>(undefined)
let scope = $state<AclScope>('target')
let privileges = $state<string[]>([])
const available = $derived(privilegesOf(scope, target.kind, supportsMaintain))
const statement = $derived(
privileges.length && role
? `GRANT ${privileges.join(', ')} ON ${scopeSql(scope, target, dbname)} TO ${role}`
: undefined
)
</script>
<div class="flex flex-col gap-2 border rounded-md p-3">
<div class="flex flex-wrap items-center gap-2 text-xs text-secondary">
<span class="font-mono text-primary">GRANT</span>
<MultiSelect
bind:value={privileges}
items={available.map((p) => ({ value: p, label: p }))}
placeholder="privileges"
{disabled}
size="sm"
class="min-w-48"
/>
<span class="font-mono text-primary">ON</span>
<Select
bind:value={
() => scope,
(s) => {
if (!s) return
scope = s
// A privilege only exists for some objects — SELECT means nothing on a function —
// so drop what the new scope cannot carry rather than send it.
const allowed = privilegesOf(s, target.kind, supportsMaintain)
privileges = privileges.filter((p) => allowed.includes(p))
}
}
items={scopesOf(target.kind)}
{disabled}
size="sm"
class="w-52"
/>
<span class="font-mono text-primary">TO</span>
<Select
bind:value={role}
items={roles.map((r) => ({ value: r, label: r }))}
placeholder="role"
{disabled}
size="sm"
class="w-40"
/>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: Plus }}
disabled={disabled || !role || privileges.length === 0}
onClick={() => {
if (!role) return
onAdd({ role, privileges: [...privileges], scope })
privileges = []
}}
>
Add
</Button>
</div>
{#if statement}
<pre class="text-2xs text-tertiary overflow-x-auto">{statement}</pre>
{/if}
</div>
@@ -0,0 +1,178 @@
import { describe, expect, it } from 'vitest'
import type { AclGrant } from '$lib/gen'
import {
blockingSources,
grantKey,
groupGrants,
revocablePrivileges,
revokeScopeOf,
uncoveredCreators
} from './aclScopes'
const table = (name: string) => ({ name, kind: 'TABLE' })
const by = (role: string, privileges: string[], reachable = true) => ({
role,
privileges,
reachable
})
const byAdmin = (grant: Omit<AclGrant, 'sources'>): AclGrant => ({
...grant,
sources: [by('admin', grant.privileges)]
})
describe('grantKey', () => {
it('tells apart a table and a function of the same name', () => {
const row = (object: { name: string; kind: string; args?: string }) => ({
grantee: 'analytics',
privileges: ['SELECT'],
objects: [object],
sources: [by('admin', ['SELECT'])]
})
expect(grantKey(row(table('orders')))).not.toBe(
grantKey(row({ name: 'orders', kind: 'FUNCTION', args: '' }))
)
})
})
describe('groupGrants', () => {
// A row's revoke names every object in it, so a row must only hold what one revoke may take.
it('folds the same privileges on objects of one kind, and nothing else', () => {
const grants: AclGrant[] = [
{ grantee: 'analytics', privileges: ['SELECT'], object: table('orders') },
{ grantee: 'analytics', privileges: ['SELECT'], object: table('salaries') },
{ grantee: 'operator', privileges: ['SELECT'], object: table('orders') },
{ grantee: 'analytics', privileges: ['INSERT', 'SELECT'], object: table('events') },
{ grantee: 'analytics', privileges: ['SELECT'], object: { name: 's', kind: 'SEQUENCE' } },
{ grantee: 'analytics', privileges: ['SELECT'], future: 'TABLES' },
{ grantee: 'analytics', privileges: ['USAGE'] }
].map(byAdmin)
const rows = groupGrants(grants)
expect(rows.map((r) => [r.grantee, r.privileges, r.objects, r.future])).toEqual([
['analytics', ['SELECT'], [table('orders'), table('salaries')], undefined],
['operator', ['SELECT'], [table('orders')], undefined],
['analytics', ['INSERT', 'SELECT'], [table('events')], undefined],
['analytics', ['SELECT'], [{ name: 's', kind: 'SEQUENCE' }], undefined],
['analytics', ['SELECT'], [], 'TABLES'],
['analytics', ['USAGE'], [], undefined]
])
})
// A revoke takes the row back from every source, so the row must name them all, with what each
// gave.
it('keeps every source of the grants it folds', () => {
const grants: AclGrant[] = [
{
grantee: 'analytics',
privileges: ['SELECT'],
object: table('orders'),
sources: [by('admin', ['SELECT'])]
},
{
grantee: 'analytics',
privileges: ['SELECT'],
object: table('salaries'),
sources: [by('admin', ['SELECT']), by('operator', ['SELECT'])]
}
]
expect(groupGrants(grants)[0].sources).toEqual([
by('admin', ['SELECT']),
by('operator', ['SELECT'])
])
})
})
describe('revoke of a row', () => {
const row = (future?: string) => ({
grantee: 'analytics',
privileges: ['SELECT'],
objects: [],
future,
sources: [by('admin', ['SELECT'])]
})
it('takes back only what the editor may revoke on the database', () => {
const database = { ...row(), privileges: ['CONNECT', 'CREATE'] }
expect(revocablePrivileges(database, { kind: 'database' })).toEqual(['CREATE'])
expect(revocablePrivileges(database, { kind: 'schema', schema: 'public' })).toEqual([
'CONNECT',
'CREATE'
])
// Set database-wide, so not one a schema's revoke could take back either.
const schemasLater = { ...row('SCHEMAS'), privileges: ['CREATE'] }
expect(revocablePrivileges(schemasLater, { kind: 'database' })).toEqual([])
})
it('maps default privileges to their scope, and refuses the ones it has none for', () => {
expect(revokeScopeOf(row())).toBe('target')
expect(revokeScopeOf(row('TABLES'))).toBe('future_tables')
expect(revokeScopeOf(row('TYPES'))).toBeUndefined()
expect(revokeScopeOf({ ...row(), objects: [{ name: 'mood', kind: 'TYPE' }] })).toBeUndefined()
})
// Postgres takes a grant back only through its source: offering the revoke would promise what
// the plan then refuses. But only the sources of what is revoked count: the catalog's CONNECT
// on the database comes from its owner, out of reach, and must not hold back a CREATE the
// editor granted.
it('is held back only by a source out of reach for what it takes', () => {
const database = {
...row(),
privileges: ['CONNECT', 'CREATE'],
sources: [by('postgres', ['CONNECT'], false), by('admin', ['CREATE'])]
}
const revocable = revocablePrivileges(database, { kind: 'database' })
expect(blockingSources(database, revocable)).toEqual([])
expect(blockingSources(database, ['CONNECT'])).toEqual(['postgres'])
const partly = {
...row('TABLES'),
sources: [by('admin', ['SELECT']), by('postgres', ['SELECT'], false)]
}
expect(blockingSources(partly, ['SELECT'])).toEqual(['postgres'])
})
// Whether a grant can be taken back depends on its object, so a row folding several objects is
// only revocable if each of its grants is.
it('is held back by a source out of reach on any of the objects it folds', () => {
const grants: AclGrant[] = [
{
grantee: 'analytics',
privileges: ['SELECT'],
object: table('orders'),
sources: [by('admin', ['SELECT'])]
},
{
grantee: 'analytics',
privileges: ['SELECT'],
object: table('salaries'),
sources: [by('admin', ['SELECT'], false)]
}
]
const [folded] = groupGrants(grants)
expect(folded.objects).toHaveLength(2)
expect(blockingSources(folded, ['SELECT'])).toEqual(['admin'])
// Folding reads the grants, never rewrites them.
expect(grants[0].sources[0].reachable).toBe(true)
})
})
describe('uncoveredCreators', () => {
// A default privilege binds only the creating roles it was granted for: a role added since is
// left out until the grant is made again.
it('names the roles a created-later row leaves out', () => {
const future = {
grantee: 'analytics',
privileges: ['SELECT'],
objects: [],
future: 'TABLES',
sources: [by('admin', ['SELECT']), by('analytics', ['SELECT'])]
}
expect(uncoveredCreators(future, ['admin', 'analytics', 'late'])).toEqual(['late'])
expect(uncoveredCreators({ ...future, future: undefined }, ['late'])).toEqual([])
// Set by a role outside the catalog, it was never meant to cover the catalog's roles.
expect(
uncoveredCreators({ ...future, sources: [by('postgres', ['SELECT'], false)] }, [
'admin',
'late'
])
).toEqual([])
})
})
@@ -0,0 +1,211 @@
import type { AclGrant, AclSource, AclTarget } from '$lib/gen'
/** The role a data table connects as without roles — `custom_instance_user` in Postgres. */
export const ADMIN_ROLE = 'admin'
/** Privileges Postgres accepts per kind of object. Mirrors the whitelist the backend validates
* against a privilege missing here just cannot be built. */
/** `CREATE` on a database is the right to create schemas in it, and the only database privilege
* handed out here: `CONNECT` is managed with the instance's role catalog. */
export const DATABASE_PRIVILEGES = ['CREATE']
export const SCHEMA_PRIVILEGES = ['USAGE', 'CREATE']
export const TABLE_PRIVILEGES = [
'SELECT',
'INSERT',
'UPDATE',
'DELETE',
'TRUNCATE',
'REFERENCES',
'TRIGGER'
]
/** Postgres 17 and later only, so it is offered from what the server reports. */
export const MAINTAIN_PRIVILEGE = 'MAINTAIN'
export const SEQUENCE_PRIVILEGES = ['USAGE', 'SELECT', 'UPDATE']
export const FUNCTION_PRIVILEGES = ['EXECUTE']
export type AclScope =
| 'target'
| 'all_tables'
| 'all_sequences'
| 'all_functions'
| 'future_tables'
| 'future_sequences'
| 'future_functions'
export type AclTargetKind = AclTarget['kind']
/** The scopes a target can grant on, in the order the builder offers them. */
export function scopesOf(kind: AclTargetKind): { value: AclScope; label: string }[] {
if (kind === 'database') return [{ value: 'target', label: 'the database itself' }]
if (kind === 'table') return [{ value: 'target', label: 'this table' }]
return [
{ value: 'target', label: 'the schema itself' },
{ value: 'all_tables', label: 'all tables in it' },
{ value: 'all_sequences', label: 'all sequences in it' },
{ value: 'all_functions', label: 'all functions in it' },
{ value: 'future_tables', label: 'tables created later' },
{ value: 'future_sequences', label: 'sequences created later' },
{ value: 'future_functions', label: 'functions created later' }
]
}
export function privilegesOf(
scope: AclScope,
kind: AclTargetKind,
supportsMaintain = false
): string[] {
const tablePrivileges = supportsMaintain
? [...TABLE_PRIVILEGES, MAINTAIN_PRIVILEGE]
: TABLE_PRIVILEGES
switch (scope) {
case 'target':
if (kind === 'database') return DATABASE_PRIVILEGES
return kind === 'schema' ? SCHEMA_PRIVILEGES : tablePrivileges
case 'all_tables':
case 'future_tables':
return tablePrivileges
case 'all_sequences':
case 'future_sequences':
return SEQUENCE_PRIVILEGES
case 'all_functions':
case 'future_functions':
return FUNCTION_PRIVILEGES
}
}
/** What a statement built at this scope reads as, for the builder's own preview. */
export function scopeSql(scope: AclScope, target: AclTarget, dbname?: string): string {
if (target.kind === 'database') return `DATABASE ${dbname ?? ''}`.trim()
const schema = target.schema
switch (scope) {
case 'target':
return target.kind === 'schema' ? `SCHEMA ${schema}` : `TABLE ${schema}.${target.table}`
case 'all_tables':
return `ALL TABLES IN SCHEMA ${schema}`
case 'all_sequences':
return `ALL SEQUENCES IN SCHEMA ${schema}`
case 'all_functions':
return `ALL FUNCTIONS IN SCHEMA ${schema}`
case 'future_tables':
return `TABLES (default privileges in ${schema})`
case 'future_sequences':
return `SEQUENCES (default privileges in ${schema})`
case 'future_functions':
return `FUNCTIONS (default privileges in ${schema})`
}
}
/** One row of the grants table: the same privileges on several objects read as one line, since
* granting them per object is what `ON ALL TABLES` does. */
export type GroupedGrant = {
grantee: string
privileges: string[]
objects: NonNullable<AclGrant['object']>[]
future?: string
/** Every role the row's grants come from, each once, with what it gave. */
sources: AclSource[]
}
export function groupGrants(grants: AclGrant[]): GroupedGrant[] {
const rows: GroupedGrant[] = []
for (const grant of grants) {
const existing = grant.object
? rows.find(
(r) =>
r.grantee === grant.grantee &&
r.future === grant.future &&
r.objects[0]?.kind === grant.object?.kind &&
r.privileges.join() === grant.privileges.join()
)
: undefined
if (existing) {
existing.objects.push(grant.object!)
for (const source of grant.sources) {
const known = existing.sources.find((s) => s.role === source.role)
// Whether a role's grant can be taken back depends on the object it is on, so a row
// holds a source as reachable only if it is on every object the row folds.
if (known) {
known.reachable &&= source.reachable
known.privileges = [...new Set([...known.privileges, ...source.privileges])].sort()
} else {
existing.sources.push({ ...source, privileges: [...source.privileges] })
}
}
} else {
rows.push({
grantee: grant.grantee,
privileges: grant.privileges,
objects: grant.object ? [grant.object] : [],
future: grant.future,
sources: grant.sources.map((s) => ({ ...s, privileges: [...s.privileges] }))
})
}
}
return rows
}
/** The roles that gave some of `privileges` and that this data table's connection cannot act for.
* Only they can take those grants back, so a revoke of `privileges` is not offered. */
export function blockingSources(grant: GroupedGrant, privileges: string[]): string[] {
return grant.sources
.filter((s) => !s.reachable && s.privileges.some((p) => privileges.includes(p)))
.map((s) => s.role)
}
/** Which of `roles` a "created later" row granted for some of them does not cover. A default
* privilege binds only the creating roles it was granted for, so what the others create stays out
* of it. A row none of `roles` set the instance's own, say was never meant to cover them, and
* names none. */
export function uncoveredCreators(grant: GroupedGrant, roles: string[]): string[] {
if (!grant.future || !grant.sources.some((s) => roles.includes(s.role))) return []
return roles.filter((r) => !grant.sources.some((s) => s.role === r))
}
/** A row's identity. Two rows may share a grantee and an object name a table `orders` and a
* function `orders()` so the kind and the privileges are part of it too. */
export function grantKey(grant: GroupedGrant): string {
return [
grant.grantee,
grant.future ?? '',
grant.privileges.join(','),
...grant.objects.map((o) => `${o.kind}:${o.name}(${o.args ?? ''})`)
].join('|')
}
/** The scope a revoke of this row takes, or `undefined` when the builder cannot express it
* Postgres also records privileges on types, present and default, which nothing here grants and
* the API has no scope for. */
export function revokeScopeOf(grant: GroupedGrant): AclScope | undefined {
if (!grant.future) return grant.objects.some((o) => o.kind === 'TYPE') ? undefined : 'target'
const scope = `future_${grant.future.toLowerCase()}`
return (['future_tables', 'future_sequences', 'future_functions'] as const).find(
(s) => s === scope
)
}
/** The privileges of a row a revoke may take back. On the database that is `CREATE` alone:
* `CONNECT` belongs to the role catalog, which would grant it again, and `TEMPORARY` is not one
* the editor hands out a row holding only those has nothing to revoke here. A database's rows
* "created later" are default privileges set database-wide, which nothing here revokes. */
export function revocablePrivileges(grant: GroupedGrant, target: AclTarget): string[] {
if (target.kind === 'database' && grant.objects.length === 0) {
if (grant.future) return []
return grant.privileges.filter((p) => DATABASE_PRIVILEGES.includes(p))
}
return grant.privileges
}
/** How a row reads back: what it covers, in one phrase. */
export function grantScopeLabel(grant: GroupedGrant): string {
if (grant.future) return `${grant.future.toLowerCase()} created later`
if (grant.objects.length === 1) {
const object = grant.objects[0]
// A routine's arguments are part of what it is, so two of the same name would otherwise
// read as one row twice.
const args = object.args !== undefined ? `(${object.args})` : ''
return `${object.kind.toLowerCase()} ${object.name}${args}`
}
if (grant.objects.length > 1)
return `${grant.objects.length} ${grant.objects[0].kind.toLowerCase()}s`
return 'itself'
}
@@ -0,0 +1,60 @@
import { describe, test, expect } from 'vitest'
import { parseMigrationRole, withMigrationRole } from './datatableMigrationRole'
describe('parseMigrationRole', () => {
test('reads every spelling the server accepts from the leading comment block', () => {
for (const line of [
'-- role analyst',
'-- Role: analyst',
'-- role=analyst',
'-- role analyst;'
]) {
expect(parseMigrationRole(`\n${line}\nBEGIN;\nEND;`)).toEqual({
kind: 'role',
role: 'analyst'
})
}
})
test('an annotation below BEGIN is not one', () => {
expect(parseMigrationRole('BEGIN;\n-- role analyst\nEND;')).toEqual({ kind: 'none' })
})
test('a malformed attempt is an error, not the default', () => {
for (const line of [
'-- role based access below',
'-- role',
'-- role:',
'-- role an;alytics'
]) {
expect(parseMigrationRole(`${line}\nBEGIN;`)).toEqual({ kind: 'malformed', line })
}
})
test('comments that do not start with the word role are ignored', () => {
expect(parseMigrationRole('-- roles analyst\n-- rolex\nBEGIN;')).toEqual({ kind: 'none' })
})
})
describe('withMigrationRole', () => {
test('leads above BEGIN, so the server reads it', () => {
const out = withMigrationRole('BEGIN;\nSELECT 1;\nEND;', 'analyst')
expect(out).toBe('-- role analyst\nBEGIN;\nSELECT 1;\nEND;')
})
test('replaces any attempt rather than stacking, malformed ones included', () => {
const out = withMigrationRole(
'-- Role: auditor\n-- role oops no\n-- keep me\nBEGIN;',
'analyst'
)
expect(out).toBe('-- role analyst\n-- keep me\nBEGIN;')
})
test('undefined strips the annotation, so it runs as admin', () => {
expect(withMigrationRole('-- role analyst\n\nBEGIN;\nEND;', undefined)).toBe('BEGIN;\nEND;')
})
test('refuses a name the server would refuse', () => {
expect(() => withMigrationRole('BEGIN;', 'bad;name')).toThrow()
})
})
@@ -0,0 +1,70 @@
import { isDatatableRoleName } from './dbTypes'
/**
* A migration carries the data table role it runs as in its own SQL, as a `-- role <name>`
* annotation. There is no separate field: the annotation is what the server reads, and keeping
* it in the SQL is what lets it survive a `wmill sync` round-trip.
*
* Mirrors `SqlAnnotations::datatable_role` on the backend. It is only read from the leading
* comment block, so an annotation below `BEGIN;` is ignored and the migration runs as admin. A
* leading comment whose first word is `role` is an annotation attempt, and a malformed one is an
* error there, so it is one here too.
*/
export type MigrationRole =
| { kind: 'none' }
| { kind: 'role'; role: string }
| { kind: 'malformed'; line: string }
/** The body of a leading comment line that attempts a role annotation, or undefined. */
function roleAttempt(line: string): string | undefined {
if (!line.startsWith('--')) return undefined
const body = line.slice(2).trimStart()
if (body.slice(0, 4).toLowerCase() !== 'role') return undefined
const after = body.slice(4)
if (after !== '' && !/^[\s:=]/.test(after)) return undefined
return after
}
function parseAttempt(after: string): string | undefined {
let rest = after.trimStart()
if (rest.startsWith(':') || rest.startsWith('=')) rest = rest.slice(1)
const tokens = rest.split(/\s+/).filter((t) => t !== '')
if (tokens.length !== 1) return undefined
const role = tokens[0].endsWith(';') ? tokens[0].slice(0, -1) : tokens[0]
return isDatatableRoleName(role) ? role : undefined
}
export function parseMigrationRole(sql: string): MigrationRole {
for (const raw of sql.split('\n')) {
const line = raw.trim()
if (line === '') continue
if (!line.startsWith('--')) break
const after = roleAttempt(line)
if (after === undefined) continue
const role = parseAttempt(after)
return role === undefined ? { kind: 'malformed', line } : { kind: 'role', role }
}
return { kind: 'none' }
}
/**
* `sql` declaring `role`: any role annotation attempt in the leading comment block is removed,
* and `-- role <role>` is prepended above everything, or nothing when `role` is undefined.
*/
export function withMigrationRole(sql: string, role: string | undefined): string {
if (role !== undefined && !isDatatableRoleName(role)) {
throw new Error(`Invalid data table role '${role}'`)
}
const lines = sql.split('\n')
const kept: string[] = []
let i = 0
for (; i < lines.length; i++) {
const line = lines[i].trim()
if (line !== '' && !line.startsWith('--')) break
if (roleAttempt(line) === undefined) kept.push(lines[i])
}
const rest = [...kept, ...lines.slice(i)]
while (rest.length > 0 && rest[0].trim() === '') rest.shift()
return role === undefined ? rest.join('\n') : [`-- role ${role}`, ...rest].join('\n')
}
@@ -0,0 +1,44 @@
import { OpenAPI, type ListUsableDatatableRolesResponse } from '$lib/gen'
import { request } from '$lib/gen/core/request'
import { isCloudHosted } from '$lib/cloud'
import { ADMIN_DATATABLE_ROLE } from './dbTypes'
// `datatable_roles_unavailable` on the server, which is a plain 400: rewording it there without
// here makes every role picker on a non-Enterprise build fail instead of reading "not under roles".
const ROLES_UNAVAILABLE = 'Data table roles are a Windmill Enterprise Edition feature'
const NOT_UNDER_ROLES: ListUsableDatatableRolesResponse = {
permissioned: false,
roles: [],
default_role: ADMIN_DATATABLE_ROLE
}
/**
* The roles the caller may connect as on a data table. Cloud has no instance database, so no data
* table there is under roles, and none of the role pickers show. Without the Enterprise Edition
* every roles route refuses, which reads the same way: the data table is then used the way it was
* before roles, and one that is under roles is refused when something connects to it.
*/
export async function listUsableDatatableRoles(
workspace: string,
datatableName: string
): Promise<ListUsableDatatableRolesResponse> {
if (isCloudHosted()) return NOT_UNDER_ROLES
try {
// The generated client encodes path params with `encodeURI`, which leaves a '?' in a
// data table name created before names were restricted to cut the path short.
return await request(
{ ...OpenAPI, ENCODE_PATH: encodeURIComponent },
{
method: 'GET',
url: '/w/{workspace}/workspaces/datatable_usable_roles/{datatable_name}',
path: { workspace, datatable_name: datatableName }
}
)
} catch (e) {
const body = (e as { body?: unknown })?.body
const detail = `${typeof body === 'string' ? body : JSON.stringify(body ?? '')} ${(e as Error)?.message ?? e}`
if (detail.includes(ROLES_UNAVAILABLE)) return NOT_UNDER_ROLES
throw e
}
}
@@ -5,7 +5,7 @@ import { isDbType } from './dbTypes'
/**
* Single URL param `dbm` encodes the full DB manager state:
* firstSegment~path~schema.table
* firstSegment~path~schema.table~role=name
*
* firstSegment:
* datatable database with datatable:// resource (resourceType always postgresql)
@@ -26,28 +26,46 @@ import { isDbType } from './dbTypes'
* datatable~main~.customers (schema "public" implied)
* ducklake~main~.orders (schema "main" implied)
* postgresql~$res:u/user/my_pg~public.customers
* datatable~main~.customers~role=analyst
* datatable~main~role=analyst (no schema/table selected)
*
* role=name (last segment, optional, data tables only): the data table role to connect as.
* Omitted means the data table's default role. A trailing segment starting with `role=` is always
* the role, whatever follows. The name is kept as written, even when invalid (a `.` included), so
* the connection refuses it visibly instead of falling back to the default.
*/
const dbManagerSchema = z.object({
dbm: z.string().nullable()
})
interface ParsedDbm {
export interface ParsedDbm {
type: 'database' | 'datatable' | 'ducklake'
path: string
resType?: string
schema?: string
table?: string
role?: string
}
function parseDbm(raw: unknown): ParsedDbm | null {
const ROLE_SEGMENT_PREFIX = 'role='
function isRoleSegment(segment: string | undefined): segment is string {
return !!segment && segment.startsWith(ROLE_SEGMENT_PREFIX)
}
export function parseDbm(raw: unknown): ParsedDbm | null {
if (!raw || typeof raw !== 'string') return null
const parts = raw.split('~')
if (parts.length < 2 || !parts[1]) return null
const firstSeg = parts[0]
const path = parts[1]
const schemaTable = parts[2] ?? ''
const rest = parts.slice(2)
const role = isRoleSegment(rest.at(-1))
? rest.pop()!.slice(ROLE_SEGMENT_PREFIX.length)
: undefined
const schemaTable = rest[0] ?? ''
let type: ParsedDbm['type']
let resType: string | undefined
@@ -81,12 +99,12 @@ function parseDbm(raw: unknown): ParsedDbm | null {
schema = defaultSchemas[type]
}
return { type, path, resType, schema, table }
return { type, path, resType, schema, table, role: type === 'datatable' ? role : undefined }
}
const defaultSchemas: Record<string, string> = { datatable: 'public', ducklake: 'main' }
function buildDbm(p: ParsedDbm): string {
export function buildDbm(p: ParsedDbm): string {
const firstSeg = p.type === 'database' ? p.resType! : p.type
const schema = p.schema === defaultSchemas[p.type] ? undefined : p.schema
let schemaTable = ''
@@ -97,7 +115,12 @@ function buildDbm(p: ParsedDbm): string {
} else if (schema) {
schemaTable = `${schema}.`
}
return schemaTable ? `${firstSeg}~${p.path}~${schemaTable}` : `${firstSeg}~${p.path}`
const segments = [firstSeg, p.path]
if (schemaTable) segments.push(schemaTable)
if (p.type === 'datatable' && p.role !== undefined) {
segments.push(`${ROLE_SEGMENT_PREFIX}${p.role}`)
}
return segments.join('~')
}
export interface DbManagerUriState {
@@ -105,6 +128,8 @@ export interface DbManagerUriState {
readonly effectiveInput: DbInput | undefined
readonly isDatatableInput: boolean
selectedDatatable: string | undefined
/** The data table role the drawer connects as; undefined means its default. */
selectedRole: string | undefined
selectedSchema: string | undefined
selectedTable: string | undefined
readonly open: boolean
@@ -137,6 +162,7 @@ export function useDbManagerUriState(): DbManagerUriState {
type: 'database' as const,
resourceType: resType as DbType,
resourcePath: parsed.type === 'datatable' ? `datatable://${parsed.path}` : parsed.path,
role: parsed.role,
specificSchema: parsed.schema,
specificTable: parsed.table
}
@@ -163,6 +189,7 @@ export function useDbManagerUriState(): DbManagerUriState {
type: isDatatable ? 'datatable' : 'database',
path: isDatatable ? nInput.resourcePath.slice('datatable://'.length) : nInput.resourcePath,
resType: isDatatable ? undefined : nInput.resourceType,
role: isDatatable ? nInput.role : undefined,
schema: nInput.specificSchema,
table: nInput.specificTable
})
@@ -194,7 +221,14 @@ export function useDbManagerUriState(): DbManagerUriState {
return parsed?.type === 'datatable' ? parsed.path : undefined
},
set selectedDatatable(v: string | undefined) {
if (v) updateField({ path: v })
// A role belongs to one data table, so it cannot carry over to another.
if (v) updateField({ path: v, role: undefined })
},
get selectedRole() {
return parsed?.role
},
set selectedRole(v: string | undefined) {
updateField({ role: v })
},
get selectedSchema() {
return parsed?.schema
@@ -0,0 +1,67 @@
import { describe, expect, it } from 'vitest'
import { buildDbm, parseDbm } from './dbManagerDrawerModel.svelte'
import { schemaCacheKey } from './dbSchemaCache'
import { datatableReference, type DbInput } from './dbTypes'
describe('dbm role segment', () => {
it('round-trips a role, with and without a table', () => {
for (const dbm of ['datatable~main~.orders~role=p4_analytics', 'datatable~main~role=p4-op']) {
expect(buildDbm(parseDbm(dbm)!)).toBe(dbm)
}
expect(parseDbm('datatable~main~sales.orders~role=analyst')).toMatchObject({
path: 'main',
schema: 'sales',
table: 'orders',
role: 'analyst'
})
})
it('reads a link without a role as the default role', () => {
const parsed = parseDbm('datatable~main~.orders')!
expect(parsed.role).toBeUndefined()
expect(parsed).toMatchObject({ schema: 'public', table: 'orders' })
expect(buildDbm(parsed)).toBe('datatable~main~.orders')
})
it('keeps an invalid role as written, so the connection refuses it', () => {
expect(parseDbm('datatable~main~role=a;b')?.role).toBe('a;b')
// A dot does not turn it into a schema.table selection read as the default role.
expect(parseDbm('datatable~main~role=bad.name')).toMatchObject({
role: 'bad.name',
schema: undefined,
table: undefined
})
})
})
describe('connecting as a role', () => {
const input = (role?: string): DbInput => ({
type: 'database',
resourceType: 'postgresql',
resourcePath: 'datatable://main',
role
})
// What `getDatabaseArg` builds every DB manager connection from.
it('appends the role to the data table reference', () => {
expect(datatableReference('main', 'p4_analytics')).toBe('datatable://main?role=p4_analytics')
expect(datatableReference('main', undefined)).toBe('datatable://main')
})
it('never appends a role to a name containing ?', () => {
// The server reads such a whole reference as the stored name first, so `?role=` would
// be taken as part of the name or refused instead of picking the role.
expect(datatableReference('legacy?x', undefined)).toBe('datatable://legacy?x')
expect(() => datatableReference('sales?role=analytics', 'admin')).toThrow(/'\?' in its name/)
})
it('refuses a role name the server would not accept', () => {
expect(() => datatableReference('main', 'a&role=admin')).toThrow(/Invalid data table role/)
expect(() => datatableReference('main', '')).toThrow(/Invalid data table role/)
})
it('keys the schema cache by role', () => {
expect(schemaCacheKey('ws', input('a'))).not.toBe(schemaCacheKey('ws', input('b')))
expect(schemaCacheKey('ws', input('a'))).not.toBe(schemaCacheKey('ws', input()))
})
})
+32 -14
View File
@@ -8,7 +8,8 @@ import { runScriptAndPollResult } from './jobs/utils'
import { writingJobOptions } from './jobs/writingJob'
import type { DBSchema, SQLSchema } from '$lib/stores'
import { stringifySchema } from './copilot/lib'
import type { DbInput, DbType } from './dbTypes'
import { datatableReference, type DbInput, type DbType } from './dbTypes'
import { withMigrationRole } from './datatableMigrationRole'
import { assert } from '$lib/utils'
import { WorkspaceService } from '$lib/gen'
import { pendingMigrations } from './workspaceSettings/datatableMigrationUtils'
@@ -70,7 +71,9 @@ export function dbTableOpsWithPreviewScripts({
}): IDbTableOps {
const dbType = getDbType(input)
const language = getLanguageByResourceType(dbType)
const dbArg = getDatabaseArg(input)
// Built per call: an invalid role throws there, as that operation's error, rather than while
// the manager renders.
const dbArg = () => getDatabaseArg(input)
const ducklake = input.type === 'ducklake' ? input.ducklake : undefined
function makeMarker(op: string, payload: Record<string, unknown>): string {
@@ -91,7 +94,7 @@ export function dbTableOpsWithPreviewScripts({
})
const result = await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg, quicksearch }, language, content, tag }
requestBody: { args: { ...dbArg(), quicksearch }, language, content, tag }
})
const count = result?.[0].count as number
return count
@@ -106,7 +109,7 @@ export function dbTableOpsWithPreviewScripts({
})
let items = (await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg, ...params }, language, content, tag }
requestBody: { args: { ...dbArg(), ...params }, language, content, tag }
})) as unknown[]
if (!items || !Array.isArray(items)) {
throw 'items is not an array'
@@ -123,7 +126,7 @@ export function dbTableOpsWithPreviewScripts({
{
workspace,
requestBody: {
args: { ...dbArg, value_to_update: newValue, ...values },
args: { ...dbArg(), value_to_update: newValue, ...values },
language,
content,
tag
@@ -135,14 +138,14 @@ export function dbTableOpsWithPreviewScripts({
onDelete: async ({ values }) => {
const content = makeMarker('DELETE', { table: tableKey, columns: colDefs })
await runScriptAndPollResult(
{ workspace, requestBody: { args: { ...dbArg, ...values }, language, content, tag } },
{ workspace, requestBody: { args: { ...dbArg(), ...values }, language, content, tag } },
writingJobOptions
)
},
onInsert: async ({ values }) => {
const content = makeMarker('INSERT', { table: tableKey, columns: colDefs })
await runScriptAndPollResult(
{ workspace, requestBody: { args: { ...dbArg, ...values }, language, content, tag } },
{ workspace, requestBody: { args: { ...dbArg(), ...values }, language, content, tag } },
writingJobOptions
)
}
@@ -246,6 +249,7 @@ export type IDbSchemaOps = {
previewAlterSql: (params: { values: AlterTableValues; schema?: string }) => Promise<string>
onCreateSchema: (params: { schema: string }) => Promise<void>
onDeleteSchema: (params: { schema: string }) => Promise<void>
onRenameSchema: (params: { schema: string; newSchema: string }) => Promise<void>
onFetchTableEditorDefinition: (params: {
table: string
schema?: string
@@ -283,7 +287,8 @@ export function dbSchemaOpsWithPreviewScripts({
tag?: string
}): IDbSchemaOps {
const dbType = getDbType(input)
const dbArg = getDatabaseArg(input)
// Built per call, for the same reason as in the table ops above.
const dbArg = () => getDatabaseArg(input)
const language = getLanguageByResourceType(dbType)
const ducklake = input.type === 'ducklake' ? input.ducklake : undefined
@@ -293,6 +298,8 @@ export function dbSchemaOpsWithPreviewScripts({
input.type === 'database' && input.resourcePath.startsWith('datatable://')
? input.resourcePath.slice('datatable://'.length)
: undefined
// A migration declaring no role runs as admin, whatever role the manager connects as.
const migrationRole = input.type === 'database' ? (input.role ?? input.migrationRole) : undefined
function makeMarker(op: string, payload: Record<string, unknown>): string {
if (ducklake) payload.ducklake = ducklake
@@ -359,7 +366,7 @@ export function dbSchemaOpsWithPreviewScripts({
: undefined
if (!datatableName || !status?.enabled) {
await runScriptAndPollResult(
{ workspace, requestBody: { args: dbArg, content, language, tag } },
{ workspace, requestBody: { args: dbArg(), content, language, tag } },
writingJobOptions
)
return
@@ -373,12 +380,16 @@ export function dbSchemaOpsWithPreviewScripts({
throw new MigrationRunCancelled()
}
}
const codeUp = wrapMigration(await expandMarker(workspace, language, content))
// Wrapped before annotating: the annotation must lead, above `BEGIN;`.
const codeUp = withMigrationRole(
wrapMigration(await expandMarker(workspace, language, content)),
migrationRole
)
// Down migrations are only generated for Postgres for now.
let codeDown: string | undefined
if (downContent && dbType === 'postgresql') {
const downSql = (await expandMarker(workspace, language, downContent)).trim()
if (downSql) codeDown = wrapMigration(downSql)
if (downSql) codeDown = withMigrationRole(wrapMigration(downSql), migrationRole)
}
const created = await WorkspaceService.createDatatableMigration({
workspace,
@@ -415,7 +426,7 @@ export function dbSchemaOpsWithPreviewScripts({
const fkContent = makeMarker('FOREIGN_KEYS', { table, schema })
const fkResult = await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content: fkContent, language, tag }
requestBody: { args: dbArg(), content: fkContent, language, tag }
})
let rawForeignKeys: RawForeignKey[]
@@ -501,6 +512,11 @@ export function dbSchemaOpsWithPreviewScripts({
const downContent = makeMarker('CREATE_SCHEMA', { schema })
await applyDdl(migrationName('drop_schema', schema), content, downContent)
},
onRenameSchema: async ({ schema, newSchema }) => {
const content = makeMarker('RENAME_SCHEMA', { schema, new_schema: newSchema })
const downContent = makeMarker('RENAME_SCHEMA', { schema: newSchema, new_schema: schema })
await applyDdl(migrationName('rename_schema', schema), content, downContent)
},
onFetchForeignKeys: fetchForeignKeys,
onFetchTableEditorDefinition: async ({ table, schema, colDefs }) => {
const foreignKeys = await fetchForeignKeys({ table, schema })
@@ -512,7 +528,7 @@ export function dbSchemaOpsWithPreviewScripts({
const pkContent = makeMarker('PRIMARY_KEY_CONSTRAINT', { table, schema })
const pkResult = (await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content: pkContent, language, tag }
requestBody: { args: dbArg(), content: pkContent, language, tag }
})) as { constraint_name?: string; CONSTRAINT_NAME?: string }[]
if (pkResult && Array.isArray(pkResult) && pkResult.length > 0) {
@@ -611,7 +627,9 @@ export function getDefaultDbTag(input: DbInput): string {
export function getDatabaseArg(input: DbInput | undefined) {
if (input?.type === 'database') {
if (input.resourcePath.startsWith('datatable://')) {
return { database: input.resourcePath }
return {
database: datatableReference(input.resourcePath.slice('datatable://'.length), input.role)
}
} else {
return { database: '$res:' + input.resourcePath }
}
@@ -0,0 +1,21 @@
import type { DbInput } from './dbTypes'
/** What identifies a database's schema, role included: two roles on one data table may reach
* different schemas, so they cannot share a cache entry. Never throws, since it keys derived
* state; the connection itself is what refuses an invalid role. */
export function getDbSchemasPath(input: DbInput): string {
switch (input.type) {
case 'database':
return input.role !== undefined && input.resourcePath.startsWith('datatable://')
? `${input.resourcePath}?role=${input.role}`
: input.resourcePath
case 'ducklake':
return 'ducklake://' + input.ducklake
}
}
/** Scoped by the acting workspace: a data table of the same name can exist in both the nav and
* the acting workspace, and one's schema must not be reused for the other. */
export function schemaCacheKey(workspace: string | undefined, input: DbInput): string {
return `${workspace}:${getDbSchemasPath(input)}`
}
+50
View File
@@ -3,6 +3,12 @@ export type DbInput =
type: 'database'
resourceType: DbType
resourcePath: string
/** The data table role to connect as; the data table's default when unset. Only
* meaningful for a `datatable://` path. */
role?: string
/** The role migrations written through this input declare when `role` is unset. A
* migration declaring none runs as admin, not as the role the manager connects as. */
migrationRole?: string
specificSchema?: string
specificTable?: string
}
@@ -23,3 +29,47 @@ export const dbTypes = [
'duckdb'
] as const
export const isDbType = (str?: string): str is DbType => !!str && dbTypes.includes(str as DbType)
/** The role every data table has: the one it connects as when it is not under roles. */
export const ADMIN_DATATABLE_ROLE = 'admin'
/** What the server accepts in `-- role <name>` and `?role=<name>`. */
export function isDatatableRoleName(name: string): boolean {
return /^[A-Za-z0-9_-]{1,63}$/.test(name)
}
/** Whether a role can be named in a reference to this data table. A name stored before names were
* restricted may contain `?`, and the server reads such a whole reference as that name first, so
* `?role=` after it would be taken as part of the name or refused. */
export function datatableNameTakesRole(name: string): boolean {
return !name.includes('?')
}
/** The `migrationRole` of a data table that cannot name a role in its reference: it connects as
* its default role, which its migrations must then declare. */
export function defaultMigrationRole(
name: string,
permissioned: boolean | undefined,
defaultRole: string | undefined
): string | undefined {
return permissioned && !datatableNameTakesRole(name) ? defaultRole : undefined
}
/** `datatable://<name>`, with `?role=<role>` when a role is named. Throws rather than build a
* reference the executor would refuse, or one that would silently mean another role. */
export function datatableReference(name: string, role: string | undefined): string {
if (role === undefined) return `datatable://${name}`
if (!isDatatableRoleName(role)) {
throw new Error(
`Invalid data table role '${role}': only letters, digits, '_' and '-' are allowed`
)
}
if (!datatableNameTakesRole(name)) {
throw new Error(
`Data table '${name}' has a '?' in its name, so it can only be used here as its default role. Rename it to connect as role '${role}'.`
)
}
return `datatable://${name}?role=${role}`
}
export type DatatableRowAction = 'migrations' | 'roles' | 'export' | 'import'
@@ -4,13 +4,14 @@
import Select from '$lib/components/select/Select.svelte'
import { workspaceStore } from '$lib/stores'
import {
createDatatableAccessResource,
createDatatablesResource,
createSchemasResource,
toDatatableItems,
toSchemaItems
} from './datatableUtils.svelte'
import { Button } from '../common'
import { getRawAppOperatingWorkspace } from './rawAppWorkspace'
import { appDatatableRole } from './dataTableRefUtils'
const getOpWs = getRawAppOperatingWorkspace()
let opWs = $derived(getOpWs?.() ?? $workspaceStore)
@@ -20,6 +21,8 @@
datatable: string | undefined
/** Currently selected schema */
schema: string | undefined
/** The role the app uses each data table through: schemas are listed as that role. */
roles?: Record<string, string>
/** Callback when either value changes */
onChange?: (datatable: string | undefined, schema: string | undefined) => void
/** Description text to show in the popover */
@@ -29,19 +32,28 @@
let {
datatable,
schema,
roles,
onChange,
description = 'Set the default datatable and schema for new tables. This is where AI will create new tables when needed.'
}: Props = $props()
const role = $derived(datatable ? appDatatableRole(roles, datatable) : undefined)
// Load available datatables and schemas using shared utilities
const datatables = createDatatablesResource(() => opWs)
const schemas = createSchemasResource(
const access = createDatatableAccessResource(
() => datatable,
() => role,
() => opWs
)
const datatableItems = $derived(toDatatableItems(datatables.current))
const schemaItems = $derived(toSchemaItems(schemas.current))
// Until the answer is for this data table and role, the schemas in hand belong to another.
const schemaItems = $derived(
access.current.datatable === datatable && access.current.role === role
? toSchemaItems(access.current.schemas)
: []
)
// Track datatable changes to reset schema
let previousDatatable = $state<string | undefined>(undefined)
@@ -82,6 +94,9 @@
placeholder="Select database"
size="sm"
/>
{#if role}
<span class="text-2xs text-tertiary">Used as role {role}</span>
{/if}
</div>
<div class="flex flex-col gap-1">
@@ -1,35 +1,51 @@
<script lang="ts">
import { workspaceStore } from '$lib/stores'
import { WorkspaceService } from '$lib/gen'
import { listUsableDatatableRoles } from '../datatableUsableRoles'
import Drawer from '../common/drawer/Drawer.svelte'
import DrawerContent from '../common/drawer/DrawerContent.svelte'
import Button from '../common/button/Button.svelte'
import Select from '../select/Select.svelte'
import { sendUserToast } from '$lib/toast'
import type { DataTableRef } from './dataTableRefUtils'
import { appDatatableRole, type DataTableRef } from './dataTableRefUtils'
import { untrack } from 'svelte'
import { resource } from 'runed'
import { ArrowLeft, Expand, LoaderCircle, Minimize, Plus, RefreshCcw } from 'lucide-svelte'
import { ArrowLeft, Expand, Minimize, Plus, RefreshCcw } from 'lucide-svelte'
import DBManagerContent from '../DBManagerContent.svelte'
import type { DbInput } from '../dbTypes'
import type { SelectedTable } from '../DBManager.svelte'
import {
ADMIN_DATATABLE_ROLE,
datatableNameTakesRole,
defaultMigrationRole,
type DbInput
} from '../dbTypes'
import type { PendingRowAction, SelectedTable } from '../DBManager.svelte'
import { getRawAppOperatingWorkspace } from './rawAppWorkspace'
import { useDbManagerTag } from '../dbManagerTag.svelte'
import DbWorkerTagButton from '../DbWorkerTagButton.svelte'
import type { DataTableTables } from '$lib/gen'
const getOpWs = getRawAppOperatingWorkspace()
let opWs = $derived(getOpWs?.() ?? $workspaceStore)
interface Props {
onAdd?: (ref: DataTableRef) => void
/** `roles` holds, for each added table's data table under roles, the role its tables were
* browsed as: the app uses the data table through it from then on. `roleChanged` names the
* data tables the app now uses through another role than before (its stored role, or the
* data table's default when it stored none): their existing refs are replaced, since the new
* role may not reach them. */
onAdd?: (refs: DataTableRef[], roles: Record<string, string>, roleChanged: Set<string>) => void
existingRefs?: DataTableRef[]
/** The role the app uses each data table through, by data table name */
roles?: Record<string, string>
/** Z-index offset for the drawer, useful when opening from within modals */
offset?: number
}
let { onAdd, existingRefs = [], offset = 0 }: Props = $props()
let { onAdd, existingRefs = [], roles = undefined, offset = 0 }: Props = $props()
let open = $state(false)
let selectedDatatable = $state<string | undefined>(undefined)
/** Role the manager connects as; undefined means the data table's default. */
let selectedRole = $state<string | undefined>(undefined)
// For DB manager
let dbManagerContent: DBManagerContent | undefined = $state()
@@ -39,10 +55,19 @@
// Multi-select mode: selected tables
let selectedTables = $state<SelectedTable[]>([])
/** The role each data table's selected tables were browsed as. */
let browsedRoles = $state<Record<string, string>>({})
// Survives the re-mount a data table switch causes.
let pendingAction = $state<PendingRowAction | undefined>(undefined)
// Selected schema/table from DBManager (for preview)
let selectedSchemaKey = $state<string | undefined>(undefined)
let selectedTableKey = $state<string | undefined>(undefined)
// What the manager opens on, set only when it (re-)mounts: the live selection above changes
// on every click, and feeding it to the input would reload the whole manager each time.
let openSchemaKey = $state<string | undefined>(undefined)
let openTableKey = $state<string | undefined>(undefined)
// Load available datatables from workspace
const datatables = resource<string[]>([], async () => {
@@ -55,32 +80,186 @@
}
})
export function openDrawer() {
// Auto-select first datatable if only one exists
if (datatables.current.length === 1) {
selectedDatatable = datatables.current[0]
} else if (datatables.current.length > 1 && datatables.current.includes('main')) {
selectedDatatable = 'main'
} else {
selectedDatatable = undefined
const usableRoles = resource(
() => [open, opWs, selectedDatatable] as const,
async ([isOpen, workspace, datatable]) => {
if (!isOpen || !workspace || !datatable) return undefined
try {
return {
datatable,
...(await listUsableDatatableRoles(workspace, datatable))
}
} catch (e) {
// Opens anyway: without a role the server connects as the default and says so if
// that is refused.
console.error('Failed to load datatable roles:', e)
return {
datatable,
permissioned: false,
roles: [] as string[],
default_role: ADMIN_DATATABLE_ROLE
}
}
}
)
// A resource keeps its previous value while it refetches, and roles are per data table.
const rolesOfCurrent = $derived(
usableRoles.current?.datatable === selectedDatatable ? usableRoles.current : undefined
)
// Mounting the manager fires its first queries, so it waits for the role: a round sent
// without one runs, and caches, as whatever the server defaults to.
const roleSettled = $derived(
selectedDatatable === undefined ||
(rolesOfCurrent !== undefined &&
(!rolesOfCurrent.permissioned ||
rolesOfCurrent.roles.length === 0 ||
selectedRole !== undefined ||
// Its reference cannot name a role, so it connects as the default one.
!datatableNameTakesRole(selectedDatatable)))
)
$effect(() => {
const current = rolesOfCurrent
if (
!current?.permissioned ||
selectedRole !== undefined ||
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))
)
return
const effective = current.roles.includes(current.default_role)
? current.default_role
: current.roles[0]
const datatable = selectedDatatable
if (effective && datatable) untrack(() => connectAs(datatable, effective))
})
// Every data table with its schemas and tables: the tree is the picker. The privileges it
// reports are the connected role's, so the role picked on the open data table is asked too.
// Waits for the role like the manager does, and drops an answer for a selection that has
// since changed: it would describe another role.
let datatableTreeRun = 0
const datatableTree = resource(
() => [open, opWs, selectedDatatable, selectedRole, roleSettled] as const,
async ([isOpen, workspace, roleFor, role, settled]): Promise<DataTableTables[]> => {
if (!isOpen || !workspace) return []
if (!settled) return untrack(() => datatableTree.current)
const run = ++datatableTreeRun
try {
const result = await WorkspaceService.listDataTableTables({
workspace,
roleFor: role ? roleFor : undefined,
role
})
return run === datatableTreeRun ? result : untrack(() => datatableTree.current)
} catch (e) {
console.error('Failed to load datatable tables:', e)
return run === datatableTreeRun ? [] : untrack(() => datatableTree.current)
}
},
{ initialValue: [] }
)
/** The role a table of `datatable` is seen through right now: the connected data table's
* picked role, or the default role the tree lists any other one as. */
function roleSeenFor(datatable: string): string | undefined {
if (!datatableNameTakesRole(datatable)) return undefined
if (datatable === selectedDatatable) {
return rolesOfCurrent?.permissioned ? selectedRole : undefined
}
const entry = datatableTree.current.find((t) => t.datatable_name === datatable)
return entry?.permissioned ? entry.default_role : undefined
}
function defaultRoleOf(datatable: string): string | undefined {
if (datatable === selectedDatatable && rolesOfCurrent?.permissioned) {
return rolesOfCurrent.default_role
}
const entry = datatableTree.current.find((t) => t.datatable_name === datatable)
return entry?.permissioned ? entry.default_role : undefined
}
const tableDatatable = (t: SelectedTable) => t.datatable ?? selectedDatatable
/** Stamps each newly selected table with the role it was seen through. A data table's
* selections all come from one role, since the app uses it through one: picking a table under
* another role drops the ones picked under the previous, which that role may not reach. */
function setSelectedTables(next: SelectedTable[]) {
const isNew = (t: SelectedTable) =>
!selectedTables.some(
(s) =>
tableDatatable(s) === tableDatatable(t) && s.schema === t.schema && s.table === t.table
)
const added = next.filter(isNew)
const nextRoles = { ...browsedRoles }
let kept = next
for (const table of added) {
const dt = tableDatatable(table)
if (!dt) continue
const role = roleSeenFor(dt)
if (role === undefined) continue
if (nextRoles[dt] !== undefined && nextRoles[dt] !== role) {
kept = kept.filter((s) => tableDatatable(s) !== dt || added.includes(s))
}
nextRoles[dt] = role
}
const stillSelected = new Set(kept.map(tableDatatable))
selectedTables = kept
browsedRoles = Object.fromEntries(
Object.entries(nextRoles).filter(([dt]) => stillSelected.has(dt))
)
}
function selectDatatable(datatable: string, role?: string) {
// A row clicked under another data table has just set the selection it should open on.
openSchemaKey = selectedSchemaKey
openTableKey = selectedTableKey
selectedDatatable = datatable
// A data table opens as the role its picked tables were browsed as, else the one the app
// already uses it through.
connectAs(
datatable,
datatableNameTakesRole(datatable)
? (role ?? browsedRoles[datatable] ?? appDatatableRole(roles, datatable))
: undefined
)
}
/** Connects to `datatable` as `role`. The tables picked on it under another role are dropped:
* they would be saved under a role other than the one on screen. */
function connectAs(datatable: string, role: string | undefined) {
selectedRole = role
const browsed = browsedRoles[datatable]
if (browsed !== undefined && role !== undefined && role !== browsed) {
selectedTables = selectedTables.filter((t) => (t.datatable ?? datatable) !== datatable)
const { [datatable]: _, ...rest } = browsedRoles
browsedRoles = rest
}
}
// Cleared before a data table is selected: a pick left unadded when the drawer last closed
// would otherwise decide the role it reopens as.
function resetSelection() {
selectedTables = []
browsedRoles = {}
selectedRole = undefined
}
export function openDrawer() {
resetSelection()
selectedSchemaKey = undefined
selectedTableKey = undefined
selectedTables = []
selectDatatable(datatables.current.includes('main') ? 'main' : datatables.current[0])
expand = false
open = true
}
let initialTableKey: string | undefined = $state<string | undefined>(undefined)
let initialSchemaKey: string | undefined = $state<string | undefined>(undefined)
export function openDrawerWithRef(ref: DataTableRef) {
selectedDatatable = ref.datatable
resetSelection()
selectedSchemaKey = ref.schema
selectedTableKey = ref.table
initialTableKey = ref.table
initialSchemaKey = ref.schema
selectedTables = []
selectDatatable(ref.datatable)
expand = false
open = true
}
@@ -88,49 +267,58 @@
export function closeDrawer() {
open = false
dbManagerContent?.clearReplResult()
// An action outlives the data table it was asked for otherwise.
pendingAction = undefined
}
function handleAddTables() {
if (!selectedDatatable) {
sendUserToast('Please select a data table first', true)
return
}
if (selectedTables.length === 0) {
sendUserToast('Please select at least one table', true)
return
}
// Add all selected tables
const refs: DataTableRef[] = []
for (const table of selectedTables) {
const ref: DataTableRef = {
datatable: selectedDatatable,
schema: table.schema,
table: table.table
}
onAdd?.(ref)
const datatable = table.datatable ?? selectedDatatable
if (!datatable) continue
refs.push({ datatable, schema: table.schema, table: table.table })
}
const added = new Set(refs.map((r) => r.datatable))
const addedRoles = Object.fromEntries(
Object.entries(browsedRoles).filter(([dt]) => added.has(dt))
)
const roleChanged = new Set(
Object.entries(addedRoles)
.filter(([dt, role]) => {
const usedAs = appDatatableRole(roles, dt) ?? defaultRoleOf(dt)
return usedAs !== undefined && usedAs !== role
})
.map(([dt]) => dt)
)
onAdd?.(refs, addedRoles, roleChanged)
const count = selectedTables.length
const count = refs.length
sendUserToast(`Added ${count} table${count > 1 ? 's' : ''} to app`)
selectedTables = []
browsedRoles = {}
}
const datatableItems = $derived(
datatables.current.map((dt) => ({
value: dt,
label: dt
}))
)
// Carries the picked schema/table, so a click on a row of another data table lands on that
// table once the manager re-mounts against it.
const dbInput: DbInput | undefined = $derived(
selectedDatatable
? {
type: 'database' as const,
resourceType: 'postgresql' as const,
resourcePath: `datatable://${selectedDatatable}`,
specificSchema: initialSchemaKey,
specificTable: initialTableKey
role: selectedRole,
migrationRole: defaultMigrationRole(
selectedDatatable,
rolesOfCurrent?.permissioned,
rolesOfCurrent?.default_role
),
specificSchema: openSchemaKey,
specificTable: openTableKey
}
: undefined
)
@@ -141,15 +329,13 @@
}
})
// Convert existingRefs to disabledTables format for the current datatable
const disabledTables = $derived(
existingRefs
.filter((ref) => ref.datatable === selectedDatatable && ref.schema && ref.table)
.map((ref) => ({ schema: ref.schema!, table: ref.table! }))
.filter((ref) => ref.schema && ref.table)
.map((ref) => ({ datatable: ref.datatable, schema: ref.schema!, table: ref.table! }))
)
// Can add: has tables selected
const canAdd = $derived(selectedDatatable && selectedTables.length > 0)
const canAdd = $derived(selectedTables.length > 0)
// Shares the drawer-set override with the Database Manager: same data table,
// same worker group needed to reach it.
@@ -175,37 +361,27 @@
noPadding
>
{#if dbInput && opWs}
{#key selectedDatatable}
<DBManagerContent
bind:this={dbManagerContent}
input={dbInput}
workspace={opWs}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey
bind:selectedTableKey
multiSelectMode={true}
bind:selectedTables
{disabledTables}
>
{#snippet dbSelector()}
{#if datatables.loading}
<div class="flex items-center gap-2 text-tertiary ml-2">
<LoaderCircle size={14} class="animate-spin" />
<span class="text-sm">Loading...</span>
</div>
{:else if datatables.current.length >= 1}
<Select
transformInputSelectedText={(s) => `Datatable: ${s}`}
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Select data table"
size="md"
/>
{/if}
{/snippet}
</DBManagerContent>
{/key}
{#if roleSettled}
{#key `${selectedDatatable}~${selectedRole ?? ''}`}
<DBManagerContent
bind:this={dbManagerContent}
input={dbInput}
workspace={opWs}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey
bind:selectedTableKey
multiSelectMode={true}
bind:selectedTables={() => selectedTables, setSelectedTables}
{disabledTables}
datatableTree={datatableTree.current}
datatableTreeLoading={datatableTree.loading}
onSelectDatatable={(dt) => selectDatatable(dt)}
onSelectRole={(dt, role) => selectDatatable(dt, role)}
bind:pendingAction
/>
{/key}
{/if}
{:else}
<div class="flex items-center justify-center h-full text-tertiary">
<span>Select a data table to explore</span>
@@ -214,12 +390,11 @@
{#snippet actions()}
<Button
variant="contained"
color="blue"
variant="accent"
disabled={!canAdd}
on:click={handleAddTables}
startIcon={{ icon: Plus }}
size="xs"
unifiedSize="sm"
>
{#if selectedTables.length > 0}
Add {selectedTables.length} table{selectedTables.length > 1 ? 's' : ''}
@@ -240,8 +415,8 @@
loading={dbManagerContent?.isLoading() ?? false}
on:click={() => dbManagerContent?.refresh()}
startIcon={{ icon: RefreshCcw }}
size="xs"
color="light"
unifiedSize="sm"
variant="default"
disabled={!selectedDatatable}
>
Refresh
@@ -250,8 +425,9 @@
<Button
on:click={() => (expand = !expand)}
startIcon={{ icon: expand ? Minimize : Expand }}
size="xs"
color="light"
unifiedSize="sm"
variant="default"
iconOnly
/>
{/snippet}
</DrawerContent>
@@ -16,6 +16,8 @@
defaultDatatable?: string | undefined
/** Default schema for new tables */
defaultSchema?: string | undefined
/** The role the app uses each data table through, by data table name */
roles?: Record<string, string>
onAdd?: () => void
onRemove?: (index: number) => void
onSelect?: (ref: DataTableRef, index: number) => void
@@ -31,6 +33,7 @@
dataTableRefs = [],
defaultDatatable = undefined,
defaultSchema = undefined,
roles = undefined,
onAdd,
onRemove,
onSelect,
@@ -95,6 +98,7 @@
<DefaultDatabaseSelector
datatable={defaultDatatable}
schema={defaultSchema}
{roles}
onChange={onDefaultChange}
/>
{/if}
@@ -124,6 +128,14 @@
<div class="flex items-center gap-1.5 px-1 py-1 text-2xs text-tertiary">
<Database size={12} class="shrink-0" />
<span class="font-medium truncate">{datatableName}</span>
{#if roles?.[datatableName]}
<span
class="truncate font-mono"
title="The app's queries on this data table run as this role"
>
as {roles[datatableName]}
</span>
{/if}
{#if isDefaultDatatable}
<span title="Default datatable">
<Star size={10} class="shrink-0 text-primary" />
@@ -70,8 +70,10 @@
formatDataTableRef,
isDatatableTableAllowed,
type RawAppData,
DEFAULT_DATA
DEFAULT_DATA,
appDatatableRole
} from './dataTableRefUtils'
import { datatableReference } from '../dbTypes'
import { randomUUID } from '$lib/utils/uuid'
interface Props {
@@ -706,11 +708,23 @@
runnables = update.runnables
}
if (update.data !== undefined) {
data = update.data
replaceData(update.data)
}
historyManager.manualSnapshot(files ?? {}, runnables, summary, data, true)
}
/** Replaces `data` from outside the editor (history, YAML). The policy sync writes the policy
* into `data`, so the policy takes the new values first or it puts the old ones straight back. */
function replaceData(next: RawAppData) {
data = next
aiChatManager.datatableCreationPolicy = {
...aiChatManager.datatableCreationPolicy,
datatable: next.datatable,
schema: next.schema,
roles: next.roles
}
}
let jobs: string[] = $state([])
let jobsById: Record<string, JobById> = $state({})
@@ -878,7 +892,8 @@
aiChatManager.datatableCreationPolicy = {
enabled: data.datatable !== undefined,
datatable: data.datatable,
schema: data.schema
schema: data.schema,
roles: data.roles
}
// Start auto-snapshot
@@ -900,9 +915,15 @@
// Read the current policy from aiChatManager
const policy = aiChatManager.datatableCreationPolicy
// Only update if different to avoid infinite loops
if (data.datatable !== policy.datatable || data.schema !== policy.schema) {
if (
data.datatable !== policy.datatable ||
data.schema !== policy.schema ||
// By value: the policy holds its own proxy of the same map.
JSON.stringify(data.roles) !== JSON.stringify(policy.roles)
) {
data.datatable = policy.datatable
data.schema = policy.schema
data.roles = policy.roles
}
})
@@ -1079,10 +1100,32 @@
return []
}
const tables = await WorkspaceService.listDataTableTables({
workspace: opWorkspace
// A data table the app uses through a role is listed as that role, so the AI sees
// what the app's own queries reach.
const workspace = opWorkspace
const tables = await WorkspaceService.listDataTableTables({ workspace })
// Only data tables that still exist: `data.roles` can outlive a removed or renamed one,
// and the server answers a `role_for` naming nothing with a 404.
const roled = Object.entries(data.roles ?? {}).filter(([dt]) =>
tables.some((t) => t.datatable_name === dt)
)
const roledTables = await Promise.all(
roled.map(([roleFor, role]) =>
WorkspaceService.listDataTableTables({
workspace,
datatableName: roleFor,
roleFor,
role
})
)
)
const merged = tables.map((entry) => {
const i = roled.findIndex(([dt]) => dt === entry.datatable_name)
return i === -1
? entry
: (roledTables[i].find((t) => t.datatable_name === entry.datatable_name) ?? entry)
})
return filterDatatableTables(tables)
return filterDatatableTables(merged)
},
getDatatableTableSchema: async (
datatableName: string,
@@ -1106,7 +1149,8 @@
workspace: opWorkspace,
datatableName,
schemaName,
tableName
tableName,
role: appDatatableRole(data.roles, datatableName)
})
return schema.columns
},
@@ -1124,13 +1168,15 @@
}
try {
// The same role the app's runnables use, so a table the AI creates belongs to it.
const role = appDatatableRole(data.roles, datatableName)
const result = await runScriptAndPollResult(
{
workspace: opWorkspace,
requestBody: {
language: 'postgresql',
content: sql,
args: { database: `datatable://${datatableName}` }
args: { database: datatableReference(datatableName, role) }
}
},
writingJobOptions
@@ -1150,6 +1196,12 @@
const resourcePath = `datatable://${datatableName}`
delete $dbSchemas[resourcePath]
delete $dbSchemas[`${opWorkspace}:${resourcePath}`]
// The DB manager keys its cache by the role it connected as too.
for (const key of Object.keys($dbSchemas)) {
if (key.startsWith(`${opWorkspace}:${resourcePath}?role=`)) {
delete $dbSchemas[key]
}
}
}
}
@@ -2137,7 +2189,7 @@
files = structuredClone($state.snapshot(entry.files))
runnables = structuredClone($state.snapshot(entry.runnables))
summary = entry.summary
data = structuredClone($state.snapshot(entry.data))
replaceData(structuredClone($state.snapshot(entry.data)))
// If the open document survives into the new files, use the combined message
if (iframeDocument && isOpenableDocument(iframeDocument)) {
@@ -2368,6 +2420,20 @@
schema
}
}}
datatableRoles={data.roles}
onDatatableRolesChange={(roles, roleChanged) => {
// The default schema was picked among what the previous role reaches: after the
// user moves the app's default data table to another role, it is picked again.
const dt = data.datatable
const schemaStale = dt !== undefined && roleChanged.has(dt)
data.roles = roles
if (schemaStale) data.schema = undefined
aiChatManager.datatableCreationPolicy = {
...aiChatManager.datatableCreationPolicy,
roles,
...(schemaStale && { schema: undefined })
}
}}
{runnables}
{modules}
{historyManager}
@@ -40,6 +40,13 @@
/** Default schema for new tables */
defaultSchema?: string | undefined
onDefaultChange?: (datatable: string | undefined, schema: string | undefined) => void
/** The role the app uses each data table through, by data table name */
datatableRoles?: Record<string, string>
/** `roleChanged` names the data tables now used through another role than before. */
onDatatableRolesChange?: (
roles: Record<string, string> | undefined,
roleChanged: Set<string>
) => void
}
let {
@@ -60,15 +67,27 @@
onDataTableRefsChange,
defaultDatatable = undefined,
defaultSchema = undefined,
onDefaultChange
onDefaultChange,
datatableRoles = undefined,
onDatatableRolesChange
}: Props = $props()
let dataTableDrawer: RawAppDataTableDrawer | undefined = $state()
let selectedDataTableIndex: number | undefined = $state(undefined)
let sharedUiDrawer: RawAppSharedUiDrawer | undefined = $state()
function handleAddDataTable(ref: DataTableRef) {
onDataTableRefsChange?.([...dataTableRefs, ref])
function handleAddDataTables(
refs: DataTableRef[],
browsedRoles: Record<string, string>,
roleChanged: Set<string>
) {
onDataTableRefsChange?.([
...dataTableRefs.filter((r) => !roleChanged.has(r.datatable)),
...refs
])
if (Object.keys(browsedRoles).length > 0) {
onDatatableRolesChange?.({ ...datatableRoles, ...browsedRoles }, roleChanged)
}
}
function handleRemoveDataTable(index: number) {
@@ -153,6 +172,7 @@
{dataTableRefs}
{defaultDatatable}
{defaultSchema}
roles={datatableRoles}
onAdd={() => dataTableDrawer?.openDrawer()}
onRemove={handleRemoveDataTable}
onSelect={handleSelectDataTable}
@@ -161,8 +181,9 @@
/>
<RawAppDataTableDrawer
bind:this={dataTableDrawer}
onAdd={handleAddDataTable}
onAdd={handleAddDataTables}
existingRefs={dataTableRefs}
roles={datatableRoles}
/>
<RawAppSharedUiDrawer bind:this={sharedUiDrawer} />
@@ -1,4 +1,5 @@
<script lang="ts">
import { untrack } from 'svelte'
import { Sparkles, Plus, List, Ban, ExternalLinkIcon, Loader2 } from 'lucide-svelte'
import type { Policy } from '$lib/gen'
import { superadmin, userStore, workspaceStore } from '$lib/stores'
@@ -19,13 +20,21 @@
import { loadCopilot } from '$lib/components/copilot/loadCopilot'
import { react18Template, react19Template, svelte5Template } from './templates'
import type { Runnable } from './rawAppPolicy'
import { type DataTableRef, type RawAppData, formatDataTableRef } from './dataTableRefUtils'
import {
type DataTableRef,
type RawAppData,
formatDataTableRef,
withAppDatatableRole
} from './dataTableRefUtils'
import {
createDatatableAccessResource,
createDatatablesResource,
createSchemasResource,
createRolesResource,
rolesWorthPicking,
toDatatableItems,
toSchemaItems
} from './datatableUtils.svelte'
import { datatableNameTakesRole, defaultMigrationRole } from '../dbTypes'
import RawAppDataTableList from './RawAppDataTableList.svelte'
import RawAppDataTableDrawer from './RawAppDataTableDrawer.svelte'
import FileEditorIcon from './FileEditorIcon.svelte'
@@ -62,19 +71,139 @@
let appSummary = $state('')
let initialPrompt = $state('')
let preWhitelistedTables = $state<DataTableRef[]>([])
/** The role each pre-whitelisted table's data table was browsed as. */
let preWhitelistedRoles = $state<Record<string, string>>({})
let dataTableDrawer: RawAppDataTableDrawer | undefined = $state()
const getOpWs = getRawAppOperatingWorkspace()
let opWs = $derived(getOpWs?.() ?? $workspaceStore)
const datatables = createDatatablesResource(() => opWs)
const schemas = createSchemasResource(
const roles = createRolesResource(
() => selectedDatatable,
() => opWs
)
let selectedRole = $state<string | undefined>(undefined)
// Every reader waits for an answer stamped with the current selection: until then `current`
// belongs to the previous data table or role.
// A data table whose name cannot carry a role in a reference is used as its default one.
const loadedRoles = $derived(
roles.current.datatable === selectedDatatable &&
selectedDatatable !== undefined &&
datatableNameTakesRole(selectedDatatable)
? roles.current.roles
: []
)
const showRolePicker = $derived(rolesWorthPicking(loadedRoles))
// Saved explicitly rather than left to resolve: "whatever the default is then" moves the app
// the day an admin changes the default.
const effectiveRole = $derived(
selectedRole !== undefined && loadedRoles.includes(selectedRole) ? selectedRole : undefined
)
// An app uses one role per data table: the one picked above is what the table drawer browses
// as and what the list shows, or tables would be added under a role the app is not saved with.
const pickerRoles = $derived(
selectedDatatable !== undefined && effectiveRole !== undefined
? withAppDatatableRole(preWhitelistedRoles, selectedDatatable, effectiveRole)
: preWhitelistedRoles
)
const availableDatatables = $derived(datatables.current)
const availableSchemas = $derived(schemas.current)
// `undefined` while the list loads, so this is false until it has answered.
const hasNoDatatables = $derived(availableDatatables?.length === 0)
const rolesSettled = $derived(
hasNoDatatables ||
(selectedDatatable !== undefined && roles.current.datatable === selectedDatatable)
)
// A role is picked on one data table: two data tables can both define an `analyst` that
// means something different, so a name surviving the switch is not the role surviving it.
let rolesPickedOn = $state<string | undefined>(undefined)
$effect(() => {
const loaded = roles.current
if (loaded.datatable !== selectedDatatable) return
const switched = untrack(() => rolesPickedOn) !== selectedDatatable
const current = untrack(() => selectedRole)
if (switched || current === undefined || !loaded.roles.includes(current)) {
// Tables already picked on this data table were browsed as a role: keep that one.
const browsed = selectedDatatable
? untrack(() => preWhitelistedRoles)[selectedDatatable]
: undefined
pickRole(
browsed !== undefined && loaded.roles.includes(browsed)
? browsed
: loaded.roles.includes(loaded.defaultRole)
? loaded.defaultRole
: loaded.roles[0]
)
rolesPickedOn = selectedDatatable
}
})
/** Picks the app's role on the selected data table. Tables picked on it under another role are
* dropped: that role may reach them where this one does not. */
function pickRole(role: string | undefined) {
selectedRole = role
const dt = selectedDatatable
const browsed = dt ? preWhitelistedRoles[dt] : undefined
if (dt === undefined || role === undefined || browsed === undefined || browsed === role) return
preWhitelistedTables = preWhitelistedTables.filter((t) => t.datatable !== dt)
const { [dt]: _, ...rest } = preWhitelistedRoles
preWhitelistedRoles = rest
}
const access = createDatatableAccessResource(
() => selectedDatatable,
() => effectiveRole,
() => opWs,
() => rolesSettled && (loadedRoles.length === 0 || effectiveRole !== undefined)
)
// Under roles, and this caller may use none of them: the app would be saved with queries the
// server refuses.
const noUsableRole = $derived(
rolesSettled &&
selectedDatatable !== undefined &&
roles.current.permissioned &&
roles.current.roles.length === 0
)
const rolesUnknown = $derived(rolesSettled && roles.current.failed)
const accessSettled = $derived(
hasNoDatatables ||
(rolesSettled &&
selectedDatatable !== undefined &&
access.current.datatable === selectedDatatable &&
access.current.role === effectiveRole)
)
const accessUnknown = $derived(accessSettled && access.current.failed)
const availableSchemas = $derived(accessSettled ? access.current.schemas : [])
const canCreateSchema = $derived(accessSettled && access.current.canCreateSchema)
// Only an app that keeps the data table is held back by it: with table creation off nothing
// saves it.
const blockedByRole = $derived(
(noUsableRole || rolesUnknown || accessUnknown) && tableCreationEnabled
)
// A role that cannot create schemas has nothing to name, so the mode goes back to the one
// every role has, once that is an answer.
$effect(() => {
if (accessSettled && !accessUnknown && schemaMode === 'new' && !canCreateSchema) {
schemaMode = 'none'
}
})
// Likewise an existing schema the current role no longer reaches is unpicked, so the select
// does not keep showing it.
$effect(() => {
if (
accessSettled &&
selectedSchema !== undefined &&
!availableSchemas.includes(selectedSchema)
) {
selectedSchema = undefined
}
})
let hasAutoSelected = false
$effect(() => {
@@ -115,12 +244,18 @@
const datatableItems = $derived(toDatatableItems(availableDatatables))
const schemaItems = $derived(toSchemaItems(availableSchemas))
// An existing schema counts only while the current role reaches it: one picked under another
// role would save an app that creates its tables where it cannot.
const effectiveSchema = $derived(
schemaMode === 'new' ? newSchemaName : schemaMode === 'existing' ? selectedSchema : undefined
schemaMode === 'new'
? newSchemaName
: schemaMode === 'existing' &&
selectedSchema !== undefined &&
availableSchemas.includes(selectedSchema)
? selectedSchema
: undefined
)
const hasNoDatatables = $derived(availableDatatables?.length === 0)
// copilotInfo is a global that stays empty until some ancestor's fetch lands, so
// `enabled` alone cannot tell "no providers" from "not loaded yet" and the modal
// would announce AI as unconfigured while it is merely unknown. Gate on the
@@ -140,7 +275,13 @@
async function start(withPrompt: boolean) {
const template = templates[selectedTemplateIndex]
if (schemaMode === 'new' && newSchemaName && selectedDatatable && opWs) {
if (
tableCreationEnabled &&
schemaMode === 'new' &&
newSchemaName &&
selectedDatatable &&
opWs
) {
try {
const { dbSchemaOpsWithPreviewScripts } = await import('$lib/components/dbOps')
const dbOps = dbSchemaOpsWithPreviewScripts({
@@ -148,7 +289,13 @@
input: {
type: 'database',
resourceType: 'postgresql',
resourcePath: `datatable://${selectedDatatable}`
resourcePath: `datatable://${selectedDatatable}`,
role: effectiveRole,
migrationRole: defaultMigrationRole(
selectedDatatable,
roles.current.permissioned,
roles.current.defaultRole
)
}
})
await dbOps.onCreateSchema({ schema: newSchemaName })
@@ -159,14 +306,20 @@
}
const formattedTables = preWhitelistedTables.map(formatDataTableRef)
const data: RawAppData =
tableCreationEnabled && selectedDatatable
? {
tables: formattedTables,
datatable: selectedDatatable,
schema: effectiveSchema
}
: { tables: formattedTables, datatable: undefined, schema: undefined }
const keepsDatatable = tableCreationEnabled && selectedDatatable !== undefined
// The roles shown, for the data tables the app ends up using.
const usedDatatables = new Set(preWhitelistedTables.map((t) => t.datatable))
if (keepsDatatable) usedDatatables.add(selectedDatatable!)
const shownRoles = Object.entries(pickerRoles ?? {}).filter(([dt]) => usedDatatables.has(dt))
const appRoles = shownRoles.length > 0 ? Object.fromEntries(shownRoles) : undefined
const data: RawAppData = keepsDatatable
? {
tables: formattedTables,
datatable: selectedDatatable,
schema: effectiveSchema,
roles: appRoles
}
: { tables: formattedTables, datatable: undefined, schema: undefined, roles: appRoles }
const policy: Policy = {
on_behalf_of: $userStore?.username.includes('@')
@@ -259,15 +412,43 @@
<label class="text-xs text-emphasis font-semibold" for="datatable"
>Datatable</label
>
<Select
id="datatable"
disablePortal
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Datatable"
size="sm"
class="w-40"
/>
<div class="flex flex-row items-center gap-2">
<Select
id="datatable"
disablePortal
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Datatable"
size="sm"
class="w-40"
/>
{#if showRolePicker}
<!-- Reads as one phrase, "main as analyst", so the role needs no label. -->
<span class="text-xs text-secondary">as</span>
<Select
id="datatable-role"
disablePortal
items={loadedRoles.map((r) => ({ value: r, label: r }))}
bind:value={() => selectedRole, pickRole}
clearable={false}
placeholder="Role"
size="sm"
class="w-40"
/>
{/if}
{#if noUsableRole || rolesUnknown || accessUnknown}
<span
class="text-xs text-red-600 dark:text-red-400"
title={access.current.error}
>
{rolesUnknown
? 'could not read its roles'
: noUsableRole
? 'no role you can use'
: 'could not reach it'}
</span>
{/if}
</div>
</div>
<div>
<span class="text-xs text-emphasis font-semibold">Schema</span>
@@ -276,7 +457,21 @@
<ToggleButtonGroup bind:selected={schemaMode} noWFull>
{#snippet children({ item })}
<ToggleButton value="none" label="None" icon={Ban} {item} size="sm" />
<ToggleButton value="new" label="New" icon={Plus} {item} size="sm" />
<ToggleButton
value="new"
label="New"
icon={Plus}
disabled={!canCreateSchema}
tooltip={canCreateSchema
? undefined
: noUsableRole
? `You can use no role of ${selectedDatatable}`
: accessUnknown
? `Could not read what may be created in ${selectedDatatable}`
: `${effectiveRole ?? 'This connection'} cannot create schemas in ${selectedDatatable}`}
{item}
size="sm"
/>
<ToggleButton
value="existing"
label="Existing"
@@ -333,6 +528,7 @@
dataTableRefs={preWhitelistedTables}
defaultDatatable={selectedDatatable}
defaultSchema={effectiveSchema}
roles={pickerRoles}
standalone
hideDefaultSelector
onAdd={() => dataTableDrawer?.openDrawer()}
@@ -418,7 +614,11 @@
variant="default"
size="sm"
on:click={() => start(false)}
disabled={!templates[selectedTemplateIndex] || newSchemaAlreadyExists}
disabled={!templates[selectedTemplateIndex] ||
newSchemaAlreadyExists ||
!rolesSettled ||
!accessSettled ||
blockedByRole}
>
{$copilotInfo.workspaceDisabled ? 'Start' : 'Start without AI'}
</Button>
@@ -426,7 +626,10 @@
<Button
variant="accent"
on:click={() => start(true)}
disabled={!templates[selectedTemplateIndex] ||
disabled={!rolesSettled ||
!accessSettled ||
blockedByRole ||
!templates[selectedTemplateIndex] ||
!initialPrompt.trim() ||
newSchemaAlreadyExists}
startIcon={{ icon: Sparkles }}
@@ -444,7 +647,15 @@
bind:this={dataTableDrawer}
offset={10000}
existingRefs={preWhitelistedTables}
onAdd={(ref) => {
preWhitelistedTables = [...preWhitelistedTables, ref]
roles={pickerRoles}
onAdd={(refs, browsedRoles, roleChanged) => {
preWhitelistedTables = [
...preWhitelistedTables.filter((t) => !roleChanged.has(t.datatable)),
...refs
]
preWhitelistedRoles = { ...preWhitelistedRoles, ...browsedRoles }
if (selectedDatatable !== undefined && browsedRoles[selectedDatatable] !== undefined) {
selectedRole = browsedRoles[selectedDatatable]
}
}}
/>
@@ -1,5 +1,31 @@
import { describe, expect, it } from 'vitest'
import { buildDataTableWhitelist, isDatatableTableAllowed } from './dataTableRefUtils'
import {
buildDataTableWhitelist,
isDatatableTableAllowed,
sdkDatatableCall,
withAppDatatableRole
} from './dataTableRefUtils'
describe('app data table roles', () => {
it('writes the role the way each SDK takes it', () => {
expect(sdkDatatableCall('main', 'analyst', 'typescript')).toBe(
"wmill.datatable('main', { role: 'analyst' })"
)
expect(sdkDatatableCall('main', 'analyst', 'python')).toBe(
"wmill.datatable('main', role='analyst')"
)
expect(sdkDatatableCall('main', undefined, 'python')).toBe('wmill.datatable()')
})
it('keeps one role per data table, and no map once none is left', () => {
const roles = withAppDatatableRole(undefined, 'main', 'analyst')
expect(withAppDatatableRole(roles, 'other', 'operator')).toEqual({
main: 'analyst',
other: 'operator'
})
expect(withAppDatatableRole(roles, 'main', undefined)).toBeUndefined()
})
})
describe('datatable whitelist helpers', () => {
it('allows every datatable table when no refs are configured', () => {
@@ -16,6 +16,43 @@ export interface RawAppData {
datatable: string | undefined
/** The schema for table creation (if specified) */
schema: string | undefined
/** The role the app uses each data table through, by data table name. A data table without
* an entry is used as its default role, then `admin`. */
roles?: Record<string, string>
}
export function appDatatableRole(
roles: Record<string, string> | undefined,
datatable: string
): string | undefined {
return roles?.[datatable]
}
/** `roles` with `datatable` set to `role`, or without it when `role` is undefined. */
export function withAppDatatableRole(
roles: Record<string, string> | undefined,
datatable: string,
role: string | undefined
): Record<string, string> | undefined {
const next = { ...roles }
if (role === undefined) delete next[datatable]
else next[datatable] = role
return Object.keys(next).length > 0 ? next : undefined
}
/** The SDK call app code uses to reach a data table, in the language it is written in. A role
* is a keyword argument in Python and an option in TypeScript. */
export function sdkDatatableCall(
datatable: string,
role: string | undefined,
language: 'typescript' | 'python'
): string {
if (role === undefined) {
return datatable === 'main' ? 'wmill.datatable()' : `wmill.datatable('${datatable}')`
}
return language === 'python'
? `wmill.datatable('${datatable}', role='${role}')`
: `wmill.datatable('${datatable}', { role: '${role}' })`
}
/** Default data configuration */
@@ -1,9 +1,27 @@
import { resource } from 'runed'
import { workspaceStore, dbSchemas } from '$lib/stores'
import { workspaceStore } from '$lib/stores'
import { WorkspaceService } from '$lib/gen'
import { getDbSchemas } from '$lib/components/apps/components/display/dbtable/metadata'
import { listUsableDatatableRoles } from '$lib/components/datatableUsableRoles'
import { ADMIN_DATATABLE_ROLE } from '$lib/components/dbTypes'
import { get } from 'svelte/store'
/**
* `fetch` wrapped so that an answer for a request a newer one has replaced resolves to
* `stale()` instead: a resource keeps whichever answer lands last, and a slow answer for the
* previous data table or role would otherwise describe a selection that no longer exists.
*/
function latestOnly<A extends unknown[], T>(
fetch: (...args: A) => Promise<T>,
stale: () => T
): (...args: A) => Promise<T> {
let run = 0
return async (...args) => {
const mine = ++run
const result = await fetch(...args)
return mine === run ? result : stale()
}
}
/**
* Creates a resource that loads available datatables from the workspace.
* Pass a getter function that returns the workspace to create a reactive dependency.
@@ -21,42 +39,142 @@ export function createDatatablesResource(getWorkspace: () => string | undefined)
})
}
export type DatatableRoles = {
/** The data table this answers for: while a switch is in flight, `current` still holds the
* previous one's roles, which say nothing about the one now selected. */
datatable: string | undefined
/** Whether the data table is under roles. Without roles `roles` is empty because there is
* nothing to pick, which is not the same as a permissioned one this caller may use no role of. */
permissioned: boolean
/** The lookup failed, so an empty `roles` means nothing was learned. */
failed: boolean
roles: string[]
defaultRole: string
}
/**
* Creates a resource that loads schemas for a given datatable.
* The getDatatable getter is used as a reactive dependency - when it changes, schemas are refetched.
* Creates a resource that loads the roles the caller may use on a datatable, and the one it
* defaults to.
*/
export function createSchemasResource(
export function createRolesResource(
getDatatable: () => string | undefined,
getWorkspace: () => string | undefined = () => get(workspaceStore)
) {
return resource<string[]>([() => getDatatable() ?? '', () => getWorkspace() ?? ''], async () => {
const datatable = getDatatable()
const workspace = getWorkspace()
if (!datatable || !workspace) return []
const initialValue: DatatableRoles = {
datatable: undefined,
permissioned: false,
failed: false,
roles: [],
defaultRole: ADMIN_DATATABLE_ROLE
}
const rolesResource = resource(
() => [getDatatable() ?? '', getWorkspace() ?? ''] as const,
latestOnly(
async ([datatableName, workspace]: readonly [string, string]): Promise<DatatableRoles> => {
const empty = { ...initialValue, datatable: datatableName || undefined }
if (!datatableName || !workspace) return empty
try {
const res = await listUsableDatatableRoles(workspace, datatableName)
return {
...empty,
permissioned: res.permissioned,
roles: res.roles,
defaultRole: res.default_role
}
} catch (e) {
console.error('Failed to load datatable roles:', e)
return { ...empty, failed: true }
}
},
() => rolesResource.current
),
{ initialValue }
)
return rolesResource
}
const resourcePath = `datatable://${datatable}`
// Key the schema cache by workspace too: a datatable of the same name can
// exist in both the nav and the acting workspace, so `datatable://<name>`
// alone would let one workspace's schema be reused for the other.
const cacheKey = `${workspace}:${resourcePath}`
const schemas = get(dbSchemas)
let dbSchema = schemas[cacheKey]
export type DatatableAccess = {
/** What this answers for. Until both match the selection, the schemas and the right to
* create one belong to another data table or another role. */
datatable: string | undefined
role: string | undefined
/** The request failed, or the server kept the entry with an error (a role this caller may
* not use, an unreachable database). `canCreateSchema: false` is then no answer at all. */
failed: boolean
error: string | undefined
schemas: string[]
canCreateSchema: boolean
}
if (!dbSchema) {
try {
schemas[cacheKey] = await getDbSchemas('postgresql', resourcePath, workspace, (msg) =>
console.error('Schema error:', msg)
)
dbSchema = get(dbSchemas)[cacheKey]
} catch (e) {
console.error(`Failed to load schema for ${datatable}:`, e)
return []
}
}
/**
* Creates a resource that loads, for one data table read as one role, the schemas that role can
* reach and whether it may create more.
*/
export function createDatatableAccessResource(
getDatatable: () => string | undefined,
getRole: () => string | undefined,
getWorkspace: () => string | undefined = () => get(workspaceStore),
/** False while the role is still being settled: a listing sent before that is read as the
* data table's default role, which is not the one about to be asked for. */
getReady: () => boolean = () => true
) {
const initialValue: DatatableAccess = {
datatable: undefined,
role: undefined,
failed: false,
error: undefined,
schemas: [],
canCreateSchema: false
}
const accessResource = resource(
() => [getDatatable() ?? '', getRole() ?? '', getWorkspace() ?? '', getReady()] as const,
latestOnly(
async ([datatable, role, workspace, ready]: readonly [
string,
string,
string,
boolean
]): Promise<DatatableAccess> => {
const asked = {
...initialValue,
datatable: datatable || undefined,
role: role || undefined
}
if (!ready) return accessResource.current
if (!datatable || !workspace) return asked
try {
const tables = await WorkspaceService.listDataTableTables({
workspace,
datatableName: datatable,
roleFor: datatable,
role: role || undefined
})
const entry = tables.find((t) => t.datatable_name === datatable)
return {
...asked,
failed: entry === undefined || entry.error !== undefined,
error: entry?.error,
schemas: Object.keys(entry?.schemas ?? {}).sort(),
canCreateSchema: !!entry?.can_create_schema
}
} catch (e) {
console.error('Failed to load datatable access:', e)
return { ...asked, failed: true, error: (e as Error)?.message }
}
},
() => accessResource.current
),
{ initialValue }
)
return accessResource
}
if (!dbSchema?.schema) return []
return Object.keys(dbSchema.schema)
})
/**
* Whether naming a role says anything: a data table without roles has none to pick, and one
* whose single usable role is `admin` offers no choice.
*/
export function rolesWorthPicking(roles: string[]): boolean {
return roles.length > 1 || (roles.length === 1 && roles[0] !== ADMIN_DATATABLE_ROLE)
}
/**
@@ -27,6 +27,7 @@
import DropdownV2 from '../DropdownV2.svelte'
import MissingWorkerTagAlert from '../jobs/MissingWorkerTagAlert.svelte'
import { superadmin, userStore } from '$lib/stores'
import { parseMigrationRole, withMigrationRole } from '../datatableMigrationRole'
let {
workspace,
@@ -84,8 +85,9 @@
function startAddDownMigration() {
// Same transaction frame the new-migration modal starts from, so the down
// applies atomically.
downDraft = DOWN_TEMPLATE
// applies atomically. It rolls back as the role the up ran as.
const upRole = viewMigration ? parseMigrationRole(viewMigration.code_up) : undefined
downDraft = withMigrationRole(DOWN_TEMPLATE, upRole?.kind === 'role' ? upRole.role : undefined)
addingDown = true
}
@@ -167,6 +169,10 @@
loadMigrations()
}
export function open() {
openList()
}
// Open the list modal and the detail view for a specific migration. Used to
// jump to a just-created migration from the "See migration" toast action.
export async function openMigration(timestamp: number) {
@@ -5,65 +5,115 @@
import Toggle from '../Toggle.svelte'
import Tooltip from '../Tooltip.svelte'
import CloseButton from '../common/CloseButton.svelte'
import { KeyRound, Plus } from 'lucide-svelte'
import Checkbox from '../common/checkbox/Checkbox.svelte'
import Cell from '../table/Cell.svelte'
import DataTable from '../table/DataTable.svelte'
import Head from '../table/Head.svelte'
import Row from '../table/Row.svelte'
import { KeyRound } from 'lucide-svelte'
import {
FolderService,
GroupService,
SettingService,
UserService,
WorkspaceService,
type DatatablePermissions,
type InstanceDatatableRole
} from '$lib/gen'
import { superadmin } from '$lib/stores'
import { sendUserToast } from '$lib/toast'
const ADMIN_ROLE = 'admin'
import { deepEqual } from 'fast-equals'
import { ADMIN_DATATABLE_ROLE, isDatatableRoleName } from '../dbTypes'
import PgAclEditor from '../datatableAcl/PgAclEditor.svelte'
import InstanceRolesButton from './InstanceRolesButton.svelte'
let {
workspace,
datatable,
disabled = false
disabled = false,
hideTrigger = false,
onSaved
}: {
workspace: string
datatable: string
disabled?: boolean
/** Mount the drawer without its button, for a caller that opens it with `open()`. */
hideTrigger?: boolean
/** Called once a save went through, so a caller showing the roles can read them again. */
onSaved?: () => void
} = $props()
let drawer: Drawer | undefined = $state(undefined)
// `id` is the instance role's catalog id (or the reserved `admin`), which is what the tenant
// lists are keyed by — so renaming a role instance-side moves nothing here. A row without an id
// names a role the instance does not define yet: it cannot be saved until a superadmin creates
// it, and takes the new role's id once they have.
type EditedRole = { id: string | undefined; name: string | undefined; tenants: string[] }
type Edited = { permissioned: boolean; roles: EditedRole[]; defaultRoleId: string }
let drawerOpen = $state(false)
let loading = $state(false)
let saving = $state(false)
let loadError = $state<string | undefined>(undefined)
let info = $state<DatatablePermissions | undefined>(undefined)
// Edited copy. `id` is the instance role's catalog id (or the reserved `admin`), which is what
// the tenant lists are keyed by — so renaming a role instance-side moves nothing here.
let permissioned = $state(false)
let defaultRole = $state(ADMIN_ROLE)
let rows = $state<{ id: string; name: string | undefined; tenants: string[] }[]>([])
let roles = $state<EditedRole[]>([])
let defaultRoleId = $state(ADMIN_DATATABLE_ROLE)
/** The last loaded state, to detect unsaved changes against. */
let saved = $state<Edited>({
permissioned: false,
roles: [],
defaultRoleId: ADMIN_DATATABLE_ROLE
})
// Tenants name principals of the workspace that governs the data table, which is not
// necessarily the one we are browsing from.
let tenantOptions = $state<{ value: string; label: string }[]>([])
let tenantItems = $state<{ value: string; label: string; group: string }[]>([])
const editable = $derived(!!info?.editable)
const governing = $derived(info?.governing_workspace_id)
const availableRoles: InstanceDatatableRole[] = $derived(info?.available_roles ?? [])
const unusedRoles = $derived(availableRoles.filter((r) => !rows.some((row) => row.id === r.id)))
// The instance catalog, read again after the instance roles drawer changes it.
let catalog = $state<InstanceDatatableRole[] | undefined>(undefined)
const availableRoles: InstanceDatatableRole[] = $derived(catalog ?? info?.available_roles ?? [])
const unusedRoles = $derived(availableRoles.filter((r) => !roles.some((row) => row.id === r.id)))
const pendingRoles = $derived(roles.filter((r) => r.id === undefined))
let instanceRoles: InstanceRolesButton | undefined = $state(undefined)
const roleKey = (role: EditedRole) => role.id ?? `pending:${role.name}`
const hasUnsavedChanges = $derived(
!deepEqual($state.snapshot(saved), {
permissioned,
roles: $state.snapshot(roles) as EditedRole[],
defaultRoleId
})
)
async function load() {
loading = true
loadError = undefined
try {
const res = await WorkspaceService.getDatatablePermissions({ workspace, datatableName: datatable })
info = res
permissioned = res.permissioned
defaultRole = res.default_role
rows = (res.roles ?? [])
.map((r) => ({ id: r.id, name: r.name, tenants: r.tenants ?? [] }))
.sort((a, b) => (a.id === ADMIN_ROLE ? -1 : b.id === ADMIN_ROLE ? 1 : 0))
if (rows.length === 0) {
rows = [{ id: ADMIN_ROLE, name: ADMIN_ROLE, tenants: [] }]
const res = await WorkspaceService.getDatatablePermissions({
workspace,
datatableName: datatable
})
const loaded: EditedRole[] = res.roles
.map((r) => ({ id: r.id, name: r.name, tenants: [...(r.tenants ?? [])] }))
.sort(
(a, b) => Number(b.id === ADMIN_DATATABLE_ROLE) - Number(a.id === ADMIN_DATATABLE_ROLE)
)
// A data table never put under roles comes back with none; admin is what turning the toggle
// on starts from.
if (!loaded.some((r) => r.id === ADMIN_DATATABLE_ROLE)) {
loaded.unshift({ id: ADMIN_DATATABLE_ROLE, name: ADMIN_DATATABLE_ROLE, tenants: [] })
}
await loadTenantOptions(res.governing_workspace_id ?? workspace)
info = res
catalog = undefined
permissioned = res.permissioned
roles = loaded
defaultRoleId = res.default_role
saved = structuredClone({ permissioned, roles: loaded, defaultRoleId })
await loadTenantItems(res.governing_workspace_id ?? workspace)
} catch (e) {
loadError = e?.body ?? e?.message ?? String(e)
} finally {
@@ -71,35 +121,70 @@
}
}
async function loadTenantOptions(ws: string) {
async function loadTenantItems(ws: string) {
try {
const [users, groups, folders] = await Promise.all([
UserService.listUsernames({ workspace: ws }),
GroupService.listGroupNames({ workspace: ws }),
FolderService.listFolderNames({ workspace: ws })
])
tenantOptions = [
{ value: '*', label: 'Everyone in the workspace' },
...users.map((u) => ({ value: `u/${u}`, label: `u/${u}` })),
...groups.map((g) => ({ value: `g/${g}`, label: `g/${g}` })),
...folders.map((f) => ({ value: `f/${f}`, label: `f/${f}` }))
tenantItems = [
{ value: '*', label: 'Everyone', group: 'Anyone in the workspace' },
...users.map((u) => ({ value: `u/${u}`, label: u, group: 'Users' })),
...groups.map((g) => ({ value: `g/${g}`, label: g, group: 'Groups' })),
...folders.map((f) => ({ value: `f/${f}`, label: f, group: 'Folders' }))
]
} catch {
// A fork member may not be able to list the governing workspace's principals. The
// tenants they cannot name are still shown, they just cannot pick new ones.
tenantOptions = []
tenantItems = []
}
}
function addRole(id: string) {
const role = availableRoles.find((r) => r.id === id)
if (!role) return
rows = [...rows, { id: role.id, name: role.name, tenants: [] }]
roles.push({ id: role.id, name: role.name, tenants: [] })
}
function removeRole(id: string) {
rows = rows.filter((r) => r.id !== id)
if (defaultRole === id) defaultRole = ADMIN_ROLE
/** Adds a role by name: the instance's role of that name, or a pending row for one it does not
* define yet. */
function addRoleByName(typed: string) {
const name = typed.trim()
if (!isDatatableRoleName(name) || name.toLowerCase() === ADMIN_DATATABLE_ROLE) {
sendUserToast(
`'${name}' cannot be a data table role name: use letters, digits, '_' and '-'`,
true
)
return
}
if (roles.some((r) => r.name === name)) return
const existing = availableRoles.find((r) => r.name === name)
roles.push({ id: existing?.id, name, tenants: [] })
}
function removeRole(role: EditedRole) {
const key = roleKey(role)
roles = roles.filter((r) => roleKey(r) !== key)
if (role.id !== undefined && defaultRoleId === role.id) defaultRoleId = ADMIN_DATATABLE_ROLE
}
/** Reads the instance catalog again and gives each pending row the id of the role now defined
* under its name. */
async function refreshCatalog() {
let fresh: InstanceDatatableRole[]
try {
fresh = await SettingService.listInstanceDatatableRoles()
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
return
}
catalog = fresh
for (const row of roles) {
if (row.id !== undefined) continue
const created = fresh.find((r) => r.name === row.name)
if (created) row.id = created.id
}
}
async function save() {
@@ -110,12 +195,13 @@
datatableName: datatable,
requestBody: {
permissioned,
default_role: defaultRole,
roles: rows.map((r) => ({ id: r.id, tenants: r.tenants }))
default_role: defaultRoleId,
roles: roles.map((r) => ({ id: r.id!, tenants: $state.snapshot(r.tenants) }))
}
})
sendUserToast(msg)
await load()
onSaved?.()
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
} finally {
@@ -124,33 +210,48 @@
}
export function open() {
drawer?.openDrawer()
drawerOpen = true
load()
}
</script>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: KeyRound }}
iconOnly
{disabled}
title={disabled ? 'Save settings first' : 'Roles: who may connect as which Postgres role'}
on:click={open}
/>
{#if !hideTrigger}
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: KeyRound }}
iconOnly
{disabled}
title={disabled ? 'Save settings first' : 'Roles: who may connect as which Postgres role'}
on:click={open}
/>
{/if}
<Drawer bind:this={drawer} size="700px">
<Drawer bind:open={drawerOpen} size="900px">
<DrawerContent
title="Roles for {datatable}"
on:close={() => drawer?.closeDrawer()}
tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant privileges with SQL. Roles are defined for the whole instance; here you say who may use each one on this data table."
title="Roles {datatable}"
on:close={() => (drawerOpen = false)}
tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant it privileges under Access. Roles are defined for the whole instance; here you say who may use each one on this data table."
>
{#if loading}
<p class="text-sm text-secondary">Loading…</p>
{:else if loadError}
{#snippet titleExtra()}
<Badge color="blue" small>Beta</Badge>
{/snippet}
{#if loadError}
<Alert type="error" title="Could not load roles" size="xs">{loadError}</Alert>
{:else if loading && !info}
<span class="text-sm text-secondary">Loading…</span>
{:else}
<div class="flex flex-col gap-4">
<Toggle
bind:checked={permissioned}
disabled={!editable || !info?.supported}
options={{
right: 'Put this data table under roles',
rightTooltip:
'Off, every job connects as admin — the connection that owns every table. On, every job resolves to a role, and a caller no tenant covers is refused.'
}}
/>
{#if !info?.supported}
<Alert type="info" title="Not available on this data table" size="xs">
A data table role is a Postgres login on the Windmill instance's own database, so only a
@@ -173,105 +274,162 @@
These data tables point at the same database with their own entry, so what you set here
does not reach them:
<ul class="mt-1 list-disc list-inside font-mono">
{#each info.ungoverned_reachers as reacher}
{#each info.ungoverned_reachers as reacher (`${reacher.workspace_id}/${reacher.datatable}`)}
<li>{reacher.workspace_id} / {reacher.datatable}</li>
{/each}
</ul>
</Alert>
{/if}
<Toggle
bind:checked={permissioned}
disabled={!editable || !info?.supported}
options={{
right: 'Put this data table under roles',
rightTooltip:
'Off, every job connects as admin — the connection that owns every table. On, every job resolves to a role, and a caller no tenant covers is refused.'
}}
/>
{#if permissioned}
{#if availableRoles.length === 0}
{#if editable && availableRoles.length === 0}
<Alert type="warning" title="No role defined on this instance" size="xs">
Only <span class="font-mono">admin</span> can be used until a superadmin adds a data table
role in the data table settings page.
Only <span class="font-mono">admin</span> can be used until a superadmin creates a data
table role. Type a name below to add one.
</Alert>
{/if}
<div class="flex flex-col gap-3">
{#each rows as row (row.id)}
<div class="flex flex-col gap-1 border rounded-md p-3 bg-surface-secondary">
<div class="flex items-center gap-2">
<Badge color={row.id === ADMIN_ROLE ? 'blue' : 'gray'}>
{row.name ?? row.id}
</Badge>
{#if row.id === ADMIN_ROLE}
<Tooltip>
The connection every data table resolved to before roles. It owns every
existing object, so it is always available and cannot be removed.
</Tooltip>
{:else if !row.name}
<span class="text-xs text-secondary italic">
no longer defined on this instance
</span>
{/if}
{#if defaultRole === row.id}
<Badge color="green">Default</Badge>
{:else if editable}
<Button
unifiedSize="2xs"
variant="subtle"
on:click={() => (defaultRole = row.id)}
>
Make default
</Button>
{/if}
<div class="grow"></div>
{#if editable && row.id !== ADMIN_ROLE}
<CloseButton small on:close={() => removeRole(row.id)} />
{/if}
</div>
<MultiSelect
items={tenantOptions}
bind:value={row.tenants}
disabled={!editable}
placeholder="Nobody yet — add a user, group or folder"
size="sm"
/>
</div>
{/each}
</div>
{#if editable && unusedRoles.length > 0}
<div class="flex items-center gap-2">
<Plus size={14} class="text-secondary" />
<Select
items={unusedRoles.map((r) => ({
value: r.id,
label: r.enabled ? r.name : `${r.name} (disabled)`
}))}
placeholder="Add a role"
bind:value={
() => undefined,
(id) => {
if (id) addRole(id)
}
}
class="w-64"
size="sm"
/>
</div>
{/if}
{/if}
{#if editable}
<div class="flex justify-end">
<Button unifiedSize="sm" variant="accent" loading={saving} on:click={save}>
Save roles
</Button>
</div>
<DataTable>
<Head>
<tr>
<Cell head first>
Role
<Tooltip>
admin is the connection the data table used before roles, so it owns every
existing object and cannot be removed. Every other role is a login defined for
the whole instance, with only the privileges granted to it under Access.
</Tooltip>
</Cell>
<Cell head>
Tenants
<Tooltip>
Users, groups and folders allowed to connect as this role. Workspace admins can
use every role.
</Tooltip>
</Cell>
<Cell head>
Default
<Tooltip>
The role a job gets when it names none — no `-- role` annotation, no `?role=` in
the reference. Callers still have to be one of its tenants.
</Tooltip>
</Cell>
<Cell head last />
</tr>
</Head>
<tbody class="divide-y bg-surface-tertiary">
{#each roles as role (roleKey(role))}
{@const isAdmin = role.id === ADMIN_DATATABLE_ROLE}
<Row>
<Cell first class="w-56 align-top">
<div class="flex flex-col gap-0.5 pt-1.5">
<span class="font-mono text-xs text-emphasis">{role.name ?? role.id}</span>
{#if !role.name}
<span class="text-2xs text-secondary italic">
no longer defined on this instance
</span>
{:else if role.id === undefined}
<Alert type="warning" title="This role does not exist yet" size="xs">
{#if $superadmin}
<div class="flex flex-col items-start gap-1">
<span>Create it on the instance to use it here.</span>
<Button
unifiedSize="xs"
variant="default"
on:click={() => instanceRoles?.open(role.name)}
>
Create it
</Button>
</div>
{:else}
Only a superadmin can create it on the instance.
{/if}
</Alert>
{/if}
</div>
</Cell>
<Cell class="align-top">
<MultiSelect
items={tenantItems}
bind:value={role.tenants}
groupBy={(item) => item.group}
disabled={!editable}
placeholder="Nobody — add users, groups or folders"
/>
</Cell>
<Cell class="w-20 align-top">
<div class="flex justify-center pt-2">
<Checkbox
checked={role.id !== undefined && defaultRoleId === role.id}
disabled={!editable || role.id === undefined}
title="Use this role when a job names none"
onChange={() => {
if (role.id !== undefined) defaultRoleId = role.id
}}
/>
</div>
</Cell>
<Cell last class="w-10 align-top">
{#if editable && !isAdmin}
<CloseButton small on:close={() => removeRole(role)} />
{/if}
</Cell>
</Row>
{/each}
{#if editable}
<Row class="!border-0">
<Cell colspan={4} class="pt-2 pb-2">
<div class="flex justify-center">
<Select
items={unusedRoles.map((r) => ({
value: r.id,
label: r.enabled ? r.name : `${r.name} (disabled)`
}))}
placeholder="+ Add a role"
bind:value={
() => undefined,
(id) => {
if (id) addRole(id)
}
}
onCreateItem={addRoleByName}
class="w-64"
/>
</div>
</Cell>
</Row>
{/if}
</tbody>
</DataTable>
{/if}
</div>
{#if info?.supported && !hasUnsavedChanges}
<div class="mt-6 pt-6 border-t">
<PgAclEditor {workspace} {datatable} target={{ kind: 'database' }} />
</div>
{/if}
{/if}
{#snippet actions()}
{#if editable}
<Button
variant="accent"
unifiedSize="md"
disabled={!hasUnsavedChanges || loading || !!loadError || pendingRoles.length > 0}
title={pendingRoles.length > 0
? 'Create the roles that do not exist yet, or remove them'
: undefined}
loading={saving}
on:click={save}
>
Save
</Button>
{/if}
{/snippet}
</DrawerContent>
</Drawer>
{#if $superadmin}
<InstanceRolesButton bind:this={instanceRoles} hideTrigger onChanged={refreshCatalog} />
{/if}
@@ -3,7 +3,6 @@
import CloseButton from '../common/CloseButton.svelte'
import TextInput from '../text_input/TextInput.svelte'
import Toggle from '../Toggle.svelte'
import Tooltip from '../Tooltip.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import Cell from '../table/Cell.svelte'
@@ -14,11 +13,22 @@
import { SettingService, type InstanceDatatableRole } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
let {
initialName = '',
onChanged
}: {
/** Prefills the name of the role to add. */
initialName?: string
/** Called after every change to the catalog, whether or not it went through. */
onChanged?: () => void
} = $props()
let roles = $state<InstanceDatatableRole[]>([])
let loading = $state(true)
let loadError = $state<string | undefined>(undefined)
let busy = $state(false)
let newName = $state('')
// svelte-ignore state_referenced_locally
let newName = $state(initialName)
/** Which role's name is being edited, and to what. */
let renaming = $state<{ id: string; name: string } | undefined>(undefined)
@@ -49,6 +59,7 @@
// holds, so a failed flip has to snap back rather than sit there claiming it landed.
await load()
busy = false
onChanged?.()
}
}
@@ -83,16 +94,6 @@
<ConfirmationModal {...confirmationModal.props} />
<div class="flex flex-col gap-2">
<div class="flex items-baseline gap-1">
<h3 class="font-semibold text-sm">Instance roles</h3>
<Tooltip>
A data table role is a real Postgres login on this instance, shared by every instance
database. A job that names one connects as it, and Postgres decides what it may touch — grant
it privileges with SQL. Which people may use a role on a given data table is set per data
table, in its roles drawer.
</Tooltip>
</div>
{#if loadError}
<Alert type="error" title="Could not load the instance roles" size="xs">{loadError}</Alert>
{:else}
@@ -64,12 +64,11 @@
</script>
<script lang="ts">
import { Plus, PlugZap } from 'lucide-svelte'
import { History, KeyRound, Plus, PlugZap, Trash2 } from 'lucide-svelte'
import DropdownV2 from '../DropdownV2.svelte'
import Button from '../common/button/Button.svelte'
import CloseButton from '../common/CloseButton.svelte'
import ResourcePicker from '../ResourcePicker.svelte'
import SettingsPageHeader from '../settings/SettingsPageHeader.svelte'
import Select from '../select/Select.svelte'
@@ -92,8 +91,7 @@
type GetSettingsResponse,
type TestDataTableConnectionResponse
} from '$lib/gen'
// `superadmin` gates the commented-out roles section at the bottom; restore it there.
import { workspaceStore } from '$lib/stores'
import { enterpriseLicense, superadmin, workspaceStore } from '$lib/stores'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { resource } from 'runed'
@@ -101,12 +99,10 @@
import { Popover } from '../meltComponents'
import ExploreAssetButton from '../ExploreAssetButton.svelte'
import DataTableMigrationsButton from './DataTableMigrationsButton.svelte'
// Both components are complete and reviewed; their call sites in this file are commented
// out until the ACL editor lands. Uncomment these with them.
// import DataTablePermissionsButton from './DataTablePermissionsButton.svelte'
// import DataTableRolesSection from './DataTableRolesSection.svelte'
import DataTablePermissionsButton from './DataTablePermissionsButton.svelte'
import InstanceRolesButton from './InstanceRolesButton.svelte'
import { deepEqual } from 'fast-equals'
import { clone } from '$lib/utils'
import { clone, onlyAlphaNumAndUnderscore } from '$lib/utils'
import SettingsFooter from './SettingsFooter.svelte'
import Alert from '../common/alert/Alert.svelte'
import MissingWorkerTagAlert from '../jobs/MissingWorkerTagAlert.svelte'
@@ -287,6 +283,9 @@
}
let confirmationModal = createAsyncConfirmationModal()
// Each mounts its own modal or drawer; the row menu opens them.
let migrationsButtons = $state<Record<string, DataTableMigrationsButton | undefined>>({})
let permissionsButtons = $state<Record<string, DataTablePermissionsButton | undefined>>({})
let dirtyMap = $derived.by(() => {
const map: Record<string, boolean> = {}
for (let i = 0; i < tempSettings.dataTables.length; i++) {
@@ -318,7 +317,13 @@
title="Data tables"
description="Relational storage the whole workspace shares under one name. Scripts, flows and apps address it as <span class='font-mono'>datatable://main</span> instead of picking a PostgreSQL resource, so nobody needs access to the credentials to query it, and you can point that name at another database without touching a line of code. Browse and edit tables, and version schema changes as migrations, from here."
link="https://www.windmill.dev/docs/core_concepts/persistent_storage/data_tables"
/>
>
{#snippet actions()}
{#if $superadmin && $enterpriseLicense && !isCloudHosted()}
<InstanceRolesButton />
{/if}
{/snippet}
</SettingsPageHeader>
{#if isCloudHosted()}
<Alert type="info" title="Instance database not available on cloud" class="mb-4" size="xs">
@@ -468,27 +473,19 @@
<Cell class="whitespace-nowrap">
<div class="flex gap-2">
<DataTableMigrationsButton
bind:this={migrationsButtons[dataTable.name]}
hideTrigger
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
<!-- Data table roles: not mounted yet. The enforcement ships first and this
drawer is what turns it on, so leaving it reachable would expose a half of the
feature whose other half (the ACL editor, which grants the privileges a role
actually needs) does not exist yet.
DataTablePermissionsButton.svelte is complete and reviewed — reuse it rather
than rewriting it, and uncomment this together with the roles section at the
bottom of this file and the two imports at the top.
{#if $enterpriseLicense}
{#if $enterpriseLicense && !isCloudHosted()}
<DataTablePermissionsButton
bind:this={permissionsButtons[dataTable.name]}
hideTrigger
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
{/if}
-->
<Button
size="xs"
color="light"
@@ -522,9 +519,41 @@
</div>
</Cell>
<Cell class="w-12">
{#if !dataTable.reference}
<CloseButton small on:close={() => removeDataTable(dataTableIndex)} />
{/if}
<DropdownV2
items={() => [
{
displayName: 'Migrations',
icon: History,
// Both act on the saved data table, which unsaved edits are not.
disabled: !!dirtyMap[dataTable.name],
tooltip: dirtyMap[dataTable.name] ? 'Save the settings first' : undefined,
action: () => migrationsButtons[dataTable.name]?.open()
},
...($enterpriseLicense && !isCloudHosted()
? [
{
displayName: 'Roles',
icon: KeyRound,
disabled: !!dirtyMap[dataTable.name],
tooltip: dirtyMap[dataTable.name] ? 'Save the settings first' : undefined,
action: () => permissionsButtons[dataTable.name]?.open()
}
]
: []),
// A fork's pointer entry is written by forking and kept by the server, not this form.
...(dataTable.reference
? []
: [
{
displayName: 'Remove',
icon: Trash2,
type: 'delete' as const,
action: () => removeDataTable(dataTableIndex)
}
])
]}
btnId={'datatable-settings-actions-' + onlyAlphaNumAndUnderscore(dataTable.name)}
/>
</Cell>
</Row>
{/each}
@@ -615,20 +644,6 @@
{/if}
{/if}
<!-- The instance role catalog, superadmin-only. Not mounted for the same reason as the
permissions drawer above: creating roles is only useful once there is a way to grant them
privileges, which arrives with the ACL editor.
DataTableRolesSection.svelte is complete and reviewed — reuse it rather than rewriting it,
and uncomment this together with the permissions button above and the two imports at the top.
{#if $superadmin && $enterpriseLicense && !isCloudHosted()}
<div class="mt-8">
<DataTableRolesSection />
</div>
{/if}
-->
<SettingsFooter
class="mt-8"
{hasUnsavedChanges}
@@ -189,11 +189,19 @@
(v) => (datatableBehaviors[dt.name] = v)
}
items={[
{ value: 'keep_original', label: 'Keep original' },
{ value: 'schema_only', label: 'Clone schema only' },
...(!isCloudHosted() && $userStore?.is_admin
? [{ value: 'schema_and_data', label: 'Clone schema and data' }]
: [])
{
value: 'keep_original',
label: dt.permissioned ? 'Keep original (under roles)' : 'Keep original'
},
// A copy of a data table under roles is refused by the server, so it is not offered.
...(dt.permissioned
? []
: [
{ value: 'schema_only', label: 'Clone schema only' },
...(!isCloudHosted() && $userStore?.is_admin
? [{ value: 'schema_and_data', label: 'Clone schema and data' }]
: [])
])
]}
/>
</div>
@@ -0,0 +1,48 @@
<script lang="ts">
import { Badge, Button, Drawer, DrawerContent } from '../common'
import { Users } from 'lucide-svelte'
import DataTableRolesSection from './DataTableRolesSection.svelte'
let {
hideTrigger = false,
onChanged
}: {
/** Mount the drawer without its button, for a caller that opens it with `open()`. */
hideTrigger?: boolean
/** Called after every change to the instance roles. */
onChanged?: () => void
} = $props()
let drawer: Drawer | undefined = $state(undefined)
let prefill = $state('')
// Remounts the section on each open, so the prefilled name is the one just asked for.
let openCount = $state(0)
/** Opens the drawer, with `name` prefilled as the role to add. */
export function open(name = '') {
prefill = name
openCount++
drawer?.openDrawer()
}
</script>
{#if !hideTrigger}
<Button unifiedSize="sm" variant="default" startIcon={{ icon: Users }} on:click={() => open()}>
Instance roles
</Button>
{/if}
<Drawer bind:this={drawer} size="700px">
<DrawerContent
title="Instance roles"
on:close={() => drawer?.closeDrawer()}
tooltip="A data table role is a real Postgres login on this instance, shared by every instance database. A job that names one connects as it, and Postgres decides what it may touch. Which people may use a role on a given data table, and what it may do there, is set per data table, in its roles drawer."
>
{#snippet titleExtra()}
<Badge color="blue" small>Beta</Badge>
{/snippet}
{#key openCount}
<DataTableRolesSection initialName={prefill} {onChanged} />
{/key}
</DrawerContent>
</Drawer>
@@ -8,12 +8,17 @@
import TextInput from '../text_input/TextInput.svelte'
import SimpleEditor from '../SimpleEditor.svelte'
import { WorkspaceService, type DatatableMigration } from '$lib/gen'
import { listUsableDatatableRoles } from '../datatableUsableRoles'
import { sendUserToast } from '$lib/toast'
import { tick } from 'svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import Portal from '$lib/components/Portal.svelte'
import { fetchPendingMigrations, outOfOrderRunMessage } from './datatableMigrationUtils'
import Select from '../select/Select.svelte'
import { parseMigrationRole, withMigrationRole } from '../datatableMigrationRole'
import { ADMIN_DATATABLE_ROLE } from '../dbTypes'
import { resource } from 'runed'
let {
workspace,
@@ -50,6 +55,8 @@
let tab = $state('up')
let name = $state('')
let nameInput = $state<TextInput>()
let upEditor = $state<SimpleEditor | undefined>()
let downEditor = $state<SimpleEditor | undefined>()
// A valid migration name is non-empty and limited to letters, digits, '_' and '-'.
const MIGRATION_NAME_RE = /^[a-zA-Z0-9_-]+$/
let nameInvalid = $derived(!MIGRATION_NAME_RE.test(name.trim()))
@@ -60,6 +67,96 @@
const confirmationModal = createAsyncConfirmationModal()
// The role lives in the SQL as its `-- role <name>` annotation, so the code is the single
// source of truth and the Select is a view onto it: reading parses, writing rewrites the
// annotation.
const usableRoles = resource(
() => [workspace, datatable] as const,
async ([ws, dt]) => {
try {
return await listUsableDatatableRoles(ws, dt)
} catch (e) {
console.error('Failed to load data table roles:', e)
return null
}
}
)
// Not a valid role name, so it cannot collide with one.
const NO_ROLE = '(no role)'
let declaredUp = $derived(parseMigrationRole(codeUp))
let declaredDown = $derived(enableDown ? parseMigrationRole(codeDown) : undefined)
let malformedLine = $derived(
declaredUp.kind === 'malformed'
? declaredUp.line
: declaredDown?.kind === 'malformed'
? declaredDown.line
: undefined
)
const roleOf = (d: typeof declaredUp) => (d.kind === 'role' ? d.role : undefined)
// A rollback runs as the role its own SQL names: under another role than the up migration it
// typically cannot touch what the up created.
let sqlProblem = $derived(
malformedLine !== undefined
? malformedMessage(malformedLine)
: declaredDown !== undefined && roleOf(declaredDown) !== roleOf(declaredUp)
? `The down migration runs as ${roleOf(declaredDown) ?? 'admin (no role)'} but the up migration as ${roleOf(declaredUp) ?? 'admin (no role)'}: make their role annotations match`
: undefined
)
let selectedRole = $derived(declaredUp.kind === 'role' ? declaredUp.role : NO_ROLE)
let permissioned = $derived(!!usableRoles.current?.permissioned)
// No annotation runs as admin, which the server allows exactly to those who may use `admin`.
let adminUsable = $derived(!!usableRoles.current?.roles.includes(ADMIN_DATATABLE_ROLE))
let roleItems = $derived.by(() => {
const usable = usableRoles.current
if (!usable?.permissioned) return []
const names = usable.roles.filter((r) => r !== ADMIN_DATATABLE_ROLE)
// A role the SQL names but the caller cannot use is still shown, or the picker would
// misreport what the migration runs as.
if (declaredUp.kind === 'role' && !names.includes(declaredUp.role)) {
names.push(declaredUp.role)
}
const items = names.map((r) => ({
value: r,
label: r === usable.default_role ? `${r} (default)` : r
}))
if (adminUsable || declaredUp.kind === 'none') {
items.push({
value: NO_ROLE,
label: 'No role — runs as admin with full access'
})
}
return items
})
function setRole(value: string | undefined) {
const role = value === NO_ROLE ? undefined : value
codeUp = withMigrationRole(codeUp, role)
// Up and down agree: a rollback run as another role could fail on objects it does not own.
if (enableDown) codeDown = withMigrationRole(codeDown, role)
// Assigning the bound value does not repaint the editor, and its next keystroke would write
// the stale text back.
upEditor?.setCode(codeUp)
if (enableDown) downEditor?.setCode(codeDown)
}
// Set by `open` when the SQL names no role yet: the data table's default is written once its
// roles are known.
let applyDefaultRole = $state(false)
$effect(() => {
// `undefined` until the first answer lands; `null` when it failed.
const usable = usableRoles.current
if (!applyDefaultRole || !isOpen || usableRoles.loading || usable === undefined) return
applyDefaultRole = false
if (!usable?.permissioned || declaredUp.kind !== 'none') return
const role =
usable.default_role !== ADMIN_DATATABLE_ROLE && usable.roles.includes(usable.default_role)
? usable.default_role
: usable.default_role === ADMIN_DATATABLE_ROLE && adminUsable
? undefined
: usable.roles.find((r) => r !== ADMIN_DATATABLE_ROLE)
if (role !== undefined) setRole(role)
})
// Frame the migration body in an explicit transaction so it applies atomically.
function wrapInTransaction(body: string): string {
return `BEGIN;\n\n${body}\n\nEND;`
@@ -72,15 +169,35 @@
}
const PLACEHOLDER = wrapInTransaction('-- Add your migration here')
function malformedMessage(line: string): string {
return `Malformed role annotation \`${line}\`: write it as \`-- role <name>\`, or pick the role above`
}
export function open(prefill?: { name?: string; codeUp?: string; codeDown?: string }) {
// Roles and the default can have changed since the last open (the roles drawer sits next
// to this modal), and the default role is written from this answer.
usableRoles.refetch()
name = prefill?.name ?? ''
// Start from the transaction template; when prefilled from detected DDL,
// wrap that DDL in the same BEGIN; ... END; frame.
codeUp = prefill?.codeUp
? wrapInTransaction(ensureTrailingSemicolon(prefill.codeUp))
: PLACEHOLDER
// wrap that DDL in the same BEGIN; ... END; frame. A role the prefill declares is taken
// out first and put back on top: below `BEGIN;` it would not be read.
const prefillRole = prefill?.codeUp ? parseMigrationRole(prefill.codeUp) : undefined
if (prefill?.codeUp) {
const wrapped = wrapInTransaction(
ensureTrailingSemicolon(withMigrationRole(prefill.codeUp, undefined))
)
codeUp =
prefillRole?.kind === 'role'
? withMigrationRole(wrapped, prefillRole.role)
: prefillRole?.kind === 'malformed'
? `${prefillRole.line}\n${wrapped}`
: wrapped
} else {
codeUp = PLACEHOLDER
}
codeDown = prefill?.codeDown ?? PLACEHOLDER
enableDown = (prefill?.codeDown ?? '') !== ''
applyDefaultRole = prefillRole === undefined || prefillRole.kind === 'none'
tab = 'up'
isOpen = true
// Focus the name field once the modal content has rendered.
@@ -96,6 +213,10 @@
sendUserToast("Invalid migration name: use only letters, digits, '_' and '-'", true)
return
}
if (sqlProblem !== undefined) {
sendUserToast(sqlProblem, true)
return
}
if (run) {
// A new migration gets the highest timestamp, so any still-pending
// migration is earlier: running only this one applies it out of order.
@@ -176,29 +297,65 @@
closeOnOutsideClick={false}
>
<div class="flex flex-col gap-3 w-full grow min-h-0">
<TextInput
bind:this={nameInput}
bind:value={name}
error={nameInvalid}
inputProps={{ placeholder: 'Migration name (e.g. add_index_to_customers)' }}
/>
<div class="flex gap-2 items-center">
<TextInput
bind:this={nameInput}
bind:value={name}
error={nameInvalid}
class="grow"
inputProps={{ placeholder: 'Migration name (e.g. add_index_to_customers)' }}
/>
{#if permissioned}
<Select
transformInputSelectedText={(s) => `Role: ${s}`}
items={roleItems}
bind:value={() => selectedRole, (r) => setRole(r)}
placeholder="Role"
class="w-72"
/>
{/if}
</div>
{#if sqlProblem !== undefined}
<p class="text-xs text-red-500">{sqlProblem}</p>
{:else if permissioned && usableRoles.current?.roles.length === 0}
<p class="text-xs text-secondary">
You can't use any role of this data table, so a migration you create can't be run.
</p>
{/if}
<Tabs bind:selected={tab} class="grow min-h-0">
<Tab value="up" label="Up" />
<Tab value="down" label="Down" />
{#snippet content()}
<TabContent value="up" class="h-80 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeUp} />
<SimpleEditor bind:this={upEditor} class="h-full" lang="sql" bind:code={codeUp} />
</TabContent>
<TabContent value="down" class="h-80">
<div class="flex flex-col gap-2 h-full">
<Toggle
bind:checked={enableDown}
bind:checked={
() => enableDown,
(checked) => {
enableDown = checked
// The down editor is created by this toggle, so it reads the rewritten text.
if (checked) {
codeDown = withMigrationRole(
codeDown,
declaredUp.kind === 'role' ? declaredUp.role : undefined
)
}
}
}
options={{ right: 'Enable down migration' }}
size="sm"
/>
{#if enableDown}
<div class="grow min-h-0 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeDown} />
<SimpleEditor
bind:this={downEditor}
class="h-full"
lang="sql"
bind:code={codeDown}
/>
</div>
{/if}
</div>
@@ -209,7 +366,7 @@
<Button
variant="accent"
size="sm"
disabled={creating}
disabled={creating || sqlProblem !== undefined}
on:click={() => create(true)}
dropdownItems={[
{
@@ -497,7 +497,8 @@
aiChatManager.datatableCreationPolicy = {
enabled: !!result.data.datatable,
datatable: result.data.datatable,
schema: result.data.schema
schema: result.data.schema,
roles: result.data.roles
}
if (withPrompt && result.prompt) {
const prompt = result.prompt
@@ -172,6 +172,8 @@ data:
tables:
- main/users # Table in public schema
- main/app_schema:items # Table in specific schema
roles: # Optional: the role the app uses each datatable through
main: analyst
```
**Table reference formats:**
@@ -179,6 +181,8 @@ data:
- `<datatable>/<table>` — Specific table in public schema
- `<datatable>/<schema>:<table>` — Table in specific schema
**Roles:** when a datatable is under roles, its queries run as a role, which only reaches what it was granted. `roles` records the role the app uses each datatable through; the app's code must pass the same role: `wmill.datatable('main', { role: 'analyst' })` in TypeScript, `wmill.datatable('main', role='analyst')` in Python. A datatable without an entry is used as its default role.
## SQL Migrations (sql_to_apply/)
The `sql_to_apply/` folder is for creating/modifying database tables during development.
+4
View File
@@ -167,6 +167,8 @@ data:
tables:
- main/users # Table in public schema
- main/app_schema:items # Table in specific schema
roles: # Optional: the role the app uses each datatable through
main: analyst
```
**Table reference formats:**
@@ -174,6 +176,8 @@ data:
- `<datatable>/<table>` — Specific table in public schema
- `<datatable>/<schema>:<table>` — Table in specific schema
**Roles:** when a datatable is under roles, its queries run as a role, which only reaches what it was granted. `roles` records the role the app uses each datatable through; the app's code must pass the same role: `wmill.datatable('main', { role: 'analyst' })` in TypeScript, `wmill.datatable('main', role='analyst')` in Python. A datatable without an entry is used as its default role.
## SQL Migrations (sql_to_apply/)
The `sql_to_apply/` folder is for creating/modifying database tables during development.