feat: data table roles in the DB manager and raw apps (#11139)

* feat(datatables): put a data table's connection under Postgres roles

A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): gate the paths that reach a whole database as admin

Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping

A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): fail loudly where a role or a pointer can be left half-recorded

Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): unbreak two operator messages and two comments that described other code

The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract

The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): give the role catalog its own row, out of reach of the config machinery

Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced

The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing

The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* refactor(datatables): put the role catalog in its own table, not in global_settings

Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): validate a rename against the save it describes, and re-check under the locks

Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically

The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse to copy a data table that is under roles

pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse the clone's database too, not only its data

A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* nit worker error msg

* fix pg_dump stuck on version 17 on nix

* fix(datatables): refuse a malformed role annotation instead of ignoring it

`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse a malformed role query string instead of ignoring it

`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): carry the role annotation into the row_to_json retry

The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore(datatables): don't mount the roles UI until the ACL editor lands

Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test

Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): clone a fork's pointer instead of failing after the copy

Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse to roll back the catalog while roles exist

The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* refactor(datatables): settle clonability in one place, before anything is created

A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): let a retried clone reclaim its own leftover database

A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "fix(datatables): let a retried clone reclaim its own leftover database"

This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): record the stale clone database as a known limitation

A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): bounce the streams reading a data table when it is deleted

Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): keep the fork schema baseline, and bounce streams on every removal

Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(datatables): keep Postgres triggers and data table roles apart

A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved

Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): disable a data table role before deleting it

Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with a stream starting

Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): wait out live listeners, and resolve stored names containing `?`

Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): warn when a settings sync strands fork pointers

A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse a save that drops a data table's roles through an undeclared rename

A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): no entry without roles may newly reach a database under roles

The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move data table role catalog and resolution to the enterprise edition

Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table permissions endpoints to the enterprise edition

The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table role catalog endpoints to the enterprise edition

The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the roles tests on the enterprise edition, refusals without it

Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): gate the roles UI mount sites on an enterprise license

Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the tenant matcher test on the enterprise edition

The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607

This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.

* fix(datatables): refuse roles while a same-workspace alias reaches the database

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(datatables): add an ACL editor for data table roles

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(datatables): data table roles in the DB manager and raw apps

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: never add a role to the reference of a data table whose name contains '?'

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: read the roles of a data table whose name contains '?'

The generated client leaves a '?' in a path param unencoded, so the lookup
404'd and the raw-app picker blocked Start on such a data table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: take every pooled connection before the ACL apply locks

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: refresh grant options only after the ACL apply validates its plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix(datatables): refuse a reference naming both a legacy data table and a role

When a workspace stores both `sales` and a legacy `sales?role=analytics`, the
reference resolved to the legacy entry without a role, so browsing `sales` as
`analytics` reached another data table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: declare the default role in migrations written for a data table whose name contains '?'

Such a data table connects as its default role without naming it, so the
migrations the manager wrote for it declared no role and ran as admin.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): let CE migrations connect as an explicitly named admin

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: add only missing grant options before an ACL apply, never default privileges

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix(datatables): serialize roles going on with aliases saved from other workspaces

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): note that legacy names with ? cannot be migrated

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: run one data table ACL apply at a time per server before it connects

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: hold the ACL connection to the database that was authorized

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: build the ACL connection from the authorized data table entry

Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: check ACL read reach against the entry it connects from

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* feat(datatables): put a data table's connection under Postgres roles

A data table backed by the instance database resolved to exactly one Postgres connection,
`custom_instance_user`, for everyone who could reach it at all. There was no way to say
this job reads, that one writes, this one never sees the salaries table.

A data table role is now a real Postgres login on the cluster, defined once for the
instance by a superadmin and named exactly as they named it. A script that declares
`-- role analytics` connects as `analytics`, and Postgres decides what it may touch —
grants are ordinary SQL. Windmill answers only "may this caller ask for this role", from
the tenant lists on the data table entry: `u/alice`, `g/analysts`, `f/finance` or `*`.
A data table with no `permissions` block behaves exactly as before.

Everything that opens a connection on someone's behalf goes through one chokepoint,
`get_datatable_resource_from_db`, which takes the identity explicitly and fails closed when
there is none. The role logs in as itself — never `SET ROLE`, which a script could
`RESET ROLE` its way out of.

A fork's data table entry becomes a pointer at the workspace that governs it rather than a
copy of it. The settings clone used to hand a fork a byte-identical entry naming the
parent's database, which a fork admin could edit to grant themselves `admin` there; a
pointer has nothing local to edit, and its tenants are evaluated as a member of the
governing workspace, by email. `permissions` is stripped from the workspace export and
ignored on import: tenants name principals of one workspace, and a settings push is not
where an access decision should be made.

Operations that see the whole database whatever the roles grant stay with the governing
workspace's admins: editing the roles, a migration that declares none, and opening a
replication stream for a Postgres trigger or capture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): gate the paths that reach a whole database as admin

Auditing what still resolved through the unchecked resolver turned up three that act for a
caller and hand back the admin connection: `resolve_pg_source_checked` (behind schema
export, the full-schema read, database creation, import and the forked-database drop), the
connection test, and the schema snapshot a fork clone takes of its parent. On a data table
under roles each let any workspace member — or a fork admin who is nobody in the governing
workspace — read or copy the whole database whatever its roles grant.

All three now require admin reach on the governing workspace. A dump taken under a
restricted role would be a silently truncated copy rather than an error, so refusing is the
only right answer for the copy paths.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): confine roles to the instance database, and stop a fork reaching the parent's bookkeeping

A data table role is a login on Windmill's own Postgres. Nothing stopped a workspace admin
putting a *resource-backed* data table under roles, at which point the executor dialled the
host that resource names — one the admin chose — with the role's real cluster password, and
`CONNECT` is granted to every registered instance database. Both ends now refuse: the
permissions endpoint rejects the save, and the chokepoint refuses to substitute credentials
on a non-instance entry rather than trusting the record it read.

Two more places reached the governing database without answering to it. The initial-migration
generator returned a `pg_dump` of the whole schema to any member. And the migration
rename/delete cascade followed a fork's pointer into the parent, so a fork admin renaming or
removing their own local entry relabelled or wiped the parent's `_wm_migrations` — after
which the parent re-runs every migration from zero. The remote half is now skipped when the
entry resolves into another workspace, which is also just correct: a fork renaming what it
calls a data table changes nothing about the data table.

Also: revoking a tenant now bounces the replication streams of every workspace holding an
entry that resolves here, not only the governing one, so a fork's trigger stops rather than
living on inside its open connection; the instance role catalog and the governing workspace's
tenant lists are no longer returned to someone who cannot edit them; and the tenant rename
dedup collapses non-adjacent duplicates, per role rather than once any role changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): fail loudly where a role or a pointer can be left half-recorded

Three ways the feature could end up in a state nobody could see or undo.

Creating a role writes the cluster first and the catalog second, but the catalog write was an
`UPDATE` that matched nothing when the instance Postgres settings row was absent — leaving a
live login with a password nobody recorded: invisible to the catalog, un-recreatable because
the name is taken, and un-deletable because there is no entry to delete. It now errors, so
the operation is retryable once the row is restored.

Deleting a workspace only nulls the fork lineage; the data table entries pointing at it are
left resolving to nothing. Sweeping them is not an option — turning a pointer back into a copy
would hand each fork the database outright — so the delete now names the data tables it
stranded, and resolving one says which workspace is missing rather than reporting a data table
this workspace never had.

`InstanceDatatableRole` derived `Debug` while holding a Postgres password; it is now
hand-written so `{:?}` on the catalog cannot put a live credential in a log line.

Adds the two branches the reviews found unpinned: a caller who is not a member of the
governing workspace at all, and `NoIdentity` — the compatibility path for an agent worker that
predates this and sends no job id.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): unbreak two operator messages and two comments that described other code

The two strings this branch added for states an operator hits once — the catalog write that
matched nothing, and the delete that stranded a pointer — were collapsed from their multi-line
form with the indentation left in, so both rendered with a fourteen-space gap mid-sentence.

`list_datatables` claimed to report a chain it cannot follow and then dropped it; it does drop
it, and the comment now says why that is the right place to stay quiet. The non-superadmin
check in `edit_datatable_config` was introduced as also covering references, which it does not
and need not: `reference` is overwritten from the stored entry for every caller before the
check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): serialize role catalog mutations, and state each helper's authorization contract

The catalog is one JSON document, so create, rename, enable and delete are all
read-modify-write. Two concurrent creates read the same snapshot, both succeed in the
cluster, and the second write drops the first — leaving a live Postgres login with a password
nobody recorded, which is the exact state the delete path exists to prevent. Every mutation
now runs in one transaction holding an advisory lock across the read, the cluster DDL and the
write, so a lost update cannot happen and a failure rolls the whole thing back. The DDL
helpers take that transaction rather than the pool, which is what makes the lock cover them.

Their statements moved off `sqlx::raw_sql`: the simple protocol is only needed for genuinely
multi-statement SQL, and its future is not `Send`, which an axum handler holding the
transaction requires. Each of these is one statement anyway.

The new cross-crate surface now says what callers must do. `read_role_catalog` returns
plaintext credentials; `create`/`rename`/`set_login`/`drop_instance_role` and
`converge_connect_grants` mutate cluster-wide state; `read_datatable_entry` reads a workspace's
raw config. All of them are superadmin-gated by their current handlers, but nothing said so at
the definition, which is where the next caller looks.

Also: the roles table reloads after a failed login toggle instead of leaving it claiming a flip
that did not land; the rename affordance is the design-system `Button`, not a raw one; and
`resolve_datatable_pg_as_caller` drops a `role` parameter no caller ever filled — browsing
resolves as the data table's default until the database manager grows a picker.

Why role passwords stay a plain `String` while the instance user's password beside them is a
`StringOrSecretRef`, asked three times across reviews: that one is a secret ref because an
operator supplies it and may want it from their own backend, while these are minted here and
never entered by anyone, so there is nothing for a ref to point at. Encrypting generated
secrets at rest is a separate change that would take the replication password with it. Now
said at the field.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): give the role catalog its own row, out of reach of the config machinery

Putting it inside `custom_instance_pg_databases` was the wrong call, and it cost two ways.
The catalog serializes a generated Postgres password per role, and that row is the
operator-facing instance config, so the passwords reached `get_instance_config` and its YAML
editor — a live cluster credential in a response body, a UI field and any log of either.
Worse in the other direction: `to_settings_map` strips the catalog, so a full-row upsert of
that key writes the row back without it and the catalog is gone, while the cluster keeps every
login it described.

`custom_instance_replication_pwd` is the precedent and says exactly why — a generated secret,
written only by the server, never operator-authored, hidden so the config machinery cannot
read, rewrite or drop it. The catalog is the same thing, so it now has the same shape:
`datatable_roles`, in `HIDDEN_SETTINGS`, `PROTECTED_SETTINGS` and the agent-worker denylist.
No redaction to keep in step with three code paths, and no way for a neighbouring write to
take it out.

Two races on the same shared documents. `edit_datatable_config` read the stored data tables
outside its transaction and then wrote the whole `datatable` document, so a permissions save
committing in between was silently rolled back; it now reads under `FOR UPDATE`. And
`set_datatable_permissions` validated role ids against the catalog before opening its
transaction, so a deletion in between let it write a deleted role back — including as the
default, which every later job then fails on; it now holds the catalog lock and the settings
row across validation and write.

Completes the authorization contracts the previous commit claimed but did not finish:
`read_datatable_entry` (which it named and missed), `resolve_governing_datatable`, whose whole
job is to answer for a workspace the caller may not belong to, and
`converge_connect_grants_with`, which had not inherited its wrapper's.

Also the generic Python SDK reference: `_format_py_params` learned the bare `*` last time, but
`extract_py_functions` is a second formatter and still rendered `datatable(name, role)`, so
code written from that page passed a keyword-only argument positionally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): make the concurrency test pin the handlers, and the contracts describe what is enforced

The concurrency test reimplemented the read-modify-write inline, so deleting the lock from all
three handlers left it green — it pinned Postgres, not the code it was written for. It now
drives `create_datatable_role` twice concurrently and asserts the catalog kept both names.
Checked the way the last one should have been: removing the lock from the handler makes it
fail with "wmtest_a_… is a live cluster login the catalog forgot".

The contracts added last commit were stricter than this PR's own callers, which is worse than
none — the next reader sees a rule already broken and learns to ignore it.
`read_role_catalog` said superadmin-only while two of its four callers are open to any
workspace member, and `converge_connect_grants` said superadmin while
`set_datatable_permissions` reaches it as a workspace admin. Both were fine on substance: the
rule that actually holds is about the credential never reaching a response, log, audit record
or export, not about who may call. They now say that. `read_datatable_entry` gets the same
treatment rather than the one the earlier message claimed for it: it is the primitive every
resolution goes through, so it is deliberately open, and what must not escape is `permissions`
— it names the governing workspace's users, groups and folders.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): close the last ways a role or a pointer can be left pointing at nothing

The raw settings readers hand back whatever is in the row, so moving the catalog into its own
`global_settings` key protected the config machinery and left `GET /settings/global/datatable_roles`
and the settings listing returning every live password. Both now filter that one key. The
neighbouring `custom_instance_replication_pwd` has the same shape and is not touched here: it
predates this and widening the fix to it is a decision about an operator workflow, not a
consequence of this change.

Three ways a save could leave something resolving to nothing:

A permissioned data table could be moved to a PostgreSQL resource. The block was carried across
as a server-owned field, the runtime refuses roles on a resource-backed table, so the save
succeeded and every job afterwards failed. Refused instead — turning roles off first is one step,
and it keeps discarding an access decision something somebody chose.

Renaming a governing data table left every fork pointing at the old name: the data table
disappears from their pickers and their jobs stop, with nothing in the renaming workspace to
suggest why. The rename now follows into the pointers in the same transaction.

Deleting one cannot be followed the same way, so it is reported instead — the response names what
it stranded, the way deleting a workspace does, and the fork's own error already says which
workspace is gone.

Also: `ensure_instance_db_grant_options_unchecked` claimed superadmin while the permissions
handler reaches it as a workspace admin (the same class fixed last commit, one instance missed);
the role entry kept an `instance_config_schema` derive it no longer needs; `write_role_catalog`
was the one writer of that table not stamping `updated_at`; and the concurrency test dropped its
roles only on success — a failing run is exactly the one that creates them without recording them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* refactor(datatables): put the role catalog in its own table, not in global_settings

Five findings across three rounds were all the same choice. A set of live Postgres credentials
was living in `global_settings`, which has generic read, list, write, config-export and CLI
round-trip paths that know nothing about what they carry: the passwords reached the instance
config and its YAML editor, a full-row upsert of a neighbouring key erased the catalog,
`GET /settings/global/{key}` and the settings listing returned them raw, and this round the
redaction that fixed the last two turned `wmill instance push` into something that wipes every
password — a fix breaking the assumption the previous fix made. `POST /settings/global/datatable_roles`
could also empty it outside the lock.

The approved plan offered a table or `global_settings`, so this is the other option it already
allowed rather than a new design. `datatable_role` is a table: no generic settings path can read
it, list it, export it, write it or round-trip it, so none of the five needs a guard. The
redaction, the hidden/protected/agent-denylist entries and the JSON document all go with it.

One row per role also removes the read-modify-write the concurrency work was about: two
concurrent creates are two inserts, and the unique index on `name` is what settles a collision.
The advisory lock stays for the one window rows do not cover — `CREATE ROLE` is invisible to
another transaction until commit, so without it both creates pass their `pg_roles` check.

Also from this round: rename mappings are checked against the configuration they claim to
describe, since fork pointers are rewritten from them — a caller could otherwise submit
`main -> missing` against an unchanged config and repoint every fork of `main` at a name nothing
has, and `A -> B` plus `B -> C` moved what pointed at `A` all the way to `C`. And the warning
naming forks a delete stranded reached the response but not the screen: both the data table
settings save and the workspace delete now show it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): validate a rename against the save it describes, and re-check under the locks

Three from the round, all about deciding on state that could already have moved.

A permission save resolved the data table and checked it was instance-backed before taking any
lock, then wrote under one. A config save committing in between could move the table onto a
PostgreSQL resource — recreating exactly what the transition guard refuses — or rename it, in
which case the write targeted a key that no longer existed and reported success having changed
nothing. It now re-resolves and re-checks on the locked state.

Rename validation checked that the source existed before and the target existed after, which
still accepts `main -> decoy` against a save that keeps both: every fork of `main` then follows
onto a different data table, silently, because it keeps resolving. The rule is now the actual
old-to-new key transition — a source may only survive if another rename took its name, and a
target may only pre-exist if another rename freed it. That also stops two sources sharing one
target, and it admits a swap, which the previous guard refused: `datatables` is keyed by name, so
a swap cannot be done one save at a time, and refusing it was a regression against main. The
pointer cascade now runs in two passes through a temporary name, the way the migration cascade
one layer down already handles the same shape, so `A -> B` with `B -> C` moves each pointer once
from what it named before the save.

The tenant mutators say what they are for: they write an access decision for any workspace named,
with an arbitrary mutation, and exist for the transaction that frees or renames a principal.
Editing a decision on purpose belongs in the permissions endpoint.

Carried in the same change: the stranded-fork list is a field rather than a phrase to grep out of
a success string; the pointer cascade matches with `EXISTS` instead of a `LIKE` over the whole
document, so a workspace whose pointers name something else is not rewritten to a byte-identical
value under an exclusive lock; and `InstanceDatatableRole` drops the serde derives left over from
the JSON document, one of which would emit `pwd`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): cascade on the leave route that is used, gate migrations before the admin connection, and drop a role atomically

The tenant cascade on leaving went onto `/users/leave`. The UI and the generated client call
`/workspaces/leave` — a different handler in a different crate with the same name — which
deleted the membership and left `u/<username>` in the tenant lists. Leaving and rejoining
therefore restored the access the leave was supposed to end, and a later account taking the
username would have inherited it. The regression test drives the route the client actually
calls; without the fix it fails with "leaving kept the tenant".

The migration endpoints authorized too late. `run_datatable_migrations` opened the data table's
admin connection, created `_wm_migrations` and read it before reaching the per-migration role
check — so with nothing pending, nothing was checked at all. Rollback returned before its check
when nothing was applied, and the status endpoint had none. All three now ask, before any
connection is opened, whether the caller can reach the data table as any role at all; which role
a given migration runs as is still decided per migration, and by the executor after that.

Deleting a role committed the cluster drop and the catalog row, then swept the tenant lists in
separate transactions. A sweep failing part-way left workspaces naming a role nothing can connect
as, while the retry answered `NotFound` because the catalog entry was already gone. The sweep now
runs in the same transaction, so the drop, the row and every tenant list commit together.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse to copy a data table that is under roles

pg_dump carries no roles and the import runs with --no-privileges, so a copied
data table arrives owned by the admin connection with no GRANT for any role.
The settings clone brings `permissions` across, so the fork's tenants pass
Windmill's check, connect as the role they were given, and are denied by
Postgres on everything: an entry that reads as configured and answers nothing.

Refuse the copy — in the import endpoint before any data moves, and in the fork
path the CLI takes. Replaying the source's owners and ACLs into the clone is
what lifts this, and is a change of its own. Dropping `permissions` from the
copy instead would be the unsafe half, since the copy holds the parent's rows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse the clone's database too, not only its data

A clone is two endpoints: `create_pg_database` then `import_pg_database`. Only
the second refused a data table under roles, so a fork asking to clone one
created and registered an empty `wm_fork_…` instance database and then failed —
and nothing collects it, since `drop_forked_datatable_databases` only drops
entries carrying `forked_from` and no entry names this one.

Refuse in both, so the clone stops before a database exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* nit worker error msg

* fix pg_dump stuck on version 17 on nix

* fix(datatables): refuse a malformed role annotation instead of ignoring it

`-- Role operator`, `-- role operator;` and `-- role operator -- why` all failed
the annotation parser's exact-match rule, so the query fell through to the data
table's default role and ran, silently, under a login the author did not choose.
Naming a role exists precisely to not do that.

A leading comment whose first word is `role` is now an annotation attempt: the
keyword matches case-insensitively, one trailing `;` is tolerated, and anything
else is an error naming the line. Only callers that already know the target is a
`datatable://` reference ever run this, so ordinary SQL keeps its comments.

Also bumps the dev shell's postgres client to 18 — it trailed the server the dev
database runs, which takes out every data table export, clone and fork-with-data.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): refuse a malformed role query string instead of ignoring it

`?Role=analytics`, `?role=` and `?x=1&role=…` all fell through the reference
parser's exact-match rule, so the connection resolved to the data table's default
role and ran under a login the caller never asked for — the URI half of the same
trap as a malformed `-- role` annotation.

The key now matches case-insensitively, and anything else in the query string is
an error naming it; `role` is the only parameter a reference takes. Callers that
only need the entry keep a lenient `datatable_ref_name`, since they never act on
the role. The DuckDB `ATTACH` parser propagates it rather than attaching under
the default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012ti5HyeTikPMYyW8YSdiHR

* fix(datatables): carry the role annotation into the row_to_json retry

The retry rebuilds its SQL from `pruneComments(code)`, so the leading comment
block never reached the second attempt — and with it the `-- role <name>` line
that decides which login the query runs as. The retry connected as the data
table's default role instead, so a query the first attempt was denied could
succeed on the second, reported as "recovered with the row_to_json fix".

Carry the leading comment block over. The retry itself is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore(datatables): don't mount the roles UI until the ACL editor lands

Enforcement ships first. The permissions drawer is what turns roles on, and the
catalog section is what creates them — both are only useful once there is a way
to grant a role the privileges it needs, which arrives with the ACL editor. Left
mounted they would offer a feature whose other half does not exist.

The two components are complete and reviewed; only their call sites here are
commented out, with a note pointing the follow-up PRs at them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): honour `-- role: x`, and fix the DuckDB attach test

Two review findings, both real.

`attach_datatable_parses_name_and_role` never compiled: `parse_attach_datatable`
returns `Result<Option<_>>` now and one call site kept a single `unwrap`. Its
`?Role=analytics` case also asserted a refusal, contradicting the parser in the
same commit, which matches the key case-insensitively. Replaced with the cases
that are genuinely malformed, and a positive one for the cased key.

`-- role: analytics` fell through to the default role — the silent fallback the
strict parser exists to remove, for the spelling most likely to be typed. The
keyword now accepts an optional colon, attached or spaced, while a word that
merely starts with it (`rolebased`) is still not an attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): clone a fork's pointer instead of failing after the copy

Forking a fork with cloning left an orphan database. The preflight resolves the
pointer and sees the governing entry, so both endpoints ran and filled the new
database; `apply_forked_datatable` then refused the inherited pointer and rolled
the fork back, stranding a registered `wm_fork_*` that no entry names and whose
name blocks the retry.

Refusing earlier would have been the smaller change, but forking a fork and
cloning worked before pointers existed, so it would trade an orphan for a
regression. Resolve what the pointer names and write the terminal entry the
clone needs: the whole `database` object rather than a patch of its
`resource_path`, since a pointer has none, and `reference` removed with it.

Also accepts `-- role=x` and `-- Role = x`, two more spellings that fell through
to the default role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse to roll back the catalog while roles exist

The down migration dropped the table and left every role behind: live Postgres
logins whose passwords only that table carried, so after a revert Windmill could
neither use, disable nor delete them, and re-applying could not recreate them
because the names were taken. Cleaning up here is not possible either — dropping
a role means reassigning what it owns in every instance database, and a
migration runs in one — so it now refuses while the catalog is non-empty and
says to delete the roles through instance settings, which does the cluster work.

Also enforces the instance-only invariant the resolved-pointer clone relies on
rather than only asserting it in a comment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* refactor(datatables): settle clonability in one place, before anything is created

A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): let a retried clone reclaim its own leftover database

A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert "fix(datatables): let a retried clone reclaim its own leftover database"

This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): record the stale clone database as a known limitation

A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): bounce the streams reading a data table when it is deleted

Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): keep the fork schema baseline, and bounce streams on every removal

Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* refactor(datatables): keep Postgres triggers and data table roles apart

A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved

Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): disable a data table role before deleting it

Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with a stream starting

Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): wait out live listeners, and resolve stored names containing `?`

Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): warn when a settings sync strands fork pointers

A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): refuse a save that drops a data table's roles through an undeclared rename

A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* fix(datatables): no entry without roles may newly reach a database under roles

The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move data table role catalog and resolution to the enterprise edition

Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table permissions endpoints to the enterprise edition

The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): move the data table role catalog endpoints to the enterprise edition

The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the roles tests on the enterprise edition, refusals without it

Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* feat(datatables): gate the roles UI mount sites on an enterprise license

Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* test(datatables): run the tenant matcher test on the enterprise edition

The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb

* chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607

This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.

* fix(datatables): refuse roles while a same-workspace alias reaches the database

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): let CE migrations connect as an explicitly named admin

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): serialize roles going on with aliases saved from other workspaces

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(datatables): note that legacy names with ? cannot be migrated

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(datatables): add an ACL editor for data table roles

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: take every pooled connection before the ACL apply locks

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: refresh grant options only after the ACL apply validates its plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: add only missing grant options before an ACL apply, never default privileges

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: run one data table ACL apply at a time per server before it connects

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: hold the ACL connection to the database that was authorized

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: build the ACL connection from the authorized data table entry

Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: check ACL read reach against the entry it connects from

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix(datatables): drop a DuckDB data table secret once its ATTACH has used it

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* perf(datatables): resolve a workspace's data tables per pointer hop, not per entry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): hold the parent's settings while a fork points at its data tables

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(datatables): add an ACL editor for data table roles

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: take every pooled connection before the ACL apply locks

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: refresh grant options only after the ACL apply validates its plan

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: add only missing grant options before an ACL apply, never default privileges

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: run one data table ACL apply at a time per server before it connects

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: hold the ACL connection to the database that was authorized

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: build the ACL connection from the authorized data table entry

Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* fix: check ACL read reach against the entry it connects from

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb

* chore: update ee-repo-ref to 7e338e4dabf91689bfd7fb0333c6534040b17b59

This commit updates the EE repository reference after PR #787 was merged in windmill-ee-private.

Previous ee-repo-ref: 0edd40979cf36bfba59323f3f6a0811ae1369cf5

New ee-repo-ref: 7e338e4dabf91689bfd7fb0333c6534040b17b59

Automated by sync-ee-ref workflow.

* feat(datatables): clone a data table under roles with its owners and grants (#11120)

Claude-Session: https://claude.ai/code/session_01UbrtwiYNfayrmqouBJHwGV

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix: open the raw app data table drawer when the workspace has none

Selecting the first data table of an empty list passed undefined to the name
check, which threw instead of opening the drawer on no data table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): offer cloning a data table under roles where its grants can be replayed

The server clones such a data table and replays the source's owners and grants,
which only the Enterprise Edition does, so the fork wizard hid both clone
options everywhere instead of on a build that cannot replay them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs: name the placeholder the empty raw app data drawer renders

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test: pin the enterprise refusal the role pickers read as 'not under roles'

The server's sentence and the frontend's copy of it were coupled by nothing,
so rewording either one turned every role picker on a community build into a
failed lookup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): read a roles answer only for the workspace it was asked in

A fork and its parent each have their own roles on a data table of the same
name, so an answer stamped with the name alone settled the role from the
workspace the editor was acting on before.

Also derive the AI table creation flag from the data replaced into the editor:
data naming no data table left the flag on from before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): keep an instance database a settings save is waiting to name

Cleanup for a database whose setup failed took the lock first, read no user,
and dropped it while a save blocked on that same lock was about to commit a
reference to it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): check the workspace stamp in the default database selector too

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): read what a save racing instance-database cleanup committed

A transaction blocked on the lock may still roll back, so keeping the database
for it stranded one whose name then blocks every retry: it is let through and
its outcome read instead. The waiter query also matches this database's locks
only, since pg_locks spans the cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): tell a waiting request apart from the workspaces using a database

Both callers render what cleanup returns as the workspaces that keep the
database, so a waiting request's pid read as one of them. Each now words that
case itself, and the give-up comment names where the kept name actually goes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): take the cleanup lock on a connection the pool cannot reclaim

A session lock outlives the future holding it, so a cancellation between
taking it and releasing it handed a locked session back to the pool, where
every later settings save waits on it. Detached, the connection closes when it
is dropped and the server releases the lock with the session.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(datatables): close the cleanup connection on drop instead of detaching it

Detaching released the pool permit while the session stayed alive, so
concurrent cleanups waiting on their locks could open as many connections as
they liked. Closing on drop covers the same cancellation and keeps them
counted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to fd5b8af748f2c985b13e18d9ea30894f3bd7e9a3

This commit updates the EE repository reference after PR #798 was merged in windmill-ee-private.

Previous ee-repo-ref: 3145e422d61d580f0a82804f075285c112879da0

New ee-repo-ref: fd5b8af748f2c985b13e18d9ea30894f3bd7e9a3

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Diego Imbert
2026-09-21 12:03:01 +00:00
committed by GitHub
co-authored by Claude Opus 5 windmill-internal-app[bot]
parent 4f7f705f8c
commit e1e3692fbc
68 changed files with 5431 additions and 1428 deletions
File diff suppressed because it is too large Load Diff
@@ -1,16 +1,21 @@
<script lang="ts">
import { dbSchemas, type DBSchema } from '$lib/stores'
import type { DataTableTables } from '$lib/gen'
import { sortArray } from '$lib/utils'
import { Loader2, RefreshCcw } from 'lucide-svelte'
import Alert from './common/alert/Alert.svelte'
import Button from './common/button/Button.svelte'
import { dbSupportsSchemas } from './apps/components/display/dbtable/utils'
import {
dbSupportsSchemas,
getLanguageByResourceType
} from './apps/components/display/dbtable/utils'
import DbManager from './DBManager.svelte'
import DbWorkerTagPicker from './DbWorkerTagPicker.svelte'
import MissingWorkerTagAlert from './jobs/MissingWorkerTagAlert.svelte'
import {
dbSchemaOpsWithPreviewScripts,
dbTableOpsWithPreviewScripts,
getDatabaseArg,
getDbType,
getDefaultDbTag,
getDucklakeSchema
@@ -18,11 +23,11 @@
import { Pane, Splitpanes } from 'svelte-splitpanes'
import SqlRepl from './SqlRepl.svelte'
import SimpleAgTable from './SimpleAgTable.svelte'
import { type Snippet } from 'svelte'
import type { DbInput } from './dbTypes'
import type { DatatableRowAction, DbInput } from './dbTypes'
import { schemaCacheKey } from './dbSchemaCache'
import { getDbSchemas, loadAllTablesMetaData } from './apps/components/display/dbtable/metadata'
import type { SelectedTable } from './DBManager.svelte'
import type { PendingRowAction, SelectedTable } from './DBManager.svelte'
import { getDbFeatures } from './apps/components/display/dbtable/dbFeatures'
import { resource } from 'runed'
import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte'
@@ -39,7 +44,15 @@
hasReplResult?: boolean
selectedSchemaKey?: string | undefined
selectedTableKey?: string | undefined
dbSelector?: Snippet<[]>
/** Every data table with its schemas and tables, for the left-pane tree. Undefined when
* the manager is not on a data table, which drops the tree's top level. */
datatableTree?: DataTableTables[]
datatableTreeLoading?: boolean
onSelectDatatable?: (datatable: string) => void
onSelectRole?: (datatable: string, role: string) => void
pendingAction?: PendingRowAction | undefined
onDatatableAction?: (datatable: string, action: DatatableRowAction) => void
canManageDatatable?: boolean
/** Enable multi-select mode with checkboxes in sidebar */
multiSelectMode?: boolean
/** Selected tables in multi-select mode */
@@ -61,7 +74,13 @@
hasReplResult = $bindable(false),
selectedSchemaKey = $bindable(undefined),
selectedTableKey = $bindable(undefined),
dbSelector,
datatableTree,
datatableTreeLoading,
onSelectDatatable,
onSelectRole,
pendingAction = $bindable(),
onDatatableAction,
canManageDatatable,
multiSelectMode = false,
selectedTables = $bindable([]),
disabledTables = [],
@@ -72,33 +91,25 @@
let ws = $derived(workspace ?? $operatingWorkspace)
let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[schemaCacheKey(input)])
let dbSchema: DBSchema | undefined = $derived(input && $dbSchemas[schemaCacheKey(ws, input)])
const outOfOrderModal = createAsyncConfirmationModal()
function getDbSchemasPath(input: DbInput): string {
switch (input.type) {
case 'database':
return input.resourcePath
case 'ducklake':
return 'ducklake://' + input.ducklake
}
}
// Scope the shared `dbSchemas` cache by the acting workspace: a datatable of
// the same name can exist in both the nav and the acting workspace, so the
// bare resource path alone would let one workspace's schema be reused for the
// other while DB operations target the acting one.
function schemaCacheKey(input: DbInput): string {
return `${ws}:${getDbSchemasPath(input)}`
}
// Reported in place of the loading spinner: both queries run as jobs, so
// anything from a bad connection to a tag no worker serves surfaces here
// instead of leaving the manager spinning with no explanation. Each query
// owns its slot so neither can clear the other's error on a refetch.
let schemaError = $state<string | undefined>(undefined)
let colDefsError = $state<string | undefined>(undefined)
function emptySchemaFor(db: DbInput): DBSchema {
return {
lang: db.type === 'ducklake' ? 'ducklake' : getLanguageByResourceType(db.resourceType),
schema: {},
publicOnly: undefined,
stringified: ''
} as DBSchema
}
let loadError = $derived(
schemaError
? { title: 'Could not load the database schema', message: schemaError }
@@ -138,14 +149,23 @@
const run = ++schemaRun
schemaError = undefined
if (!input) return
const dbSchemasPath = schemaCacheKey(input)
const dbSchemasPath = schemaCacheKey(ws, input)
if (input.type == 'database') {
let connection = input.resourcePath
try {
// The role'd reference, validated: an invalid role fails here rather than
// reading the schema as the data table's default role.
if (connection.startsWith('datatable://')) connection = getDatabaseArg(input).database!
} catch (e) {
schemaError = (e as Error)?.message ?? String(e)
return
}
// Reported through a local, not `schemaError` directly, so a superseded
// run's callback can't fail a load that already succeeded.
let queryError: string | undefined
const schema = await getDbSchemas(
input.resourceType,
input.resourcePath,
connection,
ws,
(message: string) => (queryError = message),
{ customTag: workerTag }
@@ -225,17 +245,23 @@
}}
/>
<!-- The error branch comes first on purpose: `dbSchema` is read from a cache that
survives a failed refetch, so ordering it first would hide the failure behind
stale content. -->
{#if loadError}
<!-- A load error replaces only the data pane: the tree, its role badge and menus, and the REPL
stay usable, so another data table or role can be picked and the connection tried by hand.
The tree then gets an empty schema: the cached one survives a failed refetch and would pass
stale content off as what this connection reaches. -->
{#snippet errorPane()}
<div class="h-full w-full flex flex-col items-center justify-center gap-3 p-8">
<div class="max-w-2xl w-full flex flex-col gap-3">
<Alert type="error" title={loadError.title} size="xs">
{loadError.message}
<Alert type="error" title={loadError?.title ?? ''} size="xs">
{loadError?.message}
</Alert>
<div class="self-start">
<Button size="xs" color="light" startIcon={{ icon: RefreshCcw }} on:click={() => refresh()}>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: RefreshCcw }}
on:click={() => refresh()}
>
Retry
</Button>
</div>
@@ -249,9 +275,12 @@
/>
</div>
</div>
{:else if dbSchema && ws && input}
{/snippet}
{#if (loadError || dbSchema) && ws && input}
{@const _input = input}
{@const dbType = getDbType(_input)}
{@const shownSchema = loadError || !dbSchema ? emptySchemaFor(_input) : dbSchema}
<Splitpanes horizontal>
<Pane class="relative">
<!-- svelte-ignore a11y_click_events_have_key_events -->
@@ -276,9 +305,11 @@
</div>
<DbManager
dbSupportsSchemas={dbSupportsSchemas(dbType)}
databaseIsEmpty={!Object.values(dbSchema.schema).flatMap((s) => Object.values(s)).length}
{dbSchema}
colDefs={colDefs.current}
databaseIsEmpty={!loadError &&
!Object.values(shownSchema.schema).flatMap((s) => Object.values(s)).length}
dbSchema={shownSchema}
mainPane={loadError ? errorPane : undefined}
colDefs={loadError ? undefined : colDefs.current}
dbTableOpsFactory={({ colDefs, tableKey, whereClause }) =>
dbTableOpsWithPreviewScripts({
colDefs,
@@ -308,7 +339,15 @@
: undefined}
{dbType}
refresh={() => refresh()}
{dbSelector}
{datatableTree}
{datatableTreeLoading}
{onSelectDatatable}
{onSelectRole}
workspace={ws}
currentRole={input.type === 'database' ? input.role : undefined}
bind:pendingAction
{onDatatableAction}
{canManageDatatable}
{onImport}
bind:selectedSchemaKey
bind:selectedTableKey
@@ -331,12 +370,12 @@
onSchemaChange={() => refresh()}
placeholderTableName={sortArray(
Object.keys(
dbSchema?.schema[
'public' in dbSchema?.schema
shownSchema.schema[
'public' in shownSchema.schema
? 'public'
: 'dbo' in dbSchema?.schema
: 'dbo' in shownSchema.schema
? 'dbo'
: Object.keys(dbSchema?.schema ?? {})?.[0]
: Object.keys(shownSchema.schema ?? {})?.[0]
] ?? {}
)
)?.[0]}
@@ -1,25 +1,25 @@
<script lang="ts">
import { superadmin, userStore, workspaceStore } from '$lib/stores'
import { WorkspaceService } from '$lib/gen'
import { enterpriseLicense, superadmin, userStore, workspaceStore } from '$lib/stores'
import { WorkspaceService, type DataTableTables } from '$lib/gen'
import { listUsableDatatableRoles } from './datatableUsableRoles'
import Button from './common/button/Button.svelte'
import Drawer from './common/drawer/Drawer.svelte'
import DrawerContent from './common/drawer/DrawerContent.svelte'
import Select from './select/Select.svelte'
import {
ArrowLeft,
Copy,
Download,
Expand,
LoaderCircle,
Minimize,
RefreshCcw,
Upload
} from 'lucide-svelte'
import { ArrowLeft, Copy, Download, Expand, Minimize, RefreshCcw, Upload } from 'lucide-svelte'
import DBManagerContent from './DBManagerContent.svelte'
import type { PendingRowAction } from './DBManager.svelte'
import DataTableMigrationsButton from './workspaceSettings/DataTableMigrationsButton.svelte'
import DataTablePermissionsButton from './workspaceSettings/DataTablePermissionsButton.svelte'
import { resource } from 'runed'
import { untrack } from 'svelte'
import { tick, untrack } from 'svelte'
import type { DbManagerUriState } from './dbManagerDrawerModel.svelte'
import {
ADMIN_DATATABLE_ROLE,
datatableNameTakesRole,
defaultMigrationRole,
type DatatableRowAction
} from './dbTypes'
import ResourcePicker from './ResourcePicker.svelte'
import Alert from './common/alert/Alert.svelte'
import { sendUserToast } from '$lib/toast'
@@ -41,31 +41,122 @@
// the editor that opened it (set via openDrawer), else the nav workspace.
let ws = $derived(uriState.workspace ?? $workspaceStore)
// Load available datatables when drawer opens with datatable input
const datatables = resource<string[]>([], async () => {
if (!ws) return []
try {
return (await WorkspaceService.listDataTables({ workspace: ws })).map((d) => d.name)
} catch (e) {
console.error('Failed to load datatables:', e)
return []
}
})
// A create started on a data table other than the current one: survives the
// re-mount the switch causes.
let pendingAction = $state<PendingRowAction | undefined>(undefined)
const datatableItems = $derived(
datatables.current.map((dt) => ({
value: dt,
label: dt
}))
// Read once through primitives: the getters return values of a freshly parsed URL, which
// changes on every table click, and the listings below must not refetch for that.
const selectedDatatable = $derived(uriState.selectedDatatable)
const selectedRole = $derived(uriState.selectedRole)
// Roles the caller may use, to settle the role before anything connects. Offering only
// these is a convenience: the server refuses any other.
const usableRoles = resource(
() => [ws, selectedDatatable] as const,
async ([workspace, datatable]) => {
if (!workspace || !datatable) return undefined
try {
return {
workspace,
datatable,
...(await listUsableDatatableRoles(workspace, datatable))
}
} catch (e) {
// Never leave the drawer waiting on this: fall back to the
// unpermissioned shape so it opens and the server picks the role.
console.error('Failed to load datatable roles:', e)
return {
workspace,
datatable,
permissioned: false,
roles: [],
default_role: ADMIN_DATATABLE_ROLE
}
}
}
)
// Refetch datatables when switching to a datatable input
// A resource keeps its previous value while refetching, and roles are per data table of one
// workspace: settling from another answer would connect to this data table as a role it may
// not even have.
const rolesOfCurrent = $derived(
usableRoles.current?.datatable === selectedDatatable && usableRoles.current?.workspace === ws
? usableRoles.current
: undefined
)
// Nothing that connects runs until the role is settled: a first round sent without a role
// would run — and cache — as whatever the server defaults to.
const roleSettled = $derived(
!uriState.isDatatableInput ||
(rolesOfCurrent !== undefined &&
(!rolesOfCurrent.permissioned ||
rolesOfCurrent.roles.length === 0 ||
selectedRole !== undefined ||
// Its reference cannot name a role, so it connects as the default one.
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))))
)
// Make the role explicit before anything queries the data table, so the URL, the
// cache and every migration the manager writes name it. A role already in the URL
// is kept even when it is not usable: the server refuses it, visibly.
$effect(() => {
if (uriState.isDatatableInput) {
untrack(() => datatables.refetch())
}
const roles = rolesOfCurrent
if (
!roles?.permissioned ||
selectedRole !== undefined ||
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))
)
return
const effective = roles.roles.includes(roles.default_role) ? roles.default_role : roles.roles[0]
if (effective) untrack(() => (uriState.selectedRole = effective))
})
const contentInput = $derived.by(() => {
const input = uriState.effectiveInput
if (input?.type !== 'database' || selectedDatatable === undefined) return input
const migrationRole = defaultMigrationRole(
selectedDatatable,
rolesOfCurrent?.permissioned,
rolesOfCurrent?.default_role
)
return migrationRole === undefined ? input : { ...input, migrationRole }
})
// Every data table with its schemas and tables, in one call: this is what the
// left pane's tree navigates, so it has to cover the data tables the user is
// not currently on, not just the selected one. The privileges it reports are
// the connected role's, so the role picked on the open data table is part of
// what is being asked. Gated on the drawer being open on a data table: this
// reaches every data table's database in turn, and the component is mounted on
// every logged-in page.
let datatablesRun = 0
const datatables = resource(
() =>
[
open && uriState.isDatatableInput,
ws,
selectedDatatable,
selectedRole,
roleSettled
] as const,
async ([active, workspace, roleFor, role, settled]): Promise<DataTableTables[]> => {
if (!active || !workspace) return []
if (!settled) return untrack(() => datatables.current)
const run = ++datatablesRun
try {
const result = await WorkspaceService.listDataTableTables({ workspace, roleFor, role })
// An answer for a selection that has since changed describes another role.
return run === datatablesRun ? result : untrack(() => datatables.current)
} catch (e) {
console.error('Failed to load datatables:', e)
return run === datatablesRun ? [] : untrack(() => datatables.current)
}
},
{ initialValue: [] }
)
function handleClose() {
uriState.closeDrawer()
dbManagerContent?.clearReplResult()
@@ -78,6 +169,10 @@
if (!open) {
expand = false
uriState.closeDrawer()
// An action asked for on one data table must not be waiting when the
// drawer is next opened on another database — or on no data table at
// all, where nothing would recognise it as foreign.
pendingAction = undefined
}
})
@@ -97,6 +192,8 @@
let importDrawerOpen = $state(false)
let importLoading = $state(false)
let importSource = $state<string | undefined>(undefined)
/** Which database an import writes into; set when driven from a tree row. */
let importTarget = $state<string | undefined>(undefined)
let importBehavior = $state<'schema_only' | 'schema_and_data'>('schema_only')
let isPostgresqlInput = $derived(
@@ -116,13 +213,49 @@
return toSourceIdentifier(input.resourcePath)
}
// The tree's row menus act on the data table of the row that was clicked, which
// is not necessarily the one currently open — so the target is set first and the
// headless modals are keyed on it.
let actionDatatable = $state<string | undefined>(undefined)
let migrationsModal = $state<DataTableMigrationsButton | undefined>()
let permissionsDrawer = $state<DataTablePermissionsButton | undefined>()
async function runDatatableAction(datatable: string, action: DatatableRowAction) {
actionDatatable = datatable
// Let the keyed block above mount against the new target before driving it.
await tick()
switch (action) {
case 'migrations':
migrationsModal?.open()
break
case 'roles':
permissionsDrawer?.open()
break
case 'export':
await handleExportSchema(`datatable://${datatable}`)
break
case 'import':
importTarget = `datatable://${datatable}`
importDrawerOpen = true
break
}
}
function refreshManager() {
dbManagerContent?.refresh()
dbManagerContent?.dbManager()?.dbTable()?.refresh()
refreshRoles()
}
async function handleExportSchema() {
const source = currentSourceIdentifier()
/** Re-read what the tree and the role picker show: both are answers about the
* data table's roles, which the permissions drawer can have just changed. */
function refreshRoles() {
datatables.refetch()
usableRoles.refetch()
}
async function handleExportSchema(explicitSource?: string) {
const source = explicitSource ?? currentSourceIdentifier()
if (!source || !ws) return
try {
exportResult = await WorkspaceService.exportPgSchema({
@@ -137,7 +270,7 @@
async function handleImportDatabase() {
if (!importSource || !ws) return
const target = currentSourceIdentifier()
const target = importTarget ?? currentSourceIdentifier()
if (!target) return
importLoading = true
try {
@@ -183,52 +316,48 @@
noPadding
id="db-manager-drawer"
>
{#if uriState.effectiveInput && ws}
{#key uriState.selectedDatatable}
{#if contentInput && ws && roleSettled}
{#key `${selectedDatatable}~${selectedRole ?? ''}`}
<DBManagerContent
bind:this={dbManagerContent}
input={uriState.effectiveInput}
input={contentInput}
workspace={uriState.workspace}
datatableTree={uriState.isDatatableInput ? datatables.current : undefined}
datatableTreeLoading={datatables.loading}
onSelectDatatable={(dt) => (uriState.selectedDatatable = dt)}
onSelectRole={(dt, role) => {
// Setting the data table clears the role, so the order matters.
uriState.selectedDatatable = dt
uriState.selectedRole = role
}}
bind:pendingAction
canManageDatatable={!!($superadmin || $userStore?.is_admin) &&
!!$enterpriseLicense &&
!isCloudHosted()}
onDatatableAction={runDatatableAction}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey={uriState.selectedSchema}
bind:selectedTableKey={uriState.selectedTable}
onImport={enableImportExport
? (mode) => ((importDrawerOpen = true), (importBehavior = mode))
? (mode) => (
(importTarget = undefined),
(importDrawerOpen = true),
(importBehavior = mode)
)
: undefined}
>
{#snippet dbSelector()}
{#if uriState.isDatatableInput}
{#if datatables.loading}
<div class="flex items-center gap-2 text-tertiary ml-2">
<LoaderCircle size={14} class="animate-spin" />
<span class="text-sm">Loading...</span>
</div>
{:else if datatables.current.length >= 1}
<Select
transformInputSelectedText={(s) => `Datatable: ${s}`}
items={datatableItems}
bind:value={uriState.selectedDatatable}
placeholder="Select data table"
size="md"
/>
{/if}
{/if}
{/snippet}
</DBManagerContent>
></DBManagerContent>
{/key}
{/if}
{#snippet actions()}
{#if uriState.isDatatableInput && uriState.selectedDatatable && ws}
<DataTableMigrationsButton
workspace={ws}
datatable={uriState.selectedDatatable}
onSchemaChanged={refreshManager}
/>
{/if}
{#if enableImportExport}
<Button startIcon={{ icon: Download }} onClick={handleExportSchema}>Export</Button>
<Button startIcon={{ icon: Upload }} onClick={() => (importDrawerOpen = true)}>
<!-- A data table exports and imports from its row menu in the tree; a plain
database has no tree row to hold them. -->
{#if enableImportExport && !uriState.isDatatableInput}
<Button startIcon={{ icon: Download }} onClick={() => handleExportSchema()}>Export</Button>
<Button
startIcon={{ icon: Upload }}
onClick={() => ((importTarget = undefined), (importDrawerOpen = true))}
>
Import
</Button>
{/if}
@@ -260,6 +389,27 @@
</DrawerContent>
</Drawer>
{#if actionDatatable && ws}
{#key actionDatatable}
<DataTableMigrationsButton
bind:this={migrationsModal}
hideTrigger
workspace={ws}
datatable={actionDatatable}
onSchemaChanged={refreshManager}
/>
{#if $enterpriseLicense && !isCloudHosted()}
<DataTablePermissionsButton
bind:this={permissionsDrawer}
hideTrigger
workspace={ws}
datatable={actionDatatable}
onSaved={refreshRoles}
/>
{/if}
{/key}
{/if}
<Drawer bind:open={exportDrawerOpen} size="800px" offset={offset + 1}>
<DrawerContent title="Export Schemas" on:close={() => (exportDrawerOpen = false)}>
{#if exportResult}
@@ -503,7 +503,7 @@
{/if}
{#if askingForConfirmation?.codeContent}
<div
class="bg-surface-secondary border border-surface-selected rounded-md p-2 relative group"
class="bg-surface-secondary border border-surface-selected rounded-md p-2 relative group min-w-0"
>
<button
class="absolute top-2 right-2 opacity-0 group-hover:opacity-100 transition-opacity p-1 rounded hover:bg-surface-hover"
@@ -512,9 +512,7 @@
>
<ClipboardCopy size={14} />
</button>
<pre class="whitespace-pre-wrap text-sm"
><code>{askingForConfirmation.codeContent}</code></pre
>
<pre class="overflow-x-auto text-sm"><code>{askingForConfirmation.codeContent}</code></pre>
</div>
{/if}
</ConfirmationModal>
@@ -0,0 +1,66 @@
<script lang="ts">
import { ChevronDown } from 'lucide-svelte'
import SelectDropdown from './select/SelectDropdown.svelte'
import Badge from './common/badge/Badge.svelte'
import { clickOutside } from '$lib/utils'
let {
role,
roles,
onSelect
}: {
/** The role in effect, shown on the badge. */
role: string
/** The roles the caller may switch to. */
roles: string[]
onSelect: (role: string) => void
} = $props()
let open = $state(false)
let anchorEl: HTMLSpanElement | undefined = $state()
const items = $derived(roles.map((r) => ({ label: r, value: r })))
// The table picker's drawer opens at `disposables + 10000`, which the
// dropdown's own z-index would sit under.
const dropdownClass = 'z-[20000]'
</script>
<span
bind:this={anchorEl}
class="relative flex min-w-0"
use:clickOutside={{ onClickOutside: () => (open = false) }}
>
<Badge
clickable
color="gray"
wrapperClass="min-w-0"
class="min-w-0 gap-0.5 pl-2 pr-1 bg-surface-sunken hover:bg-surface-sunken text-primary
transition-[filter,transform] hover:brightness-95 active:brightness-90 active:scale-[0.97]
{open ? 'brightness-95' : ''}"
onclick={(e) => {
// The row underneath folds on click, and picking a role is not that.
e.stopPropagation()
open = !open
}}
>
<!-- A long role name gives way rather than pushing the row's own actions
past its right edge. -->
<span class="truncate">{role}</span>
<ChevronDown
size={11}
class="shrink-0 text-secondary transition-transform {open ? 'rotate-180' : ''}"
/>
</Badge>
<SelectDropdown
processedItems={items}
value={role}
{open}
listAutoWidth={false}
class={dropdownClass}
getInputRect={anchorEl && (() => anchorEl!.getBoundingClientRect())}
onSelectValue={(item) => {
open = false
if (item.value !== role) onSelect(item.value)
}}
/>
</span>
@@ -6,8 +6,18 @@
import { joinSqlStatements, splitSqlRuns } from './sqlDdl'
import { logDdlGuardChoice } from './workspaceSettings/datatableTelemetry'
import { CornerDownLeft } from 'lucide-svelte'
import { withMigrationRole } from './datatableMigrationRole'
let { workspace, datatable }: { workspace: string; datatable: string } = $props()
let {
workspace,
datatable,
role
}: {
workspace: string
datatable: string
/** The role the editor runs as. The migration declares it, or it would run as admin. */
role?: string
} = $props()
type Choice = 'run' | 'migrate' | 'cancel'
@@ -73,7 +83,7 @@
function openMigrationModal(sql: string): Promise<boolean> {
return new Promise((resolve) => {
resolveMigrationClosed = (created: boolean) => resolve(created)
newMigrationModal?.open({ codeUp: sql })
newMigrationModal?.open({ codeUp: withMigrationRole(sql, role) })
})
}
@@ -145,6 +155,11 @@
migrations rather than run ad-hoc. Create a migration for it instead?
{/if}
</p>
{#if role}
<p class="text-sm text-secondary">
It will run as role <span class="font-mono">{role}</span>.
</p>
{/if}
<pre
class="text-xs whitespace-pre-wrap font-mono bg-surface-secondary rounded p-3 max-h-48 overflow-auto"
>{promptSql}</pre
@@ -1094,9 +1094,10 @@
the home page’s create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, whether a pre-approved trial offer was opened, whether data tables are put
under roles and whether callers name a role or take the default, and which kinds of
access change (grant, revoke, ownership, default privileges) are applied to data
tables, last 30 days)</li
under roles and whether callers name a role or take the default, which kinds of access
change (grant, revoke, ownership, default privileges) are applied to data tables, and
whether a data table under roles is cloned into a fork with its schema only or with
its data, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
@@ -1163,9 +1164,10 @@
the home page’s create menu and hub-project picker are opened and from which entry
point, the name of any public hub project imported from the home page and how far that
import got, whether a pre-approved trial offer was opened, whether data tables are put
under roles and whether callers name a role or take the default, and which kinds of
access change (grant, revoke, ownership, default privileges) are applied to data
tables, last 30 days)</li
under roles and whether callers name a role or take the default, which kinds of access
change (grant, revoke, ownership, default privileges) are applied to data tables, and
whether a data table under roles is cloned into a fork with its schema only or with
its data, last 30 days)</li
>
<li
>feature adoption (counts of which flow, script, trigger, worker and data table
+6 -1
View File
@@ -225,5 +225,10 @@
</Splitpanes>
{#if datatableName && ws}
<DdlMigrationGuard bind:this={ddlGuard} workspace={ws} datatable={datatableName} />
<DdlMigrationGuard
bind:this={ddlGuard}
workspace={ws}
datatable={datatableName}
role={input.type === 'database' ? (input.role ?? input.migrationRole) : undefined}
/>
{/if}
+5 -4
View File
@@ -9,9 +9,10 @@
kind: FavoriteKind
summary?: string
workspaceId?: string
size?: number
}
let { path, kind, workspaceId, summary }: Props = $props()
let { path, kind, workspaceId, summary, size = 16 }: Props = $props()
let buttonHover = $state(false)
let starred = $derived(favoriteManager.isStarred(path, kind))
@@ -31,14 +32,14 @@
>
{#if starred}
{#if buttonHover}
<StarOff size={16} fill="currentcolor" />
<StarOff {size} fill="currentcolor" />
{:else}
<Star size={16} fill="currentcolor" />
<Star {size} fill="currentcolor" />
{/if}
{:else}
<Star
class={!buttonHover ? 'opacity-60' : ''}
size={16}
{size}
fill={buttonHover ? 'currentcolor' : 'none'}
/>
{/if}
@@ -282,7 +282,8 @@ const scriptsV2: typeof legacyScripts = {
...legacyScripts.postgresql,
code: `
SELECT table_name, column_name, udt_name, column_default, is_nullable, nsp.nspname AS table_schema FROM information_schema.columns
RIGHT JOIN pg_namespace nsp ON table_schema = nsp.nspname WHERE nsp.nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')`
RIGHT JOIN pg_namespace nsp ON table_schema = nsp.nspname WHERE nsp.nspname NOT IN ('information_schema', 'pg_toast', 'pg_catalog')
AND NOT starts_with(nsp.nspname, 'pg_') AND has_schema_privilege(nsp.oid, 'USAGE')`
}
}
@@ -197,6 +197,7 @@
one long unbreakable string (a path list, a URL) sizes this column by
that string and pushes it out of the panel — and any `truncate` inside
never engages. -->
<div class={twMerge('ml-0 text-left flex-1 min-w-0', showIcon ? 'ml-4' : '')}>
<h3 class="text-lg font-medium text-primary">
{title}
@@ -732,12 +732,14 @@ export class AIChatManager implements ChatViewHost {
/** Every mounted flow editor. */
#flowEditors = new Set<FlowAIChatHelpers>()
appAiChatHelpers = $state<AppAIChatHelpers | undefined>(undefined)
/** Datatable creation policy: enabled flag, datatable name, and optional schema */
/** Datatable creation policy: enabled flag, datatable name, optional schema, and the role the
* app uses each data table through */
datatableCreationPolicy = $state<{
enabled: boolean
datatable: string | undefined
schema: string | undefined
}>({ enabled: false, datatable: undefined, schema: undefined })
roles?: Record<string, string>
}>({ enabled: false, datatable: undefined, schema: undefined, roles: undefined })
pendingNewCode = $state<string | undefined>(undefined)
apiTools = $state<Tool<any>[]>([])
aiChatInput = $state<AIChatInput | null>(null)
@@ -69,6 +69,7 @@
<DefaultDatabaseSelector
datatable={aiChatManager.datatableCreationPolicy.datatable}
schema={aiChatManager.datatableCreationPolicy.schema}
roles={aiChatManager.datatableCreationPolicy.roles}
onChange={handleDefaultChange}
description="Set the default datatable and schema for new tables. When table creation is enabled, AI can create tables here if needed."
/>
@@ -18,6 +18,7 @@ import {
type AppCodeSelectionElement,
type AppDatatableElement
} from '../context'
import { appDatatableRole, sdkDatatableCall } from '$lib/components/raw_apps/dataTableRefUtils'
// Backend runnable types
export type BackendRunnableType = 'script' | 'flow' | 'hubscript' | 'inline'
@@ -921,9 +922,20 @@ export function prepareAppSystemMessage(customPrompt?: string): ChatCompletionSy
const policy = aiChatManager.datatableCreationPolicy
const datatableName = policy.datatable ?? 'main'
const schemaPrefix = policy.schema ? `${policy.schema}.` : ''
// Use wmill.datatable() for 'main' (default), otherwise wmill.datatable('name')
const datatableCall =
datatableName === 'main' ? 'wmill.datatable()' : `wmill.datatable('${datatableName}')`
// A role names the privileges the app's queries run with, so it has to be in the code the
// model writes.
const datatableRole = appDatatableRole(policy.roles, datatableName)
const tsDatatableCall = sdkDatatableCall(datatableName, datatableRole, 'typescript')
const pyDatatableCall = sdkDatatableCall(datatableName, datatableRole, 'python')
const roleEntries = Object.entries(policy.roles ?? {})
const rolesNote =
roleEntries.length > 0
? `\n\nThis app uses these data tables through a role: ${roleEntries
.map(([dt, role]) => `\`${dt}\` as \`${role}\``)
.join(
', '
)}. Always pass that role when calling \`wmill.datatable\` on them, as in the examples. The role only reaches what it was granted, so a query on a table it lacks privileges on fails with \`permission denied\`.`
: ''
let content = `You are a helpful assistant that creates and edits apps on the Windmill platform. Apps are defined as a collection of files that contains both the frontend and the backend.
@@ -1024,7 +1036,7 @@ Backend runnables should only perform **data operations** (SELECT, INSERT, UPDAT
import * as wmill from 'windmill-client';
export async function main(user_id: string) {
const sql = ${datatableCall};
const sql = ${tsDatatableCall};
const user = await sql\`SELECT * FROM ${schemaPrefix}users WHERE id = \${user_id}\`.fetchOne();
return user;
}
@@ -1035,12 +1047,12 @@ export async function main(user_id: string) {
import wmill
def main(user_id: str):
db = ${datatableCall}
db = ${pyDatatableCall}
user = db.query('SELECT * FROM ${schemaPrefix}users WHERE id = $1', user_id).fetch_one()
return user
\`\`\`
Use these examples for normal datatable access.
Use these examples for normal datatable access.${rolesNote}
### Schema Modifications (DDL) - Use exec_datatable_sql tool ONLY
@@ -2,6 +2,7 @@ import { z } from 'zod'
import { WorkspaceService, type CompletedJob } from '$lib/gen'
import type { DataTableTables } from '$lib/gen/types.gen'
import { runScript } from '$lib/components/jobs/utils'
import { datatableReference } from '$lib/components/dbTypes'
import {
createToolDef,
executeTestRun,
@@ -15,9 +16,9 @@ import {
*
* Datatables are workspace-level managed PostgreSQL databases. The backend
* endpoints used here (`list_datatable_tables`, `get_datatable_table_schema`)
* and SQL execution (`datatable://<name>`) are gated only by workspace
* membership, so these tools need no app context and operate directly on the
* workspace. This is the unrestricted counterpart to the app-mode datatable
* and SQL execution (`datatable://<name>`) need no app context: the server
* decides what the caller reaches, as the datatable role they name or its
* default. This is the unrestricted counterpart to the app-mode datatable
* tools in `app/core.ts`, which additionally filter by the app's whitelist.
*/
@@ -31,9 +32,19 @@ const memo = <T>(factory: () => T): (() => T) => {
// ============= Pure workspace-scoped operations =============
/** List all datatables configured in the workspace, with their schema/table names. */
export async function listDatatables(workspace: string): Promise<DataTableTables[]> {
return await WorkspaceService.listDataTableTables({ workspace })
/** List the datatables configured in the workspace, with their schema/table names: all of them as
* their default role, or only `datatableName`, as `role` when one is given. */
export async function listDatatables(
workspace: string,
datatableName?: string,
role?: string
): Promise<DataTableTables[]> {
if (datatableName === undefined) return await WorkspaceService.listDataTableTables({ workspace })
return await WorkspaceService.listDataTableTables({
workspace,
datatableName,
...(role !== undefined && { roleFor: datatableName, role })
})
}
/** Get the columns (column_name -> compact_type) of one datatable table. */
@@ -41,13 +52,15 @@ export async function getDatatableColumns(
workspace: string,
datatableName: string,
schemaName: string,
tableName: string
tableName: string,
role?: string
): Promise<Record<string, string>> {
const schema = await WorkspaceService.getDataTableTableSchema({
workspace,
datatableName,
schemaName,
tableName
tableName,
role
})
return schema.columns
}
@@ -81,7 +94,26 @@ const NO_DATATABLES_CONFIGURED_MESSAGE =
// ============= Tool definitions =============
const getListDatatablesSchema = memo(() => z.object({}))
// The same rule the server applies to `-- role <name>`; a name it would refuse fails here instead.
const getRoleSchema = memo(() =>
z
.string()
.regex(/^[A-Za-z0-9_-]{1,63}$/)
.optional()
.describe(
"The datatable role to connect as, when the code you are working on uses one (an app's `data.roles` entry, or the `role` it passes to wmill.datatable). Omit for the datatable's default role."
)
)
const getListDatatablesSchema = memo(() =>
z.object({
datatable_name: z
.string()
.optional()
.describe('List only this datatable. Required with `role`.'),
role: getRoleSchema()
})
)
const getListDatatablesToolDef = memo(() =>
createToolDef(
getListDatatablesSchema(),
@@ -94,7 +126,8 @@ const getGetDatatableTableSchemaSchema = memo(() =>
z.object({
datatable_name: z.string().describe('The datatable name to inspect, e.g. "main".'),
schema_name: z.string().describe('The schema name, e.g. "public".'),
table_name: z.string().describe('The table name to inspect.')
table_name: z.string().describe('The table name to inspect.'),
role: getRoleSchema()
})
)
const getGetDatatableTableSchemaToolDef = memo(() =>
@@ -117,6 +150,7 @@ const getExecDatatableSqlSchema = memo(() =>
.describe(
'The SQL query to execute. Supports SELECT, INSERT, UPDATE, DELETE, CREATE TABLE, ALTER TABLE, DROP TABLE, etc. For SELECT queries, results are returned as an array of objects. A newly created table will appear in list_datatables automatically.'
),
role: getRoleSchema(),
background: z
.boolean()
.optional()
@@ -217,10 +251,18 @@ export function getDatatableTools(): Tool<{}>[] {
{
def: getListDatatablesToolDef(),
planModeSafe: true,
fn: async ({ workspace, toolId, toolCallbacks }) => {
fn: async ({ args, workspace, toolId, toolCallbacks }) => {
toolCallbacks.setToolStatus(toolId, { content: 'Listing datatables...' })
try {
const metadata = await listDatatables(workspace)
const parsedArgs = getListDatatablesSchema().parse(args ?? {})
if (parsedArgs.role !== undefined && parsedArgs.datatable_name === undefined) {
throw new Error('`role` needs `datatable_name`, the datatable it is a role of')
}
const metadata = await listDatatables(
workspace,
parsedArgs.datatable_name,
parsedArgs.role
)
if (metadata.length === 0) {
toolCallbacks.setToolStatus(toolId, {
content: 'No datatables configured — set one up in workspace settings'
@@ -236,7 +278,18 @@ export function getDatatableTools(): Tool<{}>[] {
toolCallbacks.setToolStatus(toolId, {
content: `Listed ${metadata.length} datatable(s) with ${totalTables} table(s)`
})
return JSON.stringify(metadata, null, 2)
// Only what the model acts on: the roles it may pass, not the creation privileges
// the manager's UI gates on.
return JSON.stringify(
metadata.map((d) => ({
datatable_name: d.datatable_name,
schemas: d.schemas,
...(d.error && { error: d.error }),
...(d.permissioned && { usable_roles: d.usable_roles, default_role: d.default_role })
})),
null,
2
)
} catch (e) {
const errorMsg = `Error listing datatables: ${e instanceof Error ? e.message : String(e)}`
toolCallbacks.setToolStatus(toolId, { content: errorMsg, error: errorMsg })
@@ -257,7 +310,8 @@ export function getDatatableTools(): Tool<{}>[] {
workspace,
parsedArgs.datatable_name,
parsedArgs.schema_name,
parsedArgs.table_name
parsedArgs.table_name,
parsedArgs.role
)
toolCallbacks.setToolStatus(toolId, {
content: `Retrieved schema for ${parsedArgs.schema_name}.${parsedArgs.table_name}`
@@ -300,7 +354,7 @@ export function getDatatableTools(): Tool<{}>[] {
requestBody: {
language: 'postgresql',
content: parsedArgs.sql,
args: { database: `datatable://${name}` }
args: { database: datatableReference(name, parsedArgs.role) }
}
}),
workspace,
@@ -1434,7 +1434,12 @@ Data Tables:
- Datatables are workspace-scoped managed PostgreSQL databases, shared across the workspace (not owned by any single app). They must be configured by the user in their workspace settings (Workspace settings → Data Tables); they cannot be created via SQL.
- Use list_datatables to discover the available datatables and their tables. Reuse an existing table rather than creating a duplicate. If list_datatables reports none, this is a blocking prerequisite — tell the user to set up a datatable in their workspace settings and stop; do not assume a "main" datatable exists or call exec_datatable_sql.
- Use get_datatable_table_schema only when you need a table's column names/types; list_datatables is enough for table-list or availability summaries.
- Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement — it appears in list_datatables afterward, with no registration step.
- Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement — it appears in list_datatables afterward, with no registration step.${
isCloudHosted()
? ''
: `
- A raw app may use a datatable through a role (\`data.roles\` in its raw_app.yaml). When working on such an app, pass that role to the datatable tools, and to wmill.datatable in its runnables, so you see and change only what the app itself can.`
}
- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.${
skills.length > 0
? `
@@ -1,50 +0,0 @@
<script lang="ts">
import Select from '../select/Select.svelte'
let {
schemas,
tables,
schemasLoading = false,
schema = $bindable(),
table = $bindable()
}: {
/** The database's schemas, as the editor last read them. */
schemas: string[]
/** The picked schema's tables, as the editor last read them. */
tables: string[]
/** The editor has not read the database yet, so `schemas` is not known to be empty. */
schemasLoading?: boolean
/** Unset for the database itself. */
schema?: string
/** Unset for the whole schema. */
table?: string
} = $props()
</script>
<div class="flex flex-wrap items-center gap-2">
<Select
items={schemas.map((s) => ({ value: s, label: s }))}
bind:value={
() => schema,
(s) => {
schema = s
table = undefined
}
}
placeholder="The database itself"
clearable
loading={schemasLoading}
size="sm"
class="w-56"
/>
{#if schema}
<Select
items={tables.map((t) => ({ value: t, label: t }))}
bind:value={table}
placeholder="The whole schema"
clearable
size="sm"
class="w-56"
/>
{/if}
</div>
@@ -25,30 +25,24 @@
let {
workspace,
datatable,
target,
onLoaded
target
}: {
workspace: string
datatable: string
/** What owner and grants are read and written for. */
target: AclTarget
/** Each read, with the target it was made for: it also lists what the target holds. */
onLoaded?: (target: AclTarget, info: DatatableAclInfo) => void
} = $props()
const acl = resource(
() => [workspace, datatable, target] as const,
async ([ws, dt, t]) => {
const loaded = await WorkspaceService.getDatatableAcl({
async ([ws, dt, t]) =>
await WorkspaceService.getDatatableAcl({
workspace: ws,
datatableName: dt,
kind: t.kind,
schema: t.kind === 'database' ? undefined : t.schema,
table: t.kind === 'table' ? t.table : undefined
})
onLoaded?.(t, loaded)
return loaded
}
)
// Nothing is written before its SQL has been shown, and the apply runs exactly that SQL: the
@@ -120,7 +114,12 @@
<span class="text-xs text-secondary">Loading…</span>
{:else}
<div class="flex flex-col gap-4">
{#if !info.editable}
{#if info.clone}
<span class="text-xs text-secondary">
Read only: this data table is a clone, and its owners and grants stay as they were copied
from the data table it was cloned from.
</span>
{:else if !info.editable}
<span class="text-xs text-secondary">
Read only: access is changed by the admins of the workspace that governs this data table, on
Windmill Enterprise Edition.
@@ -0,0 +1,60 @@
import { describe, test, expect } from 'vitest'
import { parseMigrationRole, withMigrationRole } from './datatableMigrationRole'
describe('parseMigrationRole', () => {
test('reads every spelling the server accepts from the leading comment block', () => {
for (const line of [
'-- role analyst',
'-- Role: analyst',
'-- role=analyst',
'-- role analyst;'
]) {
expect(parseMigrationRole(`\n${line}\nBEGIN;\nEND;`)).toEqual({
kind: 'role',
role: 'analyst'
})
}
})
test('an annotation below BEGIN is not one', () => {
expect(parseMigrationRole('BEGIN;\n-- role analyst\nEND;')).toEqual({ kind: 'none' })
})
test('a malformed attempt is an error, not the default', () => {
for (const line of [
'-- role based access below',
'-- role',
'-- role:',
'-- role an;alytics'
]) {
expect(parseMigrationRole(`${line}\nBEGIN;`)).toEqual({ kind: 'malformed', line })
}
})
test('comments that do not start with the word role are ignored', () => {
expect(parseMigrationRole('-- roles analyst\n-- rolex\nBEGIN;')).toEqual({ kind: 'none' })
})
})
describe('withMigrationRole', () => {
test('leads above BEGIN, so the server reads it', () => {
const out = withMigrationRole('BEGIN;\nSELECT 1;\nEND;', 'analyst')
expect(out).toBe('-- role analyst\nBEGIN;\nSELECT 1;\nEND;')
})
test('replaces any attempt rather than stacking, malformed ones included', () => {
const out = withMigrationRole(
'-- Role: auditor\n-- role oops no\n-- keep me\nBEGIN;',
'analyst'
)
expect(out).toBe('-- role analyst\n-- keep me\nBEGIN;')
})
test('undefined strips the annotation, so it runs as admin', () => {
expect(withMigrationRole('-- role analyst\n\nBEGIN;\nEND;', undefined)).toBe('BEGIN;\nEND;')
})
test('refuses a name the server would refuse', () => {
expect(() => withMigrationRole('BEGIN;', 'bad;name')).toThrow()
})
})
@@ -0,0 +1,70 @@
import { isDatatableRoleName } from './dbTypes'
/**
* A migration carries the data table role it runs as in its own SQL, as a `-- role <name>`
* annotation. There is no separate field: the annotation is what the server reads, and keeping
* it in the SQL is what lets it survive a `wmill sync` round-trip.
*
* Mirrors `SqlAnnotations::datatable_role` on the backend. It is only read from the leading
* comment block, so an annotation below `BEGIN;` is ignored and the migration runs as admin. A
* leading comment whose first word is `role` is an annotation attempt, and a malformed one is an
* error there, so it is one here too.
*/
export type MigrationRole =
| { kind: 'none' }
| { kind: 'role'; role: string }
| { kind: 'malformed'; line: string }
/** The body of a leading comment line that attempts a role annotation, or undefined. */
function roleAttempt(line: string): string | undefined {
if (!line.startsWith('--')) return undefined
const body = line.slice(2).trimStart()
if (body.slice(0, 4).toLowerCase() !== 'role') return undefined
const after = body.slice(4)
if (after !== '' && !/^[\s:=]/.test(after)) return undefined
return after
}
function parseAttempt(after: string): string | undefined {
let rest = after.trimStart()
if (rest.startsWith(':') || rest.startsWith('=')) rest = rest.slice(1)
const tokens = rest.split(/\s+/).filter((t) => t !== '')
if (tokens.length !== 1) return undefined
const role = tokens[0].endsWith(';') ? tokens[0].slice(0, -1) : tokens[0]
return isDatatableRoleName(role) ? role : undefined
}
export function parseMigrationRole(sql: string): MigrationRole {
for (const raw of sql.split('\n')) {
const line = raw.trim()
if (line === '') continue
if (!line.startsWith('--')) break
const after = roleAttempt(line)
if (after === undefined) continue
const role = parseAttempt(after)
return role === undefined ? { kind: 'malformed', line } : { kind: 'role', role }
}
return { kind: 'none' }
}
/**
* `sql` declaring `role`: any role annotation attempt in the leading comment block is removed,
* and `-- role <role>` is prepended above everything, or nothing when `role` is undefined.
*/
export function withMigrationRole(sql: string, role: string | undefined): string {
if (role !== undefined && !isDatatableRoleName(role)) {
throw new Error(`Invalid data table role '${role}'`)
}
const lines = sql.split('\n')
const kept: string[] = []
let i = 0
for (; i < lines.length; i++) {
const line = lines[i].trim()
if (line !== '' && !line.startsWith('--')) break
if (roleAttempt(line) === undefined) kept.push(lines[i])
}
const rest = [...kept, ...lines.slice(i)]
while (rest.length > 0 && rest[0].trim() === '') rest.shift()
return role === undefined ? rest.join('\n') : [`-- role ${role}`, ...rest].join('\n')
}
@@ -0,0 +1,29 @@
import { describe, expect, it, vi } from 'vitest'
const request = vi.fn()
vi.mock('$lib/gen/core/request', () => ({ request: (...args: unknown[]) => request(...args) }))
vi.mock('$lib/gen', () => ({ OpenAPI: {} }))
vi.mock('$lib/cloud', () => ({ isCloudHosted: () => false }))
import { listUsableDatatableRoles } from './datatableUsableRoles'
const refuse = (body: string) => () =>
Promise.reject(Object.assign(new Error('Bad Request'), { body }))
describe('listUsableDatatableRoles', () => {
// The refusal is recognised by its sentence, so rewording it on one side alone would turn every
// data table on a build without the Enterprise Edition into a failed lookup rather than one
// that is not under roles.
it('reads the enterprise refusal as not under roles, and rethrows anything else', async () => {
request.mockImplementation(refuse('Data table roles are a Windmill Enterprise Edition feature'))
expect(await listUsableDatatableRoles('ws', 'main')).toEqual({
permissioned: false,
roles: [],
default_role: 'admin'
})
request.mockImplementation(refuse('Data table not found'))
const rethrown = await listUsableDatatableRoles('ws', 'main').catch((e) => e)
expect(rethrown).toBeInstanceOf(Error)
})
})
@@ -0,0 +1,44 @@
import { OpenAPI, type ListUsableDatatableRolesResponse } from '$lib/gen'
import { request } from '$lib/gen/core/request'
import { isCloudHosted } from '$lib/cloud'
import { ADMIN_DATATABLE_ROLE } from './dbTypes'
// `datatable_roles_unavailable` on the server, which is a plain 400: rewording it there without
// here makes every role picker on a non-Enterprise build fail instead of reading "not under roles".
const ROLES_UNAVAILABLE = 'Data table roles are a Windmill Enterprise Edition feature'
const NOT_UNDER_ROLES: ListUsableDatatableRolesResponse = {
permissioned: false,
roles: [],
default_role: ADMIN_DATATABLE_ROLE
}
/**
* The roles the caller may connect as on a data table. Cloud has no instance database, so no data
* table there is under roles, and none of the role pickers show. Without the Enterprise Edition
* every roles route refuses, which reads the same way: the data table is then used the way it was
* before roles, and one that is under roles is refused when something connects to it.
*/
export async function listUsableDatatableRoles(
workspace: string,
datatableName: string
): Promise<ListUsableDatatableRolesResponse> {
if (isCloudHosted()) return NOT_UNDER_ROLES
try {
// The generated client encodes path params with `encodeURI`, which leaves a '?' in a
// data table name created before names were restricted to cut the path short.
return await request(
{ ...OpenAPI, ENCODE_PATH: encodeURIComponent },
{
method: 'GET',
url: '/w/{workspace}/workspaces/datatable_usable_roles/{datatable_name}',
path: { workspace, datatable_name: datatableName }
}
)
} catch (e) {
const body = (e as { body?: unknown })?.body
const detail = `${typeof body === 'string' ? body : JSON.stringify(body ?? '')} ${(e as Error)?.message ?? e}`
if (detail.includes(ROLES_UNAVAILABLE)) return NOT_UNDER_ROLES
throw e
}
}
@@ -5,7 +5,7 @@ import { isDbType } from './dbTypes'
/**
* Single URL param `dbm` encodes the full DB manager state:
* firstSegment~path~schema.table
* firstSegment~path~schema.table~role=name
*
* firstSegment:
* datatable – database with datatable:// resource (resourceType always postgresql)
@@ -26,28 +26,46 @@ import { isDbType } from './dbTypes'
* datatable~main~.customers (schema "public" implied)
* ducklake~main~.orders (schema "main" implied)
* postgresql~$res:u/user/my_pg~public.customers
* datatable~main~.customers~role=analyst
* datatable~main~role=analyst (no schema/table selected)
*
* role=name (last segment, optional, data tables only): the data table role to connect as.
* Omitted means the data table's default role. A trailing segment starting with `role=` is always
* the role, whatever follows. The name is kept as written, even when invalid (a `.` included), so
* the connection refuses it visibly instead of falling back to the default.
*/
const dbManagerSchema = z.object({
dbm: z.string().nullable()
})
interface ParsedDbm {
export interface ParsedDbm {
type: 'database' | 'datatable' | 'ducklake'
path: string
resType?: string
schema?: string
table?: string
role?: string
}
function parseDbm(raw: unknown): ParsedDbm | null {
const ROLE_SEGMENT_PREFIX = 'role='
function isRoleSegment(segment: string | undefined): segment is string {
return !!segment && segment.startsWith(ROLE_SEGMENT_PREFIX)
}
export function parseDbm(raw: unknown): ParsedDbm | null {
if (!raw || typeof raw !== 'string') return null
const parts = raw.split('~')
if (parts.length < 2 || !parts[1]) return null
const firstSeg = parts[0]
const path = parts[1]
const schemaTable = parts[2] ?? ''
const rest = parts.slice(2)
const role = isRoleSegment(rest.at(-1))
? rest.pop()!.slice(ROLE_SEGMENT_PREFIX.length)
: undefined
const schemaTable = rest[0] ?? ''
let type: ParsedDbm['type']
let resType: string | undefined
@@ -81,12 +99,12 @@ function parseDbm(raw: unknown): ParsedDbm | null {
schema = defaultSchemas[type]
}
return { type, path, resType, schema, table }
return { type, path, resType, schema, table, role: type === 'datatable' ? role : undefined }
}
const defaultSchemas: Record<string, string> = { datatable: 'public', ducklake: 'main' }
function buildDbm(p: ParsedDbm): string {
export function buildDbm(p: ParsedDbm): string {
const firstSeg = p.type === 'database' ? p.resType! : p.type
const schema = p.schema === defaultSchemas[p.type] ? undefined : p.schema
let schemaTable = ''
@@ -97,7 +115,12 @@ function buildDbm(p: ParsedDbm): string {
} else if (schema) {
schemaTable = `${schema}.`
}
return schemaTable ? `${firstSeg}~${p.path}~${schemaTable}` : `${firstSeg}~${p.path}`
const segments = [firstSeg, p.path]
if (schemaTable) segments.push(schemaTable)
if (p.type === 'datatable' && p.role !== undefined) {
segments.push(`${ROLE_SEGMENT_PREFIX}${p.role}`)
}
return segments.join('~')
}
export interface DbManagerUriState {
@@ -105,6 +128,8 @@ export interface DbManagerUriState {
readonly effectiveInput: DbInput | undefined
readonly isDatatableInput: boolean
selectedDatatable: string | undefined
/** The data table role the drawer connects as; undefined means its default. */
selectedRole: string | undefined
selectedSchema: string | undefined
selectedTable: string | undefined
readonly open: boolean
@@ -137,6 +162,7 @@ export function useDbManagerUriState(): DbManagerUriState {
type: 'database' as const,
resourceType: resType as DbType,
resourcePath: parsed.type === 'datatable' ? `datatable://${parsed.path}` : parsed.path,
role: parsed.role,
specificSchema: parsed.schema,
specificTable: parsed.table
}
@@ -163,6 +189,7 @@ export function useDbManagerUriState(): DbManagerUriState {
type: isDatatable ? 'datatable' : 'database',
path: isDatatable ? nInput.resourcePath.slice('datatable://'.length) : nInput.resourcePath,
resType: isDatatable ? undefined : nInput.resourceType,
role: isDatatable ? nInput.role : undefined,
schema: nInput.specificSchema,
table: nInput.specificTable
})
@@ -194,7 +221,14 @@ export function useDbManagerUriState(): DbManagerUriState {
return parsed?.type === 'datatable' ? parsed.path : undefined
},
set selectedDatatable(v: string | undefined) {
if (v) updateField({ path: v })
// A role belongs to one data table, so it cannot carry over to another.
if (v) updateField({ path: v, role: undefined })
},
get selectedRole() {
return parsed?.role
},
set selectedRole(v: string | undefined) {
updateField({ role: v })
},
get selectedSchema() {
return parsed?.schema
@@ -0,0 +1,67 @@
import { describe, expect, it } from 'vitest'
import { buildDbm, parseDbm } from './dbManagerDrawerModel.svelte'
import { schemaCacheKey } from './dbSchemaCache'
import { datatableReference, type DbInput } from './dbTypes'
describe('dbm role segment', () => {
it('round-trips a role, with and without a table', () => {
for (const dbm of ['datatable~main~.orders~role=p4_analytics', 'datatable~main~role=p4-op']) {
expect(buildDbm(parseDbm(dbm)!)).toBe(dbm)
}
expect(parseDbm('datatable~main~sales.orders~role=analyst')).toMatchObject({
path: 'main',
schema: 'sales',
table: 'orders',
role: 'analyst'
})
})
it('reads a link without a role as the default role', () => {
const parsed = parseDbm('datatable~main~.orders')!
expect(parsed.role).toBeUndefined()
expect(parsed).toMatchObject({ schema: 'public', table: 'orders' })
expect(buildDbm(parsed)).toBe('datatable~main~.orders')
})
it('keeps an invalid role as written, so the connection refuses it', () => {
expect(parseDbm('datatable~main~role=a;b')?.role).toBe('a;b')
// A dot does not turn it into a schema.table selection read as the default role.
expect(parseDbm('datatable~main~role=bad.name')).toMatchObject({
role: 'bad.name',
schema: undefined,
table: undefined
})
})
})
describe('connecting as a role', () => {
const input = (role?: string): DbInput => ({
type: 'database',
resourceType: 'postgresql',
resourcePath: 'datatable://main',
role
})
// What `getDatabaseArg` builds every DB manager connection from.
it('appends the role to the data table reference', () => {
expect(datatableReference('main', 'p4_analytics')).toBe('datatable://main?role=p4_analytics')
expect(datatableReference('main', undefined)).toBe('datatable://main')
})
it('never appends a role to a name containing ?', () => {
// The server reads such a whole reference as the stored name first, so `?role=` would
// be taken as part of the name or refused instead of picking the role.
expect(datatableReference('legacy?x', undefined)).toBe('datatable://legacy?x')
expect(() => datatableReference('sales?role=analytics', 'admin')).toThrow(/'\?' in its name/)
})
it('refuses a role name the server would not accept', () => {
expect(() => datatableReference('main', 'a&role=admin')).toThrow(/Invalid data table role/)
expect(() => datatableReference('main', '')).toThrow(/Invalid data table role/)
})
it('keys the schema cache by role', () => {
expect(schemaCacheKey('ws', input('a'))).not.toBe(schemaCacheKey('ws', input('b')))
expect(schemaCacheKey('ws', input('a'))).not.toBe(schemaCacheKey('ws', input()))
})
})
+32 -14
View File
@@ -8,7 +8,8 @@ import { runScriptAndPollResult } from './jobs/utils'
import { writingJobOptions } from './jobs/writingJob'
import type { DBSchema, SQLSchema } from '$lib/stores'
import { stringifySchema } from './copilot/lib'
import type { DbInput, DbType } from './dbTypes'
import { datatableReference, type DbInput, type DbType } from './dbTypes'
import { withMigrationRole } from './datatableMigrationRole'
import { assert } from '$lib/utils'
import { WorkspaceService } from '$lib/gen'
import { pendingMigrations } from './workspaceSettings/datatableMigrationUtils'
@@ -70,7 +71,9 @@ export function dbTableOpsWithPreviewScripts({
}): IDbTableOps {
const dbType = getDbType(input)
const language = getLanguageByResourceType(dbType)
const dbArg = getDatabaseArg(input)
// Built per call: an invalid role throws there, as that operation's error, rather than while
// the manager renders.
const dbArg = () => getDatabaseArg(input)
const ducklake = input.type === 'ducklake' ? input.ducklake : undefined
function makeMarker(op: string, payload: Record<string, unknown>): string {
@@ -91,7 +94,7 @@ export function dbTableOpsWithPreviewScripts({
})
const result = await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg, quicksearch }, language, content, tag }
requestBody: { args: { ...dbArg(), quicksearch }, language, content, tag }
})
const count = result?.[0].count as number
return count
@@ -106,7 +109,7 @@ export function dbTableOpsWithPreviewScripts({
})
let items = (await runScriptAndPollResult({
workspace,
requestBody: { args: { ...dbArg, ...params }, language, content, tag }
requestBody: { args: { ...dbArg(), ...params }, language, content, tag }
})) as unknown[]
if (!items || !Array.isArray(items)) {
throw 'items is not an array'
@@ -123,7 +126,7 @@ export function dbTableOpsWithPreviewScripts({
{
workspace,
requestBody: {
args: { ...dbArg, value_to_update: newValue, ...values },
args: { ...dbArg(), value_to_update: newValue, ...values },
language,
content,
tag
@@ -135,14 +138,14 @@ export function dbTableOpsWithPreviewScripts({
onDelete: async ({ values }) => {
const content = makeMarker('DELETE', { table: tableKey, columns: colDefs })
await runScriptAndPollResult(
{ workspace, requestBody: { args: { ...dbArg, ...values }, language, content, tag } },
{ workspace, requestBody: { args: { ...dbArg(), ...values }, language, content, tag } },
writingJobOptions
)
},
onInsert: async ({ values }) => {
const content = makeMarker('INSERT', { table: tableKey, columns: colDefs })
await runScriptAndPollResult(
{ workspace, requestBody: { args: { ...dbArg, ...values }, language, content, tag } },
{ workspace, requestBody: { args: { ...dbArg(), ...values }, language, content, tag } },
writingJobOptions
)
}
@@ -246,6 +249,7 @@ export type IDbSchemaOps = {
previewAlterSql: (params: { values: AlterTableValues; schema?: string }) => Promise<string>
onCreateSchema: (params: { schema: string }) => Promise<void>
onDeleteSchema: (params: { schema: string }) => Promise<void>
onRenameSchema: (params: { schema: string; newSchema: string }) => Promise<void>
onFetchTableEditorDefinition: (params: {
table: string
schema?: string
@@ -283,7 +287,8 @@ export function dbSchemaOpsWithPreviewScripts({
tag?: string
}): IDbSchemaOps {
const dbType = getDbType(input)
const dbArg = getDatabaseArg(input)
// Built per call, for the same reason as in the table ops above.
const dbArg = () => getDatabaseArg(input)
const language = getLanguageByResourceType(dbType)
const ducklake = input.type === 'ducklake' ? input.ducklake : undefined
@@ -293,6 +298,8 @@ export function dbSchemaOpsWithPreviewScripts({
input.type === 'database' && input.resourcePath.startsWith('datatable://')
? input.resourcePath.slice('datatable://'.length)
: undefined
// A migration declaring no role runs as admin, whatever role the manager connects as.
const migrationRole = input.type === 'database' ? (input.role ?? input.migrationRole) : undefined
function makeMarker(op: string, payload: Record<string, unknown>): string {
if (ducklake) payload.ducklake = ducklake
@@ -359,7 +366,7 @@ export function dbSchemaOpsWithPreviewScripts({
: undefined
if (!datatableName || !status?.enabled) {
await runScriptAndPollResult(
{ workspace, requestBody: { args: dbArg, content, language, tag } },
{ workspace, requestBody: { args: dbArg(), content, language, tag } },
writingJobOptions
)
return
@@ -373,12 +380,16 @@ export function dbSchemaOpsWithPreviewScripts({
throw new MigrationRunCancelled()
}
}
const codeUp = wrapMigration(await expandMarker(workspace, language, content))
// Wrapped before annotating: the annotation must lead, above `BEGIN;`.
const codeUp = withMigrationRole(
wrapMigration(await expandMarker(workspace, language, content)),
migrationRole
)
// Down migrations are only generated for Postgres for now.
let codeDown: string | undefined
if (downContent && dbType === 'postgresql') {
const downSql = (await expandMarker(workspace, language, downContent)).trim()
if (downSql) codeDown = wrapMigration(downSql)
if (downSql) codeDown = withMigrationRole(wrapMigration(downSql), migrationRole)
}
const created = await WorkspaceService.createDatatableMigration({
workspace,
@@ -415,7 +426,7 @@ export function dbSchemaOpsWithPreviewScripts({
const fkContent = makeMarker('FOREIGN_KEYS', { table, schema })
const fkResult = await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content: fkContent, language, tag }
requestBody: { args: dbArg(), content: fkContent, language, tag }
})
let rawForeignKeys: RawForeignKey[]
@@ -501,6 +512,11 @@ export function dbSchemaOpsWithPreviewScripts({
const downContent = makeMarker('CREATE_SCHEMA', { schema })
await applyDdl(migrationName('drop_schema', schema), content, downContent)
},
onRenameSchema: async ({ schema, newSchema }) => {
const content = makeMarker('RENAME_SCHEMA', { schema, new_schema: newSchema })
const downContent = makeMarker('RENAME_SCHEMA', { schema: newSchema, new_schema: schema })
await applyDdl(migrationName('rename_schema', schema), content, downContent)
},
onFetchForeignKeys: fetchForeignKeys,
onFetchTableEditorDefinition: async ({ table, schema, colDefs }) => {
const foreignKeys = await fetchForeignKeys({ table, schema })
@@ -512,7 +528,7 @@ export function dbSchemaOpsWithPreviewScripts({
const pkContent = makeMarker('PRIMARY_KEY_CONSTRAINT', { table, schema })
const pkResult = (await runScriptAndPollResult({
workspace,
requestBody: { args: dbArg, content: pkContent, language, tag }
requestBody: { args: dbArg(), content: pkContent, language, tag }
})) as { constraint_name?: string; CONSTRAINT_NAME?: string }[]
if (pkResult && Array.isArray(pkResult) && pkResult.length > 0) {
@@ -611,7 +627,9 @@ export function getDefaultDbTag(input: DbInput): string {
export function getDatabaseArg(input: DbInput | undefined) {
if (input?.type === 'database') {
if (input.resourcePath.startsWith('datatable://')) {
return { database: input.resourcePath }
return {
database: datatableReference(input.resourcePath.slice('datatable://'.length), input.role)
}
} else {
return { database: '$res:' + input.resourcePath }
}
@@ -0,0 +1,21 @@
import type { DbInput } from './dbTypes'
/** What identifies a database's schema, role included: two roles on one data table may reach
* different schemas, so they cannot share a cache entry. Never throws, since it keys derived
* state; the connection itself is what refuses an invalid role. */
export function getDbSchemasPath(input: DbInput): string {
switch (input.type) {
case 'database':
return input.role !== undefined && input.resourcePath.startsWith('datatable://')
? `${input.resourcePath}?role=${input.role}`
: input.resourcePath
case 'ducklake':
return 'ducklake://' + input.ducklake
}
}
/** Scoped by the acting workspace: a data table of the same name can exist in both the nav and
* the acting workspace, and one's schema must not be reused for the other. */
export function schemaCacheKey(workspace: string | undefined, input: DbInput): string {
return `${workspace}:${getDbSchemasPath(input)}`
}
+50
View File
@@ -3,6 +3,12 @@ export type DbInput =
type: 'database'
resourceType: DbType
resourcePath: string
/** The data table role to connect as; the data table's default when unset. Only
* meaningful for a `datatable://` path. */
role?: string
/** The role migrations written through this input declare when `role` is unset. A
* migration declaring none runs as admin, not as the role the manager connects as. */
migrationRole?: string
specificSchema?: string
specificTable?: string
}
@@ -23,3 +29,47 @@ export const dbTypes = [
'duckdb'
] as const
export const isDbType = (str?: string): str is DbType => !!str && dbTypes.includes(str as DbType)
/** The role every data table has: the one it connects as when it is not under roles. */
export const ADMIN_DATATABLE_ROLE = 'admin'
/** What the server accepts in `-- role <name>` and `?role=<name>`. */
export function isDatatableRoleName(name: string): boolean {
return /^[A-Za-z0-9_-]{1,63}$/.test(name)
}
/** Whether a role can be named in a reference to this data table. A name stored before names were
* restricted may contain `?`, and the server reads such a whole reference as that name first, so
* `?role=` after it would be taken as part of the name or refused. */
export function datatableNameTakesRole(name: string): boolean {
return !name.includes('?')
}
/** The `migrationRole` of a data table that cannot name a role in its reference: it connects as
* its default role, which its migrations must then declare. */
export function defaultMigrationRole(
name: string,
permissioned: boolean | undefined,
defaultRole: string | undefined
): string | undefined {
return permissioned && !datatableNameTakesRole(name) ? defaultRole : undefined
}
/** `datatable://<name>`, with `?role=<role>` when a role is named. Throws rather than build a
* reference the executor would refuse, or one that would silently mean another role. */
export function datatableReference(name: string, role: string | undefined): string {
if (role === undefined) return `datatable://${name}`
if (!isDatatableRoleName(role)) {
throw new Error(
`Invalid data table role '${role}': only letters, digits, '_' and '-' are allowed`
)
}
if (!datatableNameTakesRole(name)) {
throw new Error(
`Data table '${name}' has a '?' in its name, so it can only be used here as its default role. Rename it to connect as role '${role}'.`
)
}
return `datatable://${name}?role=${role}`
}
export type DatatableRowAction = 'migrations' | 'roles' | 'export' | 'import'
@@ -3,13 +3,14 @@
import Popover from '$lib/components/meltComponents/Popover.svelte'
import Select from '$lib/components/select/Select.svelte'
import {
createDatatableAccessResource,
createDatatablesResource,
createSchemasResource,
toDatatableItems,
toSchemaItems
} from './datatableUtils.svelte'
import { Button } from '../common'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
import { appDatatableRole } from './dataTableRefUtils'
const operatingWorkspace = useOperatingWorkspace()
let opWs = $derived($operatingWorkspace)
@@ -19,6 +20,8 @@
datatable: string | undefined
/** Currently selected schema */
schema: string | undefined
/** The role the app uses each data table through: schemas are listed as that role. */
roles?: Record<string, string>
/** Callback when either value changes */
onChange?: (datatable: string | undefined, schema: string | undefined) => void
/** Description text to show in the popover */
@@ -28,19 +31,31 @@
let {
datatable,
schema,
roles,
onChange,
description = 'Set the default datatable and schema for new tables. This is where AI will create new tables when needed.'
}: Props = $props()
const role = $derived(datatable ? appDatatableRole(roles, datatable) : undefined)
// Load available datatables and schemas using shared utilities
const datatables = createDatatablesResource(() => opWs)
const schemas = createSchemasResource(
const access = createDatatableAccessResource(
() => datatable,
() => role,
() => opWs
)
const datatableItems = $derived(toDatatableItems(datatables.current))
const schemaItems = $derived(toSchemaItems(schemas.current))
// Until the answer is for this workspace, data table and role, the schemas in hand belong to
// another.
const schemaItems = $derived(
access.current.workspace === opWs &&
access.current.datatable === datatable &&
access.current.role === role
? toSchemaItems(access.current.schemas)
: []
)
// Track datatable changes to reset schema
let previousDatatable = $state<string | undefined>(undefined)
@@ -81,6 +96,9 @@
placeholder="Select database"
size="sm"
/>
{#if role}
<span class="text-2xs text-tertiary">Used as role {role}</span>
{/if}
</div>
<div class="flex flex-col gap-1">
@@ -1,34 +1,50 @@
<script lang="ts">
import { WorkspaceService } from '$lib/gen'
import { listUsableDatatableRoles } from '../datatableUsableRoles'
import Drawer from '../common/drawer/Drawer.svelte'
import DrawerContent from '../common/drawer/DrawerContent.svelte'
import Button from '../common/button/Button.svelte'
import Select from '../select/Select.svelte'
import { sendUserToast } from '$lib/toast'
import type { DataTableRef } from './dataTableRefUtils'
import { appDatatableRole, type DataTableRef } from './dataTableRefUtils'
import { untrack } from 'svelte'
import { resource } from 'runed'
import { ArrowLeft, Expand, LoaderCircle, Minimize, Plus, RefreshCcw } from 'lucide-svelte'
import { ArrowLeft, Expand, Minimize, Plus, RefreshCcw } from 'lucide-svelte'
import DBManagerContent from '../DBManagerContent.svelte'
import type { DbInput } from '../dbTypes'
import type { SelectedTable } from '../DBManager.svelte'
import {
ADMIN_DATATABLE_ROLE,
datatableNameTakesRole,
defaultMigrationRole,
type DbInput
} from '../dbTypes'
import type { PendingRowAction, SelectedTable } from '../DBManager.svelte'
import { useDbManagerTag } from '../dbManagerTag.svelte'
import DbWorkerTagButton from '../DbWorkerTagButton.svelte'
import type { DataTableTables } from '$lib/gen'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
const operatingWorkspace = useOperatingWorkspace()
let opWs = $derived($operatingWorkspace)
interface Props {
onAdd?: (ref: DataTableRef) => void
/** `roles` holds, for each added table's data table under roles, the role its tables were
* browsed as: the app uses the data table through it from then on. `roleChanged` names the
* data tables the app now uses through another role than before (its stored role, or the
* data table's default when it stored none): their existing refs are replaced, since the new
* role may not reach them. */
onAdd?: (refs: DataTableRef[], roles: Record<string, string>, roleChanged: Set<string>) => void
existingRefs?: DataTableRef[]
/** The role the app uses each data table through, by data table name */
roles?: Record<string, string>
/** Z-index offset for the drawer, useful when opening from within modals */
offset?: number
}
let { onAdd, existingRefs = [], offset = 0 }: Props = $props()
let { onAdd, existingRefs = [], roles = undefined, offset = 0 }: Props = $props()
let open = $state(false)
let selectedDatatable = $state<string | undefined>(undefined)
/** Role the manager connects as; undefined means the data table's default. */
let selectedRole = $state<string | undefined>(undefined)
// For DB manager
let dbManagerContent: DBManagerContent | undefined = $state()
@@ -38,10 +54,19 @@
// Multi-select mode: selected tables
let selectedTables = $state<SelectedTable[]>([])
/** The role each data table's selected tables were browsed as. */
let browsedRoles = $state<Record<string, string>>({})
// Survives the re-mount a data table switch causes.
let pendingAction = $state<PendingRowAction | undefined>(undefined)
// Selected schema/table from DBManager (for preview)
let selectedSchemaKey = $state<string | undefined>(undefined)
let selectedTableKey = $state<string | undefined>(undefined)
// What the manager opens on, set only when it (re-)mounts: the live selection above changes
// on every click, and feeding it to the input would reload the whole manager each time.
let openSchemaKey = $state<string | undefined>(undefined)
let openTableKey = $state<string | undefined>(undefined)
// Load available datatables from workspace
const datatables = resource<string[]>([], async () => {
@@ -54,32 +79,197 @@
}
})
export function openDrawer() {
// Auto-select first datatable if only one exists
if (datatables.current.length === 1) {
selectedDatatable = datatables.current[0]
} else if (datatables.current.length > 1 && datatables.current.includes('main')) {
selectedDatatable = 'main'
} else {
selectedDatatable = undefined
const usableRoles = resource(
() => [open, opWs, selectedDatatable] as const,
async ([isOpen, workspace, datatable]) => {
if (!isOpen || !workspace || !datatable) return undefined
try {
return {
workspace,
datatable,
...(await listUsableDatatableRoles(workspace, datatable))
}
} catch (e) {
// Opens anyway: without a role the server connects as the default and says so if
// that is refused.
console.error('Failed to load datatable roles:', e)
return {
workspace,
datatable,
permissioned: false,
roles: [] as string[],
default_role: ADMIN_DATATABLE_ROLE
}
}
}
)
// A resource keeps its previous value while it refetches, and roles are per data table of one
// workspace: a fork and its parent both have a `main`, with their own roles on it.
const rolesOfCurrent = $derived(
usableRoles.current?.datatable === selectedDatatable && usableRoles.current?.workspace === opWs
? usableRoles.current
: undefined
)
// Mounting the manager fires its first queries, so it waits for the role: a round sent
// without one runs, and caches, as whatever the server defaults to.
const roleSettled = $derived(
selectedDatatable === undefined ||
(rolesOfCurrent !== undefined &&
(!rolesOfCurrent.permissioned ||
rolesOfCurrent.roles.length === 0 ||
selectedRole !== undefined ||
// Its reference cannot name a role, so it connects as the default one.
!datatableNameTakesRole(selectedDatatable)))
)
$effect(() => {
const current = rolesOfCurrent
if (
!current?.permissioned ||
selectedRole !== undefined ||
(selectedDatatable !== undefined && !datatableNameTakesRole(selectedDatatable))
)
return
const effective = current.roles.includes(current.default_role)
? current.default_role
: current.roles[0]
const datatable = selectedDatatable
if (effective && datatable) untrack(() => connectAs(datatable, effective))
})
// Every data table with its schemas and tables: the tree is the picker. The privileges it
// reports are the connected role's, so the role picked on the open data table is asked too.
// Waits for the role like the manager does, and drops an answer for a selection that has
// since changed: it would describe another role.
let datatableTreeRun = 0
const datatableTree = resource(
() => [open, opWs, selectedDatatable, selectedRole, roleSettled] as const,
async ([isOpen, workspace, roleFor, role, settled]): Promise<DataTableTables[]> => {
if (!isOpen || !workspace) return []
if (!settled) return untrack(() => datatableTree.current)
const run = ++datatableTreeRun
try {
const result = await WorkspaceService.listDataTableTables({
workspace,
roleFor: role ? roleFor : undefined,
role
})
return run === datatableTreeRun ? result : untrack(() => datatableTree.current)
} catch (e) {
console.error('Failed to load datatable tables:', e)
return run === datatableTreeRun ? [] : untrack(() => datatableTree.current)
}
},
{ initialValue: [] }
)
/** The role a table of `datatable` is seen through right now: the connected data table's
* picked role, or the default role the tree lists any other one as. */
function roleSeenFor(datatable: string): string | undefined {
if (!datatableNameTakesRole(datatable)) return undefined
if (datatable === selectedDatatable) {
return rolesOfCurrent?.permissioned ? selectedRole : undefined
}
const entry = datatableTree.current.find((t) => t.datatable_name === datatable)
return entry?.permissioned ? entry.default_role : undefined
}
function defaultRoleOf(datatable: string): string | undefined {
if (datatable === selectedDatatable && rolesOfCurrent?.permissioned) {
return rolesOfCurrent.default_role
}
const entry = datatableTree.current.find((t) => t.datatable_name === datatable)
return entry?.permissioned ? entry.default_role : undefined
}
const tableDatatable = (t: SelectedTable) => t.datatable ?? selectedDatatable
/** Stamps each newly selected table with the role it was seen through. A data table's
* selections all come from one role, since the app uses it through one: picking a table under
* another role drops the ones picked under the previous, which that role may not reach. */
function setSelectedTables(next: SelectedTable[]) {
const isNew = (t: SelectedTable) =>
!selectedTables.some(
(s) =>
tableDatatable(s) === tableDatatable(t) && s.schema === t.schema && s.table === t.table
)
const added = next.filter(isNew)
const nextRoles = { ...browsedRoles }
let kept = next
for (const table of added) {
const dt = tableDatatable(table)
if (!dt) continue
const role = roleSeenFor(dt)
if (role === undefined) continue
if (nextRoles[dt] !== undefined && nextRoles[dt] !== role) {
kept = kept.filter((s) => tableDatatable(s) !== dt || added.includes(s))
}
nextRoles[dt] = role
}
const stillSelected = new Set(kept.map(tableDatatable))
selectedTables = kept
browsedRoles = Object.fromEntries(
Object.entries(nextRoles).filter(([dt]) => stillSelected.has(dt))
)
}
// `undefined` where the workspace has no data table at all: the drawer opens on none and
// shows its placeholder, rather than on a name that does not exist.
function selectDatatable(datatable: string | undefined, role?: string) {
// A row clicked under another data table has just set the selection it should open on.
openSchemaKey = selectedSchemaKey
openTableKey = selectedTableKey
selectedDatatable = datatable
if (datatable === undefined) {
selectedRole = undefined
return
}
// A data table opens as the role its picked tables were browsed as, else the one the app
// already uses it through.
connectAs(
datatable,
datatableNameTakesRole(datatable)
? (role ?? browsedRoles[datatable] ?? appDatatableRole(roles, datatable))
: undefined
)
}
/** Connects to `datatable` as `role`. The tables picked on it under another role are dropped:
* they would be saved under a role other than the one on screen. */
function connectAs(datatable: string, role: string | undefined) {
selectedRole = role
const browsed = browsedRoles[datatable]
if (browsed !== undefined && role !== undefined && role !== browsed) {
selectedTables = selectedTables.filter((t) => (t.datatable ?? datatable) !== datatable)
const { [datatable]: _, ...rest } = browsedRoles
browsedRoles = rest
}
}
// Cleared before a data table is selected: a pick left unadded when the drawer last closed
// would otherwise decide the role it reopens as.
function resetSelection() {
selectedTables = []
browsedRoles = {}
selectedRole = undefined
}
export function openDrawer() {
resetSelection()
selectedSchemaKey = undefined
selectedTableKey = undefined
selectedTables = []
selectDatatable(datatables.current.includes('main') ? 'main' : datatables.current[0])
expand = false
open = true
}
let initialTableKey: string | undefined = $state<string | undefined>(undefined)
let initialSchemaKey: string | undefined = $state<string | undefined>(undefined)
export function openDrawerWithRef(ref: DataTableRef) {
selectedDatatable = ref.datatable
resetSelection()
selectedSchemaKey = ref.schema
selectedTableKey = ref.table
initialTableKey = ref.table
initialSchemaKey = ref.schema
selectedTables = []
selectDatatable(ref.datatable)
expand = false
open = true
}
@@ -87,49 +277,58 @@
export function closeDrawer() {
open = false
dbManagerContent?.clearReplResult()
// An action outlives the data table it was asked for otherwise.
pendingAction = undefined
}
function handleAddTables() {
if (!selectedDatatable) {
sendUserToast('Please select a data table first', true)
return
}
if (selectedTables.length === 0) {
sendUserToast('Please select at least one table', true)
return
}
// Add all selected tables
const refs: DataTableRef[] = []
for (const table of selectedTables) {
const ref: DataTableRef = {
datatable: selectedDatatable,
schema: table.schema,
table: table.table
}
onAdd?.(ref)
const datatable = table.datatable ?? selectedDatatable
if (!datatable) continue
refs.push({ datatable, schema: table.schema, table: table.table })
}
const added = new Set(refs.map((r) => r.datatable))
const addedRoles = Object.fromEntries(
Object.entries(browsedRoles).filter(([dt]) => added.has(dt))
)
const roleChanged = new Set(
Object.entries(addedRoles)
.filter(([dt, role]) => {
const usedAs = appDatatableRole(roles, dt) ?? defaultRoleOf(dt)
return usedAs !== undefined && usedAs !== role
})
.map(([dt]) => dt)
)
onAdd?.(refs, addedRoles, roleChanged)
const count = selectedTables.length
const count = refs.length
sendUserToast(`Added ${count} table${count > 1 ? 's' : ''} to app`)
selectedTables = []
browsedRoles = {}
}
const datatableItems = $derived(
datatables.current.map((dt) => ({
value: dt,
label: dt
}))
)
// Carries the picked schema/table, so a click on a row of another data table lands on that
// table once the manager re-mounts against it.
const dbInput: DbInput | undefined = $derived(
selectedDatatable
? {
type: 'database' as const,
resourceType: 'postgresql' as const,
resourcePath: `datatable://${selectedDatatable}`,
specificSchema: initialSchemaKey,
specificTable: initialTableKey
role: selectedRole,
migrationRole: defaultMigrationRole(
selectedDatatable,
rolesOfCurrent?.permissioned,
rolesOfCurrent?.default_role
),
specificSchema: openSchemaKey,
specificTable: openTableKey
}
: undefined
)
@@ -140,15 +339,13 @@
}
})
// Convert existingRefs to disabledTables format for the current datatable
const disabledTables = $derived(
existingRefs
.filter((ref) => ref.datatable === selectedDatatable && ref.schema && ref.table)
.map((ref) => ({ schema: ref.schema!, table: ref.table! }))
.filter((ref) => ref.schema && ref.table)
.map((ref) => ({ datatable: ref.datatable, schema: ref.schema!, table: ref.table! }))
)
// Can add: has tables selected
const canAdd = $derived(selectedDatatable && selectedTables.length > 0)
const canAdd = $derived(selectedTables.length > 0)
// Shares the drawer-set override with the Database Manager: same data table,
// same worker group needed to reach it.
@@ -174,37 +371,27 @@
noPadding
>
{#if dbInput && opWs}
{#key selectedDatatable}
<DBManagerContent
bind:this={dbManagerContent}
input={dbInput}
workspace={opWs}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey
bind:selectedTableKey
multiSelectMode={true}
bind:selectedTables
{disabledTables}
>
{#snippet dbSelector()}
{#if datatables.loading}
<div class="flex items-center gap-2 text-tertiary ml-2">
<LoaderCircle size={14} class="animate-spin" />
<span class="text-sm">Loading...</span>
</div>
{:else if datatables.current.length >= 1}
<Select
transformInputSelectedText={(s) => `Datatable: ${s}`}
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Select data table"
size="md"
/>
{/if}
{/snippet}
</DBManagerContent>
{/key}
{#if roleSettled}
{#key `${selectedDatatable}~${selectedRole ?? ''}`}
<DBManagerContent
bind:this={dbManagerContent}
input={dbInput}
workspace={opWs}
bind:workerTag={() => workerTag.tag, (v) => (workerTag.tag = v)}
bind:hasReplResult
bind:selectedSchemaKey
bind:selectedTableKey
multiSelectMode={true}
bind:selectedTables={() => selectedTables, setSelectedTables}
{disabledTables}
datatableTree={datatableTree.current}
datatableTreeLoading={datatableTree.loading}
onSelectDatatable={(dt) => selectDatatable(dt)}
onSelectRole={(dt, role) => selectDatatable(dt, role)}
bind:pendingAction
/>
{/key}
{/if}
{:else}
<div class="flex items-center justify-center h-full text-tertiary">
<span>Select a data table to explore</span>
@@ -213,12 +400,11 @@
{#snippet actions()}
<Button
variant="contained"
color="blue"
variant="accent"
disabled={!canAdd}
on:click={handleAddTables}
startIcon={{ icon: Plus }}
size="xs"
unifiedSize="sm"
>
{#if selectedTables.length > 0}
Add {selectedTables.length} table{selectedTables.length > 1 ? 's' : ''}
@@ -239,8 +425,8 @@
loading={dbManagerContent?.isLoading() ?? false}
on:click={() => dbManagerContent?.refresh()}
startIcon={{ icon: RefreshCcw }}
size="xs"
color="light"
unifiedSize="sm"
variant="default"
disabled={!selectedDatatable}
>
Refresh
@@ -249,8 +435,9 @@
<Button
on:click={() => (expand = !expand)}
startIcon={{ icon: expand ? Minimize : Expand }}
size="xs"
color="light"
unifiedSize="sm"
variant="default"
iconOnly
/>
{/snippet}
</DrawerContent>
@@ -16,6 +16,8 @@
defaultDatatable?: string | undefined
/** Default schema for new tables */
defaultSchema?: string | undefined
/** The role the app uses each data table through, by data table name */
roles?: Record<string, string>
onAdd?: () => void
onRemove?: (index: number) => void
onSelect?: (ref: DataTableRef, index: number) => void
@@ -31,6 +33,7 @@
dataTableRefs = [],
defaultDatatable = undefined,
defaultSchema = undefined,
roles = undefined,
onAdd,
onRemove,
onSelect,
@@ -95,6 +98,7 @@
<DefaultDatabaseSelector
datatable={defaultDatatable}
schema={defaultSchema}
{roles}
onChange={onDefaultChange}
/>
{/if}
@@ -124,6 +128,14 @@
<div class="flex items-center gap-1.5 px-1 py-1 text-2xs text-tertiary">
<Database size={12} class="shrink-0" />
<span class="font-medium truncate">{datatableName}</span>
{#if roles?.[datatableName]}
<span
class="truncate font-mono"
title="The app's queries on this data table run as this role"
>
as {roles[datatableName]}
</span>
{/if}
{#if isDefaultDatatable}
<span title="Default datatable">
<Star size={10} class="shrink-0 text-primary" />
@@ -69,8 +69,10 @@
formatDataTableRef,
isDatatableTableAllowed,
type RawAppData,
DEFAULT_DATA
DEFAULT_DATA,
appDatatableRole
} from './dataTableRefUtils'
import { datatableReference } from '../dbTypes'
import { randomUUID } from '$lib/utils/uuid'
import { editorFontSize } from '$lib/editorFontSize.svelte'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
@@ -734,11 +736,25 @@
runnables = update.runnables
}
if (update.data !== undefined) {
data = update.data
replaceData(update.data)
}
historyManager.manualSnapshot(files ?? {}, runnables, summary, data, true)
}
/** Replaces `data` from outside the editor (history, YAML). The policy sync writes the policy
* into `data`, so the policy takes the new values first or it puts the old ones straight back. */
function replaceData(next: RawAppData) {
data = next
aiChatManager.datatableCreationPolicy = {
...aiChatManager.datatableCreationPolicy,
// As on load: data that names no data table leaves nothing to create tables in.
enabled: next.datatable !== undefined,
datatable: next.datatable,
schema: next.schema,
roles: next.roles
}
}
let jobs: string[] = $state([])
let jobsById: Record<string, JobById> = $state({})
@@ -906,7 +922,8 @@
aiChatManager.datatableCreationPolicy = {
enabled: data.datatable !== undefined,
datatable: data.datatable,
schema: data.schema
schema: data.schema,
roles: data.roles
}
// Start auto-snapshot
@@ -928,9 +945,15 @@
// Read the current policy from aiChatManager
const policy = aiChatManager.datatableCreationPolicy
// Only update if different to avoid infinite loops
if (data.datatable !== policy.datatable || data.schema !== policy.schema) {
if (
data.datatable !== policy.datatable ||
data.schema !== policy.schema ||
// By value: the policy holds its own proxy of the same map.
JSON.stringify(data.roles) !== JSON.stringify(policy.roles)
) {
data.datatable = policy.datatable
data.schema = policy.schema
data.roles = policy.roles
}
})
@@ -1107,10 +1130,32 @@
return []
}
const tables = await WorkspaceService.listDataTableTables({
workspace: opWorkspace
// A data table the app uses through a role is listed as that role, so the AI sees
// what the app's own queries reach.
const workspace = opWorkspace
const tables = await WorkspaceService.listDataTableTables({ workspace })
// Only data tables that still exist: `data.roles` can outlive a removed or renamed one,
// and the server answers a `role_for` naming nothing with a 404.
const roled = Object.entries(data.roles ?? {}).filter(([dt]) =>
tables.some((t) => t.datatable_name === dt)
)
const roledTables = await Promise.all(
roled.map(([roleFor, role]) =>
WorkspaceService.listDataTableTables({
workspace,
datatableName: roleFor,
roleFor,
role
})
)
)
const merged = tables.map((entry) => {
const i = roled.findIndex(([dt]) => dt === entry.datatable_name)
return i === -1
? entry
: (roledTables[i].find((t) => t.datatable_name === entry.datatable_name) ?? entry)
})
return filterDatatableTables(tables)
return filterDatatableTables(merged)
},
getDatatableTableSchema: async (
datatableName: string,
@@ -1134,7 +1179,8 @@
workspace: opWorkspace,
datatableName,
schemaName,
tableName
tableName,
role: appDatatableRole(data.roles, datatableName)
})
return schema.columns
},
@@ -1152,13 +1198,15 @@
}
try {
// The same role the app's runnables use, so a table the AI creates belongs to it.
const role = appDatatableRole(data.roles, datatableName)
const result = await runScriptAndPollResult(
{
workspace: opWorkspace,
requestBody: {
language: 'postgresql',
content: sql,
args: { database: `datatable://${datatableName}` }
args: { database: datatableReference(datatableName, role) }
}
},
writingJobOptions
@@ -1178,6 +1226,12 @@
const resourcePath = `datatable://${datatableName}`
delete $dbSchemas[resourcePath]
delete $dbSchemas[`${opWorkspace}:${resourcePath}`]
// The DB manager keys its cache by the role it connected as too.
for (const key of Object.keys($dbSchemas)) {
if (key.startsWith(`${opWorkspace}:${resourcePath}?role=`)) {
delete $dbSchemas[key]
}
}
}
}
@@ -2151,7 +2205,7 @@
files = structuredClone($state.snapshot(entry.files))
runnables = structuredClone($state.snapshot(entry.runnables))
summary = entry.summary
data = structuredClone($state.snapshot(entry.data))
replaceData(structuredClone($state.snapshot(entry.data)))
// If the open document survives into the new files, use the combined message
if (iframeDocument && isOpenableDocument(iframeDocument)) {
@@ -2386,6 +2440,20 @@
schema
}
}}
datatableRoles={data.roles}
onDatatableRolesChange={(roles, roleChanged) => {
// The default schema was picked among what the previous role reaches: after the
// user moves the app's default data table to another role, it is picked again.
const dt = data.datatable
const schemaStale = dt !== undefined && roleChanged.has(dt)
data.roles = roles
if (schemaStale) data.schema = undefined
aiChatManager.datatableCreationPolicy = {
...aiChatManager.datatableCreationPolicy,
roles,
...(schemaStale && { schema: undefined })
}
}}
{runnables}
{modules}
{historyManager}
@@ -40,6 +40,13 @@
/** Default schema for new tables */
defaultSchema?: string | undefined
onDefaultChange?: (datatable: string | undefined, schema: string | undefined) => void
/** The role the app uses each data table through, by data table name */
datatableRoles?: Record<string, string>
/** `roleChanged` names the data tables now used through another role than before. */
onDatatableRolesChange?: (
roles: Record<string, string> | undefined,
roleChanged: Set<string>
) => void
}
let {
@@ -60,15 +67,27 @@
onDataTableRefsChange,
defaultDatatable = undefined,
defaultSchema = undefined,
onDefaultChange
onDefaultChange,
datatableRoles = undefined,
onDatatableRolesChange
}: Props = $props()
let dataTableDrawer: RawAppDataTableDrawer | undefined = $state()
let selectedDataTableIndex: number | undefined = $state(undefined)
let sharedUiDrawer: RawAppSharedUiDrawer | undefined = $state()
function handleAddDataTable(ref: DataTableRef) {
onDataTableRefsChange?.([...dataTableRefs, ref])
function handleAddDataTables(
refs: DataTableRef[],
browsedRoles: Record<string, string>,
roleChanged: Set<string>
) {
onDataTableRefsChange?.([
...dataTableRefs.filter((r) => !roleChanged.has(r.datatable)),
...refs
])
if (Object.keys(browsedRoles).length > 0) {
onDatatableRolesChange?.({ ...datatableRoles, ...browsedRoles }, roleChanged)
}
}
function handleRemoveDataTable(index: number) {
@@ -153,6 +172,7 @@
{dataTableRefs}
{defaultDatatable}
{defaultSchema}
roles={datatableRoles}
onAdd={() => dataTableDrawer?.openDrawer()}
onRemove={handleRemoveDataTable}
onSelect={handleSelectDataTable}
@@ -161,8 +181,9 @@
/>
<RawAppDataTableDrawer
bind:this={dataTableDrawer}
onAdd={handleAddDataTable}
onAdd={handleAddDataTables}
existingRefs={dataTableRefs}
roles={datatableRoles}
/>
<RawAppSharedUiDrawer bind:this={sharedUiDrawer} />
@@ -1,4 +1,5 @@
<script lang="ts">
import { untrack } from 'svelte'
import { Sparkles, Plus, List, Ban, ExternalLinkIcon, Loader2 } from 'lucide-svelte'
import type { Policy } from '$lib/gen'
import { superadmin, userStore } from '$lib/stores'
@@ -18,13 +19,21 @@
import { loadCopilot } from '$lib/components/copilot/loadCopilot'
import { react18Template, react19Template, svelte5Template } from './templates'
import type { Runnable } from './rawAppPolicy'
import { type DataTableRef, type RawAppData, formatDataTableRef } from './dataTableRefUtils'
import {
type DataTableRef,
type RawAppData,
formatDataTableRef,
withAppDatatableRole
} from './dataTableRefUtils'
import {
createDatatableAccessResource,
createDatatablesResource,
createSchemasResource,
createRolesResource,
rolesWorthPicking,
toDatatableItems,
toSchemaItems
} from './datatableUtils.svelte'
import { datatableNameTakesRole, defaultMigrationRole } from '../dbTypes'
import RawAppDataTableList from './RawAppDataTableList.svelte'
import RawAppDataTableDrawer from './RawAppDataTableDrawer.svelte'
import FileEditorIcon from './FileEditorIcon.svelte'
@@ -62,19 +71,142 @@
let appSummary = $state('')
let initialPrompt = $state('')
let preWhitelistedTables = $state<DataTableRef[]>([])
/** The role each pre-whitelisted table's data table was browsed as. */
let preWhitelistedRoles = $state<Record<string, string>>({})
let dataTableDrawer: RawAppDataTableDrawer | undefined = $state()
const operatingWorkspace = useOperatingWorkspace()
let opWs = $derived($operatingWorkspace)
const datatables = createDatatablesResource(() => opWs)
const schemas = createSchemasResource(
const roles = createRolesResource(
() => selectedDatatable,
() => opWs
)
let selectedRole = $state<string | undefined>(undefined)
// Every reader waits for an answer stamped with the current selection: until then `current`
// belongs to the previous workspace, data table or role.
// A data table whose name cannot carry a role in a reference is used as its default one.
const rolesAnswered = $derived(
roles.current.workspace === opWs && roles.current.datatable === selectedDatatable
)
const loadedRoles = $derived(
rolesAnswered &&
selectedDatatable !== undefined &&
datatableNameTakesRole(selectedDatatable)
? roles.current.roles
: []
)
const showRolePicker = $derived(rolesWorthPicking(loadedRoles))
// Saved explicitly rather than left to resolve: "whatever the default is then" moves the app
// the day an admin changes the default.
const effectiveRole = $derived(
selectedRole !== undefined && loadedRoles.includes(selectedRole) ? selectedRole : undefined
)
// An app uses one role per data table: the one picked above is what the table drawer browses
// as and what the list shows, or tables would be added under a role the app is not saved with.
const pickerRoles = $derived(
selectedDatatable !== undefined && effectiveRole !== undefined
? withAppDatatableRole(preWhitelistedRoles, selectedDatatable, effectiveRole)
: preWhitelistedRoles
)
const availableDatatables = $derived(datatables.current)
const availableSchemas = $derived(schemas.current)
// `undefined` while the list loads, so this is false until it has answered.
const hasNoDatatables = $derived(availableDatatables?.length === 0)
const rolesSettled = $derived(
hasNoDatatables || (selectedDatatable !== undefined && rolesAnswered)
)
// A role is picked on one data table: two data tables can both define an `analyst` that
// means something different, so a name surviving the switch is not the role surviving it.
let rolesPickedOn = $state<string | undefined>(undefined)
$effect(() => {
const loaded = roles.current
if (!rolesAnswered) return
const switched = untrack(() => rolesPickedOn) !== selectedDatatable
const current = untrack(() => selectedRole)
if (switched || current === undefined || !loaded.roles.includes(current)) {
// Tables already picked on this data table were browsed as a role: keep that one.
const browsed = selectedDatatable
? untrack(() => preWhitelistedRoles)[selectedDatatable]
: undefined
pickRole(
browsed !== undefined && loaded.roles.includes(browsed)
? browsed
: loaded.roles.includes(loaded.defaultRole)
? loaded.defaultRole
: loaded.roles[0]
)
rolesPickedOn = selectedDatatable
}
})
/** Picks the app's role on the selected data table. Tables picked on it under another role are
* dropped: that role may reach them where this one does not. */
function pickRole(role: string | undefined) {
selectedRole = role
const dt = selectedDatatable
const browsed = dt ? preWhitelistedRoles[dt] : undefined
if (dt === undefined || role === undefined || browsed === undefined || browsed === role) return
preWhitelistedTables = preWhitelistedTables.filter((t) => t.datatable !== dt)
const { [dt]: _, ...rest } = preWhitelistedRoles
preWhitelistedRoles = rest
}
const access = createDatatableAccessResource(
() => selectedDatatable,
() => effectiveRole,
() => opWs,
() => rolesSettled && (loadedRoles.length === 0 || effectiveRole !== undefined)
)
// Under roles, and this caller may use none of them: the app would be saved with queries the
// server refuses.
const noUsableRole = $derived(
rolesSettled &&
selectedDatatable !== undefined &&
roles.current.permissioned &&
roles.current.roles.length === 0
)
const rolesUnknown = $derived(rolesSettled && roles.current.failed)
const accessSettled = $derived(
hasNoDatatables ||
(rolesSettled &&
selectedDatatable !== undefined &&
access.current.workspace === opWs &&
access.current.datatable === selectedDatatable &&
access.current.role === effectiveRole)
)
const accessUnknown = $derived(accessSettled && access.current.failed)
const availableSchemas = $derived(accessSettled ? access.current.schemas : [])
const canCreateSchema = $derived(accessSettled && access.current.canCreateSchema)
// Only an app that keeps the data table is held back by it: with table creation off nothing
// saves it.
const blockedByRole = $derived(
(noUsableRole || rolesUnknown || accessUnknown) && tableCreationEnabled
)
// A role that cannot create schemas has nothing to name, so the mode goes back to the one
// every role has, once that is an answer.
$effect(() => {
if (accessSettled && !accessUnknown && schemaMode === 'new' && !canCreateSchema) {
schemaMode = 'none'
}
})
// Likewise an existing schema the current role no longer reaches is unpicked, so the select
// does not keep showing it.
$effect(() => {
if (
accessSettled &&
selectedSchema !== undefined &&
!availableSchemas.includes(selectedSchema)
) {
selectedSchema = undefined
}
})
let hasAutoSelected = false
$effect(() => {
@@ -115,12 +247,18 @@
const datatableItems = $derived(toDatatableItems(availableDatatables))
const schemaItems = $derived(toSchemaItems(availableSchemas))
// An existing schema counts only while the current role reaches it: one picked under another
// role would save an app that creates its tables where it cannot.
const effectiveSchema = $derived(
schemaMode === 'new' ? newSchemaName : schemaMode === 'existing' ? selectedSchema : undefined
schemaMode === 'new'
? newSchemaName
: schemaMode === 'existing' &&
selectedSchema !== undefined &&
availableSchemas.includes(selectedSchema)
? selectedSchema
: undefined
)
const hasNoDatatables = $derived(availableDatatables?.length === 0)
// copilotInfo is a global that stays empty until some ancestor's fetch lands, so
// `enabled` alone cannot tell "no providers" from "not loaded yet" and the modal
// would announce AI as unconfigured while it is merely unknown. Gate on the
@@ -140,7 +278,13 @@
async function start(withPrompt: boolean) {
const template = templates[selectedTemplateIndex]
if (schemaMode === 'new' && newSchemaName && selectedDatatable && opWs) {
if (
tableCreationEnabled &&
schemaMode === 'new' &&
newSchemaName &&
selectedDatatable &&
opWs
) {
try {
const { dbSchemaOpsWithPreviewScripts } = await import('$lib/components/dbOps')
const dbOps = dbSchemaOpsWithPreviewScripts({
@@ -148,7 +292,13 @@
input: {
type: 'database',
resourceType: 'postgresql',
resourcePath: `datatable://${selectedDatatable}`
resourcePath: `datatable://${selectedDatatable}`,
role: effectiveRole,
migrationRole: defaultMigrationRole(
selectedDatatable,
roles.current.permissioned,
roles.current.defaultRole
)
}
})
await dbOps.onCreateSchema({ schema: newSchemaName })
@@ -159,14 +309,20 @@
}
const formattedTables = preWhitelistedTables.map(formatDataTableRef)
const data: RawAppData =
tableCreationEnabled && selectedDatatable
? {
tables: formattedTables,
datatable: selectedDatatable,
schema: effectiveSchema
}
: { tables: formattedTables, datatable: undefined, schema: undefined }
const keepsDatatable = tableCreationEnabled && selectedDatatable !== undefined
// The roles shown, for the data tables the app ends up using.
const usedDatatables = new Set(preWhitelistedTables.map((t) => t.datatable))
if (keepsDatatable) usedDatatables.add(selectedDatatable!)
const shownRoles = Object.entries(pickerRoles ?? {}).filter(([dt]) => usedDatatables.has(dt))
const appRoles = shownRoles.length > 0 ? Object.fromEntries(shownRoles) : undefined
const data: RawAppData = keepsDatatable
? {
tables: formattedTables,
datatable: selectedDatatable,
schema: effectiveSchema,
roles: appRoles
}
: { tables: formattedTables, datatable: undefined, schema: undefined, roles: appRoles }
const policy: Policy = {
on_behalf_of: $userStore?.username.includes('@')
@@ -259,15 +415,43 @@
<label class="text-xs text-emphasis font-semibold" for="datatable"
>Datatable</label
>
<Select
id="datatable"
disablePortal
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Datatable"
size="sm"
class="w-40"
/>
<div class="flex flex-row items-center gap-2">
<Select
id="datatable"
disablePortal
items={datatableItems}
bind:value={selectedDatatable}
placeholder="Datatable"
size="sm"
class="w-40"
/>
{#if showRolePicker}
<!-- Reads as one phrase, "main as analyst", so the role needs no label. -->
<span class="text-xs text-secondary">as</span>
<Select
id="datatable-role"
disablePortal
items={loadedRoles.map((r) => ({ value: r, label: r }))}
bind:value={() => selectedRole, pickRole}
clearable={false}
placeholder="Role"
size="sm"
class="w-40"
/>
{/if}
{#if noUsableRole || rolesUnknown || accessUnknown}
<span
class="text-xs text-red-600 dark:text-red-400"
title={access.current.error}
>
{rolesUnknown
? 'could not read its roles'
: noUsableRole
? 'no role you can use'
: 'could not reach it'}
</span>
{/if}
</div>
</div>
<div>
<span class="text-xs text-emphasis font-semibold">Schema</span>
@@ -276,7 +460,21 @@
<ToggleButtonGroup bind:selected={schemaMode} noWFull>
{#snippet children({ item })}
<ToggleButton value="none" label="None" icon={Ban} {item} size="sm" />
<ToggleButton value="new" label="New" icon={Plus} {item} size="sm" />
<ToggleButton
value="new"
label="New"
icon={Plus}
disabled={!canCreateSchema}
tooltip={canCreateSchema
? undefined
: noUsableRole
? `You can use no role of ${selectedDatatable}`
: accessUnknown
? `Could not read what may be created in ${selectedDatatable}`
: `${effectiveRole ?? 'This connection'} cannot create schemas in ${selectedDatatable}`}
{item}
size="sm"
/>
<ToggleButton
value="existing"
label="Existing"
@@ -333,6 +531,7 @@
dataTableRefs={preWhitelistedTables}
defaultDatatable={selectedDatatable}
defaultSchema={effectiveSchema}
roles={pickerRoles}
standalone
hideDefaultSelector
onAdd={() => dataTableDrawer?.openDrawer()}
@@ -418,7 +617,11 @@
variant="default"
size="sm"
on:click={() => start(false)}
disabled={!templates[selectedTemplateIndex] || newSchemaAlreadyExists}
disabled={!templates[selectedTemplateIndex] ||
newSchemaAlreadyExists ||
!rolesSettled ||
!accessSettled ||
blockedByRole}
>
{$copilotInfo.workspaceDisabled ? 'Start' : 'Start without AI'}
</Button>
@@ -426,7 +629,10 @@
<Button
variant="accent"
on:click={() => start(true)}
disabled={!templates[selectedTemplateIndex] ||
disabled={!rolesSettled ||
!accessSettled ||
blockedByRole ||
!templates[selectedTemplateIndex] ||
!initialPrompt.trim() ||
newSchemaAlreadyExists}
startIcon={{ icon: Sparkles }}
@@ -444,7 +650,15 @@
bind:this={dataTableDrawer}
offset={10000}
existingRefs={preWhitelistedTables}
onAdd={(ref) => {
preWhitelistedTables = [...preWhitelistedTables, ref]
roles={pickerRoles}
onAdd={(refs, browsedRoles, roleChanged) => {
preWhitelistedTables = [
...preWhitelistedTables.filter((t) => !roleChanged.has(t.datatable)),
...refs
]
preWhitelistedRoles = { ...preWhitelistedRoles, ...browsedRoles }
if (selectedDatatable !== undefined && browsedRoles[selectedDatatable] !== undefined) {
selectedRole = browsedRoles[selectedDatatable]
}
}}
/>
@@ -1,5 +1,31 @@
import { describe, expect, it } from 'vitest'
import { buildDataTableWhitelist, isDatatableTableAllowed } from './dataTableRefUtils'
import {
buildDataTableWhitelist,
isDatatableTableAllowed,
sdkDatatableCall,
withAppDatatableRole
} from './dataTableRefUtils'
describe('app data table roles', () => {
it('writes the role the way each SDK takes it', () => {
expect(sdkDatatableCall('main', 'analyst', 'typescript')).toBe(
"wmill.datatable('main', { role: 'analyst' })"
)
expect(sdkDatatableCall('main', 'analyst', 'python')).toBe(
"wmill.datatable('main', role='analyst')"
)
expect(sdkDatatableCall('main', undefined, 'python')).toBe('wmill.datatable()')
})
it('keeps one role per data table, and no map once none is left', () => {
const roles = withAppDatatableRole(undefined, 'main', 'analyst')
expect(withAppDatatableRole(roles, 'other', 'operator')).toEqual({
main: 'analyst',
other: 'operator'
})
expect(withAppDatatableRole(roles, 'main', undefined)).toBeUndefined()
})
})
describe('datatable whitelist helpers', () => {
it('allows every datatable table when no refs are configured', () => {
@@ -16,6 +16,43 @@ export interface RawAppData {
datatable: string | undefined
/** The schema for table creation (if specified) */
schema: string | undefined
/** The role the app uses each data table through, by data table name. A data table without
* an entry is used as its default role, then `admin`. */
roles?: Record<string, string>
}
export function appDatatableRole(
roles: Record<string, string> | undefined,
datatable: string
): string | undefined {
return roles?.[datatable]
}
/** `roles` with `datatable` set to `role`, or without it when `role` is undefined. */
export function withAppDatatableRole(
roles: Record<string, string> | undefined,
datatable: string,
role: string | undefined
): Record<string, string> | undefined {
const next = { ...roles }
if (role === undefined) delete next[datatable]
else next[datatable] = role
return Object.keys(next).length > 0 ? next : undefined
}
/** The SDK call app code uses to reach a data table, in the language it is written in. A role
* is a keyword argument in Python and an option in TypeScript. */
export function sdkDatatableCall(
datatable: string,
role: string | undefined,
language: 'typescript' | 'python'
): string {
if (role === undefined) {
return datatable === 'main' ? 'wmill.datatable()' : `wmill.datatable('${datatable}')`
}
return language === 'python'
? `wmill.datatable('${datatable}', role='${role}')`
: `wmill.datatable('${datatable}', { role: '${role}' })`
}
/** Default data configuration */
@@ -1,9 +1,27 @@
import { resource } from 'runed'
import { workspaceStore, dbSchemas } from '$lib/stores'
import { workspaceStore } from '$lib/stores'
import { WorkspaceService } from '$lib/gen'
import { getDbSchemas } from '$lib/components/apps/components/display/dbtable/metadata'
import { listUsableDatatableRoles } from '$lib/components/datatableUsableRoles'
import { ADMIN_DATATABLE_ROLE } from '$lib/components/dbTypes'
import { get } from 'svelte/store'
/**
* `fetch` wrapped so that an answer for a request a newer one has replaced resolves to
* `stale()` instead: a resource keeps whichever answer lands last, and a slow answer for the
* previous data table or role would otherwise describe a selection that no longer exists.
*/
function latestOnly<A extends unknown[], T>(
fetch: (...args: A) => Promise<T>,
stale: () => T
): (...args: A) => Promise<T> {
let run = 0
return async (...args) => {
const mine = ++run
const result = await fetch(...args)
return mine === run ? result : stale()
}
}
/**
* Creates a resource that loads available datatables from the workspace.
* Pass a getter function that returns the workspace to create a reactive dependency.
@@ -21,42 +39,151 @@ export function createDatatablesResource(getWorkspace: () => string | undefined)
})
}
export type DatatableRoles = {
/** The workspace and data table this answers for: while a switch is in flight, `current` still
* holds the previous ones' roles, and a fork and its parent both have a `main` of their own. */
workspace: string | undefined
datatable: string | undefined
/** Whether the data table is under roles. Without roles `roles` is empty because there is
* nothing to pick, which is not the same as a permissioned one this caller may use no role of. */
permissioned: boolean
/** The lookup failed, so an empty `roles` means nothing was learned. */
failed: boolean
roles: string[]
defaultRole: string
}
/**
* Creates a resource that loads schemas for a given datatable.
* The getDatatable getter is used as a reactive dependency - when it changes, schemas are refetched.
* Creates a resource that loads the roles the caller may use on a datatable, and the one it
* defaults to.
*/
export function createSchemasResource(
export function createRolesResource(
getDatatable: () => string | undefined,
getWorkspace: () => string | undefined = () => get(workspaceStore)
) {
return resource<string[]>([() => getDatatable() ?? '', () => getWorkspace() ?? ''], async () => {
const datatable = getDatatable()
const workspace = getWorkspace()
if (!datatable || !workspace) return []
const initialValue: DatatableRoles = {
workspace: undefined,
datatable: undefined,
permissioned: false,
failed: false,
roles: [],
defaultRole: ADMIN_DATATABLE_ROLE
}
const rolesResource = resource(
() => [getDatatable() ?? '', getWorkspace() ?? ''] as const,
latestOnly(
async ([datatableName, workspace]: readonly [string, string]): Promise<DatatableRoles> => {
const empty = {
...initialValue,
workspace: workspace || undefined,
datatable: datatableName || undefined
}
if (!datatableName || !workspace) return empty
try {
const res = await listUsableDatatableRoles(workspace, datatableName)
return {
...empty,
permissioned: res.permissioned,
roles: res.roles,
defaultRole: res.default_role
}
} catch (e) {
console.error('Failed to load datatable roles:', e)
return { ...empty, failed: true }
}
},
() => rolesResource.current
),
{ initialValue }
)
return rolesResource
}
const resourcePath = `datatable://${datatable}`
// Key the schema cache by workspace too: a datatable of the same name can
// exist in both the nav and the acting workspace, so `datatable://<name>`
// alone would let one workspace's schema be reused for the other.
const cacheKey = `${workspace}:${resourcePath}`
const schemas = get(dbSchemas)
let dbSchema = schemas[cacheKey]
export type DatatableAccess = {
/** What this answers for. Until all three match the selection, the schemas and the right to
* create one belong to another workspace, data table or role. */
workspace: string | undefined
datatable: string | undefined
role: string | undefined
/** The request failed, or the server kept the entry with an error (a role this caller may
* not use, an unreachable database). `canCreateSchema: false` is then no answer at all. */
failed: boolean
error: string | undefined
schemas: string[]
canCreateSchema: boolean
}
if (!dbSchema) {
try {
schemas[cacheKey] = await getDbSchemas('postgresql', resourcePath, workspace, (msg) =>
console.error('Schema error:', msg)
)
dbSchema = get(dbSchemas)[cacheKey]
} catch (e) {
console.error(`Failed to load schema for ${datatable}:`, e)
return []
}
}
/**
* Creates a resource that loads, for one data table read as one role, the schemas that role can
* reach and whether it may create more.
*/
export function createDatatableAccessResource(
getDatatable: () => string | undefined,
getRole: () => string | undefined,
getWorkspace: () => string | undefined = () => get(workspaceStore),
/** False while the role is still being settled: a listing sent before that is read as the
* data table's default role, which is not the one about to be asked for. */
getReady: () => boolean = () => true
) {
const initialValue: DatatableAccess = {
workspace: undefined,
datatable: undefined,
role: undefined,
failed: false,
error: undefined,
schemas: [],
canCreateSchema: false
}
const accessResource = resource(
() => [getDatatable() ?? '', getRole() ?? '', getWorkspace() ?? '', getReady()] as const,
latestOnly(
async ([datatable, role, workspace, ready]: readonly [
string,
string,
string,
boolean
]): Promise<DatatableAccess> => {
const asked = {
...initialValue,
workspace: workspace || undefined,
datatable: datatable || undefined,
role: role || undefined
}
if (!ready) return accessResource.current
if (!datatable || !workspace) return asked
try {
const tables = await WorkspaceService.listDataTableTables({
workspace,
datatableName: datatable,
roleFor: datatable,
role: role || undefined
})
const entry = tables.find((t) => t.datatable_name === datatable)
return {
...asked,
failed: entry === undefined || entry.error !== undefined,
error: entry?.error,
schemas: Object.keys(entry?.schemas ?? {}).sort(),
canCreateSchema: !!entry?.can_create_schema
}
} catch (e) {
console.error('Failed to load datatable access:', e)
return { ...asked, failed: true, error: (e as Error)?.message }
}
},
() => accessResource.current
),
{ initialValue }
)
return accessResource
}
if (!dbSchema?.schema) return []
return Object.keys(dbSchema.schema)
})
/**
* Whether naming a role says anything: a data table without roles has none to pick, and one
* whose single usable role is `admin` offers no choice.
*/
export function rolesWorthPicking(roles: string[]): boolean {
return roles.length > 1 || (roles.length === 1 && roles[0] !== ADMIN_DATATABLE_ROLE)
}
/**
@@ -380,7 +380,7 @@
failedSyncJobs = []
forkCreationError = ''
// Clone datatables BEFORE creating the workspace fork
// Each data table copy is confirmed first; the fork request makes the copies
if (forkDatatableSection) {
const queue = forkDatatableSection.buildCloneQueue(prefixed_id)
if (queue.length > 0) {
@@ -396,6 +396,15 @@
}
async function completeFork(prefixed_id: string): Promise<void> {
// The fork request makes these copies, and drops them if the fork is not created
const forkedDatatables = forkDatatableSection
? forkDatatableSection.getConfirmedCloneJobs().map((job) => ({
name: job.name,
new_dbname: job._newDbName,
fork_behavior: job.behavior
}))
: []
let gitSyncJobIds: string[]
try {
gitSyncJobIds = await WorkspaceService.createWorkspaceForkGitBranch({
@@ -404,6 +413,9 @@
id: prefixed_id,
name,
color: colorEnabled && workspaceColor ? workspaceColor : undefined,
// Sent in this first phase too, so a clone the fork would refuse is refused before
// any branch is created.
forked_datatables: forkedDatatables,
is_dev_workspace: createAsDevWorkspace,
dev_workspace_label: createAsDevWorkspace ? devWorkspaceLabel : undefined,
// Send the lock intent in this first phase too so the backend can reject a non-admin's
@@ -457,14 +469,6 @@
return
}
// Build forked_datatables info from completed clone jobs
const forkedDatatables = forkDatatableSection
? forkDatatableSection.getCompletedCloneJobs().map((job) => ({
name: job.name,
new_dbname: job._newDbName
}))
: []
try {
await WorkspaceService.createWorkspaceFork({
workspace: baseWorkspaceId!,
@@ -28,6 +28,7 @@
import MissingWorkerTagAlert from '../jobs/MissingWorkerTagAlert.svelte'
import { superadmin } from '$lib/stores'
import { useOperatingUser } from '$lib/components/operatingWorkspace.svelte'
import { parseMigrationRole, withMigrationRole } from '../datatableMigrationRole'
const operatingUser = useOperatingUser()
const actingUser = $derived(operatingUser.current)
@@ -88,8 +89,9 @@
function startAddDownMigration() {
// Same transaction frame the new-migration modal starts from, so the down
// applies atomically.
downDraft = DOWN_TEMPLATE
// applies atomically. It rolls back as the role the up ran as.
const upRole = viewMigration ? parseMigrationRole(viewMigration.code_up) : undefined
downDraft = withMigrationRole(DOWN_TEMPLATE, upRole?.kind === 'role' ? upRole.role : undefined)
addingDown = true
}
@@ -171,6 +173,10 @@
loadMigrations()
}
export function open() {
openList()
}
// Open the list modal and the detail view for a specific migration. Used to
// jump to a just-created migration from the "See migration" toast action.
export async function openMigration(timestamp: number) {
@@ -5,77 +5,89 @@
import Toggle from '../Toggle.svelte'
import Tooltip from '../Tooltip.svelte'
import CloseButton from '../common/CloseButton.svelte'
import { KeyRound, Plus } from 'lucide-svelte'
import Checkbox from '../common/checkbox/Checkbox.svelte'
import Cell from '../table/Cell.svelte'
import DataTable from '../table/DataTable.svelte'
import Head from '../table/Head.svelte'
import Row from '../table/Row.svelte'
import { KeyRound } from 'lucide-svelte'
import {
FolderService,
GroupService,
SettingService,
UserService,
WorkspaceService,
type AclTarget,
type DatatableAclInfo,
type DatatablePermissions,
type InstanceDatatableRole
} from '$lib/gen'
import { superadmin } from '$lib/stores'
import { sendUserToast } from '$lib/toast'
import AclTargetPicker from '../datatableAcl/AclTargetPicker.svelte'
import { deepEqual } from 'fast-equals'
import { ADMIN_DATATABLE_ROLE, isDatatableRoleName } from '../dbTypes'
import PgAclEditor from '../datatableAcl/PgAclEditor.svelte'
const ADMIN_ROLE = 'admin'
import InstanceRolesButton from './InstanceRolesButton.svelte'
let {
workspace,
datatable,
disabled = false
disabled = false,
hideTrigger = false,
onSaved
}: {
workspace: string
datatable: string
disabled?: boolean
/** Mount the drawer without its button, for a caller that opens it with `open()`. */
hideTrigger?: boolean
/** Called once a save went through, so a caller showing the roles can read them again. */
onSaved?: () => void
} = $props()
let drawer: Drawer | undefined = $state(undefined)
// `id` is the instance role's catalog id (or the reserved `admin`), which is what the tenant
// lists are keyed by — so renaming a role instance-side moves nothing here. A row without an id
// names a role the instance does not define yet: it cannot be saved until a superadmin creates
// it, and takes the new role's id once they have.
type EditedRole = { id: string | undefined; name: string | undefined; tenants: string[] }
type Edited = { permissioned: boolean; roles: EditedRole[]; defaultRoleId: string }
let drawerOpen = $state(false)
let loading = $state(false)
let saving = $state(false)
let loadError = $state<string | undefined>(undefined)
let info = $state<DatatablePermissions | undefined>(undefined)
// Edited copy. `id` is the instance role's catalog id (or the reserved `admin`), which is what
// the tenant lists are keyed by — so renaming a role instance-side moves nothing here.
let permissioned = $state(false)
let defaultRole = $state(ADMIN_ROLE)
let rows = $state<{ id: string; name: string | undefined; tenants: string[] }[]>([])
let roles = $state<EditedRole[]>([])
let defaultRoleId = $state(ADMIN_DATATABLE_ROLE)
/** The last loaded state, to detect unsaved changes against. */
let saved = $state<Edited>({
permissioned: false,
roles: [],
defaultRoleId: ADMIN_DATATABLE_ROLE
})
// Tenants name principals of the workspace that governs the data table, which is not
// necessarily the one we are browsing from.
let tenantOptions = $state<{ value: string; label: string }[]>([])
let tenantItems = $state<{ value: string; label: string; group: string }[]>([])
const editable = $derived(!!info?.editable)
const governing = $derived(info?.governing_workspace_id)
const availableRoles: InstanceDatatableRole[] = $derived(info?.available_roles ?? [])
const unusedRoles = $derived(availableRoles.filter((r) => !rows.some((row) => row.id === r.id)))
// The instance catalog, read again after the instance roles drawer changes it.
let catalog = $state<InstanceDatatableRole[] | undefined>(undefined)
const availableRoles: InstanceDatatableRole[] = $derived(catalog ?? info?.available_roles ?? [])
const unusedRoles = $derived(availableRoles.filter((r) => !roles.some((row) => row.id === r.id)))
const pendingRoles = $derived(roles.filter((r) => r.id === undefined))
let instanceRoles: InstanceRolesButton | undefined = $state(undefined)
let aclSchema = $state<string | undefined>(undefined)
let aclTable = $state<string | undefined>(undefined)
const aclTarget: AclTarget = $derived(
aclSchema
? aclTable
? { kind: 'table', schema: aclSchema, table: aclTable }
: { kind: 'schema', schema: aclSchema }
: { kind: 'database' }
const roleKey = (role: EditedRole) => role.id ?? `pending:${role.name}`
const hasUnsavedChanges = $derived(
!deepEqual($state.snapshot(saved), {
permissioned,
roles: $state.snapshot(roles) as EditedRole[],
defaultRoleId
})
)
let aclSchemas = $state<string[]>([])
let aclSchemasLoaded = $state(false)
let aclTables = $state<string[]>([])
// The editor's read of a database lists its schemas, and of a schema its tables — which is what
// the picker offers, so the picker reads nothing of its own. A read for a target since left
// behind is dropped.
function onAclLoaded(target: AclTarget, loaded: DatatableAclInfo) {
if (JSON.stringify(target) !== JSON.stringify(aclTarget)) return
if (target.kind === 'database') {
aclSchemas = loaded.children
aclSchemasLoaded = true
} else if (target.kind === 'schema') aclTables = loaded.children
}
async function load() {
loading = true
@@ -85,16 +97,23 @@
workspace,
datatableName: datatable
})
info = res
permissioned = res.permissioned
defaultRole = res.default_role
rows = (res.roles ?? [])
.map((r) => ({ id: r.id, name: r.name, tenants: r.tenants ?? [] }))
.sort((a, b) => (a.id === ADMIN_ROLE ? -1 : b.id === ADMIN_ROLE ? 1 : 0))
if (rows.length === 0) {
rows = [{ id: ADMIN_ROLE, name: ADMIN_ROLE, tenants: [] }]
const loaded: EditedRole[] = res.roles
.map((r) => ({ id: r.id, name: r.name, tenants: [...(r.tenants ?? [])] }))
.sort(
(a, b) => Number(b.id === ADMIN_DATATABLE_ROLE) - Number(a.id === ADMIN_DATATABLE_ROLE)
)
// A data table never put under roles comes back with none; admin is what turning the toggle
// on starts from.
if (!loaded.some((r) => r.id === ADMIN_DATATABLE_ROLE)) {
loaded.unshift({ id: ADMIN_DATATABLE_ROLE, name: ADMIN_DATATABLE_ROLE, tenants: [] })
}
await loadTenantOptions(res.governing_workspace_id ?? workspace)
info = res
catalog = undefined
permissioned = res.permissioned
roles = loaded
defaultRoleId = res.default_role
saved = structuredClone({ permissioned, roles: loaded, defaultRoleId })
await loadTenantItems(res.governing_workspace_id ?? workspace)
} catch (e) {
loadError = e?.body ?? e?.message ?? String(e)
} finally {
@@ -102,35 +121,70 @@
}
}
async function loadTenantOptions(ws: string) {
async function loadTenantItems(ws: string) {
try {
const [users, groups, folders] = await Promise.all([
UserService.listUsernames({ workspace: ws }),
GroupService.listGroupNames({ workspace: ws }),
FolderService.listFolderNames({ workspace: ws })
])
tenantOptions = [
{ value: '*', label: 'Everyone in the workspace' },
...users.map((u) => ({ value: `u/${u}`, label: `u/${u}` })),
...groups.map((g) => ({ value: `g/${g}`, label: `g/${g}` })),
...folders.map((f) => ({ value: `f/${f}`, label: `f/${f}` }))
tenantItems = [
{ value: '*', label: 'Everyone', group: 'Anyone in the workspace' },
...users.map((u) => ({ value: `u/${u}`, label: u, group: 'Users' })),
...groups.map((g) => ({ value: `g/${g}`, label: g, group: 'Groups' })),
...folders.map((f) => ({ value: `f/${f}`, label: f, group: 'Folders' }))
]
} catch {
// A fork member may not be able to list the governing workspace's principals. The
// tenants they cannot name are still shown, they just cannot pick new ones.
tenantOptions = []
tenantItems = []
}
}
function addRole(id: string) {
const role = availableRoles.find((r) => r.id === id)
if (!role) return
rows = [...rows, { id: role.id, name: role.name, tenants: [] }]
roles.push({ id: role.id, name: role.name, tenants: [] })
}
function removeRole(id: string) {
rows = rows.filter((r) => r.id !== id)
if (defaultRole === id) defaultRole = ADMIN_ROLE
/** Adds a role by name: the instance's role of that name, or a pending row for one it does not
* define yet. */
function addRoleByName(typed: string) {
const name = typed.trim()
if (!isDatatableRoleName(name) || name.toLowerCase() === ADMIN_DATATABLE_ROLE) {
sendUserToast(
`'${name}' cannot be a data table role name: use letters, digits, '_' and '-'`,
true
)
return
}
if (roles.some((r) => r.name === name)) return
const existing = availableRoles.find((r) => r.name === name)
roles.push({ id: existing?.id, name, tenants: [] })
}
function removeRole(role: EditedRole) {
const key = roleKey(role)
roles = roles.filter((r) => roleKey(r) !== key)
if (role.id !== undefined && defaultRoleId === role.id) defaultRoleId = ADMIN_DATATABLE_ROLE
}
/** Reads the instance catalog again and gives each pending row the id of the role now defined
* under its name. */
async function refreshCatalog() {
let fresh: InstanceDatatableRole[]
try {
fresh = await SettingService.listInstanceDatatableRoles()
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
return
}
catalog = fresh
for (const row of roles) {
if (row.id !== undefined) continue
const created = fresh.find((r) => r.name === row.name)
if (created) row.id = created.id
}
}
async function save() {
@@ -141,12 +195,13 @@
datatableName: datatable,
requestBody: {
permissioned,
default_role: defaultRole,
roles: rows.map((r) => ({ id: r.id, tenants: r.tenants }))
default_role: defaultRoleId,
roles: roles.map((r) => ({ id: r.id!, tenants: $state.snapshot(r.tenants) }))
}
})
sendUserToast(msg)
await load()
onSaved?.()
} catch (e) {
sendUserToast(e?.body ?? e?.message ?? String(e), true)
} finally {
@@ -155,47 +210,60 @@
}
export function open() {
aclSchema = undefined
aclTable = undefined
aclSchemas = []
aclSchemasLoaded = false
aclTables = []
drawer?.openDrawer()
drawerOpen = true
load()
}
</script>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: KeyRound }}
iconOnly
{disabled}
title={disabled ? 'Save settings first' : 'Roles: who may connect as which Postgres role'}
on:click={open}
/>
{#if !hideTrigger}
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: KeyRound }}
iconOnly
{disabled}
title={disabled ? 'Save settings first' : 'Roles: who may connect as which Postgres role'}
on:click={open}
/>
{/if}
<Drawer bind:this={drawer} size="700px">
<Drawer bind:open={drawerOpen} size="900px">
<DrawerContent
title="Roles for {datatable}"
on:close={() => drawer?.closeDrawer()}
title="Roles — {datatable}"
on:close={() => (drawerOpen = false)}
tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant it privileges under Access. Roles are defined for the whole instance; here you say who may use each one on this data table."
>
{#snippet titleExtra()}
<Badge color="blue" small>Beta</Badge>
{/snippet}
{#if loading}
<p class="text-sm text-secondary">Loading…</p>
{:else if loadError}
{#if loadError}
<Alert type="error" title="Could not load roles" size="xs">{loadError}</Alert>
{:else if loading && !info}
<span class="text-sm text-secondary">Loading…</span>
{:else}
<div class="flex flex-col gap-4">
<Toggle
bind:checked={permissioned}
disabled={!editable || !info?.supported}
options={{
right: 'Put this data table under roles',
rightTooltip:
'Off, every job connects as admin — the connection that owns every table. On, every job resolves to a role, and a caller no tenant covers is refused.'
}}
/>
{#if !info?.supported}
<Alert type="info" title="Not available on this data table" size="xs">
A data table role is a Postgres login on the Windmill instance's own database, so only a
data table backed by that database can use one. This one is backed by a PostgreSQL
resource — grant access on that server directly.
</Alert>
{:else if info?.clone_of}
<Alert type="info" title="Governed by {info.clone_of.workspace_id}" size="xs">
This data table is a clone of <span class="font-mono">{info.clone_of.datatable}</span>
in workspace <span class="font-mono">{info.clone_of.workspace_id}</span>, so it takes
its roles from there. You are evaluated as a member of that workspace.
</Alert>
{:else if governing}
<Alert type="info" title="Governed by {governing}" size="xs">
This data table points at the one in workspace <span class="font-mono">{governing}</span
@@ -219,126 +287,155 @@
</Alert>
{/if}
<Toggle
bind:checked={permissioned}
disabled={!editable || !info?.supported}
options={{
right: 'Put this data table under roles',
rightTooltip:
'Off, every job connects as admin — the connection that owns every table. On, every job resolves to a role, and a caller no tenant covers is refused.'
}}
/>
{#if permissioned}
{#if availableRoles.length === 0}
{#if editable && availableRoles.length === 0}
<Alert type="warning" title="No role defined on this instance" size="xs">
Only <span class="font-mono">admin</span> can be used until a superadmin adds a data table
role, from Instance roles at the top of the data tables settings page.
Only <span class="font-mono">admin</span> can be used until a superadmin creates a data
table role. Type a name below to add one.
</Alert>
{/if}
<div class="flex flex-col gap-3">
{#each rows as row (row.id)}
<div class="flex flex-col gap-1 border rounded-md p-3 bg-surface-secondary">
<div class="flex items-center gap-2">
<Badge color={row.id === ADMIN_ROLE ? 'blue' : 'gray'}>
{row.name ?? row.id}
</Badge>
{#if row.id === ADMIN_ROLE}
<Tooltip>
The connection every data table resolved to before roles. It owns every
existing object, so it is always available and cannot be removed.
</Tooltip>
{:else if !row.name}
<span class="text-xs text-secondary italic">
no longer defined on this instance
</span>
{/if}
{#if defaultRole === row.id}
<Badge color="green">Default</Badge>
{:else if editable}
<Button
unifiedSize="2xs"
variant="subtle"
on:click={() => (defaultRole = row.id)}
>
Make default
</Button>
{/if}
<div class="grow"></div>
{#if editable && row.id !== ADMIN_ROLE}
<CloseButton small on:close={() => removeRole(row.id)} />
{/if}
</div>
<MultiSelect
items={tenantOptions}
bind:value={row.tenants}
disabled={!editable}
placeholder="Nobody yet — add a user, group or folder"
size="sm"
/>
</div>
{/each}
</div>
{#if editable && unusedRoles.length > 0}
<div class="flex items-center gap-2">
<Plus size={14} class="text-secondary" />
<Select
items={unusedRoles.map((r) => ({
value: r.id,
label: r.enabled ? r.name : `${r.name} (disabled)`
}))}
placeholder="Add a role"
bind:value={
() => undefined,
(id) => {
if (id) addRole(id)
}
}
class="w-64"
size="sm"
/>
</div>
{/if}
{/if}
{#if editable}
<div class="flex justify-end">
<Button unifiedSize="sm" variant="accent" loading={saving} on:click={save}>
Save roles
</Button>
</div>
{/if}
{#if info?.supported}
<div class="flex flex-col gap-3 border-t pt-4">
<div class="flex flex-col gap-0.5">
<span class="text-sm font-semibold text-emphasis">Access</span>
<span class="text-xs text-secondary">
What each role may do in Postgres, on the database, a schema or a table. Every
change shows the SQL it runs before running it.
</span>
</div>
<AclTargetPicker
schemas={aclSchemas}
schemasLoading={!aclSchemasLoaded}
tables={aclTables}
bind:schema={
() => aclSchema,
(s) => {
aclSchema = s
aclTables = []
}
}
bind:table={aclTable}
/>
{#key JSON.stringify(aclTarget)}
<PgAclEditor {workspace} {datatable} target={aclTarget} onLoaded={onAclLoaded} />
{/key}
</div>
<DataTable>
<Head>
<tr>
<Cell head first>
Role
<Tooltip>
admin is the connection the data table used before roles, so it owns every
existing object and cannot be removed. Every other role is a login defined for
the whole instance, with only the privileges granted to it under Access.
</Tooltip>
</Cell>
<Cell head>
Tenants
<Tooltip>
Users, groups and folders allowed to connect as this role. Workspace admins can
use every role.
</Tooltip>
</Cell>
<Cell head>
Default
<Tooltip>
The role a job gets when it names none — no `-- role` annotation, no `?role=` in
the reference. Callers still have to be one of its tenants.
</Tooltip>
</Cell>
<Cell head last />
</tr>
</Head>
<tbody class="divide-y bg-surface-tertiary">
{#each roles as role (roleKey(role))}
{@const isAdmin = role.id === ADMIN_DATATABLE_ROLE}
<Row>
<Cell first class="w-56 align-top">
<div class="flex flex-col gap-0.5 pt-1.5">
<span class="font-mono text-xs text-emphasis">{role.name ?? role.id}</span>
{#if !role.name}
<span class="text-2xs text-secondary italic">
no longer defined on this instance
</span>
{:else if role.id === undefined}
<Alert type="warning" title="This role does not exist yet" size="xs">
{#if $superadmin}
<div class="flex flex-col items-start gap-1">
<span>Create it on the instance to use it here.</span>
<Button
unifiedSize="xs"
variant="default"
on:click={() => instanceRoles?.open(role.name)}
>
Create it
</Button>
</div>
{:else}
Only a superadmin can create it on the instance.
{/if}
</Alert>
{/if}
</div>
</Cell>
<Cell class="align-top">
<MultiSelect
items={tenantItems}
bind:value={role.tenants}
groupBy={(item) => item.group}
disabled={!editable}
placeholder="Nobody — add users, groups or folders"
/>
</Cell>
<Cell class="w-20 align-top">
<div class="flex justify-center pt-2">
<Checkbox
checked={role.id !== undefined && defaultRoleId === role.id}
disabled={!editable || role.id === undefined}
title="Use this role when a job names none"
onChange={() => {
if (role.id !== undefined) defaultRoleId = role.id
}}
/>
</div>
</Cell>
<Cell last class="w-10 align-top">
{#if editable && !isAdmin}
<CloseButton small on:close={() => removeRole(role)} />
{/if}
</Cell>
</Row>
{/each}
{#if editable}
<Row class="!border-0">
<Cell colspan={4} class="pt-2 pb-2">
<div class="flex justify-center">
<Select
items={unusedRoles.map((r) => ({
value: r.id,
label: r.enabled ? r.name : `${r.name} (disabled)`
}))}
placeholder="+ Add a role"
bind:value={
() => undefined,
(id) => {
if (id) addRole(id)
}
}
onCreateItem={addRoleByName}
class="w-64"
/>
</div>
</Cell>
</Row>
{/if}
</tbody>
</DataTable>
{/if}
</div>
{#if info?.supported && !hasUnsavedChanges}
<div class="mt-6 pt-6 border-t">
<PgAclEditor {workspace} {datatable} target={{ kind: 'database' }} />
</div>
{/if}
{/if}
{#snippet actions()}
{#if editable}
<Button
variant="accent"
unifiedSize="md"
disabled={!hasUnsavedChanges || loading || !!loadError || pendingRoles.length > 0}
title={pendingRoles.length > 0
? 'Create the roles that do not exist yet, or remove them'
: undefined}
loading={saving}
on:click={save}
>
Save
</Button>
{/if}
{/snippet}
</DrawerContent>
</Drawer>
{#if $superadmin}
<InstanceRolesButton bind:this={instanceRoles} hideTrigger onChanged={refreshCatalog} />
{/if}
@@ -13,11 +13,22 @@
import { SettingService, type InstanceDatatableRole } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
let {
initialName = '',
onChanged
}: {
/** Prefills the name of the role to add. */
initialName?: string
/** Called after every change to the catalog, whether or not it went through. */
onChanged?: () => void
} = $props()
let roles = $state<InstanceDatatableRole[]>([])
let loading = $state(true)
let loadError = $state<string | undefined>(undefined)
let busy = $state(false)
let newName = $state('')
// svelte-ignore state_referenced_locally
let newName = $state(initialName)
/** Which role's name is being edited, and to what. */
let renaming = $state<{ id: string; name: string } | undefined>(undefined)
@@ -48,6 +59,7 @@
// holds, so a failed flip has to snap back rather than sit there claiming it landed.
await load()
busy = false
onChanged?.()
}
}
@@ -64,12 +64,11 @@
</script>
<script lang="ts">
import { Plus, PlugZap } from 'lucide-svelte'
import { History, KeyRound, Plus, PlugZap, Trash2 } from 'lucide-svelte'
import DropdownV2 from '../DropdownV2.svelte'
import Button from '../common/button/Button.svelte'
import CloseButton from '../common/CloseButton.svelte'
import ResourcePicker from '../ResourcePicker.svelte'
import SettingsPageHeader from '../settings/SettingsPageHeader.svelte'
import Select from '../select/Select.svelte'
@@ -103,7 +102,7 @@
import DataTablePermissionsButton from './DataTablePermissionsButton.svelte'
import InstanceRolesButton from './InstanceRolesButton.svelte'
import { deepEqual } from 'fast-equals'
import { clone } from '$lib/utils'
import { clone, onlyAlphaNumAndUnderscore } from '$lib/utils'
import SettingsFooter from './SettingsFooter.svelte'
import Alert from '../common/alert/Alert.svelte'
import MissingWorkerTagAlert from '../jobs/MissingWorkerTagAlert.svelte'
@@ -284,6 +283,9 @@
}
let confirmationModal = createAsyncConfirmationModal()
// Each mounts its own modal or drawer; the row menu opens them.
let migrationsButtons = $state<Record<string, DataTableMigrationsButton | undefined>>({})
let permissionsButtons = $state<Record<string, DataTablePermissionsButton | undefined>>({})
let dirtyMap = $derived.by(() => {
const map: Record<string, boolean> = {}
for (let i = 0; i < tempSettings.dataTables.length; i++) {
@@ -471,15 +473,17 @@
<Cell class="whitespace-nowrap">
<div class="flex gap-2">
<DataTableMigrationsButton
bind:this={migrationsButtons[dataTable.name]}
hideTrigger
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
{#if $enterpriseLicense}
{#if $enterpriseLicense && !isCloudHosted()}
<DataTablePermissionsButton
bind:this={permissionsButtons[dataTable.name]}
hideTrigger
workspace={$workspaceStore ?? ''}
datatable={dataTable.name}
disabled={!!dirtyMap[dataTable.name]}
/>
{/if}
<Button
@@ -515,9 +519,41 @@
</div>
</Cell>
<Cell class="w-12">
{#if !dataTable.reference}
<CloseButton small on:close={() => removeDataTable(dataTableIndex)} />
{/if}
<DropdownV2
items={() => [
{
displayName: 'Migrations',
icon: History,
// Both act on the saved data table, which unsaved edits are not.
disabled: !!dirtyMap[dataTable.name],
tooltip: dirtyMap[dataTable.name] ? 'Save the settings first' : undefined,
action: () => migrationsButtons[dataTable.name]?.open()
},
...($enterpriseLicense && !isCloudHosted()
? [
{
displayName: 'Roles',
icon: KeyRound,
disabled: !!dirtyMap[dataTable.name],
tooltip: dirtyMap[dataTable.name] ? 'Save the settings first' : undefined,
action: () => permissionsButtons[dataTable.name]?.open()
}
]
: []),
// A fork's pointer entry is written by forking and kept by the server, not this form.
...(dataTable.reference
? []
: [
{
displayName: 'Remove',
icon: Trash2,
type: 'delete' as const,
action: () => removeDataTable(dataTableIndex)
}
])
]}
btnId={'datatable-settings-actions-' + onlyAlphaNumAndUnderscore(dataTable.name)}
/>
</Cell>
</Row>
{/each}
@@ -1,33 +1,21 @@
<script module lang="ts">
export type ForkStep = {
label: string
status: 'pending' | 'running' | 'done' | 'error'
error?: string
}
export type DatatableCloneJob = {
name: string
resourceType: string
behavior: 'schema_only' | 'schema_and_data'
steps: ForkStep[]
_newDbName: string
_isInstance: boolean
_sourceWorkspace: string
_targetWorkspace: string
_resourcePath: string
}
</script>
<script lang="ts">
import { WorkspaceService } from '$lib/gen'
import { workspaceStore, userStore } from '$lib/stores'
import { workspaceStore, userStore, enterpriseLicense } from '$lib/stores'
import { isCloudHosted } from '$lib/cloud'
import { resource } from 'runed'
import Select from '../select/Select.svelte'
import Label from '../Label.svelte'
import Alert from '../common/alert/Alert.svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { Check, X, Loader2 } from 'lucide-svelte'
interface Props {
// Workspace whose datatables are cloned into the fork (the fork's base). Falls back to the
@@ -52,111 +40,46 @@
let cloneModalOpen = $state(false)
let currentCloneJob: DatatableCloneJob | undefined = $state(undefined)
let cloneQueue: DatatableCloneJob[] = $state([])
let cloneRunning = $state(false)
export function hasDatatables(): boolean {
return (allDatatables.current?.length ?? 0) > 0
}
export function buildCloneQueue(targetWorkspaceId: string): DatatableCloneJob[] {
// A fork attempt starts here: what an earlier attempt confirmed is not this one's to send.
confirmedJobs = []
return (allDatatables.current ?? [])
.filter((dt) => {
const behavior = datatableBehaviors[dt.name] ?? 'keep_original'
return behavior !== 'keep_original'
})
.map((dt) => {
const behavior = datatableBehaviors[dt.name] as 'schema_only' | 'schema_and_data'
const isInstance = dt.resource_type === 'instance'
const newDbName = `${targetWorkspaceId.replace(/-/g, '_')}__${dt.name}`
const steps: ForkStep[] = [
{
label: `CREATE DATABASE "${newDbName}"`,
status: 'pending'
},
{
label: `pg_dump → pg_import (${behavior === 'schema_only' ? 'schema only' : 'schema + data'})`,
status: 'pending'
}
]
return {
name: dt.name,
resourceType: dt.resource_type,
behavior,
steps,
_newDbName: newDbName,
_isInstance: isInstance,
_sourceWorkspace: effectiveSource!,
_targetWorkspace: targetWorkspaceId,
_resourcePath: dt.resource_path
}
})
.map((dt) => ({
name: dt.name,
resourceType: dt.resource_type,
behavior: datatableBehaviors[dt.name] as 'schema_only' | 'schema_and_data',
// A dev workspace's id has no `wm-fork-` prefix; its copies are named like a fork's
_newDbName: `wm_fork_${targetWorkspaceId.replace(/^wm-fork-/, '').replace(/-/g, '_')}__${dt.name}`
}))
}
let completedJobs: DatatableCloneJob[] = $state([])
let confirmedJobs: DatatableCloneJob[] = $state([])
// Each clone is only confirmed here: the fork request makes the copies, and drops them if the
// fork is not created.
export function startCloning(queue: DatatableCloneJob[]) {
completedJobs = []
confirmedJobs = []
cloneQueue = queue
currentCloneJob = cloneQueue[0]
cloneModalOpen = true
}
export function getCompletedCloneJobs(): DatatableCloneJob[] {
return completedJobs
}
async function executeCloneJob(job: DatatableCloneJob) {
cloneRunning = true
let stepIdx = 0
// Step 1: Create the database
job.steps[stepIdx].status = 'running'
try {
await WorkspaceService.createPgDatabase({
workspace: job._sourceWorkspace,
requestBody: {
source: `datatable://${job.name}`,
target_dbname: job._newDbName
}
})
job.steps[stepIdx].status = 'done'
} catch (e: any) {
job.steps[stepIdx].status = 'error'
job.steps[stepIdx].error = e?.body ?? e?.message ?? String(e)
cloneRunning = false
return
}
stepIdx++
// Step 2: Import data
job.steps[stepIdx].status = 'running'
try {
await WorkspaceService.importPgDatabase({
workspace: job._sourceWorkspace,
requestBody: {
source: `datatable://${job.name}`,
target: `datatable://${job.name}`,
target_dbname_override: job._newDbName,
fork_behavior: job.behavior
}
})
job.steps[stepIdx].status = 'done'
} catch (e: any) {
job.steps[stepIdx].status = 'error'
job.steps[stepIdx].error = e?.body ?? e?.message ?? String(e)
cloneRunning = false
return
}
stepIdx++
cloneRunning = false
export function getConfirmedCloneJobs(): DatatableCloneJob[] {
return confirmedJobs
}
function advanceCloneQueue() {
if (currentCloneJob) {
completedJobs.push(currentCloneJob)
confirmedJobs.push(currentCloneJob)
}
const idx = cloneQueue.indexOf(currentCloneJob!)
if (idx < cloneQueue.length - 1) {
@@ -189,11 +112,20 @@
(v) => (datatableBehaviors[dt.name] = v)
}
items={[
{ value: 'keep_original', label: 'Keep original' },
{ value: 'schema_only', label: 'Clone schema only' },
...(!isCloudHosted() && $userStore?.is_admin
? [{ value: 'schema_and_data', label: 'Clone schema and data' }]
: [])
{
value: 'keep_original',
label: dt.permissioned ? 'Keep original (under roles)' : 'Keep original'
},
// A copy of a data table under roles carries the source's owners and grants,
// which only the Enterprise Edition replays: elsewhere the server refuses it.
...(dt.permissioned && !$enterpriseLicense
? []
: [
{ value: 'schema_only', label: 'Clone schema only' },
...(!isCloudHosted() && $userStore?.is_admin
? [{ value: 'schema_and_data', label: 'Clone schema and data' }]
: [])
])
]}
/>
</div>
@@ -205,13 +137,9 @@
{#if cloneModalOpen && currentCloneJob}
<ConfirmationModal
title="Clone datatable: {currentCloneJob.name}"
confirmationText={cloneRunning ? 'Running...' : 'Start'}
confirmationText="Confirm"
open={cloneModalOpen}
loading={cloneRunning}
onConfirmed={async () => {
await executeCloneJob(currentCloneJob!)
advanceCloneQueue()
}}
onConfirmed={advanceCloneQueue}
onCanceled={() => {
cloneModalOpen = false
currentCloneJob = undefined
@@ -233,39 +161,16 @@
{#if currentCloneJob.resourceType === 'instance'}
<p class="text-xs text-secondary mt-2">
This will run <code
>CREATE DATABASE {currentCloneJob.steps[0]?.label.match(/"([^"]+)"/)?.[1] ?? ''}</code
> on the Windmill PostgreSQL instance.
Creating the fork will run <code>CREATE DATABASE {currentCloneJob._newDbName}</code> on the Windmill
PostgreSQL instance. A data table under roles keeps its owners and grants in the copy, and its
roles stay decided where they are decided today.
</p>
{:else}
<p class="text-xs text-secondary mt-2">
This will run <code>CREATE DATABASE</code> on the resource's PostgreSQL server.
Creating the fork will run <code>CREATE DATABASE {currentCloneJob._newDbName}</code> on the resource's
PostgreSQL server.
</p>
{/if}
<div class="mt-4 flex flex-col gap-2">
{#each currentCloneJob.steps as step}
<div class="flex items-center gap-2 text-xs">
{#if step.status === 'done'}
<Check class="w-4 h-4 shrink-0 text-green-500" />
{:else if step.status === 'running'}
<Loader2 class="w-4 h-4 shrink-0 animate-spin text-blue-500" />
{:else if step.status === 'error'}
<X class="w-4 h-4 shrink-0 text-red-500" />
{:else}
<div class="w-4 h-4 shrink-0 rounded-full border border-gray-300"></div>
{/if}
<span
class:text-tertiary={step.status === 'pending'}
class:font-medium={step.status === 'running'}
>
{step.label}
</span>
</div>
{#if step.error}
<p class="text-2xs text-red-500 ml-6">{step.error}</p>
{/if}
{/each}
</div>
<p class="text-xs text-secondary mt-2"> If the fork cannot be created, the copy is dropped. </p>
</ConfirmationModal>
{/if}
@@ -3,17 +3,34 @@
import { Users } from 'lucide-svelte'
import DataTableRolesSection from './DataTableRolesSection.svelte'
let {
hideTrigger = false,
onChanged
}: {
/** Mount the drawer without its button, for a caller that opens it with `open()`. */
hideTrigger?: boolean
/** Called after every change to the instance roles. */
onChanged?: () => void
} = $props()
let drawer: Drawer | undefined = $state(undefined)
let prefill = $state('')
// Remounts the section on each open, so the prefilled name is the one just asked for.
let openCount = $state(0)
/** Opens the drawer, with `name` prefilled as the role to add. */
export function open(name = '') {
prefill = name
openCount++
drawer?.openDrawer()
}
</script>
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: Users }}
on:click={() => drawer?.openDrawer()}
>
Instance roles
</Button>
{#if !hideTrigger}
<Button unifiedSize="sm" variant="default" startIcon={{ icon: Users }} on:click={() => open()}>
Instance roles
</Button>
{/if}
<Drawer bind:this={drawer} size="700px">
<DrawerContent
@@ -24,6 +41,8 @@
{#snippet titleExtra()}
<Badge color="blue" small>Beta</Badge>
{/snippet}
<DataTableRolesSection />
{#key openCount}
<DataTableRolesSection initialName={prefill} {onChanged} />
{/key}
</DrawerContent>
</Drawer>
@@ -8,12 +8,17 @@
import TextInput from '../text_input/TextInput.svelte'
import SimpleEditor from '../SimpleEditor.svelte'
import { WorkspaceService, type DatatableMigration } from '$lib/gen'
import { listUsableDatatableRoles } from '../datatableUsableRoles'
import { sendUserToast } from '$lib/toast'
import { tick } from 'svelte'
import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
import Portal from '$lib/components/Portal.svelte'
import { fetchPendingMigrations, outOfOrderRunMessage } from './datatableMigrationUtils'
import Select from '../select/Select.svelte'
import { parseMigrationRole, withMigrationRole } from '../datatableMigrationRole'
import { ADMIN_DATATABLE_ROLE } from '../dbTypes'
import { resource } from 'runed'
let {
workspace,
@@ -50,6 +55,8 @@
let tab = $state('up')
let name = $state('')
let nameInput = $state<TextInput>()
let upEditor = $state<SimpleEditor | undefined>()
let downEditor = $state<SimpleEditor | undefined>()
// A valid migration name is non-empty and limited to letters, digits, '_' and '-'.
const MIGRATION_NAME_RE = /^[a-zA-Z0-9_-]+$/
let nameInvalid = $derived(!MIGRATION_NAME_RE.test(name.trim()))
@@ -60,6 +67,96 @@
const confirmationModal = createAsyncConfirmationModal()
// The role lives in the SQL as its `-- role <name>` annotation, so the code is the single
// source of truth and the Select is a view onto it: reading parses, writing rewrites the
// annotation.
const usableRoles = resource(
() => [workspace, datatable] as const,
async ([ws, dt]) => {
try {
return await listUsableDatatableRoles(ws, dt)
} catch (e) {
console.error('Failed to load data table roles:', e)
return null
}
}
)
// Not a valid role name, so it cannot collide with one.
const NO_ROLE = '(no role)'
let declaredUp = $derived(parseMigrationRole(codeUp))
let declaredDown = $derived(enableDown ? parseMigrationRole(codeDown) : undefined)
let malformedLine = $derived(
declaredUp.kind === 'malformed'
? declaredUp.line
: declaredDown?.kind === 'malformed'
? declaredDown.line
: undefined
)
const roleOf = (d: typeof declaredUp) => (d.kind === 'role' ? d.role : undefined)
// A rollback runs as the role its own SQL names: under another role than the up migration it
// typically cannot touch what the up created.
let sqlProblem = $derived(
malformedLine !== undefined
? malformedMessage(malformedLine)
: declaredDown !== undefined && roleOf(declaredDown) !== roleOf(declaredUp)
? `The down migration runs as ${roleOf(declaredDown) ?? 'admin (no role)'} but the up migration as ${roleOf(declaredUp) ?? 'admin (no role)'}: make their role annotations match`
: undefined
)
let selectedRole = $derived(declaredUp.kind === 'role' ? declaredUp.role : NO_ROLE)
let permissioned = $derived(!!usableRoles.current?.permissioned)
// No annotation runs as admin, which the server allows exactly to those who may use `admin`.
let adminUsable = $derived(!!usableRoles.current?.roles.includes(ADMIN_DATATABLE_ROLE))
let roleItems = $derived.by(() => {
const usable = usableRoles.current
if (!usable?.permissioned) return []
const names = usable.roles.filter((r) => r !== ADMIN_DATATABLE_ROLE)
// A role the SQL names but the caller cannot use is still shown, or the picker would
// misreport what the migration runs as.
if (declaredUp.kind === 'role' && !names.includes(declaredUp.role)) {
names.push(declaredUp.role)
}
const items = names.map((r) => ({
value: r,
label: r === usable.default_role ? `${r} (default)` : r
}))
if (adminUsable || declaredUp.kind === 'none') {
items.push({
value: NO_ROLE,
label: 'No role — runs as admin with full access'
})
}
return items
})
function setRole(value: string | undefined) {
const role = value === NO_ROLE ? undefined : value
codeUp = withMigrationRole(codeUp, role)
// Up and down agree: a rollback run as another role could fail on objects it does not own.
if (enableDown) codeDown = withMigrationRole(codeDown, role)
// Assigning the bound value does not repaint the editor, and its next keystroke would write
// the stale text back.
upEditor?.setCode(codeUp)
if (enableDown) downEditor?.setCode(codeDown)
}
// Set by `open` when the SQL names no role yet: the data table's default is written once its
// roles are known.
let applyDefaultRole = $state(false)
$effect(() => {
// `undefined` until the first answer lands; `null` when it failed.
const usable = usableRoles.current
if (!applyDefaultRole || !isOpen || usableRoles.loading || usable === undefined) return
applyDefaultRole = false
if (!usable?.permissioned || declaredUp.kind !== 'none') return
const role =
usable.default_role !== ADMIN_DATATABLE_ROLE && usable.roles.includes(usable.default_role)
? usable.default_role
: usable.default_role === ADMIN_DATATABLE_ROLE && adminUsable
? undefined
: usable.roles.find((r) => r !== ADMIN_DATATABLE_ROLE)
if (role !== undefined) setRole(role)
})
// Frame the migration body in an explicit transaction so it applies atomically.
function wrapInTransaction(body: string): string {
return `BEGIN;\n\n${body}\n\nEND;`
@@ -72,15 +169,35 @@
}
const PLACEHOLDER = wrapInTransaction('-- Add your migration here')
function malformedMessage(line: string): string {
return `Malformed role annotation \`${line}\`: write it as \`-- role <name>\`, or pick the role above`
}
export function open(prefill?: { name?: string; codeUp?: string; codeDown?: string }) {
// Roles and the default can have changed since the last open (the roles drawer sits next
// to this modal), and the default role is written from this answer.
usableRoles.refetch()
name = prefill?.name ?? ''
// Start from the transaction template; when prefilled from detected DDL,
// wrap that DDL in the same BEGIN; ... END; frame.
codeUp = prefill?.codeUp
? wrapInTransaction(ensureTrailingSemicolon(prefill.codeUp))
: PLACEHOLDER
// wrap that DDL in the same BEGIN; ... END; frame. A role the prefill declares is taken
// out first and put back on top: below `BEGIN;` it would not be read.
const prefillRole = prefill?.codeUp ? parseMigrationRole(prefill.codeUp) : undefined
if (prefill?.codeUp) {
const wrapped = wrapInTransaction(
ensureTrailingSemicolon(withMigrationRole(prefill.codeUp, undefined))
)
codeUp =
prefillRole?.kind === 'role'
? withMigrationRole(wrapped, prefillRole.role)
: prefillRole?.kind === 'malformed'
? `${prefillRole.line}\n${wrapped}`
: wrapped
} else {
codeUp = PLACEHOLDER
}
codeDown = prefill?.codeDown ?? PLACEHOLDER
enableDown = (prefill?.codeDown ?? '') !== ''
applyDefaultRole = prefillRole === undefined || prefillRole.kind === 'none'
tab = 'up'
isOpen = true
// Focus the name field once the modal content has rendered.
@@ -96,6 +213,10 @@
sendUserToast("Invalid migration name: use only letters, digits, '_' and '-'", true)
return
}
if (sqlProblem !== undefined) {
sendUserToast(sqlProblem, true)
return
}
if (run) {
// A new migration gets the highest timestamp, so any still-pending
// migration is earlier: running only this one applies it out of order.
@@ -176,29 +297,65 @@
closeOnOutsideClick={false}
>
<div class="flex flex-col gap-3 w-full grow min-h-0">
<TextInput
bind:this={nameInput}
bind:value={name}
error={nameInvalid}
inputProps={{ placeholder: 'Migration name (e.g. add_index_to_customers)' }}
/>
<div class="flex gap-2 items-center">
<TextInput
bind:this={nameInput}
bind:value={name}
error={nameInvalid}
class="grow"
inputProps={{ placeholder: 'Migration name (e.g. add_index_to_customers)' }}
/>
{#if permissioned}
<Select
transformInputSelectedText={(s) => `Role: ${s}`}
items={roleItems}
bind:value={() => selectedRole, (r) => setRole(r)}
placeholder="Role"
class="w-72"
/>
{/if}
</div>
{#if sqlProblem !== undefined}
<p class="text-xs text-red-500">{sqlProblem}</p>
{:else if permissioned && usableRoles.current?.roles.length === 0}
<p class="text-xs text-secondary">
You can't use any role of this data table, so a migration you create can't be run.
</p>
{/if}
<Tabs bind:selected={tab} class="grow min-h-0">
<Tab value="up" label="Up" />
<Tab value="down" label="Down" />
{#snippet content()}
<TabContent value="up" class="h-80 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeUp} />
<SimpleEditor bind:this={upEditor} class="h-full" lang="sql" bind:code={codeUp} />
</TabContent>
<TabContent value="down" class="h-80">
<div class="flex flex-col gap-2 h-full">
<Toggle
bind:checked={enableDown}
bind:checked={
() => enableDown,
(checked) => {
enableDown = checked
// The down editor is created by this toggle, so it reads the rewritten text.
if (checked) {
codeDown = withMigrationRole(
codeDown,
declaredUp.kind === 'role' ? declaredUp.role : undefined
)
}
}
}
options={{ right: 'Enable down migration' }}
size="sm"
/>
{#if enableDown}
<div class="grow min-h-0 border rounded-md overflow-hidden">
<SimpleEditor class="h-full" lang="sql" bind:code={codeDown} />
<SimpleEditor
bind:this={downEditor}
class="h-full"
lang="sql"
bind:code={codeDown}
/>
</div>
{/if}
</div>
@@ -209,7 +366,7 @@
<Button
variant="accent"
size="sm"
disabled={creating}
disabled={creating || sqlProblem !== undefined}
on:click={() => create(true)}
dropdownItems={[
{
@@ -535,7 +535,8 @@
aiChatManager.datatableCreationPolicy = {
enabled: !!result.data.datatable,
datatable: result.data.datatable,
schema: result.data.schema
schema: result.data.schema,
roles: result.data.roles
}
if (withPrompt && result.prompt) {
const prompt = result.prompt