Commit Graph
14906 Commits
Author SHA1 Message Date
Diego Imbert a296ca016a Merge commit '7c16d482fd' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:34:50 +02:00
Diego ImbertandClaude Opus 5 7c16d482fd fix(datatables): refuse fork cleanup of an external database another workspace uses
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:34:14 +02:00
Diego ImbertandClaude Opus 5 5eec203099 fix(datatables): keep certificate verification when DuckDB attaches an external data table
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:17:15 +02:00
Diego ImbertandClaude Opus 5 53d62dd399 fix(datatables): register external fork catalogs under the lifecycle lock, and keep certificate verification in DuckDB attaches
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:07:40 +02:00
Diego Imbert e17b76c255 Merge commit '5ddaee5b78' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:07:40 +02:00
Diego ImbertandClaude Opus 5 5ddaee5b78 fix(datatables): write the external cluster setting under the lifecycle lock, and check fork targets are registered
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:04:18 +02:00
Diego ImbertandClaude Opus 5 b62d8b246a fix(datatables): protect external databases pending fork cleanup, and describe Ducklake usage in the API
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:30:25 +02:00
Diego ImbertandClaude Opus 5 3981b2eb1b fix(datatables): refuse repointing the external cluster while it is in use, and keep verify-ca working for pg_dump
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:28:31 +02:00
Diego Imbert 5bc2f13565 Merge commit '3981b2eb1b3bf77cdc1a5cbf43090e6b9706aa1f' into datatable-external-instance-3 2026-09-17 15:28:31 +02:00
Diego Imbert 0c53990d18 Merge commit '9b0046d8671697e6f6568c89170052367139d157' into datatable-external-instance-3
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-common/src/external_instance_pg.rs
2026-09-17 11:50:21 +02:00
Diego ImbertandClaude Opus 5 9b0046d867 docs(datatables): state the authorization contract of external database usage lookups
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 11:50:10 +02:00
Diego Imbert 1f78ce77f5 Merge commit 'dbe0230f483bbe10aeca055c4d39506fea41459d' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 11:50:00 +02:00
Diego ImbertandClaude Opus 5 dbe0230f48 fix(datatables): never grant CREATEROLE to custom_instance_user on the external cluster
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 11:49:20 +02:00
Diego ImbertandClaude Opus 5 67f488c676 feat(datatables): Ducklake catalogs on the external instance cluster
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 11:47:25 +02:00
Diego Imbert ced4913b33 Merge commit 'b3a3f1cf5a0e6f97af2679c8abf640c6b832d7db' into datatable-external-instance-2
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 11:34:32 +02:00
Diego Imbert b3a3f1cf5a Merge remote-tracking branch 'origin/datatable-roles-redesign-part-4' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 11:34:28 +02:00
Diego Imbert 92e51f19aa Merge remote-tracking branch 'origin/datatable-roles-redesign-part-2' into datatable-roles-redesign-part-4
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-api-integration-tests/tests/datatable_roles.rs
#	backend/windmill-api-workspaces/src/datatable_permissions_oss.rs
#	backend/windmill-api-workspaces/src/workspaces.rs
#	backend/windmill-common/src/datatable_roles_oss.rs
#	backend/windmill-common/src/workspaces.rs
#	frontend/src/lib/components/datatableAcl/PgAclEditor.svelte
#	frontend/src/lib/components/workspaceSettings/DataTablePermissionsButton.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableRolesSection.svelte
#	frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte
#	frontend/src/lib/components/workspaceSettings/InstanceRolesButton.svelte
2026-09-17 10:03:05 +02:00
Diego ImbertandClaude Opus 5 36532c35d8 fix: check ACL read reach against the entry it connects from
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 79481a2f02 fix: build the ACL connection from the authorized data table entry
Resolving the settings again could land on a resource with the same
database name on another server, which the later entry checks never see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 c884597e56 fix: hold the ACL connection to the database that was authorized
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 115448d3d1 fix: run one data table ACL apply at a time per server before it connects
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 1ece278fd2 fix: add only missing grant options before an ACL apply, never default privileges
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 7819d48df9 fix: refresh grant options only after the ACL apply validates its plan
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 ce1f720d85 fix: take every pooled connection before the ACL apply locks
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRoYE5ZeAVvrDYdfhDAYXb
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 93d346e54b feat(datatables): add an ACL editor for data table roles
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:19 +02:00
Diego ImbertandClaude Opus 5 9378828472 docs(datatables): note that legacy names with ? cannot be migrated
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 79bb761c5d fix(datatables): serialize roles going on with aliases saved from other workspaces
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 d191cfe4eb fix(datatables): let CE migrations connect as an explicitly named admin
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 31b5b0a062 fix(datatables): refuse roles while a same-workspace alias reaches the database
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
windmill-internal-app[bot] d1c2f801f4 chore: update ee-repo-ref to a1873dbb67f2302b85ff5362f8387b48eccdb607
This commit updates the EE repository reference after PR #783 was merged in windmill-ee-private.

Previous ee-repo-ref: 5c853e2c20eca6b748415fc0d6862a6ebfb5fec4

New ee-repo-ref: a1873dbb67f2302b85ff5362f8387b48eccdb607

Automated by sync-ee-ref workflow.
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 d3ee67c9bc test(datatables): run the tenant matcher test on the enterprise edition
The matcher it covers is enterprise code now, so without the enterprise
edition the test hit the stub and failed the default windmill-common run. It
runs with private and enterprise, and a counterpart without them asserts that
no tenant list covers anyone, the wildcard and a workspace admin included.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 eafc551651 feat(datatables): gate the roles UI mount sites on an enterprise license
Both mount sites are still commented out; the gate travels with them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 db05c8b7c9 test(datatables): run the roles tests on the enterprise edition, refusals without it
Each test that exercises roles runs with private and enterprise. Two tests run
without them: every roles route answers the Enterprise refusal, and a data
table saved under roles, or a named role, is refused a connection while one
not under roles resolves as before.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 11ba3cd477 feat(datatables): move the data table role catalog endpoints to the enterprise edition
The superadmin list, create, update and delete handlers move to
windmill-ee-private. The routes stay registered and, without the enterprise
edition, refuse after authentication.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 7ae3335f03 feat(datatables): move the data table permissions endpoints to the enterprise edition
The permissions read, save and usable-roles handlers move to
windmill-ee-private; the routes stay registered and, without the enterprise
edition, answer that data table roles are an Enterprise Edition feature.
ensure_governs_datatable and ensure_reaches_datatable keep their paths: the
first refuses, the second passes a data table not under roles.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 c7269e975f feat(datatables): move data table role catalog and resolution to the enterprise edition
Roles are an Enterprise Edition feature. The catalog, the Postgres logins,
CONNECT convergence, tenant evaluation and the role half of connection
resolution move to windmill-ee-private. Every public function keeps its path
and signature and forwards through datatable_roles_oss, which re-exports the
enterprise implementation or, without it, refuses.

Without the enterprise edition a data table under roles, or a caller naming a
role, is refused a connection rather than resolved as admin, and the reach and
admin-access checks refuse one under roles. A data table not under roles
resolves as before in every edition, and an instance database keeps the
CONNECT grants it was created with. The catalog lock, the stream lock, the
tenant cascades and the permissions stripping stay in OSS: they only restrict.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 8a0babd7a6 fix(datatables): no entry without roles may newly reach a database under roles
The previous guard only caught a new name replacing an entry under roles.
A whole-map save could also repoint an existing entry without roles at
that database, or another workspace could point one there, and every
caller of that entry would connect as admin. The rule is now stated on
the saved entries: one that carries no roles and newly points at an
instance database any entry under roles uses, in this workspace or
another, is refused. A declared rename carries its roles and passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 6d9ef5f6b7 fix(datatables): refuse a save that drops a data table's roles through an undeclared rename
A data table's roles follow its entry only through a declared rename. A
settings sync sends the whole map and never declares one, so renaming a
data table under roles there read as a delete and a new entry on the same
database: the new entry carried no roles, and every caller connected as
admin. Such a save is now refused, naming both entries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 1dbc1e7759 fix(datatables): warn when a settings sync strands fork pointers
A settings save reported the fork pointers left resolving to nothing only
for the names in `deleted_datatables`, which `wmill sync push` never sends.
The save now works out what it removed from the locked entries, and the
CLI prints the stranded pointers it returns.

Also correct the replication helper's contract: no role or admin check
makes a replication connection safe, so a data table under roles is
refused outright rather than gated as an admin operation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 df471e37d8 fix(datatables): wait out live listeners, and resolve stored names containing ?
Turning roles on counted a trigger as gone once disabled, and a capture
once its client stopped pinging, but the listener keeps its replication
connection until its next heartbeat notices. A trigger or capture whose
listener pinged in the last 15 seconds, the window a server holds a
listener for, now still counts as streaming.

Data table names could contain `?` before they were restricted, and such
entries are still stored. Splitting `?role=` off a reference misread them:
`a?b` became `a` with an unknown parameter, and the clone checks looked at
a different entry than the one copied. An entry stored under the whole
reference is now looked up first, in the Postgres executor, DuckDB ATTACH
and the clone checks. Agent workers cannot read the workspace and keep
the strict parse, which refuses such a name rather than misreading it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 122a3bd164 fix(datatables): serialize roles going on with a stream starting
Turning roles on looked for enabled triggers and live captures once,
without a lock anything starting a stream also took. A trigger enabled in
that window could have its listener connect before roles committed, and a
healthy listener never checks again. Both transitions now serialize on one
advisory lock: roles going on hold it exclusive while they look, and
trigger create, edit and enable, and capture setup and ping hold it shared
while they commit. Either the look sees the stream, or the listener
connects after roles are committed and refuses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 2f95c22242 fix(datatables): disable a data table role before deleting it
Deleting a role reassigns and drops what it owns in each registered
database on its own connection, and each of those passes commits as it
goes. A database failing part-way left the role enabled in the catalog and
able to log in, but already stripped in the databases reached before it.
The role is now disabled in its own commit first, so a failed delete
leaves a disabled role to retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 7279ee15e2 fix(datatables): refuse a Postgres trigger on a data table under roles when it is saved
Creating or editing a trigger that points at a data table under roles was
accepted, and its listener then retried the refused connection every 30
seconds forever. The save is now refused, and a trigger that reaches such a
data table anyway (re-enabled, or cloned into a fork) is disabled by its
listener with the reason, as a missing replication slot is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 fa1e8bd3aa refactor(datatables): keep Postgres triggers and data table roles apart
A replication stream reads every row of every table whatever the data table's
roles grant, and its listener checks access only when it connects. Rather than
chase every way access can change and bounce the streams each one affects, a
data table now carries one or the other:

- a Postgres trigger or capture cannot be created on, or connect to, a data
  table under roles;
- roles cannot be turned on while an enabled trigger or a live capture reads
  the data table, its own or a fork's through its pointer. The refusal names
  each one to disable.

This removes the stream bounces on roles edits and on data table and workspace
deletion, and the trigger gate that admitted admins. The fork schema baseline
fix from the same review round is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 dc3ebfbe74 fix(datatables): keep the fork schema baseline, and bounce streams on every removal
Three fixes from review.

`edit_datatable_config` took `forked_from` wholesale from the stored entry, so
the fork schema diff's save of an advanced baseline was silently discarded and
an applied change was offered again. Whether an entry carries a clone stamp is
still carried from the store, since that is what marks its database droppable,
but the baseline inside it is now taken from the request.

The stranded-pointer warning and the stream bounce ran over the optional
`deleted_datatables` hint, which the settings-sync CLI never sends, so removing
a governing data table through `wmill` bounced nothing. Removals are now derived
from the stored configuration against the saved one.

`delete_workspace` read the pointers to bounce before its transaction, so a fork
committing a pointer during the deletion was missed. The read now happens inside
the transaction, after the workspace row is deleted: a fork's insert key-share
locks that row through its parent foreign key, so it is either seen or fails on
the missing parent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 8255f2ad2b fix(datatables): bounce the streams reading a data table when it is deleted
Deleting a governing data table, or the workspace that holds it, only collected
the fork pointers it stranded, for the warning. A Postgres trigger or capture
already streaming through one of those pointers kept the replication connection
it opened while the pointer still resolved, so it went on dispatching the
governing database's rows after the fork lost access — until its connection
happened to restart. The governing workspace's own streams on a deleted entry
did the same.

Both deletion paths now bounce the affected listeners inside their own
transaction, through the helper a permission change already uses, so a
listener that reconnects re-resolves the entry and finds it gone. The helper is
split so a caller can pass the (workspace, local name) pairs it already holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 ebe076cbe6 docs(datatables): record the stale clone database as a known limitation
A clone is three requests and `CREATE DATABASE` is not transactional, so a
failure after the first leaves a registered `wm_fork_*` behind, as it did
before data table roles. Accepted for this PR: it is harmless to data and goes
away once the clone is a single server-side operation.

The comment also records why the obvious fix is wrong: reclaiming the leftover
on retry, without durable clone ownership, can drop another workspace's fully
copied database between its import and its final fork request.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 643bc1b51a Revert "fix(datatables): let a retried clone reclaim its own leftover database"
This reverts commit 7dd3275a10.

The reclaim tied the caller to the source they administer, but not to the
database it dropped. Between another workspace's import and its final fork
request, that workspace's target is full, registered, unnamed and has no open
connection, so an admin of any instance data table could name it and have it
dropped and recreated empty. The victim's fork would then commit pointing at
the empty copy. Safe reclaim needs durable clone ownership and serialization
with the request that names the database; until then the leftover stays, as it
did before this PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 29fccb0978 fix(datatables): let a retried clone reclaim its own leftover database
A clone creates its target database one request before it copies into it, and
the fork that would name it is written a request after that. Any failure in
between — a pg_dump error, a bad restore, a dropped connection, the source's
roles changing mid-flow — left a registered `wm_fork_*` that no entry names,
and every retry then failed on its name. This predates data table roles.

`create_pg_database` now reclaims such a leftover before creating: only a
`wm_fork_*` database Windmill registered as a data table database and that no
data table or ducklake entry names, in any workspace, archived ones included.
The drop never terminates connections, so a clone still copying into it makes
the reclaim fail instead of being cut off. It is limited to callers who
administer the source — reaching it is not enough, since on a data table
without roles every member reaches it — and anyone else gets the refusal an
existing database always got.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 10:01:15 +02:00
Diego ImbertandClaude Opus 5 432ccc0fbd refactor(datatables): settle clonability in one place, before anything is created
A clone is three stages a workspace apart — `create_pg_database`, then
`import_pg_database`, then `apply_forked_datatable` inside the fork transaction.
Only the third can roll back, and `CREATE DATABASE` is not transactional, so any
refusal that lives there strands a registered `wm_fork_*` that no entry names
and whose name blocks the retry.

That orphan has now been fixed three times, most recently reintroduced by a
guard added one commit ago. Patching each new refusal into the first endpoint is
not the fix; having two places that can refuse is. `ensure_datatable_is_clonable`
now answers every reason a copy can be refused and returns what it resolved, and
the stage that writes the entry only does the work.

Also takes an ACCESS EXCLUSIVE lock before the rollback guard counts, so a role
created concurrently cannot slip between the check and the drop.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BjfMkJyKzodxkobqGZ6Lqb
2026-09-17 10:01:15 +02:00