Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

AGENTS.md

@@ -29,10 +29,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Agent workers**: `docs/agent-worker-e2e.md` — building and running one locally. An agent
   reaches the DB only through the API, so `Connection::Http` paths are never taken by a plain
   `cargo run`; a normal build cannot start one at all.
-- **External instance data tables**: `docs/external-instance-datatables.md` — the cluster Windmill
-  administers behind `external_instance` data tables and Ducklake catalogs: its invariants (one
-  lifecycle lock, managed-object markers, the setup gate, per-cluster roles, fork copy ownership)
-  and how to run one locally
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Auth surface**: `docs/auth-surface.md` — credential precedence, session/cache invalidation
   scope, which token labels email their owner at expiry, how OAuth login matches `login_type`, and

docs/external-instance-datatables.md

@@ -1,88 +0,0 @@
-# External instance data tables
-
-A data table is backed by one of three things: a Postgres resource a workspace brings
-(`postgresql`), a database on Windmill's own cluster (`instance`), or a database on a separate
-cluster Windmill administers (`external_instance`, Enterprise Edition). The third is what this
-document covers; Ducklake catalogs take the same three shapes.
-
-Windmill administers the external cluster the way it administers its own: it creates and drops
-databases there, owns `custom_instance_user` and `custom_instance_replication_user`, and creates
-the data table roles of that cluster. It logs in as the admin in the `external_instance_pg`
-instance setting, and keeps what it generates in the hidden `external_instance_pg_state` setting.
-
-## Code
-
-| Where | What |
-|---|---|
-| `windmill-common/src/external_instance_pg.rs` | Setting, state, usage accounting, the lifecycle lock, the OSS forwarders |
-| `windmill-common/src/external_instance_pg_ee.rs` | Setup, database create and drop, the admin connection |
-| `windmill-common/src/datatable_roles.rs` | Per-cluster role catalogs (`DatatableRoleCluster`) |
-| `windmill-common/src/workspaces.rs` | Resolution (`resolve_datatable_connection_unchecked`), `managed_database_uses` |
-| `windmill-api-settings/src/lib.rs` | `/settings/external_instance_pg/*`, `/settings/datatable_roles` |
-
-## What holds it together
-
-- **One lifecycle lock.** `lock_external_instance_pg_state` serializes everything that changes
-  which databases exist on the cluster or which entries name them: setup, create, drop, data table
-  and Ducklake saves, external role DDL, and writes to the setting itself. Anything reading the
-  configuration to reach the cluster reads it under that lock, so a database is never created on
-  one cluster and registered while the setting names another.
-- **Windmill only touches what it made.** Databases it creates carry a comment, and a drop
-  requires it. The two managed roles and every data table role carry their own comment, and setup
-  refuses a `custom_instance_user` without it rather than resetting the password of someone else's
-  role.
-- **Creation needs a successful setup.** `set_up_for` records the `host:port` the last successful
-  setup converged. Creating a database on a cluster that setup has not succeeded on is refused.
-- **Nothing is dropped from under a user.** `managed_database_uses` lists every data table naming
-  a database, every fork pointing at those, every Ducklake catalog on it, and every fork Ducklake
-  metadata schema still to be dropped. Fork cleanup exempts exactly the entry it is cleaning up.
-- **Fork copies belong to a workspace.** `wm_fork_*` is a name, not an authorization: every
-  database of a cluster answers to the same `custom_instance_user`. The registry records the
-  workspace a copy was created for, and a member can only import into or fork onto a copy of their
-  own workspace.
-- **Roles are per cluster.** `datatable_role.cluster` splits the catalog, so the same role name can
-  exist on both clusters. Role names are unique per cluster, as they are in Postgres.
-
-## Running one locally
-
-```bash
-docker run -d --name wm-external-pg -e POSTGRES_PASSWORD=external -p 5497:5432 postgres:18 \
-  -c wal_level=logical
-psql "postgresql://postgres:external@127.0.0.1:5497/postgres" \
-  -c "CREATE ROLE wm_admin LOGIN PASSWORD 'adminpw' CREATEDB CREATEROLE REPLICATION"
-```
-
-A non-superuser admin with `CREATEDB` and `CREATEROLE` is the realistic case: managed Postgres
-gives nothing more. `REPLICATION` is only needed for Postgres triggers on external data tables.
-
-Then, as superadmin (`$T` is a token):
-
-```bash
-api=http://localhost:8000/api
-curl -s -X POST $api/settings/global/external_instance_pg -H "Authorization: Bearer $T" \
-  -H 'Content-Type: application/json' \
-  --data '{"value":{"host":"127.0.0.1","port":5497,"user":"wm_admin","password":"adminpw","sslmode":"disable"}}'
-curl -s -X POST $api/settings/external_instance_pg/setup -H "Authorization: Bearer $T" \
-  -H 'Content-Type: application/json' --data '{}'          # report per step
-curl -s -X POST $api/settings/external_instance_pg/databases/dt_demo -H "Authorization: Bearer $T" \
-  -H 'Content-Type: application/json' --data '{}'
-curl -s -X POST $api/w/admins/workspaces/edit_datatable_config -H "Authorization: Bearer $T" \
-  -H 'Content-Type: application/json' \
-  --data '{"settings":{"datatables":{"demo":{"database":{"resource_type":"external_instance","resource_path":"dt_demo"}}}}}'
-```
-
-`sslmode` defaults to `verify-full`; `disable` is for a local container only. With `verify-full`
-against a server with a private CA, put the CA in `root_certificate_pem` — `pg_dump`, `psql` and
-DuckDB attaches all verify against the system trust store plus that certificate.
-
-Jobs then reach it as any data table: `ATTACH 'datatable://demo' AS d` from DuckDB, or
-`datatable://demo` as the database of a PostgreSQL script, with `-- role <name>` to connect as a
-data table role of that cluster.
-
-Worth knowing while testing:
-
-- A worker needs the `postgresql` and `duckdb` tags for those jobs
-  (`update config set config = jsonb_set(config, '{worker_tags}', …) where name = 'worker__default'`).
-- DuckDB jobs load `libwindmill_duckdb_ffi_internal.so` by name, so a binary built into its own
-  `CARGO_TARGET_DIR` needs that library on `LD_LIBRARY_PATH`.
-- Setup holds the lifecycle lock for its whole run, so a settings save during it waits.

frontend/src/lib/components/InstanceSetting.svelte

@@ -27,6 +27,7 @@
 	import WebhookBaseUrlSetting from './instanceSettings/WebhookBaseUrlSetting.svelte'
 	import WsConnectivityTest from './instanceSettings/WsConnectivityTest.svelte'
 	import InstanceBannerSetting from './instanceSettings/InstanceBannerSetting.svelte'
+	import ExternalInstancePgSettings from './instanceSettings/ExternalInstancePgSettings.svelte'
 	import IndexerMemorySettings from './instanceSettings/IndexerMemorySettings.svelte'
 	import IndexerJobIndexSettings from './instanceSettings/IndexerJobIndexSettings.svelte'
 	import IndexerLogIndexSettings from './instanceSettings/IndexerLogIndexSettings.svelte'
@@ -781,10 +782,10 @@
 								/>
 								<p class="text-xs text-tertiary">
 									Comma-separated host/IP patterns the proxy still traces but for which it skips
-									upstream TLS certificate verification. Use for internal endpoints with
-									self-signed or otherwise untrusted certificates — unlike NO_PROXY above, these
-									requests stay traced. Same matching as NO_PROXY (<code>example.com</code> matches
-									subdomains; <code>.example.com</code> matches subdomains only).
+									upstream TLS certificate verification. Use for internal endpoints with self-signed
+									or otherwise untrusted certificates — unlike NO_PROXY above, these requests stay
+									traced. Same matching as NO_PROXY (<code>example.com</code> matches subdomains;
+									<code>.example.com</code> matches subdomains only).
 								</p>
 							</div>
 							<div class="flex flex-col gap-1">
@@ -872,6 +873,8 @@
 					<WsConnectivityTest {values} />
 				{:else if setting.fieldType == 'instance_banner'}
 					<InstanceBannerSetting {values} disabled={loading} />
+				{:else if setting.fieldType == 'external_instance_pg'}
+					<ExternalInstancePgSettings {values} disabled={loading} />
 				{/if}
 				{#if hasError}
 					<span class="text-red-600 dark:text-red-400 text-xs">

frontend/src/lib/components/InstanceSettings.svelte

@@ -730,6 +730,7 @@
 		secret_backend: ['token'],
 		object_store_cache_config: ['secret_key', 'serviceAccountKey'],
 		custom_instance_pg_databases: ['user_pwd'],
+		external_instance_pg: ['password'],
 		rsa_keys: ['private_key'],
 		github_enterprise_app: ['private_key']
 	}
@@ -1234,6 +1235,11 @@
 				description="Configure a self-managed GitHub App for git sync on GitHub.com, GHE Cloud or GitHub Enterprise Server."
 				link="https://www.windmill.dev/docs/integrations/git_repository#self-managed-github-app"
 			/>
+		{:else if category == 'External Postgres'}
+			<SettingsPageHeader
+				title="External Postgres"
+				description="Store data tables and Ducklake catalogs on a PostgreSQL cluster outside Windmill's own database. Save the connection, set the cluster up, then create databases for workspaces to use."
+			/>
 		{:else if category == 'DB Health'}
 			<SettingsPageHeader
 				title="DB Health"

frontend/src/lib/components/SaveButton.svelte

@@ -49,7 +49,11 @@
 			})
 			.catch((error) => {
 				console.error('Save failed:', error)
-				sendUserToast(error?.message ?? 'Save failed', true)
+				// An ApiError's message is only the HTTP status text; the server's reason is its body.
+				sendUserToast(
+					(typeof error?.body === 'string' && error.body) || error?.message || 'Save failed',
+					true
+				)
 				saveStatus = 'error'
 				statusTimeout = setTimeout(() => {
 					saveStatus = null

frontend/src/lib/components/instanceSettings.ts

@@ -71,6 +71,7 @@ export interface Setting {
 		| 'ws_connectivity'
 		| 'retention_overrides'
 		| 'instance_banner'
+		| 'external_instance_pg'
 	storage: SettingStorage
 	advancedToggle?: {
 		label: string
@@ -718,6 +719,17 @@ export const settings: Record<string, Setting[]> = {
 		}
 	],
 	'DB Health': [],
+	'External Postgres': [
+		{
+			label: 'External instance cluster',
+			description:
+				'A PostgreSQL cluster Windmill administers for data tables and Ducklake catalogs of the External instance type. The admin user needs CREATEDB and CREATEROLE, plus REPLICATION for Postgres triggers. Windmill creates its own roles and databases there and leaves everything else on the cluster alone.',
+			key: 'external_instance_pg',
+			fieldType: 'external_instance_pg',
+			storage: 'setting',
+			ee_only: 'External instance databases are an Enterprise Edition feature'
+		}
+	],
 	Registries: [
 		{
 			label: 'Instance Python Version',
@@ -1241,6 +1253,14 @@ export const instanceSettingsNavigationGroups = [
 				aiId: 'instance-settings-object-storage',
 				aiDescription: 'Instance object storage settings',
 				isEE: true
+			},
+			{
+				id: 'external_postgres',
+				label: 'External Postgres',
+				aiId: 'instance-settings-external-postgres',
+				aiDescription:
+					'External PostgreSQL cluster Windmill manages for external instance data tables and Ducklake catalogs',
+				isEE: true
 			}
 		]
 	},
@@ -1365,6 +1385,7 @@ export const tabToCategoryMap: Record<string, string> = {
 	github_enterprise_app: 'GitHub App',
 	websocket: 'WebSocket',
 	db_health: 'DB Health',
+	external_postgres: 'External Postgres',
 	lsp: 'LSP'
 }
 
@@ -1401,6 +1422,7 @@ export const categoryToTabMap: Record<string, string> = {
 	'GitHub App': 'github_enterprise_app',
 	WebSocket: 'websocket',
 	'DB Health': 'db_health',
+	'External Postgres': 'external_postgres',
 	LSP: 'lsp'
 }
 

frontend/src/lib/components/instanceSettings/ExternalInstancePgSettings.svelte

@@ -0,0 +1,400 @@
+<script lang="ts">
+	import { Alert, Button } from '$lib/components/common'
+	import TextInput from '../text_input/TextInput.svelte'
+	import Password from '../Password.svelte'
+	import Select from '../select/Select.svelte'
+	import DataTable from '../table/DataTable.svelte'
+	import Head from '../table/Head.svelte'
+	import Row from '../table/Row.svelte'
+	import Cell from '../table/Cell.svelte'
+	import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte'
+	import { createAsyncConfirmationModal } from '../common/confirmationModal/asyncConfirmationModal.svelte'
+	import {
+		SettingService,
+		type CustomInstanceDbTag,
+		type ExternalInstancePgSetupReport
+	} from '$lib/gen'
+	import { instanceSettingsSaved } from '../instanceSettings'
+	import { enterpriseLicense } from '$lib/stores'
+	import { sendUserToast } from '$lib/toast'
+	import { resource } from 'runed'
+	import { deepEqual } from 'fast-equals'
+	import {
+		CircleCheck,
+		CircleX,
+		KeyRound,
+		Plus,
+		Trash2,
+		TriangleAlert,
+		Wrench
+	} from 'lucide-svelte'
+	import type { Writable } from 'svelte/store'
+
+	interface Props {
+		values: Writable<Record<string, any>>
+		disabled?: boolean
+	}
+
+	let { values, disabled = false }: Props = $props()
+
+	const KEY = 'external_instance_pg'
+	const SSLMODES = ['verify-full', 'verify-ca', 'require', 'prefer', 'disable']
+
+	let isDisabled = $derived(disabled || !$enterpriseLicense)
+
+	function field(key: string): any {
+		return $values[KEY]?.[key]
+	}
+
+	// Empty inputs are removed rather than sent: the backend reads every field as optional, and an
+	// empty string would be a real (and invalid) host, port or sslmode.
+	function setField(key: string, value: any) {
+		const next = { ...($values[KEY] ?? {}) }
+		if (value === '' || value === undefined || value === null) {
+			delete next[key]
+		} else {
+			next[key] = key === 'port' ? Number(value) : value
+		}
+		$values[KEY] = Object.keys(next).length > 0 ? next : undefined
+	}
+
+	// Setup reads the saved settings, never the form, so it waits until the form is saved.
+	const saved = resource(
+		() => $instanceSettingsSaved,
+		async () => {
+			try {
+				return (await SettingService.getGlobal({ key: KEY })) ?? undefined
+			} catch {
+				return undefined
+			}
+		}
+	)
+	let unsaved = $derived(!saved.loading && !deepEqual(saved.current ?? undefined, $values[KEY]))
+
+	let refreshKey = $state(0)
+	const status = resource(
+		() => [$instanceSettingsSaved, refreshKey],
+		async () => {
+			try {
+				return await SettingService.getExternalInstancePgStatus()
+			} catch {
+				return undefined
+			}
+		}
+	)
+	const databases = resource(
+		() => [$instanceSettingsSaved, refreshKey],
+		async () => {
+			try {
+				return await SettingService.listExternalInstancePgDatabases()
+			} catch {
+				return {}
+			}
+		}
+	)
+
+	let freshReport: ExternalInstancePgSetupReport | undefined = $state(undefined)
+	let report = $derived(freshReport ?? status.current?.last_setup)
+	let runningSetup: 'setup' | 'rotate' | undefined = $state(undefined)
+
+	const confirmationModal = createAsyncConfirmationModal()
+
+	async function runSetup(rotate: boolean) {
+		if (rotate) {
+			const ok = await confirmationModal.ask({
+				title: 'Rotate passwords',
+				children:
+					'Windmill generates new passwords for its two roles on the external cluster. Jobs that connect afterwards use the new ones.',
+				confirmationText: 'Rotate'
+			})
+			if (!ok) return
+		}
+		runningSetup = rotate ? 'rotate' : 'setup'
+		try {
+			freshReport = await SettingService.setupExternalInstancePg({
+				requestBody: { rotate_passwords: rotate }
+			})
+			sendUserToast(
+				freshReport.success ? 'External cluster is set up' : 'Setup failed, see the report below',
+				!freshReport.success
+			)
+		} catch (e) {
+			sendUserToast(e?.body ?? e?.message ?? String(e), true)
+		} finally {
+			runningSetup = undefined
+			refreshKey++
+		}
+	}
+
+	let newDbName = $state('')
+	let newDbTag: CustomInstanceDbTag = $state('datatable')
+	let creating = $state(false)
+
+	async function createDatabase() {
+		creating = true
+		try {
+			await SettingService.createExternalInstancePgDatabase({
+				name: newDbName.trim(),
+				requestBody: { tag: newDbTag }
+			})
+			sendUserToast(`Created database ${newDbName.trim()}`)
+			newDbName = ''
+		} catch (e) {
+			sendUserToast(e?.body ?? e?.message ?? String(e), true)
+		} finally {
+			creating = false
+			refreshKey++
+		}
+	}
+
+	async function dropDatabase(name: string) {
+		const ok = await confirmationModal.ask({
+			title: `Drop database ${name}`,
+			children:
+				'The database and everything in it is deleted from the external cluster. This cannot be undone.',
+			confirmationText: 'Drop database'
+		})
+		if (!ok) return
+		try {
+			await SettingService.dropExternalInstancePgDatabase({ name })
+			sendUserToast(`Dropped database ${name}`)
+		} catch (e) {
+			sendUserToast(e?.body ?? e?.message ?? String(e), true)
+		} finally {
+			refreshKey++
+		}
+	}
+
+	let databaseEntries = $derived(Object.entries(databases.current ?? {}))
+	let setUp = $derived(!!status.current?.last_setup?.success)
+</script>
+
+<div class="flex flex-col gap-6">
+	{#if !$enterpriseLicense}
+		<Alert
+			type="info"
+			title="External instance databases are an Enterprise Edition feature"
+			size="xs"
+		/>
+	{/if}
+
+	<div class="grid grid-cols-2 gap-x-2 gap-y-4">
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_host" class="text-xs font-semibold text-emphasis">Host</label>
+			<TextInput
+				inputProps={{ id: 'external_pg_host', placeholder: 'db.example.com', disabled: isDisabled }}
+				bind:value={() => field('host'), (v) => setField('host', v)}
+			/>
+		</div>
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_port" class="text-xs font-semibold text-emphasis">Port</label>
+			<TextInput
+				inputProps={{
+					id: 'external_pg_port',
+					type: 'number',
+					placeholder: '5432',
+					disabled: isDisabled
+				}}
+				bind:value={() => field('port'), (v) => setField('port', v)}
+			/>
+		</div>
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_user" class="text-xs font-semibold text-emphasis">Admin user</label>
+			<TextInput
+				inputProps={{ id: 'external_pg_user', placeholder: 'windmill_admin', disabled: isDisabled }}
+				bind:value={() => field('user'), (v) => setField('user', v)}
+			/>
+		</div>
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_password" class="text-xs font-semibold text-emphasis">Password</label>
+			<Password
+				id="external_pg_password"
+				small
+				disabled={isDisabled}
+				bind:password={() => field('password'), (v) => setField('password', v)}
+			/>
+		</div>
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_dbname" class="text-xs font-semibold text-emphasis">
+				Maintenance database
+			</label>
+			<TextInput
+				inputProps={{ id: 'external_pg_dbname', placeholder: 'postgres', disabled: isDisabled }}
+				bind:value={() => field('dbname'), (v) => setField('dbname', v)}
+			/>
+		</div>
+		<div class="flex flex-col gap-1">
+			<label for="external_pg_sslmode" class="text-xs font-semibold text-emphasis">SSL mode</label>
+			<Select
+				id="external_pg_sslmode"
+				items={SSLMODES.map((m) => ({ value: m, label: m }))}
+				placeholder="verify-full (default)"
+				clearable
+				disabled={isDisabled}
+				bind:value={() => field('sslmode'), (v) => setField('sslmode', v)}
+			/>
+		</div>
+		<div class="col-span-2 flex flex-col gap-1">
+			<label for="external_pg_root_cert" class="text-xs font-semibold text-emphasis">
+				Root certificate (PEM)
+			</label>
+			<TextInput
+				underlyingInputEl="textarea"
+				inputProps={{
+					id: 'external_pg_root_cert',
+					placeholder: '-----BEGIN CERTIFICATE-----',
+					rows: 3,
+					disabled: isDisabled
+				}}
+				bind:value={() => field('root_certificate_pem'), (v) => setField('root_certificate_pem', v)}
+			/>
+			<span class="text-2xs text-secondary">
+				Leave empty to verify against the system trust store.
+			</span>
+		</div>
+	</div>
+
+	<div class="flex flex-col gap-2">
+		<div class="flex items-center gap-2">
+			<Button
+				unifiedSize="md"
+				variant="accent"
+				startIcon={{ icon: Wrench }}
+				disabled={isDisabled || unsaved || !saved.current || !!runningSetup}
+				loading={runningSetup === 'setup'}
+				onclick={() => runSetup(false)}
+			>
+				Set up cluster
+			</Button>
+			<Button
+				unifiedSize="md"
+				variant="default"
+				startIcon={{ icon: KeyRound }}
+				disabled={isDisabled || unsaved || !saved.current || !setUp || !!runningSetup}
+				loading={runningSetup === 'rotate'}
+				onclick={() => runSetup(true)}
+			>
+				Rotate passwords
+			</Button>
+			{#if unsaved}
+				<span class="text-xs text-secondary">Save the settings before setting the cluster up.</span>
+			{:else if !saved.current}
+				<span class="text-xs text-secondary">Fill in the connection and save to set it up.</span>
+			{/if}
+		</div>
+
+		{#if report}
+			<div class="flex flex-col gap-1 rounded-md border p-3 bg-surface-secondary">
+				<div class="flex items-center justify-between text-xs">
+					<span class="font-semibold text-emphasis">
+						{report.success ? 'Last setup succeeded' : 'Last setup failed'}
+					</span>
+					<span class="text-secondary">{new Date(report.finished_at).toLocaleString()}</span>
+				</div>
+				<ul class="flex flex-col gap-1.5 mt-1">
+					{#each report.steps as step, i (i)}
+						<li class="flex gap-2 text-xs">
+							{#if step.status === 'ok'}
+								<CircleCheck size={14} class="text-green-600 dark:text-green-400 shrink-0 mt-0.5" />
+							{:else if step.status === 'warning'}
+								<TriangleAlert
+									size={14}
+									class="text-yellow-600 dark:text-yellow-400 shrink-0 mt-0.5"
+								/>
+							{:else}
+								<CircleX size={14} class="text-red-600 dark:text-red-400 shrink-0 mt-0.5" />
+							{/if}
+							<span class="font-mono text-secondary shrink-0">{step.name}</span>
+							<span class="text-primary break-words">{step.message}</span>
+						</li>
+					{/each}
+				</ul>
+			</div>
+		{/if}
+	</div>
+
+	<div class="flex flex-col gap-2">
+		<div class="flex flex-col gap-0.5">
+			<span class="text-xs font-semibold text-emphasis">Databases</span>
+			<span class="text-xs text-secondary">
+				Databases Windmill created on this cluster. Workspaces use them by picking the
+				<span class="font-semibold">External instance</span> type in their data table or Ducklake settings.
+			</span>
+		</div>
+		<DataTable>
+			<Head>
+				<tr>
+					<Cell head first>Name</Cell>
+					<Cell head>Used for</Cell>
+					<Cell head>Used by</Cell>
+					<Cell head last></Cell>
+				</tr>
+			</Head>
+			<tbody class="divide-y bg-surface-tertiary">
+				{#if databaseEntries.length === 0}
+					<Row>
+						<Cell colspan={4} class="text-center text-xs text-secondary py-4">No database yet</Cell>
+					</Row>
+				{/if}
+				{#each databaseEntries as [name, db] (name)}
+					<Row>
+						<Cell first class="font-mono text-xs">{name}</Cell>
+						<Cell class="text-xs">{db.tag === 'ducklake' ? 'Ducklake' : 'Data table'}</Cell>
+						<Cell class="text-xs">
+							{(db.used_by_workspaces ?? []).join(', ') || '—'}
+						</Cell>
+						<Cell last class="text-right">
+							<Button
+								unifiedSize="sm"
+								variant="subtle"
+								startIcon={{ icon: Trash2 }}
+								iconOnly
+								disabled={isDisabled || (db.used_by_workspaces ?? []).length > 0}
+								title={(db.used_by_workspaces ?? []).length > 0
+									? 'Still used by a workspace'
+									: `Drop ${name}`}
+								onclick={() => dropDatabase(name)}
+							/>
+						</Cell>
+					</Row>
+				{/each}
+			</tbody>
+		</DataTable>
+		<div class="flex items-center gap-2">
+			<TextInput
+				class="flex-1"
+				inputProps={{
+					id: 'external_pg_new_db',
+					placeholder: 'New database name',
+					disabled: isDisabled || !setUp
+				}}
+				bind:value={newDbName}
+			/>
+			<Select
+				id="external_pg_new_db_tag"
+				class="w-36"
+				items={[
+					{ value: 'datatable', label: 'Data table' },
+					{ value: 'ducklake', label: 'Ducklake' }
+				]}
+				disabled={isDisabled || !setUp}
+				bind:value={newDbTag}
+			/>
+			<Button
+				unifiedSize="md"
+				variant="default"
+				startIcon={{ icon: Plus }}
+				disabled={isDisabled || !setUp || !newDbName.trim() || creating}
+				loading={creating}
+				onclick={createDatabase}
+			>
+				Create database
+			</Button>
+		</div>
+		{#if !setUp}
+			<span class="text-xs text-secondary">Set the cluster up before creating databases.</span>
+		{/if}
+	</div>
+</div>
+
+<ConfirmationModal {...confirmationModal.props} />

frontend/src/lib/components/workspaceSettings/DataTablePermissionsButton.svelte

@@ -78,6 +78,9 @@
 	const unusedRoles = $derived(availableRoles.filter((r) => !roles.some((row) => row.id === r.id)))
 	const pendingRoles = $derived(roles.filter((r) => r.id === undefined))
 	let instanceRoles: InstanceRolesButton | undefined = $state(undefined)
+	const clusterLabel = $derived(
+		info?.cluster === 'external_instance' ? 'the external instance' : 'this instance'
+	)
 
 	const roleKey = (role: EditedRole) => role.id ?? `pending:${role.name}`
 
@@ -174,7 +177,7 @@
 	async function refreshCatalog() {
 		let fresh: InstanceDatatableRole[]
 		try {
-			fresh = await SettingService.listInstanceDatatableRoles()
+			fresh = await SettingService.listInstanceDatatableRoles({ cluster: info?.cluster })
 		} catch (e) {
 			sendUserToast(e?.body ?? e?.message ?? String(e), true)
 			return
@@ -231,7 +234,7 @@
 	<DrawerContent
 		title="Roles — {datatable}"
 		on:close={() => (drawerOpen = false)}
-		tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant it privileges under Access. Roles are defined for the whole instance; here you say who may use each one on this data table."
+		tooltip="A data table role is a Postgres login. A job that names one connects as it, and Postgres decides what it may touch — grant it privileges under Access. Roles are defined once per cluster, for every data table on it; here you say who may use each one on this data table."
 	>
 		{#snippet titleExtra()}
 			<Badge color="blue" small>Beta</Badge>
@@ -254,9 +257,9 @@
 
 				{#if !info?.supported}
 					<Alert type="info" title="Not available on this data table" size="xs">
-						A data table role is a Postgres login on the Windmill instance's own database, so only a
-						data table backed by that database can use one. This one is backed by a PostgreSQL
-						resource — grant access on that server directly.
+						A data table role is a Postgres login on a cluster Windmill manages, so only a data table
+						on the Windmill instance or the external instance can use one. This one is backed by a
+						PostgreSQL resource — grant access on that server directly.
 					</Alert>
 				{:else if governing}
 					<Alert type="info" title="Governed by {governing}" size="xs">
@@ -283,7 +286,7 @@
 
 				{#if permissioned}
 					{#if editable && availableRoles.length === 0}
-						<Alert type="warning" title="No role defined on this instance" size="xs">
+						<Alert type="warning" title="No role defined on {clusterLabel}" size="xs">
 							Only <span class="font-mono">admin</span> can be used until a superadmin creates a data
 							table role. Type a name below to add one.
 						</Alert>
@@ -297,7 +300,7 @@
 									<Tooltip>
 										admin is the connection the data table used before roles, so it owns every
 										existing object and cannot be removed. Every other role is a login defined for
-										the whole instance, with only the privileges granted to it under Access.
+										the whole cluster, with only the privileges granted to it under Access.
 									</Tooltip>
 								</Cell>
 								<Cell head>
@@ -326,23 +329,23 @@
 											<span class="font-mono text-xs text-emphasis">{role.name ?? role.id}</span>
 											{#if !role.name}
 												<span class="text-2xs text-secondary italic">
-													no longer defined on this instance
+													no longer defined on {clusterLabel}
 												</span>
 											{:else if role.id === undefined}
 												<Alert type="warning" title="This role does not exist yet" size="xs">
 													{#if $superadmin}
 														<div class="flex flex-col items-start gap-1">
-															<span>Create it on the instance to use it here.</span>
+															<span>Create it on {clusterLabel} to use it here.</span>
 															<Button
 																unifiedSize="xs"
 																variant="default"
-																on:click={() => instanceRoles?.open(role.name)}
+																on:click={() => instanceRoles?.open(role.name, info?.cluster)}
 															>
 																Create it
 															</Button>
 														</div>
 													{:else}
-														Only a superadmin can create it on the instance.
+														Only a superadmin can create it on {clusterLabel}.
 													{/if}
 												</Alert>
 											{/if}

frontend/src/lib/components/workspaceSettings/DataTableRolesSection.svelte

@@ -10,13 +10,16 @@
 	import Head from '../table/Head.svelte'
 	import Row from '../table/Row.svelte'
 	import { Pencil, Plus } from 'lucide-svelte'
-	import { SettingService, type InstanceDatatableRole } from '$lib/gen'
+	import { SettingService, type DatatableRoleCluster, type InstanceDatatableRole } from '$lib/gen'
 	import { sendUserToast } from '$lib/toast'
 
 	let {
+		cluster = 'instance',
 		initialName = '',
 		onChanged
 	}: {
+		/** The cluster whose role catalog is shown. Fixed for the component's lifetime. */
+		cluster?: DatatableRoleCluster
 		/** Prefills the name of the role to add. */
 		initialName?: string
 		/** Called after every change to the catalog, whether or not it went through. */
@@ -38,7 +41,7 @@
 		loading = true
 		loadError = undefined
 		try {
-			roles = await SettingService.listInstanceDatatableRoles()
+			roles = await SettingService.listInstanceDatatableRoles({ cluster })
 		} catch (e) {
 			loadError = e?.body ?? e?.message ?? String(e)
 		} finally {
@@ -69,7 +72,7 @@
 		await run(
 			() =>
 				SettingService.createInstanceDatatableRole({
-					requestBody: { name }
+					requestBody: { name, cluster }
 				}),
 			`Created the data table role ${name}`
 		)
@@ -80,7 +83,7 @@
 		const confirmed = await confirmationModal.ask({
 			title: `Delete the role ${role.name}?`,
 			children:
-				'Everything it owns in every instance database is handed back to the admin connection, its grants are dropped, and it is removed from every data table that named it. This cannot be undone.',
+				'Everything it owns in every database of its cluster is handed back to the admin connection, its grants are dropped, and it is removed from every data table that named it. This cannot be undone.',
 			confirmationText: 'Delete role'
 		})
 		if (!confirmed) return
@@ -95,7 +98,7 @@
 
 <div class="flex flex-col gap-2">
 	{#if loadError}
-		<Alert type="error" title="Could not load the instance roles" size="xs">{loadError}</Alert>
+		<Alert type="error" title="Could not load the data table roles" size="xs">{loadError}</Alert>
 	{:else}
 		<DataTable>
 			<Head>

frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte

@@ -9,7 +9,7 @@
 			id: string
 			name: string
 			database: {
-				resource_type: 'postgresql' | 'instance'
+				resource_type: 'postgresql' | 'instance' | 'external_instance'
 				resource_path?: string | undefined
 			}
 			/** Set on a fork's entry: it names the workspace whose data table governs this one, and
@@ -81,8 +81,10 @@
 	import {
 		isCustomInstanceDbEnabled,
 		getUnusedInstanceDbName,
-		isDataTableWizardEnabled
+		isDataTableWizardEnabled,
+		externalInstanceDbUnavailableReason
 	} from './utils.svelte'
+	import ExternalInstanceDbSelect from './ExternalInstanceDbSelect.svelte'
 	import { random_adj } from '../random_positive_adjetive'
 	import { sendUserToast } from '$lib/toast'
 	import {
@@ -420,6 +422,13 @@
 									>
 										Use Windmill's PostgreSQL instance
 									</Tooltip>
+								{:else if dataTable.database.resource_type === 'external_instance'}
+									<Tooltip
+										wrapperClass="absolute mt-[0.6rem] right-2 z-20"
+										placement="bottom-start"
+									>
+										Use a database on the external PostgreSQL cluster set in instance settings
+									</Tooltip>
 								{/if}
 								<Select
 									items={[
@@ -433,6 +442,12 @@
 												: isCloudHosted()
 													? 'Not available on cloud'
 													: 'Superadmin only'
+										},
+										{
+											value: 'external_instance',
+											label: 'External instance',
+											disabled: !!$externalInstanceDbUnavailableReason,
+											subtitle: $externalInstanceDbUnavailableReason
 										}
 									]}
 									bind:value={
@@ -446,16 +461,22 @@
 										}
 									}
 									id="database-type-select"
-									class="w-28"
+									class="w-40"
 								/>
 							</div>
 							<div class="flex items-center gap-1 w-80 relative">
-								{#if dataTable.database.resource_type !== 'instance'}
+								{#if dataTable.database.resource_type === 'postgresql'}
 									<ResourcePicker
 										class="flex-1"
 										bind:value={dataTable.database.resource_path}
 										resourceType={dataTable.database.resource_type}
 									/>
+								{:else if dataTable.database.resource_type === 'external_instance'}
+									<ExternalInstanceDbSelect
+										class="flex-1"
+										bind:value={dataTable.database.resource_path}
+										tag="datatable"
+									/>
 								{:else}
 									<CustomInstanceDbSelect
 										class="flex-1"

frontend/src/lib/components/workspaceSettings/DucklakeSettings.svelte

@@ -13,7 +13,7 @@
 		ducklakes: {
 			name: string
 			catalog: {
-				resource_type: 'postgresql' | 'mysql' | 'instance'
+				resource_type: 'postgresql' | 'mysql' | 'instance' | 'external_instance'
 				resource_path?: string // Name of the database when resource_type is instance
 			}
 			storage: {
@@ -105,7 +105,12 @@
 	import Popover from '../meltComponents/Popover.svelte'
 	import TextInput from '../text_input/TextInput.svelte'
 	import { slide } from 'svelte/transition'
-	import { isCustomInstanceDbEnabled, getUnusedInstanceDbName } from './utils.svelte'
+	import {
+		isCustomInstanceDbEnabled,
+		getUnusedInstanceDbName,
+		externalInstanceDbUnavailableReason
+	} from './utils.svelte'
+	import ExternalInstanceDbSelect from './ExternalInstanceDbSelect.svelte'
 	import { resource } from 'runed'
 	import CustomInstanceDbSelect from './CustomInstanceDbSelect.svelte'
 	import Label from '../Label.svelte'
@@ -283,10 +288,9 @@
 			This workspace is a fork, and these settings are its own copy. Lakes marked
 			<span class="font-semibold">isolated</span> read the parent's tables through defer views and
 			write to a fork-scoped namespace that is cleaned up when the fork is deleted. Lakes marked
-			<span class="font-semibold">shared with parent</span> read and write the parent's physical
-			lake directly — editing their catalog or storage here repoints the shared lake for this
-			fork's jobs. The choice is made per lake when the fork is created and cannot be changed
-			here.
+			<span class="font-semibold">shared with parent</span> read and write the parent's physical lake
+			directly — editing their catalog or storage here repoints the shared lake for this fork's jobs.
+			The choice is made per lake when the fork is created and cannot be changed here.
 		</Alert>
 	</div>
 {/if}
@@ -359,8 +363,8 @@
 									isolated
 								</span>
 								<Tooltip>
-									Writes go to a fork-scoped namespace; reads of tables not yet materialized in
-									this fork defer to the parent. Deleting the fork cleans the namespace up.
+									Writes go to a fork-scoped namespace; reads of tables not yet materialized in this
+									fork defer to the parent. Deleting the fork cleans the namespace up.
 								</Tooltip>
 							{/if}
 						</div>
@@ -373,6 +377,11 @@
 								<Tooltip wrapperClass="absolute mt-[0.6rem] right-2 z-20" placement="bottom-start">
 									Use Windmill's PostgreSQL instance as a catalog
 								</Tooltip>
+							{:else if ducklake.catalog.resource_type === 'external_instance'}
+								<Tooltip wrapperClass="absolute mt-[0.6rem] right-2 z-20" placement="bottom-start">
+									Use a database on the external PostgreSQL cluster set in instance settings as a
+									catalog
+								</Tooltip>
 							{/if}
 							<Select
 								items={[
@@ -382,6 +391,12 @@
 										value: 'instance',
 										label: 'Instance',
 										subtitle: $isCustomInstanceDbEnabled ? undefined : 'Superadmin only'
+									},
+									{
+										value: 'external_instance',
+										label: 'External instance',
+										disabled: !!$externalInstanceDbUnavailableReason,
+										subtitle: $externalInstanceDbUnavailableReason
 									}
 								]}
 								bind:value={
@@ -394,16 +409,22 @@
 										}
 									}
 								}
-								class="w-24"
+								class="w-40"
 							/>
 						</div>
 						<div class="flex flex-1">
-							{#if ducklake.catalog.resource_type !== 'instance'}
+							{#if ducklake.catalog.resource_type === 'postgresql' || ducklake.catalog.resource_type === 'mysql'}
 								<ResourcePicker
 									class="flex-1 min-w-32"
 									bind:value={ducklake.catalog.resource_path}
 									resourceType={ducklake.catalog.resource_type}
 								/>
+							{:else if ducklake.catalog.resource_type === 'external_instance'}
+								<ExternalInstanceDbSelect
+									class="flex-1 min-w-32"
+									bind:value={ducklake.catalog.resource_path}
+									tag="ducklake"
+								/>
 							{:else}
 								<CustomInstanceDbSelect
 									class="flex-1 min-w-32"

frontend/src/lib/components/workspaceSettings/ExternalInstanceDbSelect.svelte

@@ -0,0 +1,84 @@
+<script lang="ts">
+	import { SettingService, type CustomInstanceDbTag } from '$lib/gen'
+	import { resource } from 'runed'
+	import Select from '../select/Select.svelte'
+	import { safeSelectItems } from '../select/utils.svelte'
+	import Button from '../common/button/Button.svelte'
+	import { sendUserToast } from '$lib/toast'
+	import { isExternalInstanceDbEnabled } from './utils.svelte'
+	import { Plus } from 'lucide-svelte'
+
+	type Props = {
+		value: string | undefined
+		tag: CustomInstanceDbTag
+		class?: string
+	}
+	let { value = $bindable(), tag, class: className }: Props = $props()
+
+	let refreshKey = $state(0)
+	const databases = resource(
+		() => refreshKey,
+		async () => {
+			try {
+				return await SettingService.listExternalInstancePgDatabases()
+			} catch {
+				return {}
+			}
+		}
+	)
+
+	// Every database Windmill created is offered, whatever it was created for: the tag only
+	// sorts the ones made for this kind of storage first.
+	let items = $derived(
+		safeSelectItems(
+			Object.entries(databases.current ?? {})
+				.sort(([, a], [, b]) => Number(b.tag === tag) - Number(a.tag === tag))
+				.map(([name]) => name)
+		)
+	)
+	let exists = $derived(!!value && !!databases.current?.[value])
+	let creating = $state(false)
+
+	async function create() {
+		if (!value) return
+		creating = true
+		try {
+			await SettingService.createExternalInstancePgDatabase({
+				name: value,
+				requestBody: { tag }
+			})
+			sendUserToast(`Created database ${value} on the external cluster`)
+		} catch (e) {
+			sendUserToast(e?.body ?? e?.message ?? String(e), true)
+		} finally {
+			creating = false
+			refreshKey++
+		}
+	}
+</script>
+
+<div class="flex items-center gap-1 {className}">
+	<Select
+		class="flex-1"
+		bind:value
+		onCreateItem={(i) => (value = i)}
+		placeholder="Search or create..."
+		showPlaceholderOnOpen
+		{items}
+		id="external-instance-db-select"
+		disabled={!$isExternalInstanceDbEnabled}
+	/>
+	{#if value && !databases.loading && !exists}
+		<Button
+			unifiedSize="sm"
+			variant="default"
+			startIcon={{ icon: Plus }}
+			loading={creating}
+			disabled={!$isExternalInstanceDbEnabled}
+			title="Create this database on the external cluster"
+			onclick={create}
+		>
+			Create
+		</Button>
+	{/if}
+</div>

frontend/src/lib/components/workspaceSettings/InstanceRolesButton.svelte

@@ -1,6 +1,9 @@
 <script lang="ts">
 	import { Badge, Button, Drawer, DrawerContent } from '../common'
+	import ToggleButtonGroup from '../common/toggleButton-v2/ToggleButtonGroup.svelte'
+	import ToggleButton from '../common/toggleButton-v2/ToggleButton.svelte'
 	import { Users } from 'lucide-svelte'
+	import { SettingService, type DatatableRoleCluster } from '$lib/gen'
 	import DataTableRolesSection from './DataTableRolesSection.svelte'
 
 	let {
@@ -15,14 +18,20 @@
 
 	let drawer: Drawer | undefined = $state(undefined)
 	let prefill = $state('')
+	let cluster = $state<DatatableRoleCluster>('instance')
+	let externalConfigured = $state(false)
 	// Remounts the section on each open, so the prefilled name is the one just asked for.
 	let openCount = $state(0)
 
-	/** Opens the drawer, with `name` prefilled as the role to add. */
-	export function open(name = '') {
+	/** Opens the drawer on `targetCluster`'s catalog, with `name` prefilled as the role to add. */
+	export function open(name = '', targetCluster: DatatableRoleCluster = 'instance') {
 		prefill = name
+		cluster = targetCluster
 		openCount++
 		drawer?.openDrawer()
+		SettingService.getExternalInstancePgStatus()
+			.then((s) => (externalConfigured = s.configured))
+			.catch(() => (externalConfigured = false))
 	}
 </script>
 
@@ -36,13 +45,30 @@
 	<DrawerContent
 		title="Instance roles"
 		on:close={() => drawer?.closeDrawer()}
-		tooltip="A data table role is a real Postgres login on this instance, shared by every instance database. A job that names one connects as it, and Postgres decides what it may touch. Which people may use a role on a given data table, and what it may do there, is set per data table, in its roles drawer."
+		tooltip="A data table role is a real Postgres login on one cluster Windmill manages: this instance's own, shared by every instance database, or the external instance cluster, shared by every external instance database. A job that names one connects as it, and Postgres decides what it may touch. Which people may use a role on a given data table, and what it may do there, is set per data table, in its roles drawer."
 	>
 		{#snippet titleExtra()}
 			<Badge color="blue" small>Beta</Badge>
 		{/snippet}
-		{#key openCount}
-			<DataTableRolesSection initialName={prefill} {onChanged} />
-		{/key}
+		<div class="flex flex-col gap-4">
+			{#if externalConfigured || cluster === 'external_instance'}
+				<ToggleButtonGroup
+					noWFull
+					selected={cluster}
+					onSelected={(v) => {
+						prefill = ''
+						cluster = v
+					}}
+				>
+					{#snippet children({ item })}
+						<ToggleButton value="instance" label="Windmill instance" {item} />
+						<ToggleButton value="external_instance" label="External instance" {item} />
+					{/snippet}
+				</ToggleButtonGroup>
+			{/if}
+			{#key `${openCount}:${cluster}`}
+				<DataTableRolesSection {cluster} initialName={prefill} {onChanged} />
+			{/key}
+		</div>
 	</DrawerContent>
 </Drawer>

frontend/src/lib/components/workspaceSettings/utils.svelte.ts

@@ -1,6 +1,6 @@
 import { isCloudHosted } from '$lib/cloud'
 
-import { superadmin } from '$lib/stores'
+import { enterpriseLicense, superadmin } from '$lib/stores'
 import { getLocalSetting } from '$lib/utils'
 import { derived } from 'svelte/store'
 
@@ -20,6 +20,24 @@ export let isCustomInstanceDbEnabled = derived(
 	([superadmin_]) => superadmin_ && !isCloudHosted()
 )
 
+export let isExternalInstanceDbEnabled = derived(
+	[superadmin, enterpriseLicense],
+	([superadmin_, enterpriseLicense_]) => superadmin_ && !!enterpriseLicense_ && !isCloudHosted()
+)
+
+/** Why the External instance option cannot be picked, or undefined when it can. */
+export let externalInstanceDbUnavailableReason = derived(
+	[superadmin, enterpriseLicense],
+	([superadmin_, enterpriseLicense_]) =>
+		isCloudHosted()
+			? 'Not available on cloud'
+			: !enterpriseLicense_
+				? 'Enterprise Edition only'
+				: !superadmin_
+					? 'Superadmin only'
+					: undefined
+)
+
 // Postgres caps identifiers at 63 bytes; the backend rejects longer db names.
 const MAX_INSTANCE_DB_NAME_LEN = 63