Merge origin/main

.agents/skills/ai-chat/SKILL.md

@@ -11,6 +11,15 @@ authoring and the full run reference.
 
 Run the affected mode **before** your change and **after**, same model(s), same cases.
 
+## Where guidance goes
+
+What the model should know about Windmill itself (flow shapes, groups, data tables, app
+access, secrets…) belongs in `system_prompts/base/` or `system_prompts/languages/`, which also
+feed the CLI's skills — not in a `*/core.ts` prompt builder, where only this chat would see it.
+The builders add tool plumbing and runtime values only. Scope a sentence to one consumer with
+`<!-- chat-only -->` / `<!-- cli-only -->`, keep chat tool names out of the shared files, and
+rerun `python system_prompts/generate.py`. `system_prompts/README.md` has the mechanics.
+
 ## Measure the window first, and cumulative second
 
 Optimize **`finalContextTokens`** (window occupancy — what drives overflow and

.agents/skills/svelte-frontend/SKILL.md

@@ -119,8 +119,8 @@ Form components (TextInput, Toggle, Select, etc.) should use the unified size sy
 New user-facing UX is the main source of `feature_usage` counters — propose them in the plan, not
 as a separate question, and read `docs/feature-telemetry.md` first. `logFeatureUsage()` from
 `$lib/utils/featureUsage` is only half the change: the `(feature, kind)` pair must also be
-registered in the backend allowlist or every event is silently discarded, and the disclosure copy
-in `InstanceSettings.svelte` must name what you added.
+registered in the backend allowlist or every event is silently discarded. The disclosure copy in
+`InstanceSettings.svelte` stays at the category level; see the recipe's step 3.
 
 ## Svelte MCP Server
 

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.818.0"
+  ".": "1.819.0"
 }

AGENTS.md

@@ -29,6 +29,10 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Agent workers**: `docs/agent-worker-e2e.md` — building and running one locally. An agent
   reaches the DB only through the API, so `Connection::Http` paths are never taken by a plain
   `cargo run`; a normal build cannot start one at all.
+- **External instance data tables**: `docs/external-instance-datatables.md` — the cluster Windmill
+  administers behind `external_instance` data tables and Ducklake catalogs: its invariants (one
+  lifecycle lock, managed-object markers, the setup gate, per-cluster roles, fork copy ownership)
+  and how to run one locally
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Operator write rights**: `docs/operator-write-rights.md` — which `operator_settings` flags are
   enforced rather than cosmetic, and why a right that is granted-unless-withdrawn needs `Option`
@@ -48,6 +52,7 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Brand/UI guidelines**: `frontend/brand-guidelines.md`
 - **Domain vocabulary**: `CONTEXT.md` — the words this codebase uses for its own concepts (step, step setting, trigger step, …). Name things the way it does.
 - **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
+- **AI guidance** (chat prompts and CLI skills): what the AI knows about Windmill itself — how flows, scripts, apps, pipelines and resources work — is written once, in `system_prompts/base/` and `system_prompts/languages/`, and `system_prompts/generate.py` builds both the chat's `$system_prompts` exports and the CLI's skills from it. Add or change such guidance there, never inline in one consumer: a sentence only one side should see goes in a `<!-- cli-only -->` / `<!-- chat-only -->` block, and a new file reaches both through the `TOPICS` table. The chat prompt builders under `frontend/src/lib/components/copilot/chat/` add only tool plumbing and runtime values, and shared markdown names no chat tool, since it reaches every chat mode. Rerun `generate.py` after editing; `system_prompts/README.md` has the details.
 - **Session recorder**: `frontend/src/lib/components/recording/` is also the recorder `wmill app dev --recording` serves, vendored into the CLI as `cli/src/commands/app/devRecorderBundle.gen.ts`. After changing `rawAppSnapshot.ts` or `rawAppRecording.svelte.ts`, run `bun run gen:dev-recorder` from `cli/` (`cli/test/dev_recorder_bundle_unit.test.ts` fails otherwise).
 - **Raw-app policy**: `frontend/src/lib/components/raw_apps/rawAppPolicy.ts` also derives the policy the server's raw-app deploy stores, vendored into the bundle job as `backend/windmill-api/src/apps_raw_policy.gen.js`. After changing it or anything it imports, run `bun run gen:app-policy` from `cli/` (`cli/test/app_policy_bundle_unit.test.ts` fails otherwise). It rides in the job rather than being read from the CLI the job runs because the images install `windmill-cli` unpinned, so an image can carry one older than its server.
 

CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## [1.819.0](https://github.com/windmill-labs/windmill/compare/v1.818.0...v1.819.0) (2026-09-28)
+
+
+### Features
+
+* add hub sync button to the resource types tab ([#11375](https://github.com/windmill-labs/windmill/issues/11375)) ([d76a962](https://github.com/windmill-labs/windmill/commit/d76a962331784672934749e2237f83259cbbc633))
+* add provenance claims to job OIDC tokens ([#11369](https://github.com/windmill-labs/windmill/issues/11369)) ([990a726](https://github.com/windmill-labs/windmill/commit/990a72640928336a4583e55376f37aa3c9eb17ea))
+
+
+### Bug Fixes
+
+* complete a canceled flow whose worker died between two steps ([#11366](https://github.com/windmill-labs/windmill/issues/11366)) ([c2d8997](https://github.com/windmill-labs/windmill/commit/c2d899754977ab95936b1084817da98d7e5d68f4))
+* **frontend:** apply operator write locks from the session's operating workspace ([#11395](https://github.com/windmill-labs/windmill/issues/11395)) ([5e59cef](https://github.com/windmill-labs/windmill/commit/5e59cefd1fa579148a06bb687aa60d1de21826b5))
+* gate batch rerun on job read access, scope started_at to workspace ([#11387](https://github.com/windmill-labs/windmill/issues/11387)) ([14a2619](https://github.com/windmill-labs/windmill/commit/14a2619ad24d98592821b12b2e6a54cbcfbc3d09))
+* hold interpolated references and captures to the token path scopes ([#11391](https://github.com/windmill-labs/windmill/issues/11391)) ([cedd6dc](https://github.com/windmill-labs/windmill/commit/cedd6dc901918d3001229fa95a5fe86e369b7bc4))
+* judge IPv4 embedded in IPv6 and pin the object storage test connect ([#11389](https://github.com/windmill-labs/windmill/issues/11389)) ([ec6ec1b](https://github.com/windmill-labs/windmill/commit/ec6ec1b06febd235e8a0d1ae8f5175fd929b79e7))
+* keep smtp_clicktracking_off when syncing instance config ([#11372](https://github.com/windmill-labs/windmill/issues/11372)) ([60ef821](https://github.com/windmill-labs/windmill/commit/60ef82196e014d39b0f497a9b8a88048d6e9792b))
+* keep test panel controls off the args form in debug mode ([#11382](https://github.com/windmill-labs/windmill/issues/11382)) ([22b5a1c](https://github.com/windmill-labs/windmill/commit/22b5a1cd62a483ab42ce7192b3b8dabe7738b99e))
+* let custom workspace error handlers send email with the instance SMTP ([#11365](https://github.com/windmill-labs/windmill/issues/11365)) ([e2be584](https://github.com/windmill-labs/windmill/commit/e2be584ca51bfa7bbe927e7c81c5355817649e7b))
+* list only the paths the caller can read in path autocomplete ([#11388](https://github.com/windmill-labs/windmill/issues/11388)) ([f4dcaf3](https://github.com/windmill-labs/windmill/commit/f4dcaf3e45a90951b6c2da71ac0d7af51eb65d89))
+* never double-process a slow canceled flow in the zombie sweep ([#11368](https://github.com/windmill-labs/windmill/issues/11368)) ([d7a61de](https://github.com/windmill-labs/windmill/commit/d7a61de23f98a09c881121d9595ec3c11101d04f))
+* only let a job's own token claim run lineage ([#11367](https://github.com/windmill-labs/windmill/issues/11367)) ([90f9e59](https://github.com/windmill-labs/windmill/commit/90f9e59321a16ce65ae25b49bda7e632beb313e8))
+* only restart a flow on a version of its own path and workspace ([#11376](https://github.com/windmill-labs/windmill/issues/11376)) ([893e64f](https://github.com/windmill-labs/windmill/commit/893e64f63056ba42b0fe0bcf07c6078830d3669c))
+* redesign run not found page and fix switching to the right workspace ([#11374](https://github.com/windmill-labs/windmill/issues/11374)) ([d9c7d71](https://github.com/windmill-labs/windmill/commit/d9c7d71f072b0562bb3b1642cfdc961a8948671b))
+* run an AI agent tool on the worker its own tag selects ([#11370](https://github.com/windmill-labs/windmill/issues/11370)) ([649c43e](https://github.com/windmill-labs/windmill/commit/649c43e7c162ee47f4896330595647e73d4e8888))
+* scope flow resume to its workspace and minting to the job's run ([#11392](https://github.com/windmill-labs/windmill/issues/11392)) ([163a4ff](https://github.com/windmill-labs/windmill/commit/163a4ffa4e60b1eb312476da617665a71bdae040))
+* scope workspace dependencies create to the path workspace ([#11385](https://github.com/windmill-labs/windmill/issues/11385)) ([3eaf288](https://github.com/windmill-labs/windmill/commit/3eaf2888c002e7697b105fe56d8358e099809e8f))
+* stop the schedule enabled toggle from showing unsaved changes ([#11390](https://github.com/windmill-labs/windmill/issues/11390)) ([3838cd6](https://github.com/windmill-labs/windmill/commit/3838cd6ee08c16556abdf62d0bbf4c73542a6d9c))
+
+
+### Performance Improvements
+
+* advance a flow step with one v2_job_status update ([#11357](https://github.com/windmill-labs/windmill/issues/11357)) ([65cba2d](https://github.com/windmill-labs/windmill/commit/65cba2dbb7bf5de88ee474f3849eba3b1bcd1232))
+* complete a job in one statement on the common path ([#11355](https://github.com/windmill-labs/windmill/issues/11355)) ([bebd762](https://github.com/windmill-labs/windmill/commit/bebd762194b2db740d4989d8925e1efb5b348e08))
+* shrink the module graph that gates first paint in dev ([#11373](https://github.com/windmill-labs/windmill/issues/11373)) ([47525b2](https://github.com/windmill-labs/windmill/commit/47525b211a37c2edcccda558d3f2b2300b5cc280))
+
 ## [1.818.0](https://github.com/windmill-labs/windmill/compare/v1.817.0...v1.818.0) (2026-09-25)
 
 

CONTEXT.md

@@ -48,8 +48,26 @@ _Avoid_: thread, session (that names an AI session, a different thing), chat (th
 One question and the answer to it: the run the question started, the handle that stops it,
 and the rows it is writing. At most one per conversation, and the chat is held for its whole
 length — from the moment the question takes the chat, before it has a job, until it is ended.
+The server holds the same rule: a question sent to a conversation whose turn is still running
+is refused, whoever sends it. Several conversations of one flow can each have a turn running.
 _Avoid_: request, exchange, message round
 
+**Running turn**:
+A turn whose run has not finished. Which conversations have one is the server's to say, so a
+chat opened after the turn started — a reload, another tab — still sees it and follows it.
+_Avoid_: busy, active, in flight
+
+**Queued message**:
+A question typed into a conversation while its turn runs, sent when that turn ends answered.
+At most one per conversation; typing another adds to it. A turn that fails, or is stopped,
+hands it back to the composer instead of sending it.
+_Avoid_: pending message (a pending message is one already sent and not yet confirmed)
+
+**Unread**:
+The answers that arrived in a conversation while it was not the one shown. Counted per open
+chat and forgotten on reload.
+_Avoid_: new messages, notifications
+
 **Transcript**:
 The rows a conversation's chat holds. Not the conversation: it is the newest page plus
 whatever older pages the reader has scrolled back through, so a question it cannot answer

ai_evals/cases/cli.yaml

@@ -11,12 +11,6 @@
     forbiddenSkills:
       - write-script-python3
       - write-flow
-    orderedAssistantMentions:
-      - wmill generate-metadata
-      - wmill sync push
-    orderedProposedCommands:
-      - wmill generate-metadata
-      - wmill sync push
     forbiddenExecutedCommands:
       - ^wmill generate-metadata
       - ^wmill sync push
@@ -38,12 +32,6 @@
       - write-flow
     forbiddenSkills:
       - write-script-python3
-    orderedAssistantMentions:
-      - wmill generate-metadata
-      - wmill sync push
-    orderedProposedCommands:
-      - wmill generate-metadata
-      - wmill sync push
     forbiddenExecutedCommands:
       - ^wmill generate-metadata
       - ^wmill sync push
@@ -65,12 +53,6 @@
     forbiddenSkills:
       - write-script-bun
       - write-flow
-    orderedAssistantMentions:
-      - wmill generate-metadata
-      - wmill sync push
-    orderedProposedCommands:
-      - wmill generate-metadata
-      - wmill sync push
     forbiddenExecutedCommands:
       - ^wmill generate-metadata
       - ^wmill sync push
@@ -114,12 +96,6 @@
       - write-flow
     requiredSkillsBeforeFirstMutation:
       - write-flow
-    orderedAssistantMentions:
-      - wmill generate-metadata
-      - wmill sync push
-    orderedProposedCommands:
-      - wmill generate-metadata
-      - wmill sync push
     forbiddenExecutedCommands:
       - ^wmill generate-metadata
       - ^wmill sync push

ai_evals/modes/cli.ts

@@ -17,8 +17,12 @@ import type { BenchmarkArtifactFile, CliTrace, ModeRunner } from "../core/types"
 
 const IGNORE_WORKSPACE_FILES = new Set([
   ".claude",
+  ".agents",
   "AGENTS.md",
+  "AGENTS.wmill.md",
   "CLAUDE.md",
+  // the guidance has the agent create one for every new folder
+  "folder.meta.yaml",
   "rt.d.ts",
   ".wmill-benchmark-bin",
   ".wmill-benchmark-wmill-invocations.log",
@@ -89,7 +93,7 @@ export function createCliModeRunner(
         const run = await runPromptAndCapture(
           renderedPrompt,
           workspaceDir,
-          context.evalCase?.runtime?.maxTurns ?? 6,
+          context.evalCase?.runtime?.maxTurns ?? 12,
           modelConfig
         );
         const workspaceFiles = await readDirectoryFiles(workspaceDir, { ignore: IGNORE_WORKSPACE_FILES });

backend/.sqlx/query-0a13981484dc657a3d314d8b791250b053f4aa3dd728268cffff7c0b43fafef7.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT id, started_at FROM v2_job_queue WHERE id = ANY($1)",
+  "query": "SELECT id, started_at FROM v2_job_queue WHERE id = ANY($1) AND workspace_id = $2",
   "describe": {
     "columns": [
       {
@@ -16,7 +16,8 @@
     ],
     "parameters": {
       "Left": [
-        "UuidArray"
+        "UuidArray",
+        "Text"
       ]
     },
     "nullable": [
@@ -24,5 +25,5 @@
       true
     ]
   },
-  "hash": "8167af4650c99dc66de483d72a675a7b57946b430ad0839685797a10ac1adfd8"
+  "hash": "0a13981484dc657a3d314d8b791250b053f4aa3dd728268cffff7c0b43fafef7"
 }

backend/.sqlx/query-0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946.json

@@ -1,31 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT\n            schedule.path, t.jobs FROM schedule,\n            LATERAL(SELECT ARRAY(\n                SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n                FROM v2_job_completed c JOIN v2_job j USING (id)\n                WHERE trigger_kind = 'schedule'\n                    AND trigger = schedule.path\n                    AND c.workspace_id = $1\n                    AND j.workspace_id = $1\n                    AND parent_job IS NULL AND runnable_path = schedule.script_path\n                    AND status <> 'skipped'\n                ORDER BY completed_at DESC\n                LIMIT 20\n            ) AS jobs) t\n        WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)\n        ORDER BY edited_at DESC\n        LIMIT $2 OFFSET $3",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "path",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "jobs",
-        "type_info": "JsonArray"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Int8",
-        "Int8",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      null
-    ]
-  },
-  "hash": "0f141ca6a58901dee1ddcf25694d76a4e5702c0324a27ef3a5740ef5d98d8946"
-}

backend/.sqlx/query-11b5d006acf040b2461b9539863b1e511c23e1a5f9be97da066a6808a49f2092.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        WITH job_info AS (\n            SELECT id, kind::text AS kind, parent_job\n            FROM v2_job\n            WHERE id = $1\n        )\n        SELECT\n            q.id AS \"id!\",\n            s.flow_status,\n            q.suspend AS \"suspend!\",\n            j.runnable_path AS script_path,\n            j.permissioned_as_email AS email,\n            (ji.kind IN ('flow', 'flowpreview', 'singlestepflow')) AS \"is_flow_level!\",\n            (ji.kind NOT IN ('flow', 'flowpreview', 'singlestepflow') AND q.id = ji.id) AS \"is_wac!\"\n        FROM job_info ji\n        JOIN v2_job_queue q ON q.id = CASE\n            WHEN ji.kind IN ('flow', 'flowpreview', 'singlestepflow') THEN ji.id\n            ELSE COALESCE(ji.parent_job, ji.id)\n        END\n        JOIN v2_job j ON j.id = q.id\n        LEFT JOIN v2_job_status s ON s.id = q.id\n        FOR UPDATE OF q\n        ",
+  "query": "\n        WITH job_info AS (\n            SELECT id, kind::text AS kind, parent_job\n            FROM v2_job\n            WHERE id = $1 AND workspace_id = $2\n        )\n        SELECT\n            q.id AS \"id!\",\n            s.flow_status,\n            q.suspend AS \"suspend!\",\n            j.runnable_path AS script_path,\n            j.permissioned_as_email AS email,\n            (ji.kind IN ('flow', 'flowpreview', 'singlestepflow')) AS \"is_flow_level!\",\n            (ji.kind NOT IN ('flow', 'flowpreview', 'singlestepflow') AND q.id = ji.id) AS \"is_wac!\"\n        FROM job_info ji\n        JOIN v2_job_queue q ON q.id = CASE\n            WHEN ji.kind IN ('flow', 'flowpreview', 'singlestepflow') THEN ji.id\n            ELSE COALESCE(ji.parent_job, ji.id)\n        END\n        JOIN v2_job j ON j.id = q.id\n        LEFT JOIN v2_job_status s ON s.id = q.id\n        FOR UPDATE OF q\n        ",
   "describe": {
     "columns": [
       {
@@ -41,7 +41,8 @@
     ],
     "parameters": {
       "Left": [
-        "Uuid"
+        "Uuid",
+        "Text"
       ]
     },
     "nullable": [
@@ -54,5 +55,5 @@
       null
     ]
   },
-  "hash": "3e68b73a1c2fdcb2ef538de1addbb096921e26fed398d5c0966b60503ca97fdb"
+  "hash": "11b5d006acf040b2461b9539863b1e511c23e1a5f9be97da066a6808a49f2092"
 }

backend/.sqlx/query-1c51aab269efb558640ea502e63ebed4c3d8b025947becd1c614757e7e29a5d8.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        INSERT INTO schedule (\n            workspace_id, path, schedule, timezone, edited_by, script_path,\n            is_flow, args, enabled, email, permissioned_as,\n            on_failure, on_failure_times, on_failure_exact, on_failure_extra_args,\n            on_recovery, on_recovery_times, on_recovery_extra_args,\n            on_success, on_success_extra_args,\n            ws_error_handler_muted, retry, summary, no_flow_overlap,\n            tag, paused_until, cron_version, description, dynamic_skip, labels\n        ) VALUES (\n            $1, $2, $3, $4, $5, $6,\n            $7, $8, $9, $10, $11,\n            $12, $13, $14, $15,\n            $16, $17, $18,\n            $19, $20,\n            $21, $22, $23, $24,\n            $25, $26, $27, $28, $29, $30\n        )\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args  AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels\n        ",
+  "query": "\n        INSERT INTO schedule (\n            workspace_id, path, schedule, timezone, edited_by, script_path,\n            is_flow, args, enabled, email, permissioned_as,\n            on_failure, on_failure_times, on_failure_exact, on_failure_extra_args,\n            on_recovery, on_recovery_times, on_recovery_extra_args,\n            on_success, on_success_extra_args,\n            ws_error_handler_muted, retry, summary, no_flow_overlap,\n            tag, paused_until, cron_version, description, dynamic_skip, labels\n        ) VALUES (\n            $1, $2, $3, $4, $5, $6,\n            $7, $8, $9, $10, $11,\n            $12, $13, $14, $15,\n            $16, $17, $18,\n            $19, $20,\n            $21, $22, $23, $24,\n            $25, $26, $27, $28, $29, $30\n        )\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args  AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels,\n            late_run_streak\n        ",
   "describe": {
     "columns": [
       {
@@ -167,6 +167,11 @@
         "ordinal": 32,
         "name": "labels",
         "type_info": "TextArray"
+      },
+      {
+        "ordinal": 33,
+        "name": "late_run_streak",
+        "type_info": "Int4"
       }
     ],
     "parameters": {
@@ -236,8 +241,9 @@
       true,
       true,
       true,
-      true
+      true,
+      false
     ]
   },
-  "hash": "2effe885aafaa0c5fc1f969b8c6ffbd38c53fb586251ba780b75a007e5a47ed6"
+  "hash": "1c51aab269efb558640ea502e63ebed4c3d8b025947becd1c614757e7e29a5d8"
 }

backend/.sqlx/query-1d346a14ad5586af347b8e7ac413500a39efa20e4915ffa56fd40537597db36e.json

@@ -1,26 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n        SELECT 'schedule' AS \"kind!\", COUNT(*)::BIGINT AS \"count!\" FROM schedule\n        UNION ALL SELECT 'http', COUNT(*)::BIGINT FROM http_trigger\n        UNION ALL SELECT 'websocket', COUNT(*)::BIGINT FROM websocket_trigger\n        UNION ALL SELECT 'kafka', COUNT(*)::BIGINT FROM kafka_trigger\n        UNION ALL SELECT 'nats', COUNT(*)::BIGINT FROM nats_trigger\n        UNION ALL SELECT 'postgres', COUNT(*)::BIGINT FROM postgres_trigger\n        UNION ALL SELECT 'mqtt', COUNT(*)::BIGINT FROM mqtt_trigger\n        UNION ALL SELECT 'sqs', COUNT(*)::BIGINT FROM sqs_trigger\n        UNION ALL SELECT 'gcp', COUNT(*)::BIGINT FROM gcp_trigger\n        UNION ALL SELECT 'azure', COUNT(*)::BIGINT FROM azure_trigger\n        UNION ALL SELECT 'amqp', COUNT(*)::BIGINT FROM amqp_trigger\n        UNION ALL SELECT 'email', COUNT(*)::BIGINT FROM email_trigger\n        -- Grouped, not a single 'native' key: these fire as nextcloud/google/github,\n        -- so a lone key would not line up with the `trigger`/`fired` series.\n        UNION ALL SELECT service_name::text, COUNT(*)::BIGINT FROM native_trigger GROUP BY service_name\n        ",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "kind!",
-        "type_info": "Text"
-      },
-      {
-        "ordinal": 1,
-        "name": "count!",
-        "type_info": "Int8"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null,
-      null
-    ]
-  },
-  "hash": "1d346a14ad5586af347b8e7ac413500a39efa20e4915ffa56fd40537597db36e"
-}

backend/.sqlx/query-201c4d014dd71da5a38ba832f836ca71b0fe6dbee8645449e97517aeb4ef9655.json

@@ -0,0 +1,83 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "WITH RECURSIVE lineage AS (\n            SELECT id, parent_job, kind, runnable_path, runnable_id, trigger_kind, trigger,\n                0 AS depth\n            FROM v2_job WHERE id = $1 AND workspace_id = $2\n          UNION ALL\n            SELECT p.id, p.parent_job, p.kind, p.runnable_path, p.runnable_id, p.trigger_kind,\n                p.trigger, l.depth + 1\n            FROM v2_job p JOIN lineage l ON p.id = l.parent_job\n            WHERE p.workspace_id = $2 AND l.depth < 100\n        )\n        SELECT id AS \"id!\", parent_job, kind AS \"kind!: JobKind\", runnable_path,\n            trigger_kind::text AS trigger_kind,\n            -- A restart takes its flow version from the request, so a trusted flow path\n            -- can carry another flow's code: the version must belong to that path.\n            CASE kind\n                WHEN 'flow' THEN EXISTS (SELECT 1 FROM flow_version fv\n                    WHERE fv.id = runnable_id AND fv.path = runnable_path AND fv.workspace_id = $2)\n                WHEN 'script' THEN EXISTS (SELECT 1 FROM script s\n                    WHERE s.hash = runnable_id AND s.path = runnable_path AND s.workspace_id = $2)\n                -- An app editor preview also runs an app script, at an app path it does not\n                -- have to own; only deployed-app runs are stamped with their app.\n                WHEN 'appscript' THEN COALESCE(trigger_kind = 'app'\n                    AND starts_with(runnable_path, trigger || '/'), false)\n                ELSE true\n            END AS \"origin_verified!\"\n        FROM lineage ORDER BY depth",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id!",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "parent_job",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 2,
+        "name": "kind!: JobKind",
+        "type_info": {
+          "Custom": {
+            "name": "job_kind",
+            "kind": {
+              "Enum": [
+                "script",
+                "preview",
+                "flow",
+                "dependencies",
+                "flowpreview",
+                "script_hub",
+                "identity",
+                "flowdependencies",
+                "http",
+                "graphql",
+                "postgresql",
+                "noop",
+                "appdependencies",
+                "deploymentcallback",
+                "singlestepflow",
+                "flowscript",
+                "flownode",
+                "appscript",
+                "aiagent",
+                "unassigned_script",
+                "unassigned_flow",
+                "unassigned_singlestepflow"
+              ]
+            }
+          }
+        }
+      },
+      {
+        "ordinal": 3,
+        "name": "runnable_path",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 4,
+        "name": "trigger_kind",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 5,
+        "name": "origin_verified!",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null,
+      null,
+      null,
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "201c4d014dd71da5a38ba832f836ca71b0fe6dbee8645449e97517aeb4ef9655"
+}

backend/.sqlx/query-2ab8038052a8fe8ff0e7648909a2a50fb1d5072f7177e86bbf0ca942aac0223b.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id FROM v2_job WHERE id = ANY($1) AND workspace_id = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Uuid"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "UuidArray",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "2ab8038052a8fe8ff0e7648909a2a50fb1d5072f7177e86bbf0ca942aac0223b"
+}

backend/.sqlx/query-2b40bde4c0f78a689ec7e0f5b2600311ac84b3f0a0229729b770bcf3cc55f03c.json

@@ -0,0 +1,35 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT parent_job, root_job, flow_innermost_root_job FROM v2_job\n                WHERE id = $1 AND workspace_id = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "parent_job",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "root_job",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 2,
+        "name": "flow_innermost_root_job",
+        "type_info": "Uuid"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Uuid",
+        "Text"
+      ]
+    },
+    "nullable": [
+      true,
+      true,
+      true
+    ]
+  },
+  "hash": "2b40bde4c0f78a689ec7e0f5b2600311ac84b3f0a0229729b770bcf3cc55f03c"
+}

backend/.sqlx/query-2bf039a2880555c5e6eec9a29c4a380aa832d72d2aaf5b109ee9554f1e9da6a4.json

@@ -0,0 +1,34 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT c.id AS \"id!\", u.job_id AS \"job_id!\", u.created_seq AS \"user_seq!\"\n         FROM unnest($1::uuid[]) AS c(id)\n         CROSS JOIN LATERAL (\n             SELECT job_id, created_seq\n             FROM flow_conversation_message\n             WHERE conversation_id = c.id AND message_type = 'user'\n             ORDER BY created_seq DESC\n             LIMIT 1\n         ) u\n         WHERE u.job_id IS NOT NULL\n           AND EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = u.job_id)",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id!",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "job_id!",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 2,
+        "name": "user_seq!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "UuidArray"
+      ]
+    },
+    "nullable": [
+      null,
+      true,
+      false
+    ]
+  },
+  "hash": "2bf039a2880555c5e6eec9a29c4a380aa832d72d2aaf5b109ee9554f1e9da6a4"
+}

backend/.sqlx/query-2da4b7cf1549494945c9761885dfd18ae25e0ee4810f1c9638e4400ab0c914a7.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT path FROM capture WHERE id = $1 AND workspace_id = $2",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path",
+        "type_info": "Varchar"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "2da4b7cf1549494945c9761885dfd18ae25e0ee4810f1c9638e4400ab0c914a7"
+}

backend/.sqlx/query-39793465641124401a732744777bc3553985b757b48962e9976b5da5b812bd05.json

@@ -0,0 +1,49 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT\n            schedule.path, t.jobs, schedule.late_run_streak, schedule.missed_occurrences, schedule.last_missed_at FROM schedule,\n            LATERAL(SELECT ARRAY(\n                SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)\n                FROM v2_job_completed c JOIN v2_job j USING (id)\n                WHERE trigger_kind = 'schedule'\n                    AND trigger = schedule.path\n                    AND c.workspace_id = $1\n                    AND j.workspace_id = $1\n                    AND parent_job IS NULL AND runnable_path = schedule.script_path\n                    AND status <> 'skipped'\n                ORDER BY completed_at DESC\n                LIMIT 20\n            ) AS jobs) t\n        WHERE workspace_id = $1 AND NOT starts_with(schedule.path, $4)\n        ORDER BY edited_at DESC\n        LIMIT $2 OFFSET $3",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "path",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "jobs",
+        "type_info": "JsonArray"
+      },
+      {
+        "ordinal": 2,
+        "name": "late_run_streak",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 3,
+        "name": "missed_occurrences",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 4,
+        "name": "last_missed_at",
+        "type_info": "Timestamptz"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Int8",
+        "Int8",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "39793465641124401a732744777bc3553985b757b48962e9976b5da5b812bd05"
+}

backend/.sqlx/query-466b2c68074202146da10d4b0757128fa332a52e3b451aac307234d02a51de8f.json

@@ -0,0 +1,15 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE schedule SET late_run_streak = 0\n                WHERE workspace_id = $1 AND path = $2",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "466b2c68074202146da10d4b0757128fa332a52e3b451aac307234d02a51de8f"
+}

backend/.sqlx/query-4a6fb702794ce250a8217034ae9b0b719f61fe0cb8a7a2c76be4b7422a7138a0.json

@@ -0,0 +1,23 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT late_run_streak FROM schedule WHERE workspace_id = $1 AND path = $2 FOR SHARE",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "late_run_streak",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "4a6fb702794ce250a8217034ae9b0b719f61fe0cb8a7a2c76be4b7422a7138a0"
+}

backend/.sqlx/query-535882853f7fd4365134c5caf1ca987970808b8b73a65e2bf0b3b209eae5b63a.json

@@ -0,0 +1,95 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "WITH norm AS (\n                        SELECT\n                            j.id, j.workspace_id, j.runnable_path, j.runnable_id, j.kind, j.args, j.created_by,\n                            -- Project effective kind for dispatch: pass script/flow through;\n                            -- for singlestepflow, read the wrapped runnable's type from\n                            -- raw_flow.modules[id='a'].value.type (always 'script' or 'flow').\n                            CASE\n                                WHEN j.kind IN ('script', 'flow') THEN j.kind::text\n                                WHEN j.kind = 'singlestepflow' THEN\n                                    COALESCE(\n                                        (SELECT m->'value'->>'type'\n                                            FROM jsonb_array_elements(j.raw_flow->'modules') m\n                                            WHERE m->>'id' = 'a'\n                                            LIMIT 1),\n                                        'script'\n                                    )\n                            END AS norm_kind,\n                            -- Pinned script hash for script-wrapped singlestepflow lives in\n                            -- raw_flow.modules[id='a'].value.hash. Flow-wrapped doesn't pin a\n                            -- version, so this is NULL there (Flow rerun pushes by path).\n                            (CASE WHEN j.kind = 'singlestepflow' THEN\n                                (SELECT ('x' || lpad(m->'value'->>'hash', 16, '0'))::bit(64)::bigint\n                                    FROM jsonb_array_elements(j.raw_flow->'modules') m\n                                    WHERE m->>'id' = 'a'\n                                      AND m->'value'->>'hash' IS NOT NULL\n                                    LIMIT 1)\n                            END) AS ssf_hash\n                        FROM v2_job j\n                        WHERE j.id = ANY($1)\n                            AND j.workspace_id = $2\n                            AND j.kind IN ('script', 'flow', 'singlestepflow')\n                    )\n                    SELECT\n                        n.id,\n                        n.norm_kind::JOB_KIND AS \"kind!: _\",\n                        COALESCE(s.path, f.path, n.runnable_path) AS \"script_path!\",\n                        -- script_hash is unused on the Flow rerun path (path-based push), so\n                        -- 0 is a safe placeholder when no version is pinned.\n                        COALESCE(s.hash, f.id, n.ssf_hash, 0::bigint) AS \"script_hash!: _\",\n                        COALESCE(jc.started_at, jq.scheduled_for, make_date(1970, 1, 1)) AS \"scheduled_for!: _\",\n                        n.args AS input,\n                        n.created_by,\n                        -- Pinned schema for script/flow; latest-by-path fallback for\n                        -- singlestepflow so input_transforms still resolve at rerun time.\n                        COALESCE(\n                            s.schema,\n                            f.schema,\n                            (CASE WHEN n.kind = 'singlestepflow' AND n.norm_kind = 'script' THEN\n                                (SELECT s2.schema FROM script s2\n                                    WHERE s2.workspace_id = $2 AND s2.path = n.runnable_path\n                                    ORDER BY s2.created_at DESC LIMIT 1)\n                            END),\n                            (CASE WHEN n.kind = 'singlestepflow' AND n.norm_kind = 'flow' THEN\n                                (SELECT fv.schema FROM flow\n                                    LEFT JOIN flow_version fv ON fv.id = flow.versions[array_upper(flow.versions, 1)]\n                                    WHERE flow.workspace_id = $2 AND flow.path = n.runnable_path)\n                            END)\n                        ) AS \"schema: _\"\n                    FROM norm n\n                    LEFT JOIN script s ON s.hash = n.runnable_id AND n.kind = 'script'\n                    LEFT JOIN flow_version f ON f.id = n.runnable_id AND f.path = n.runnable_path AND n.kind = 'flow'\n                    LEFT JOIN v2_job_completed jc ON jc.id = n.id\n                    LEFT JOIN v2_job_queue jq ON jq.id = n.id\n                    WHERE n.norm_kind IS NOT NULL\n                        AND COALESCE(s.path, f.path, n.runnable_path) IS NOT NULL\n                        AND (\n                            n.kind = 'singlestepflow'\n                            OR (COALESCE(s.hash, f.id) IS NOT NULL AND COALESCE(s.path, f.path) IS NOT NULL)\n                        )",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Uuid"
+      },
+      {
+        "ordinal": 1,
+        "name": "kind!: _",
+        "type_info": {
+          "Custom": {
+            "name": "job_kind",
+            "kind": {
+              "Enum": [
+                "script",
+                "preview",
+                "flow",
+                "dependencies",
+                "flowpreview",
+                "script_hub",
+                "identity",
+                "flowdependencies",
+                "http",
+                "graphql",
+                "postgresql",
+                "noop",
+                "appdependencies",
+                "deploymentcallback",
+                "singlestepflow",
+                "flowscript",
+                "flownode",
+                "appscript",
+                "aiagent",
+                "unassigned_script",
+                "unassigned_flow",
+                "unassigned_singlestepflow"
+              ]
+            }
+          }
+        }
+      },
+      {
+        "ordinal": 2,
+        "name": "script_path!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 3,
+        "name": "script_hash!: _",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 4,
+        "name": "scheduled_for!: _",
+        "type_info": "Timestamptz"
+      },
+      {
+        "ordinal": 5,
+        "name": "input",
+        "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 6,
+        "name": "created_by",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "schema: _",
+        "type_info": "Json"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "UuidArray",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null,
+      null,
+      null,
+      null,
+      true,
+      false,
+      null
+    ]
+  },
+  "hash": "535882853f7fd4365134c5caf1ca987970808b8b73a65e2bf0b3b209eae5b63a"
+}

backend/.sqlx/query-640482b37fe2aa63a5082250170406bcbe888ddece7dec3330778591573031fb.json

@@ -0,0 +1,31 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE schedule SET late_run_streak = late_run_streak + 1,\n                    missed_occurrences = CASE WHEN late_run_streak = 0 THEN $3\n                        ELSE missed_occurrences + $3 END,\n                    last_missed_at = $4\n                WHERE workspace_id = $1 AND path = $2\n                RETURNING late_run_streak, missed_occurrences",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "late_run_streak",
+        "type_info": "Int4"
+      },
+      {
+        "ordinal": 1,
+        "name": "missed_occurrences",
+        "type_info": "Int4"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Int4",
+        "Timestamptz"
+      ]
+    },
+    "nullable": [
+      false,
+      false
+    ]
+  },
+  "hash": "640482b37fe2aa63a5082250170406bcbe888ddece7dec3330778591573031fb"
+}

backend/.sqlx/query-6cda14bc33e3144b78b2147ac1153c516b5962b5648ae07c247fbd57b1b16ada.json

@@ -1,89 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "WITH norm AS (\n                        SELECT\n                            j.id, j.workspace_id, j.runnable_path, j.runnable_id, j.kind, j.args,\n                            -- Project effective kind for dispatch: pass script/flow through;\n                            -- for singlestepflow, read the wrapped runnable's type from\n                            -- raw_flow.modules[id='a'].value.type (always 'script' or 'flow').\n                            CASE\n                                WHEN j.kind IN ('script', 'flow') THEN j.kind::text\n                                WHEN j.kind = 'singlestepflow' THEN\n                                    COALESCE(\n                                        (SELECT m->'value'->>'type'\n                                            FROM jsonb_array_elements(j.raw_flow->'modules') m\n                                            WHERE m->>'id' = 'a'\n                                            LIMIT 1),\n                                        'script'\n                                    )\n                            END AS norm_kind,\n                            -- Pinned script hash for script-wrapped singlestepflow lives in\n                            -- raw_flow.modules[id='a'].value.hash. Flow-wrapped doesn't pin a\n                            -- version, so this is NULL there (Flow rerun pushes by path).\n                            (CASE WHEN j.kind = 'singlestepflow' THEN\n                                (SELECT ('x' || lpad(m->'value'->>'hash', 16, '0'))::bit(64)::bigint\n                                    FROM jsonb_array_elements(j.raw_flow->'modules') m\n                                    WHERE m->>'id' = 'a'\n                                      AND m->'value'->>'hash' IS NOT NULL\n                                    LIMIT 1)\n                            END) AS ssf_hash\n                        FROM v2_job j\n                        WHERE j.id = ANY($1)\n                            AND j.workspace_id = $2\n                            AND j.kind IN ('script', 'flow', 'singlestepflow')\n                    )\n                    SELECT\n                        n.id,\n                        n.norm_kind::JOB_KIND AS \"kind!: _\",\n                        COALESCE(s.path, f.path, n.runnable_path) AS \"script_path!\",\n                        -- script_hash is unused on the Flow rerun path (path-based push), so\n                        -- 0 is a safe placeholder when no version is pinned.\n                        COALESCE(s.hash, f.id, n.ssf_hash, 0::bigint) AS \"script_hash!: _\",\n                        COALESCE(jc.started_at, jq.scheduled_for, make_date(1970, 1, 1)) AS \"scheduled_for!: _\",\n                        n.args AS input,\n                        -- Pinned schema for script/flow; latest-by-path fallback for\n                        -- singlestepflow so input_transforms still resolve at rerun time.\n                        COALESCE(\n                            s.schema,\n                            f.schema,\n                            (CASE WHEN n.kind = 'singlestepflow' AND n.norm_kind = 'script' THEN\n                                (SELECT s2.schema FROM script s2\n                                    WHERE s2.workspace_id = $2 AND s2.path = n.runnable_path\n                                    ORDER BY s2.created_at DESC LIMIT 1)\n                            END),\n                            (CASE WHEN n.kind = 'singlestepflow' AND n.norm_kind = 'flow' THEN\n                                (SELECT fv.schema FROM flow\n                                    LEFT JOIN flow_version fv ON fv.id = flow.versions[array_upper(flow.versions, 1)]\n                                    WHERE flow.workspace_id = $2 AND flow.path = n.runnable_path)\n                            END)\n                        ) AS \"schema: _\"\n                    FROM norm n\n                    LEFT JOIN script s ON s.hash = n.runnable_id AND n.kind = 'script'\n                    LEFT JOIN flow_version f ON f.id = n.runnable_id AND f.path = n.runnable_path AND n.kind = 'flow'\n                    LEFT JOIN v2_job_completed jc ON jc.id = n.id\n                    LEFT JOIN v2_job_queue jq ON jq.id = n.id\n                    WHERE n.norm_kind IS NOT NULL\n                        AND COALESCE(s.path, f.path, n.runnable_path) IS NOT NULL\n                        AND (\n                            n.kind = 'singlestepflow'\n                            OR (COALESCE(s.hash, f.id) IS NOT NULL AND COALESCE(s.path, f.path) IS NOT NULL)\n                        )",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "id",
-        "type_info": "Uuid"
-      },
-      {
-        "ordinal": 1,
-        "name": "kind!: _",
-        "type_info": {
-          "Custom": {
-            "name": "job_kind",
-            "kind": {
-              "Enum": [
-                "script",
-                "preview",
-                "flow",
-                "dependencies",
-                "flowpreview",
-                "script_hub",
-                "identity",
-                "flowdependencies",
-                "http",
-                "graphql",
-                "postgresql",
-                "noop",
-                "appdependencies",
-                "deploymentcallback",
-                "singlestepflow",
-                "flowscript",
-                "flownode",
-                "appscript",
-                "aiagent",
-                "unassigned_script",
-                "unassigned_flow",
-                "unassigned_singlestepflow"
-              ]
-            }
-          }
-        }
-      },
-      {
-        "ordinal": 2,
-        "name": "script_path!",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 3,
-        "name": "script_hash!: _",
-        "type_info": "Int8"
-      },
-      {
-        "ordinal": 4,
-        "name": "scheduled_for!: _",
-        "type_info": "Timestamptz"
-      },
-      {
-        "ordinal": 5,
-        "name": "input",
-        "type_info": "Jsonb"
-      },
-      {
-        "ordinal": 6,
-        "name": "schema: _",
-        "type_info": "Json"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "UuidArray",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      null,
-      null,
-      null,
-      null,
-      true,
-      null
-    ]
-  },
-  "hash": "6cda14bc33e3144b78b2147ac1153c516b5962b5648ae07c247fbd57b1b16ada"
-}

backend/.sqlx/query-71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0.json

@@ -1,38 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT id, name, enabled, pwd FROM datatable_role",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "id",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "name",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 2,
-        "name": "enabled",
-        "type_info": "Bool"
-      },
-      {
-        "ordinal": 3,
-        "name": "pwd",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      false,
-      false,
-      false,
-      true
-    ]
-  },
-  "hash": "71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0"
-}

backend/.sqlx/query-793810c58d9e872da52d478db61c0a20126a207397cb9ed2dc18cd0b2baa57fa.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        UPDATE schedule SET\n            schedule                = $1,\n            timezone                = $2,\n            args                    = $3,\n            on_failure              = $4,\n            on_failure_times        = $5,\n            on_failure_exact        = $6,\n            on_failure_extra_args   = $7,\n            on_recovery             = $8,\n            on_recovery_times       = $9,\n            on_recovery_extra_args  = $10,\n            on_success              = $11,\n            on_success_extra_args   = $12,\n            ws_error_handler_muted  = $13,\n            retry                   = $14,\n            summary                 = $15,\n            no_flow_overlap         = $16,\n            tag                     = $17,\n            paused_until            = $18,\n            path                    = $19,\n            workspace_id            = $20,\n            cron_version            = COALESCE($21, cron_version),\n            description             = $22,\n            dynamic_skip            = $23,\n            email                   = $24,\n            edited_by               = $25,\n            edited_at               = now(),\n            permissioned_as         = $26,\n            labels                  = COALESCE($27, labels)\n        WHERE path = $19 AND workspace_id = $20\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels\n        ",
+  "query": "\n        UPDATE schedule SET\n            schedule                = $1,\n            timezone                = $2,\n            args                    = $3,\n            on_failure              = $4,\n            on_failure_times        = $5,\n            on_failure_exact        = $6,\n            on_failure_extra_args   = $7,\n            on_recovery             = $8,\n            on_recovery_times       = $9,\n            on_recovery_extra_args  = $10,\n            on_success              = $11,\n            on_success_extra_args   = $12,\n            ws_error_handler_muted  = $13,\n            retry                   = $14,\n            summary                 = $15,\n            no_flow_overlap         = $16,\n            tag                     = $17,\n            paused_until            = $18,\n            path                    = $19,\n            workspace_id            = $20,\n            cron_version            = COALESCE($21, cron_version),\n            description             = $22,\n            dynamic_skip            = $23,\n            email                   = $24,\n            edited_by               = $25,\n            edited_at               = now(),\n            permissioned_as         = $26,\n            labels                  = COALESCE($27, labels),\n            late_run_streak            = 0,\n            missed_occurrences     = 0,\n            last_missed_at              = NULL\n        WHERE path = $19 AND workspace_id = $20\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels,\n            late_run_streak\n        ",
   "describe": {
     "columns": [
       {
@@ -167,6 +167,11 @@
         "ordinal": 32,
         "name": "labels",
         "type_info": "TextArray"
+      },
+      {
+        "ordinal": 33,
+        "name": "late_run_streak",
+        "type_info": "Int4"
       }
     ],
     "parameters": {
@@ -233,8 +238,9 @@
       true,
       true,
       true,
-      true
+      true,
+      false
     ]
   },
-  "hash": "4d2c91937d87d385290b918af0238825cdd7784d0d611cd0069b15d11039ac56"
+  "hash": "793810c58d9e872da52d478db61c0a20126a207397cb9ed2dc18cd0b2baa57fa"
 }

backend/.sqlx/query-79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629.json

@@ -1,29 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "\n        SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n        FROM workspace_settings ws\n        JOIN workspace w ON w.id = ws.workspace_id AND w.deleted = false\n        CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n        WHERE ws.workspace_id <> $1\n          AND dt.value->'database'->>'resource_type' = 'instance'\n          AND dt.value->'database'->>'resource_path' = $2\n        ORDER BY ws.workspace_id, dt.key\n        ",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "workspace_id!",
-        "type_info": "Varchar"
-      },
-      {
-        "ordinal": 1,
-        "name": "datatable!",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      false,
-      null
-    ]
-  },
-  "hash": "79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629"
-}

backend/.sqlx/query-86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba.json

@@ -1,17 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "INSERT INTO datatable_role (id, name, enabled, pwd) VALUES ($1, $2, $3, $4)",
-  "describe": {
-    "columns": [],
-    "parameters": {
-      "Left": [
-        "Varchar",
-        "Varchar",
-        "Bool",
-        "Text"
-      ]
-    },
-    "nullable": []
-  },
-  "hash": "86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba"
-}

backend/.sqlx/query-872c06b3098cf59797bcaa90d6da5c6acbf5468c190fa72b6b6707d950267752.json

@@ -0,0 +1,32 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT 'schedule' AS \"kind!\", COUNT(*)::BIGINT AS \"count!\", 0::BIGINT AS \"suspended!\" FROM schedule\n        UNION ALL SELECT 'http', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM http_trigger\n        UNION ALL SELECT 'websocket', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM websocket_trigger\n        UNION ALL SELECT 'kafka', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM kafka_trigger\n        UNION ALL SELECT 'nats', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM nats_trigger\n        UNION ALL SELECT 'postgres', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM postgres_trigger\n        UNION ALL SELECT 'mqtt', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM mqtt_trigger\n        UNION ALL SELECT 'sqs', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM sqs_trigger\n        UNION ALL SELECT 'gcp', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM gcp_trigger\n        UNION ALL SELECT 'azure', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM azure_trigger\n        UNION ALL SELECT 'amqp', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM amqp_trigger\n        UNION ALL SELECT 'email', COUNT(*)::BIGINT, COUNT(*) FILTER (WHERE mode = 'suspended')::BIGINT FROM email_trigger\n        -- Grouped, not a single 'native' key: these fire as nextcloud/google/github,\n        -- so a lone key would not line up with the `trigger`/`fired` series.\n        UNION ALL SELECT service_name::text, COUNT(*)::BIGINT, 0::BIGINT FROM native_trigger GROUP BY service_name\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "kind!",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 1,
+        "name": "count!",
+        "type_info": "Int8"
+      },
+      {
+        "ordinal": 2,
+        "name": "suspended!",
+        "type_info": "Int8"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null,
+      null,
+      null
+    ]
+  },
+  "hash": "872c06b3098cf59797bcaa90d6da5c6acbf5468c190fa72b6b6707d950267752"
+}

backend/.sqlx/query-b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6.json

@@ -1,20 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT jsonb_object_keys(value->'databases') FROM global_settings\n         WHERE name = 'custom_instance_pg_databases'",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "jsonb_object_keys",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": []
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6"
-}

backend/.sqlx/query-ca54e732d521416cb2692c8c4406c63922f0eebcba2b2cefc4290c4baee5c071.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        UPDATE schedule SET\n            enabled = $1,\n            email = $2\n        WHERE path = $3 AND workspace_id = $4\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels\n        ",
+  "query": "\n        UPDATE schedule SET\n            enabled = $1,\n            email = $2,\n            late_run_streak = 0,\n            missed_occurrences = 0,\n            last_missed_at = NULL\n        WHERE path = $3 AND workspace_id = $4\n        RETURNING\n            workspace_id,\n            path,\n            edited_by,\n            edited_at,\n            schedule,\n            timezone,\n            enabled,\n            script_path,\n            is_flow,\n            args AS \"args: _\",\n            extra_perms,\n            email,\n            permissioned_as,\n            error,\n            on_failure,\n            on_failure_times,\n            on_failure_exact,\n            on_failure_extra_args AS \"on_failure_extra_args: _\",\n            on_recovery,\n            on_recovery_times,\n            on_recovery_extra_args AS \"on_recovery_extra_args: _\",\n            on_success,\n            on_success_extra_args AS \"on_success_extra_args: _\",\n            ws_error_handler_muted,\n            retry,\n            no_flow_overlap,\n            summary,\n            description,\n            tag,\n            paused_until,\n            cron_version,\n            dynamic_skip,\n            labels,\n            late_run_streak\n        ",
   "describe": {
     "columns": [
       {
@@ -167,6 +167,11 @@
         "ordinal": 32,
         "name": "labels",
         "type_info": "TextArray"
+      },
+      {
+        "ordinal": 33,
+        "name": "late_run_streak",
+        "type_info": "Int4"
       }
     ],
     "parameters": {
@@ -210,8 +215,9 @@
       true,
       true,
       true,
-      true
+      true,
+      false
     ]
   },
-  "hash": "642ca096c1c151fc01445593c8d2ed879a9a7be6ff044a9ce1b581ba725665fe"
+  "hash": "ca54e732d521416cb2692c8c4406c63922f0eebcba2b2cefc4290c4baee5c071"
 }

backend/.sqlx/query-cd8a933c10eefc76ad5be2316c83e413174d3ae4d639d50fc4a3344c02af3f95.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT EXISTS(SELECT 1 FROM flow_version WHERE id = $1 AND workspace_id = $2 AND path = $3) AS \"exists!\"",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "exists!",
+        "type_info": "Bool"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Int8",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "cd8a933c10eefc76ad5be2316c83e413174d3ae4d639d50fc4a3344c02af3f95"
+}

backend/.sqlx/query-d267e03115ee74ab7ca7edd8a010e424819133ce38dab7300a5fca4c6f54536d.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2,\n         occupancy_rate = $3, memory_usage = $4, wm_memory_usage = $5, vcpus = COALESCE($7, vcpus),\n         memory = COALESCE($8, memory), occupancy_rate_15s = $9, occupancy_rate_5m = $10, occupancy_rate_30m = $11, native_mode = $12, ip = COALESCE($13, ip) WHERE worker = $6",
+  "query": "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2,\n         occupancy_rate = $3, memory_usage = $4, wm_memory_usage = $5, vcpus = COALESCE($7, vcpus),\n         memory = COALESCE($8, memory), occupancy_rate_15s = $9, occupancy_rate_5m = $10, occupancy_rate_30m = $11, native_mode = $12, ip = COALESCE($13, ip),\n         current_job_id = COALESCE($14, current_job_id), current_job_workspace_id = COALESCE($15, current_job_workspace_id) WHERE worker = $6",
   "describe": {
     "columns": [],
     "parameters": {
@@ -17,10 +17,12 @@
         "Float4",
         "Float4",
         "Bool",
+        "Varchar",
+        "Uuid",
         "Varchar"
       ]
     },
     "nullable": []
   },
-  "hash": "0c18351237816fe0c56e23801fcb8e70dbffcf08ed121e55c871f727c4ddf626"
+  "hash": "d267e03115ee74ab7ca7edd8a010e424819133ce38dab7300a5fca4c6f54536d"
 }

backend/.sqlx/query-d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40.json

@@ -1,24 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT dt.key AS \"datatable!\"\n           FROM workspace_settings ws\n           CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n           WHERE ws.workspace_id = $1\n             AND dt.key <> $2\n             AND NOT dt.value ? 'permissions'\n             AND dt.value->'database'->>'resource_type' = 'instance'\n             AND dt.value->'database'->>'resource_path' = $3\n           ORDER BY dt.key",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "datatable!",
-        "type_info": "Text"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Text",
-        "Text",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40"
-}

backend/.sqlx/query-f3bae157b92cbc1b3741593a620c9203012a7d3754f56af665e2f34a969eabd7.json

@@ -1,23 +0,0 @@
-{
-  "db_name": "PostgreSQL",
-  "query": "SELECT EXISTS (SELECT 1 FROM v2_job WHERE id = $1 AND workspace_id = $2)",
-  "describe": {
-    "columns": [
-      {
-        "ordinal": 0,
-        "name": "exists",
-        "type_info": "Bool"
-      }
-    ],
-    "parameters": {
-      "Left": [
-        "Uuid",
-        "Text"
-      ]
-    },
-    "nullable": [
-      null
-    ]
-  },
-  "hash": "f3bae157b92cbc1b3741593a620c9203012a7d3754f56af665e2f34a969eabd7"
-}

backend/Cargo.lock

@@ -781,7 +781,7 @@ dependencies = [
  "thiserror 1.0.69",
  "time",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-util",
  "tokio-websockets",
  "tracing",
@@ -1353,7 +1353,7 @@ dependencies = [
  "rustls-native-certs 0.8.4",
  "rustls-pki-types",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tower 0.5.3",
  "tracing",
 ]
@@ -1703,9 +1703,9 @@ dependencies = [
 
 [[package]]
 name = "bigdecimal"
-version = "0.4.10"
+version = "0.4.11"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695"
+checksum = "8fa3f3d8cbf4dffcfe4991de61d012bef509a409ecbe9dd41049bfe32b4d4653"
 dependencies = [
  "autocfg",
  "libm",
@@ -4010,7 +4010,7 @@ dependencies = [
  "serde_json",
  "thiserror 2.0.21",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-socks",
  "tokio-util",
  "tokio-vsock",
@@ -6265,7 +6265,7 @@ dependencies = [
  "hyper-util",
  "pin-project-lite",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tower-service",
 ]
 
@@ -6331,7 +6331,7 @@ dependencies = [
  "rustls 0.23.35",
  "rustls-native-certs 0.8.4",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tower-service",
  "webpki-roots 1.0.9",
 ]
@@ -7570,7 +7570,7 @@ dependencies = [
  "rustls-pki-types",
  "smtp-proto",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "webpki-roots 0.26.11",
 ]
 
@@ -10356,7 +10356,7 @@ dependencies = [
  "sync_wrapper",
  "tokio",
  "tokio-native-tls",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-util",
  "tower 0.5.3",
  "tower-http",
@@ -10402,7 +10402,7 @@ dependencies = [
  "serde_urlencoded",
  "sync_wrapper",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-util",
  "tower 0.5.3",
  "tower-http",
@@ -10569,9 +10569,9 @@ dependencies = [
 
 [[package]]
 name = "rmcp-macros"
-version = "3.4.1"
+version = "3.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d2ee366004e5f87c103be91c6c09337811517ae8d2f919ece9da7c21444b85c5"
+checksum = "c7e66877136e969d5a2d561ee5841b17717a0619aebd64abb47a2a2f46e34683"
 dependencies = [
  "darling 0.24.1",
  "proc-macro2",
@@ -11529,9 +11529,9 @@ dependencies = [
 
 [[package]]
 name = "serde_with"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "935177bb8c0cd8ca1a4e6d1a2ac8988bea69cab4f9d3a31311e012ad27868ea4"
+checksum = "df9adc193c780ef8f159aee8b61e2d5801aaa555e6eb0947fe45530ec506296f"
 dependencies = [
  "base64 0.23.1",
  "bs58",
@@ -11550,9 +11550,9 @@ dependencies = [
 
 [[package]]
 name = "serde_with_macros"
-version = "3.23.0"
+version = "3.24.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1d607aa01a3cb0ad757d6fd216136910db3c97b102fe686585689615a02dbcdc"
+checksum = "3e17bbc68e28663bbbb90df47e058aa7eda4fb445b89fe70457bb94fbccf6e49"
 dependencies = [
  "darling 0.24.1",
  "proc-macro2",
@@ -13449,9 +13449,9 @@ dependencies = [
 
 [[package]]
 name = "tokio-rustls"
-version = "0.26.5"
+version = "0.26.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b0c85f2c3ef0b1cd58b36682f4b17aaa995f0e5db534d85692b4903abce21f67"
+checksum = "c9cc2678c2cdd569ef8215e2afd7954ada2ae20b4fdd2c5fe6139a3b02d105db"
 dependencies = [
  "rustls 0.23.35",
  "tokio",
@@ -13558,7 +13558,7 @@ dependencies = [
  "rustls-native-certs 0.8.4",
  "rustls-pki-types",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-util",
 ]
 
@@ -13651,7 +13651,7 @@ dependencies = [
  "rustls-pemfile 2.2.0",
  "socket2 0.5.10",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-stream",
  "tower 0.4.13",
  "tower-layer",
@@ -13684,7 +13684,7 @@ dependencies = [
  "rustls-native-certs 0.8.4",
  "socket2 0.5.10",
  "tokio",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-stream",
  "tower 0.5.3",
  "tower-layer",
@@ -14792,7 +14792,7 @@ dependencies = [
 
 [[package]]
 name = "windmill"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -14880,7 +14880,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-ai"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "async-stream",
  "async-trait",
@@ -14914,7 +14914,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-alerting"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -14927,7 +14927,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "argon2",
@@ -15067,7 +15067,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-agent-workers"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15090,7 +15090,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-assets"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15107,7 +15107,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-auth"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15133,7 +15133,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-client"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "reqwest 0.12.28",
  "serde",
@@ -15143,7 +15143,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-configs"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15160,7 +15160,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-debug"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "base64 0.22.1",
@@ -15182,7 +15182,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-embeddings"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15205,7 +15205,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-flow-conversations"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15221,7 +15221,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-flows"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15243,7 +15243,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-groups"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15265,7 +15265,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-inputs"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15279,7 +15279,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-integration-tests"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -15315,7 +15315,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-jobs"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15340,7 +15340,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-npm-proxy"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15368,7 +15368,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-openapi"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15390,7 +15390,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-schedule"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15410,7 +15410,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-scripts"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15452,7 +15452,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-settings"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15481,7 +15481,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-sse"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "lazy_static",
  "serde",
@@ -15493,7 +15493,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-users"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "argon2",
  "axum 0.8.9",
@@ -15517,7 +15517,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-workers"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15531,7 +15531,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-workspaces"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15551,6 +15551,7 @@ dependencies = [
  "strum",
  "tokio",
  "tokio-postgres",
+ "tokio-util",
  "tracing",
  "url",
  "uuid",
@@ -15569,7 +15570,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-audit"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "chrono",
  "lazy_static",
@@ -15583,7 +15584,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-autoscaling"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15602,7 +15603,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-common"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "aes-gcm",
  "aho-corasick",
@@ -15660,6 +15661,7 @@ dependencies = [
  "pin-project-lite",
  "pkcs1",
  "postgres-native-tls 0.5.3",
+ "postgres-protocol",
  "prometheus",
  "quick_cache",
  "rand 0.9.5",
@@ -15691,7 +15693,7 @@ dependencies = [
  "tikv-jemalloc-ctl",
  "tokio",
  "tokio-postgres",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-stream",
  "tokio-util",
  "tonic 0.13.1",
@@ -15714,7 +15716,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-dep-map"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "chrono",
  "futures",
@@ -15734,7 +15736,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-git-sync"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "regex",
  "serde",
@@ -15751,7 +15753,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-indexer"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "astral-tokio-tar",
@@ -15778,7 +15780,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-jseval"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "futures",
@@ -15795,7 +15797,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-macros"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "itertools 0.14.0",
  "lazy_static",
@@ -15811,7 +15813,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-mcp"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -15832,7 +15834,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-native-triggers"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -15863,7 +15865,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-oauth"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "arc-swap",
@@ -15888,7 +15890,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-object-store"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-stream",
@@ -15906,6 +15908,7 @@ dependencies = [
  "lazy_static",
  "object_store",
  "quick_cache",
+ "reqwest 0.12.28",
  "reqwest 0.13.5",
  "serde",
  "serde_json",
@@ -15923,7 +15926,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-operator"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "futures",
@@ -15941,7 +15944,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "convert_case 0.6.0",
  "serde",
@@ -15950,7 +15953,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-bash"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -15962,7 +15965,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-csharp"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -15974,7 +15977,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-go"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "gosyn",
@@ -15986,7 +15989,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-graphql"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -15998,7 +16001,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-java"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -16010,7 +16013,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-nu"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "nu-parser",
@@ -16021,7 +16024,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-php"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -16032,7 +16035,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -16044,7 +16047,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -16055,7 +16058,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16077,7 +16080,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-r"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -16089,7 +16092,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ruby"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16103,7 +16106,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-rust"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "convert_case 0.6.0",
@@ -16120,7 +16123,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16133,7 +16136,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde",
@@ -16145,7 +16148,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16163,7 +16166,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde-wasm-bindgen",
@@ -16179,7 +16182,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wac"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -16195,7 +16198,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-yaml"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16209,7 +16212,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-queue"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16248,7 +16251,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-runtime-nativets"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "const_format",
@@ -16288,7 +16291,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-sql-datatype-parser-wasm"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "getrandom 0.3.4",
  "wasm-bindgen",
@@ -16299,7 +16302,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-store"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16334,7 +16337,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-test-utils"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16358,7 +16361,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16391,7 +16394,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-amqp"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16418,7 +16421,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-azure"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16451,7 +16454,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-email"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16471,7 +16474,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-gcp"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16505,7 +16508,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-http"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16542,7 +16545,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-kafka"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16565,7 +16568,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-mqtt"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16589,7 +16592,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-nats"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -16613,7 +16616,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-postgres"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16648,7 +16651,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-sqs"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16676,7 +16679,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-websocket"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16701,7 +16704,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-types"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "bitflags 2.13.2",
@@ -16720,7 +16723,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-worker"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-once-cell",
@@ -16793,7 +16796,7 @@ dependencies = [
  "tiberius",
  "tokio",
  "tokio-postgres",
- "tokio-rustls 0.26.5",
+ "tokio-rustls 0.26.6",
  "tokio-stream",
  "tokio-util",
  "tracing",
@@ -16838,7 +16841,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-worker-volumes"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "bytes",
  "futures",

backend/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.818.0"
+version = "1.819.0"
 authors.workspace = true
 edition.workspace = true
 
@@ -88,7 +88,7 @@ members = [
 exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
 
 [workspace.package]
-version = "1.818.0"
+version = "1.819.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"
 
@@ -511,13 +511,15 @@ quick-xml = { version = "^0.37", features = ["serialize"] }
 url = { version = "^2" , features = ["serde"]}
 async-oauth2 = "0.5.1"
 reqwest = { version = "^0.13", features = ["json", "stream", "gzip", "multipart", "query", "form"] }
+# The reqwest object_store builds its HTTP clients with, for a custom `HttpConnector`.
+reqwest_object_store = { package = "reqwest", version = "0.12", default-features = false, features = ["rustls-tls-native-roots"] }
 eventsource-stream = "0.2.3"
 time = "^0"
 serde_urlencoded = "^0"
 astral-tokio-tar = "^0.6.2"
 tempfile = "^3"
 x509-parser = "^0.16"
-tokio-util = { version = "=0.7.17", features = ["io"] }
+tokio-util = { version = "=0.7.17", features = ["io", "rt"] }
 json-pointer = "^0"
 itertools = "^0.14.0"
 regex = "^1"
@@ -624,6 +626,7 @@ wasm-bindgen-test = "^0"
 convert_case = "0.6.0"
 getrandom = "0.2"
 tokio-postgres = {version = "^0.7", features = ["array-impls", "with-serde_json-1", "with-chrono-0_4", "with-uuid-1", "with-bit-vec-0_6"]}
+postgres-protocol = "0.6"
 rust-postgres = { package = "tokio-postgres", git = "https://github.com/MaterializeInc/rust-postgres", rev = "78c1222577bb091d69bc22b1bc7ad01c14675abe"}
 rust-postgres-native-tls = { package = "postgres-native-tls", git = "https://github.com/MaterializeInc/rust-postgres", features = ["runtime"], rev = "78c1222577bb091d69bc22b1bc7ad01c14675abe" }
 bit-vec = "=0.6.3"

backend/ee-repo-ref.txt

@@ -1 +1 @@
-7a256cf353db7cf64a60a09fa0de7f3a8b27f626
+dfe7b8b9204d21e0264fbea1c6f6eedf9e738d56

backend/migrations/20260917133824_datatable_role_cluster.down.sql

@@ -0,0 +1,22 @@
+-- Roles on the external cluster are live logins there; dropping the column would forget them.
+LOCK TABLE datatable_role;
+DO $$
+BEGIN
+    IF EXISTS (SELECT 1 FROM datatable_role WHERE cluster <> 'instance') THEN
+        RAISE EXCEPTION 'datatable_role holds roles on the external instance cluster. Delete them in instance settings first.';
+    END IF;
+    -- Before this, only data tables on Windmill's own cluster could be under roles, and a role
+    -- block left with just `admin` survives deleting every external role.
+    IF EXISTS (
+        SELECT 1 FROM workspace_settings ws,
+            jsonb_each(CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
+                THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
+        WHERE dt.value->'database'->>'resource_type' = 'external_instance'
+          AND dt.value ? 'permissions'
+    ) THEN
+        RAISE EXCEPTION 'external instance data tables are still under roles. Turn their roles off first.';
+    END IF;
+END $$;
+ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_cluster_name_key;
+ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_name_key UNIQUE (name);
+ALTER TABLE datatable_role DROP COLUMN cluster;

backend/migrations/20260917133824_datatable_role_cluster.up.sql

@@ -0,0 +1,8 @@
+-- A data table role is a Postgres login on one cluster: Windmill's own ('instance'), or the external
+-- instance cluster ('external_instance'). Role names are the cluster's own key, so they are unique
+-- per cluster rather than across the instance.
+ALTER TABLE datatable_role
+    ADD COLUMN cluster VARCHAR(20) NOT NULL DEFAULT 'instance'
+        CHECK (cluster IN ('instance', 'external_instance'));
+ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_name_key;
+ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_cluster_name_key UNIQUE (cluster, name);

backend/migrations/20260925151151_schedule_missed_occurrences.down.sql

@@ -0,0 +1,3 @@
+ALTER TABLE schedule DROP COLUMN last_missed_at;
+ALTER TABLE schedule DROP COLUMN missed_occurrences;
+ALTER TABLE schedule DROP COLUMN late_run_streak;

backend/migrations/20260925151151_schedule_missed_occurrences.up.sql

@@ -0,0 +1,7 @@
+-- A late run finished (or, for flows, started) after the next occurrence was due, so that
+-- occurrence was missed. late_run_streak counts late runs in a row and is reset by a run on
+-- time; missed_occurrences is what the latest streak missed and last_missed_at when, both
+-- kept after the streak ends so a recent one stays visible. An edit or toggle clears all.
+ALTER TABLE schedule ADD COLUMN late_run_streak INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE schedule ADD COLUMN missed_occurrences INTEGER NOT NULL DEFAULT 0;
+ALTER TABLE schedule ADD COLUMN last_missed_at TIMESTAMPTZ;

backend/migrations/20260929092707_workspace_fork_creation.down.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS workspace_fork_creation;

backend/migrations/20260929092707_workspace_fork_creation.up.sql

@@ -0,0 +1,20 @@
+-- A fork being created in the background, and how that ended. The fork's own workspace row only
+-- exists once the copy commits, so this is where a failure is reported to whoever polls for it.
+-- `heartbeat_at` is refreshed while the copy runs: an unfinished row that stops being refreshed
+-- belongs to a server that went away mid-copy. `step` names the part of the copy it is in.
+-- `creation_id` names one attempt: the row is reused when a failed or abandoned one is retried, and
+-- whoever polls reads only the attempt it started.
+CREATE TABLE workspace_fork_creation (
+    fork_workspace_id VARCHAR(50) PRIMARY KEY,
+    parent_workspace_id VARCHAR(50) NOT NULL,
+    creation_id UUID NOT NULL,
+    created_by VARCHAR(255) NOT NULL,
+    started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    heartbeat_at TIMESTAMPTZ NOT NULL DEFAULT now(),
+    step TEXT,
+    finished_at TIMESTAMPTZ,
+    error TEXT
+);
+
+GRANT ALL ON workspace_fork_creation TO windmill_user;
+GRANT ALL ON workspace_fork_creation TO windmill_admin;

backend/parsers/windmill-parser-wasm/Cargo.lock

@@ -6216,7 +6216,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
 
 [[package]]
 name = "windmill-common"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "aho-corasick",
  "anyhow",
@@ -6302,7 +6302,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-macros"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -6314,7 +6314,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "convert_case",
  "serde",
@@ -6323,7 +6323,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-bash"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6335,7 +6335,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-csharp"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6347,7 +6347,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-go"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "gosyn",
@@ -6359,7 +6359,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-graphql"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6371,7 +6371,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-java"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6383,7 +6383,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-nu"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "nu-parser",
@@ -6394,7 +6394,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-php"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6405,7 +6405,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6417,7 +6417,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6428,7 +6428,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -6450,7 +6450,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-r"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6462,7 +6462,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ruby"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6476,7 +6476,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-rust"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "convert_case",
@@ -6493,7 +6493,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6506,7 +6506,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde",
@@ -6518,7 +6518,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6536,7 +6536,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "serde-wasm-bindgen",
@@ -6552,7 +6552,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wac"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6568,7 +6568,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wasm"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "getrandom 0.2.17",
@@ -6600,7 +6600,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-yaml"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6614,7 +6614,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-types"
-version = "1.818.0"
+version = "1.819.0"
 dependencies = [
  "anyhow",
  "bitflags",

backend/parsers/windmill-parser-wasm/Cargo.toml

@@ -12,7 +12,7 @@ resolver = "2"
 members = ["."]
 
 [workspace.package]
-version = "1.818.0"
+version = "1.819.0"
 edition = "2021"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 

backend/tests/app_run_mode_lock_strip.rs

@@ -238,3 +238,65 @@ async fn test_run_mode_rejects_traversal_component(db: Pool<Postgres>) -> anyhow
 
     Ok(())
 }
+
+/// A deployed app's inline script (`app_script` id) runs as an `appscript` job whose
+/// path job identity reads (OIDC `sub`), so it is derived server-side there too.
+#[sqlx::test(fixtures("base"))]
+async fn test_app_script_run_derives_path(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let ws = format!("http://localhost:{port}/api/w/test-workspace");
+
+    let app_path = "u/test-user/appscript_path";
+    let pin = rawscript_pin(CONTENT);
+    let resp = authed(client().post(format!("{ws}/apps/create")), "SECRET_TOKEN")
+        .json(&json!({
+            "path": app_path,
+            "summary": "",
+            "value": {},
+            "policy": {
+                "execution_mode": "viewer",
+                "triggerables_v2": { format!("comp:{pin}"): { "static_inputs": {}, "one_of_inputs": {} } }
+            }
+        }))
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 201, "create app: {}", resp.text().await?);
+
+    let sha = pin.trim_start_matches("rawscript/").to_string();
+    let script_id: i64 = sqlx::query_scalar(
+        "INSERT INTO app_script (app, hash, code, code_sha256)
+         SELECT id, $2, $3, $2 FROM app WHERE path = $1 RETURNING id",
+    )
+    .bind(app_path)
+    .bind(&sha)
+    .bind(CONTENT)
+    .fetch_one(&db)
+    .await?;
+
+    let resp = authed(
+        client().post(format!("{ws}/apps_u/execute_component/{app_path}")),
+        "SECRET_TOKEN_2",
+    )
+    .json(&json!({
+        "args": {},
+        "component": "comp",
+        "id": script_id,
+        "raw_code": { "language": "python3", "content": "", "path": CALLER_PATH }
+    }))
+    .send()
+    .await?;
+    let status = resp.status();
+    let body = resp.text().await?;
+    assert_eq!(status, 200, "deployed app script run must be accepted: {body}");
+    let uuid = uuid::Uuid::parse_str(body.trim())?;
+    let (_, _, _, _, runnable_path) = job_fields(&db, uuid).await?;
+    assert_eq!(
+        runnable_path.as_deref(),
+        Some(format!("{app_path}/comp").as_str()),
+        "an app_script run must not take the caller's path"
+    );
+
+    Ok(())
+}

backend/tests/batch_rerun.rs

@@ -615,3 +615,46 @@ async fn list_groups_ssf_flow_has_schema_via_path(db: Pool<Postgres>) -> anyhow:
     );
     Ok(())
 }
+
+/// A member who may run the script but cannot see another user's run of it must not
+/// be able to rerun that run: the rerun would hand them a job carrying its args.
+#[sqlx::test(fixtures("base", "batch_rerun"))]
+async fn batch_rerun_denies_invisible_source_job(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let member = windmill_api_client::create_client(
+        &format!("http://localhost:{}", server.addr.port()),
+        "SECRET_TOKEN_2".to_string(),
+    );
+    // `false` is a read (run) grant, not a denial: test-user-2 can run the script.
+    sqlx::query("UPDATE script SET extra_perms = '{\"u/test-user-2\": false}' WHERE hash = $1")
+        .bind(SCRIPT_HASH)
+        .execute(&db)
+        .await?;
+
+    let original = push_completed(&db, script_payload(), vec![("name", json!("secret"))]).await?;
+
+    let results = batch_rerun(
+        &member,
+        json!({
+            "job_ids": [original],
+            "script_options_by_path": {},
+            "flow_options_by_path": {},
+        }),
+    )
+    .await?;
+    assert_eq!(results.len(), 1, "the denied job must be reported");
+    assert!(
+        results[0]
+            .as_ref()
+            .is_err_and(|e| e.starts_with("Permission denied")),
+        "rerun must be denied by the read gate: {:?}",
+        results[0]
+    );
+    let pushed: i64 =
+        sqlx::query_scalar("SELECT count(*) FROM v2_job WHERE created_by = 'test-user-2'")
+            .fetch_one(&db)
+            .await?;
+    assert_eq!(pushed, 0);
+    Ok(())
+}

backend/tests/fixtures/job_provenance.sql

@@ -0,0 +1,32 @@
+-- Add-on fixture for job_provenance.rs (combine with `base.sql`). A deployed script
+-- f/t/tool run as a step under three parents: the deployed flow f/t/agent, a flow
+-- preview claiming that path, and f/t/agent running the version of another flow.
+
+INSERT INTO script (workspace_id, hash, path, content, language, kind, created_by, schema, summary, description, lock)
+VALUES ('test-workspace', 1111111111, 'f/t/tool', 'echo tool', 'bash', 'script', 'test-user', '{}', '', '', '');
+
+INSERT INTO flow (workspace_id, path, summary, description, value, edited_by, edited_at, schema, extra_perms, versions)
+VALUES
+    ('test-workspace', 'f/t/agent', '', '', '{"modules":[]}', 'test-user', NOW(), '{}', '{}', ARRAY[2222222222::bigint]),
+    ('test-workspace', 'u/test-user/evil', '', '', '{"modules":[]}', 'test-user', NOW(), '{}', '{}', ARRAY[3333333333::bigint]);
+
+INSERT INTO flow_version (id, workspace_id, path, value, schema, created_by, created_at)
+VALUES
+    (2222222222, 'test-workspace', 'f/t/agent', '{"modules":[]}', '{}', 'test-user', NOW()),
+    (3333333333, 'test-workspace', 'u/test-user/evil', '{"modules":[]}', '{}', 'test-user', NOW());
+
+INSERT INTO v2_job (id, workspace_id, kind, runnable_path, runnable_id, parent_job, created_by, permissioned_as, permissioned_as_email)
+VALUES
+    ('3bb0c0de-0000-4000-8000-000000000001', 'test-workspace', 'flow', 'f/t/agent', 2222222222, NULL, 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000002', 'test-workspace', 'script', 'f/t/tool', 1111111111, '3bb0c0de-0000-4000-8000-000000000001', 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000003', 'test-workspace', 'flowpreview', 'f/t/agent', NULL, NULL, 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000004', 'test-workspace', 'script', 'f/t/tool', 1111111111, '3bb0c0de-0000-4000-8000-000000000003', 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000005', 'test-workspace', 'flow', 'f/t/agent', 3333333333, NULL, 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000006', 'test-workspace', 'script', 'f/t/tool', 1111111111, '3bb0c0de-0000-4000-8000-000000000005', 'test-user', 'u/test-user', 'test@windmill.dev');
+
+-- An app script run of f/t/app stamped by a deployed-app run, and one that is not
+-- (an app editor preview).
+INSERT INTO v2_job (id, workspace_id, kind, runnable_path, trigger_kind, trigger, created_by, permissioned_as, permissioned_as_email)
+VALUES
+    ('3bb0c0de-0000-4000-8000-000000000007', 'test-workspace', 'appscript', 'f/t/app/comp', 'app', 'f/t/app', 'test-user', 'u/test-user', 'test@windmill.dev'),
+    ('3bb0c0de-0000-4000-8000-000000000008', 'test-workspace', 'appscript', 'f/t/app/comp', NULL, NULL, 'test-user', 'u/test-user', 'test@windmill.dev');

backend/tests/flow_conversation_running_turn.rs

@@ -0,0 +1,87 @@
+//! A conversation answers one message at a time: while the run its newest user message started
+//! is queued or running, the list reports that turn and a run into the conversation is refused.
+//!
+//! Uses runtime `sqlx::query` (not the compile-time macros) so no offline query cache is
+//! needed, matching v2_job_delete_orphans.rs.
+
+use sqlx::{Pool, Postgres};
+use uuid::Uuid;
+use windmill_common::error::Error;
+use windmill_common::flow_conversations::{get_or_create_conversation_with_id, running_turns};
+use windmill_test_utils::*;
+
+const WS: &str = "test-workspace";
+const CONV: Uuid = Uuid::from_u128(0x5eed);
+
+/// Takes the conversation for a turn, as a run does, without keeping what it writes.
+async fn take_conversation(db: &Pool<Postgres>) -> windmill_common::error::Result<Uuid> {
+    let mut tx = db.begin().await?;
+    let result =
+        get_or_create_conversation_with_id(&mut tx, WS, "f/flow", "test-user", "t", CONV, false)
+            .await;
+    tx.rollback().await?;
+    result.map(|c| c.id)
+}
+
+#[sqlx::test(fixtures("base"))]
+async fn test_a_conversation_refuses_a_turn_while_one_runs(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    initialize_tracing().await;
+
+    let job_id = Uuid::new_v4();
+    sqlx::query("INSERT INTO v2_job (id, workspace_id, kind) VALUES ($1, $2, 'flow')")
+        .bind(job_id)
+        .bind(WS)
+        .execute(&db)
+        .await?;
+    sqlx::query(
+        "INSERT INTO v2_job_queue (id, workspace_id, scheduled_for) VALUES ($1, $2, now())",
+    )
+    .bind(job_id)
+    .bind(WS)
+    .execute(&db)
+    .await?;
+    sqlx::query(
+        "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by)
+         VALUES ($1, $2, 'f/flow', 'test-user')",
+    )
+    .bind(CONV)
+    .bind(WS)
+    .execute(&db)
+    .await?;
+    let user_seq: i64 = sqlx::query_scalar(
+        "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id)
+         VALUES ($1, 'user', 'hi', $2) RETURNING created_seq",
+    )
+    .bind(CONV)
+    .bind(job_id)
+    .fetch_one(&db)
+    .await?;
+
+    let running = running_turns(&db, &[CONV]).await?;
+    let turn = running
+        .get(&CONV)
+        .expect("the queued run is the conversation's running turn");
+    assert_eq!((turn.job_id, turn.user_seq), (job_id, user_seq));
+
+    match take_conversation(&db).await {
+        Err(Error::Generic(status, body)) => {
+            assert_eq!(status.as_u16(), 409);
+            assert!(
+                body.contains(&job_id.to_string()),
+                "the refusal names the running job: {body}"
+            );
+        }
+        other => panic!("expected a 409 while the first turn runs, got {other:?}"),
+    }
+
+    // The run ends: it leaves the queue, and the conversation takes the next message.
+    sqlx::query("DELETE FROM v2_job_queue WHERE id = $1")
+        .bind(job_id)
+        .execute(&db)
+        .await?;
+    assert!(running_turns(&db, &[CONV]).await?.is_empty());
+    assert_eq!(take_conversation(&db).await?, CONV);
+    Ok(())
+}

backend/tests/instance_config.rs

@@ -1609,6 +1609,52 @@ async fn declarative_sync_rejects_an_unusable_default_allowed_origins(db: Pool<P
     .expect("a valid origin list must sync");
 }
 
+/// While Windmill has databases on the external cluster, a sync may change its login but not
+/// point it at another cluster: the databases would be stranded there, and their registry and
+/// role passwords applied to a cluster that has neither.
+#[cfg(all(feature = "enterprise", feature = "private"))]
+#[sqlx::test(fixtures("base"))]
+async fn declarative_sync_keeps_the_external_cluster_while_it_holds_databases(db: Pool<Postgres>) {
+    clear_settings_and_configs(&db).await;
+    let cluster = |host: &str, password: &str| serde_json::json!({ "host": host, "port": 5432, "user": "wm_admin", "password": password });
+    sqlx::query(
+        "INSERT INTO global_settings (name, value) VALUES
+             ('external_instance_pg', $1),
+             ('external_instance_pg_state', '{\"databases\": {\"dt_a\": {\"success\": true}}}')",
+    )
+    .bind(cluster("pg-a.internal", "one"))
+    .execute(&db)
+    .await
+    .unwrap();
+    let current = BTreeMap::from([(
+        "external_instance_pg".to_string(),
+        cluster("pg-a.internal", "one"),
+    )]);
+    let sync = |value: serde_json::Value| {
+        let desired = BTreeMap::from([("external_instance_pg".to_string(), value)]);
+        let (db, current) = (db.clone(), current.clone());
+        async move {
+            windmill_common::instance_config::sync_global_settings_declarative(
+                &db, &current, &desired,
+            )
+            .await
+        }
+    };
+
+    let err = sync(cluster("pg-b.internal", "one"))
+        .await
+        .expect_err("another host must be refused while dt_a is registered");
+    assert!(err.to_string().contains("dt_a"), "got: {err}");
+    assert_eq!(
+        get_global_setting(&db, "external_instance_pg").await,
+        Some(cluster("pg-a.internal", "one"))
+    );
+
+    sync(cluster("PG-A.internal ", "two"))
+        .await
+        .expect("a new login on the same cluster must sync");
+}
+
 /// The accent color is interpolated into a stylesheet every user loads, so the operator
 /// path must refuse anything but `#rrggbb` just like the settings API does.
 #[sqlx::test(fixtures("base"))]

backend/tests/job_provenance.rs

@@ -0,0 +1,44 @@
+//! `deployed` gates what job OIDC tokens let external trust policies accept, so a job
+//! whose code or path the caller chose must never come out deployed.
+
+use sqlx::{Pool, Postgres};
+use uuid::Uuid;
+use windmill_common::job_provenance::job_provenance;
+
+async fn deployed(db: &Pool<Postgres>, job: &str) -> bool {
+    job_provenance(db, &Uuid::parse_str(job).unwrap(), "test-workspace")
+        .await
+        .unwrap()
+        .unwrap()
+        .deployed
+}
+
+#[sqlx::test(fixtures("base", "job_provenance"))]
+async fn step_of_deployed_flow_is_deployed(db: Pool<Postgres>) {
+    let p = job_provenance(
+        &db,
+        &Uuid::parse_str("3bb0c0de-0000-4000-8000-000000000002").unwrap(),
+        "test-workspace",
+    )
+    .await
+    .unwrap()
+    .unwrap();
+    assert!(p.deployed);
+    assert_eq!(p.root_path.as_deref(), Some("f/t/agent"));
+}
+
+#[sqlx::test(fixtures("base", "job_provenance"))]
+async fn step_under_flow_preview_is_not_deployed(db: Pool<Postgres>) {
+    assert!(!deployed(&db, "3bb0c0de-0000-4000-8000-000000000004").await);
+}
+
+#[sqlx::test(fixtures("base", "job_provenance"))]
+async fn flow_running_another_flows_version_is_not_deployed(db: Pool<Postgres>) {
+    assert!(!deployed(&db, "3bb0c0de-0000-4000-8000-000000000006").await);
+}
+
+#[sqlx::test(fixtures("base", "job_provenance"))]
+async fn app_script_is_deployed_only_from_a_deployed_app_run(db: Pool<Postgres>) {
+    assert!(deployed(&db, "3bb0c0de-0000-4000-8000-000000000007").await);
+    assert!(!deployed(&db, "3bb0c0de-0000-4000-8000-000000000008").await);
+}

backend/tests/jobs_read_auth.rs

@@ -513,12 +513,12 @@ async fn test_single_job_read_authorization(db: Pool<Postgres>) -> anyhow::Resul
     let (status, secret) = get(
         &authed_base,
         &format!("job_signature/{STEP_JOB}/0"),
-        Some("SECRET_TOKEN_2"),
+        Some("SECRET_TOKEN"),
     )
     .await;
     assert!(
         status.is_success(),
-        "owner must mint a resume secret: {secret}"
+        "an admin must mint a resume secret: {secret}"
     );
     let secret = secret.trim().trim_matches('"').to_string();
     let approval_result = format!(

backend/tests/path_autocomplete_scope.rs

@@ -0,0 +1,62 @@
+//! `GET /path_autocomplete/list_paths` returns only the paths the caller can read,
+//! and its cache never serves one caller's list to another.
+
+use sqlx::{Pool, Postgres};
+use windmill_test_utils::*;
+
+async fn list_paths(port: u16, token: &str) -> anyhow::Result<Vec<String>> {
+    let resp = reqwest::Client::new()
+        .get(format!(
+            "http://localhost:{port}/api/w/test-workspace/path_autocomplete/list_paths"
+        ))
+        .header("Authorization", format!("Bearer {token}"))
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 200, "list_paths: {}", resp.text().await?);
+    let body: serde_json::Value = resp.json().await?;
+    Ok(serde_json::from_value(body["paths"].clone())?)
+}
+
+#[sqlx::test(fixtures("base"))]
+async fn test_list_paths_is_scoped_to_caller(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+
+    sqlx::query(
+        "INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms)
+         VALUES ('test-workspace', 'secret', 'secret', '{}', '{}')",
+    )
+    .execute(&db)
+    .await?;
+    for path in [
+        "f/secret/hidden",
+        "u/test-user/private",
+        "u/test-user-2/mine",
+    ] {
+        sqlx::query(
+            "INSERT INTO variable (workspace_id, path, value, is_secret, description)
+             VALUES ('test-workspace', $1, 'x', false, '')",
+        )
+        .bind(path)
+        .execute(&db)
+        .await?;
+    }
+
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    // The admin's call fills the cache first, so a cache shared across callers
+    // would hand the admin's list to the non-admin.
+    assert_eq!(
+        list_paths(port, "SECRET_TOKEN").await?,
+        [
+            "f/secret/hidden",
+            "u/test-user-2/mine",
+            "u/test-user/private"
+        ]
+    );
+    assert_eq!(
+        list_paths(port, "SECRET_TOKEN_2").await?,
+        ["u/test-user-2/mine"]
+    );
+    Ok(())
+}

backend/tests/preview_restart_source_access.rs

@@ -0,0 +1,77 @@
+//! A flow preview restarted from an earlier run copies that run's step results, so it must be
+//! refused unless the caller can see every run the request names.
+
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+use windmill_common::jobs::JobPayload;
+use windmill_test_utils::*;
+
+fn identity_flow() -> serde_json::Value {
+    json!({ "modules": [
+        { "id": "a", "value": { "type": "identity" } },
+        { "id": "b", "value": { "type": "identity" } },
+    ]})
+}
+
+async fn completed_run(db: &Pool<Postgres>, port: u16, username: &str, email: &str) -> String {
+    RunJob::from(JobPayload::RawFlow {
+        value: serde_json::from_value(identity_flow()).unwrap(),
+        path: None,
+        restarted_from: None,
+    })
+    .as_user(username, email)
+    .run_until_complete(db, false, port)
+    .await
+    .id
+    .to_string()
+}
+
+#[sqlx::test(fixtures("base"))]
+async fn test_preview_restart_requires_source_read_access(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+
+    let admin_run = completed_run(&db, port, "test-user", "test@windmill.dev").await;
+    let own_run = completed_run(&db, port, "test-user-2", "test2@windmill.dev").await;
+
+    let restart = |restarted_from: serde_json::Value| {
+        reqwest::Client::new()
+            .post(format!(
+                "http://localhost:{port}/api/w/test-workspace/jobs/run/preview_flow"
+            ))
+            .bearer_auth("SECRET_TOKEN_2")
+            .json(
+                &json!({ "args": {}, "value": identity_flow(), "restarted_from": restarted_from }),
+            )
+            .send()
+    };
+
+    let resp = restart(json!({ "flow_job_id": admin_run, "step_id": "b" })).await?;
+    assert_eq!(resp.status(), 403, "{}", resp.text().await?);
+
+    let resp = restart(json!({
+        "flow_job_id": own_run, "step_id": "b",
+        "nested": { "flow_job_id": admin_run, "step_id": "b" },
+    }))
+    .await?;
+    assert_eq!(resp.status(), 403, "{}", resp.text().await?);
+
+    let resp = restart(json!({ "flow_job_id": own_run, "step_id": "b" })).await?;
+    assert_eq!(resp.status(), 201, "{}", resp.text().await?);
+
+    // A run hidden on its own but inside a flow the caller can read is readable, as on the run page.
+    let child_run = completed_run(&db, port, "test-user", "test@windmill.dev").await;
+    sqlx::query("UPDATE v2_job SET parent_job = $1::uuid WHERE id = $2::uuid")
+        .bind(&own_run)
+        .bind(&child_run)
+        .execute(&db)
+        .await?;
+    let resp = restart(json!({
+        "flow_job_id": own_run, "step_id": "b",
+        "nested": { "flow_job_id": child_run, "step_id": "b" },
+    }))
+    .await?;
+    assert_eq!(resp.status(), 201, "{}", resp.text().await?);
+    Ok(())
+}

backend/tests/restart_flow_version_scope.rs

@@ -0,0 +1,135 @@
+//! A restart's `flow_version` is caller-supplied while the restarted job keeps the original
+//! job's path, so only versions of that same flow may be restarted on.
+
+use sqlx::{Pool, Postgres};
+use windmill_common::cache;
+use windmill_common::error::Error;
+use windmill_common::flow_status::FlowStatus;
+use windmill_common::jobs::JobPayload;
+use windmill_queue::PushIsolationLevel;
+
+const WS: &str = "test-workspace";
+
+async fn push_restart(
+    db: &Pool<Postgres>,
+    completed_job_id: uuid::Uuid,
+    flow_version: i64,
+) -> Result<uuid::Uuid, Error> {
+    let args = std::collections::HashMap::new();
+    let (uuid, tx) = windmill_queue::push(
+        db,
+        PushIsolationLevel::IsolatedRoot(db.clone()),
+        WS,
+        JobPayload::RestartedFlow {
+            completed_job_id,
+            step_id: "a".into(),
+            branch_or_iteration_n: None,
+            flow_version: Some(flow_version),
+            branch_chosen: None,
+            nested: None,
+        },
+        windmill_queue::PushArgs::from(&args),
+        "test-user",
+        "test@windmill.dev",
+        "u/test-user".to_string(),
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        false,
+        false,
+        None,
+        true,
+        None,
+        None,
+        None,
+        None,
+        None,
+        false,
+        None,
+        None,
+        None,
+    )
+    .await?;
+    tx.commit().await?;
+    Ok(uuid)
+}
+
+#[sqlx::test(fixtures("base", "hello"))]
+async fn test_restart_rejects_flow_version_of_another_path(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    // A completed run of f/system/hello_flow (version 1443253234253453).
+    let version = 1443253234253453;
+    let flow = cache::flow::fetch_version(&db, version).await?;
+    let completed_job_id = uuid::Uuid::new_v4();
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, runnable_id, created_by,
+                             permissioned_as, permissioned_as_email, tag)
+         VALUES ($1, $2, 'flow', 'f/system/hello_flow', $3, 'test-user', 'u/test-user',
+                 'test@windmill.dev', 'flow')",
+    )
+    .bind(completed_job_id)
+    .bind(WS)
+    .bind(version)
+    .execute(&db)
+    .await?;
+    sqlx::query(
+        "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status, flow_status)
+         VALUES ($1, $2, 0, 'success', $3)",
+    )
+    .bind(completed_job_id)
+    .bind(WS)
+    .bind(serde_json::to_value(FlowStatus::new(flow.value()))?)
+    .execute(&db)
+    .await?;
+
+    // f/system/hello_with_nodes_flow's version, in the same workspace.
+    let err = push_restart(&db, completed_job_id, 1443253234253454)
+        .await
+        .expect_err("a version of another flow must be rejected");
+    assert!(matches!(err, Error::BadRequest(_)), "{err:?}");
+
+    // A version at the same path in another workspace.
+    sqlx::query(
+        "INSERT INTO workspace (id, name, owner) VALUES ('other-workspace', 'other-workspace', 'test-user')",
+    )
+    .execute(&db)
+    .await?;
+    sqlx::query(
+        "INSERT INTO flow (workspace_id, path, summary, description, value, edited_by)
+         SELECT 'other-workspace', path, summary, description, value, edited_by FROM flow
+         WHERE workspace_id = $1 AND path = 'f/system/hello_flow'",
+    )
+    .bind(WS)
+    .execute(&db)
+    .await?;
+    let other_ws_version: i64 = sqlx::query_scalar(
+        "INSERT INTO flow_version (workspace_id, path, schema, value, created_by)
+         SELECT 'other-workspace', path, schema, value, created_by FROM flow_version WHERE id = $1
+         RETURNING id",
+    )
+    .bind(version)
+    .fetch_one(&db)
+    .await?;
+    let err = push_restart(&db, completed_job_id, other_ws_version)
+        .await
+        .expect_err("a version from another workspace must be rejected");
+    assert!(matches!(err, Error::BadRequest(_)), "{err:?}");
+
+    // Another version of the restarted flow itself stays allowed.
+    let same_path_version: i64 = sqlx::query_scalar(
+        "INSERT INTO flow_version (workspace_id, path, schema, value, created_by)
+         SELECT workspace_id, path, schema, value, created_by FROM flow_version WHERE id = $1
+         RETURNING id",
+    )
+    .bind(version)
+    .fetch_one(&db)
+    .await?;
+    push_restart(&db, completed_job_id, same_path_version).await?;
+    Ok(())
+}

backend/tests/run_lineage_auth.rs

@@ -0,0 +1,136 @@
+//! `parent_job` / `root_job` on the run endpoints set the lineage a job is identified by
+//! (`WM_FLOW_JOB_ID`, `WM_ROOT_FLOW_JOB_ID`, the OIDC token's flow path). Only a job's own
+//! `WM_TOKEN` may claim that job or its ancestors, and a workspace admin any job of the
+//! workspace; a lineage anyone else names is dropped from the pushed job.
+
+use reqwest::StatusCode;
+use sqlx::{Pool, Postgres};
+use uuid::Uuid;
+use windmill_test_utils::*;
+
+const ADMIN_JOB: &str = "a0000000-0000-0000-0000-000000000001";
+const ROOT_JOB: &str = "a0000000-0000-0000-0000-000000000002";
+const USER_JOB: &str = "a0000000-0000-0000-0000-000000000003";
+
+async fn insert_job(
+    db: &Pool<Postgres>,
+    id: &str,
+    user: &str,
+    email: &str,
+    root: Option<&str>,
+) -> anyhow::Result<()> {
+    let root = root.map(Uuid::parse_str).transpose()?;
+    sqlx::query(
+        "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, permissioned_as_email,
+            kind, script_lang, runnable_path, tag, parent_job, root_job, flow_innermost_root_job)
+        VALUES ($1, 'test-workspace', $2, 'u/' || $2, $3, 'script', 'deno', 'u/x/y', 'deno',
+            $4, $4, $4)",
+    )
+    .bind(Uuid::parse_str(id)?)
+    .bind(user)
+    .bind(email)
+    .bind(root)
+    .execute(db)
+    .await?;
+    Ok(())
+}
+
+#[sqlx::test(fixtures("base"))]
+async fn test_run_lineage_must_be_the_callers_own(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+
+    sqlx::query(
+        "INSERT INTO script (workspace_id, created_by, content, schema, summary, description,
+            path, hash, language, lock, kind)
+        VALUES ('test-workspace', 'test-user-3', 'export function main() {}', '{}', '', '',
+            'u/test-user-3/probe', 424242, 'deno', '', 'script')",
+    )
+    .execute(&db)
+    .await?;
+    insert_job(&db, ADMIN_JOB, "test-user", "test@windmill.dev", None).await?;
+    insert_job(&db, ROOT_JOB, "test-user-3", "test3@windmill.dev", None).await?;
+    insert_job(
+        &db,
+        USER_JOB,
+        "test-user-3",
+        "test3@windmill.dev",
+        Some(ROOT_JOB),
+    )
+    .await?;
+
+    let server = ApiServer::start(db.clone()).await?;
+    set_jwt_secret().await;
+    let job_token = windmill_common::auth::create_token_for_owner(
+        &db,
+        "test-workspace",
+        "u/test-user-3",
+        "ephemeral-script",
+        300,
+        "test3@windmill.dev",
+        &Uuid::parse_str(USER_JOB)?,
+        None,
+        None,
+    )
+    .await?;
+
+    let client = reqwest::Client::new();
+    let cases = [
+        // a user's own token claims neither someone else's job nor even its own
+        (
+            "SECRET_TOKEN_3",
+            Some(ADMIN_JOB),
+            Some(ADMIN_JOB),
+            None,
+            None,
+        ),
+        ("SECRET_TOKEN_3", Some(USER_JOB), None, None, None),
+        // what the SDKs send from inside a job: that job and its root flow
+        (
+            &job_token,
+            Some(USER_JOB),
+            Some(ROOT_JOB),
+            Some(USER_JOB),
+            Some(ROOT_JOB),
+        ),
+        (
+            &job_token,
+            Some(USER_JOB),
+            Some(ADMIN_JOB),
+            Some(USER_JOB),
+            None,
+        ),
+        ("SECRET_TOKEN", Some(USER_JOB), None, Some(USER_JOB), None),
+    ];
+    for (token, parent, root, kept_parent, kept_root) in cases {
+        let query = [("parent_job", parent), ("root_job", root)]
+            .into_iter()
+            .filter_map(|(k, v)| v.map(|v| format!("{k}={v}")))
+            .collect::<Vec<_>>()
+            .join("&");
+        let resp = client
+            .post(format!(
+                "http://localhost:{}/api/w/test-workspace/jobs/run/p/u/test-user-3/probe?{query}",
+                server.addr.port()
+            ))
+            .bearer_auth(token)
+            .json(&serde_json::json!({}))
+            .send()
+            .await?;
+        let status = resp.status();
+        let body = resp.text().await?;
+        assert_eq!(status, StatusCode::CREATED, "{query}: {body}");
+        let (stored_parent, stored_root): (Option<Uuid>, Option<Uuid>) =
+            sqlx::query_as("SELECT parent_job, flow_innermost_root_job FROM v2_job WHERE id = $1")
+                .bind(Uuid::parse_str(&body)?)
+                .fetch_one(&db)
+                .await?;
+        let expected = |id: Option<&str>| id.map(Uuid::parse_str).transpose();
+        assert_eq!(
+            stored_parent,
+            expected(kept_parent)?,
+            "parent_job for {query}"
+        );
+        assert_eq!(stored_root, expected(kept_root)?, "root_job for {query}");
+    }
+    Ok(())
+}

backend/tests/wac_approval_urls.rs

@@ -201,7 +201,10 @@ async fn wac_approval_urls_mint_guards(db: Pool<Postgres>) -> anyhow::Result<()>
     let client = reqwest::Client::new();
     let mint = |ws: &str, job: &str, key: &str| {
         let url = format!("{base}/w/{ws}/jobs/wac_approval_urls/{job}/{key}");
-        client.get(url).header("Authorization", "Bearer SECRET_TOKEN").send()
+        client
+            .get(url)
+            .header("Authorization", "Bearer SECRET_TOKEN")
+            .send()
     };
 
     assert_eq!(
@@ -227,8 +230,15 @@ async fn wac_approval_urls_mint_guards(db: Pool<Postgres>) -> anyhow::Result<()>
     let resp = mint("test-workspace", WAC_JOB, APPROVAL_COLLISION_B).await?;
     let status = resp.status();
     let body = resp.text().await.unwrap_or_default();
-    assert_eq!(status, reqwest::StatusCode::BAD_REQUEST, "colliding key: {body}");
-    assert!(body.contains(APPROVAL_COLLISION_A), "error must name the other key: {body}");
+    assert_eq!(
+        status,
+        reqwest::StatusCode::BAD_REQUEST,
+        "colliding key: {body}"
+    );
+    assert!(
+        body.contains(APPROVAL_COLLISION_A),
+        "error must name the other key: {body}"
+    );
 
     Ok(())
 }
@@ -236,7 +246,9 @@ async fn wac_approval_urls_mint_guards(db: Pool<Postgres>) -> anyhow::Result<()>
 /// Colliding keys share one resume_job row and one capability, so the mint that
 /// records them must let exactly one through however the requests interleave.
 #[sqlx::test(fixtures("base", "wac_approval_urls"))]
-async fn wac_concurrent_colliding_mints_admit_exactly_one(db: Pool<Postgres>) -> anyhow::Result<()> {
+async fn wac_concurrent_colliding_mints_admit_exactly_one(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
     initialize_tracing().await;
     let server = ApiServer::start(db.clone()).await?;
     let base = format!("http://localhost:{}/api", server.addr.port());
@@ -262,6 +274,92 @@ async fn wac_concurrent_colliding_mints_admit_exactly_one(db: Pool<Postgres>) ->
             other => anyhow::bail!("unexpected status {other}"),
         }
     }
-    assert_eq!((ok, rejected), (1, 1), "exactly one colliding key may be minted");
+    assert_eq!(
+        (ok, rejected),
+        (1, 1),
+        "exactly one colliding key may be minted"
+    );
+    Ok(())
+}
+
+/// A resume signature skips the step's approval_conditions, so only the suspended job's own
+/// run (its `WM_TOKEN`) or a workspace admin may mint one, and a signature made with another
+/// workspace's key must not reach a job outside that workspace.
+#[sqlx::test(fixtures("base", "wac_approval_urls"))]
+async fn resume_signatures_are_bound_to_the_run_and_its_workspace(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    use hmac::Mac;
+    initialize_tracing().await;
+
+    let server = ApiServer::start(db.clone()).await?;
+    let base = format!("http://localhost:{}/api", server.addr.port());
+    let client = reqwest::Client::new();
+    set_jwt_secret().await;
+    let job_token = windmill_common::auth::create_token_for_owner(
+        &db,
+        "test-workspace",
+        "u/test-user-2",
+        "ephemeral-script",
+        300,
+        "test2@windmill.dev",
+        &uuid::Uuid::parse_str(WAC_JOB)?,
+        None,
+        None,
+    )
+    .await?;
+
+    for route in ["job_signature", "resume_urls"] {
+        for (token, allowed) in [
+            ("SECRET_TOKEN_2", false),
+            (&job_token, true),
+            ("SECRET_TOKEN", true),
+        ] {
+            let status = client
+                .get(format!("{base}/w/test-workspace/jobs/{route}/{WAC_JOB}/7"))
+                .bearer_auth(token)
+                .send()
+                .await?
+                .status();
+            assert_eq!(
+                status.is_success(),
+                allowed,
+                "{route} with a {} token: {status}",
+                if token == job_token { "job" } else { token }
+            );
+        }
+    }
+
+    let mut mac = hmac::Hmac::<sha2::Sha256>::new_from_slice(b"test-key-2")?;
+    mac.update(uuid::Uuid::parse_str(WAC_JOB)?.as_bytes());
+    mac.update(&5u32.to_be_bytes());
+    let foreign = hex::encode(mac.finalize().into_bytes());
+    for op in ["resume", "cancel"] {
+        let status = client
+            .post(format!(
+                "{base}/w/test-workspace-2/jobs_u/{op}/{WAC_JOB}/5/{foreign}"
+            ))
+            .json(&serde_json::json!({}))
+            .send()
+            .await?
+            .status();
+        assert_eq!(
+            status,
+            reqwest::StatusCode::NOT_FOUND,
+            "{op} from test-workspace-2"
+        );
+    }
+    let (suspend, resumes): (i32, i64) = sqlx::query_as(
+        "SELECT suspend, (SELECT count(*) FROM resume_job WHERE job = $1::uuid)
+         FROM v2_job_queue WHERE id = $1::uuid",
+    )
+    .bind(WAC_JOB)
+    .fetch_one(&db)
+    .await?;
+    assert_eq!(
+        (suspend, resumes),
+        (1, 0),
+        "the job must still be suspended"
+    );
     Ok(())
 }

backend/tests/worker_ping_ip.rs

@@ -1,5 +1,9 @@
 use sqlx::{Pool, Postgres};
-use windmill_common::{external_ip::UNKNOWN_IP, worker::insert_ping_query};
+use uuid::Uuid;
+use windmill_common::{
+    external_ip::UNKNOWN_IP,
+    worker::{insert_ping_query, update_worker_ping_main_loop_query},
+};
 
 async fn insert_ping(db: &Pool<Postgres>, worker: &str, ip: Option<&str>) -> anyhow::Result<()> {
     insert_ping_query(
@@ -42,3 +46,47 @@ async fn unresolved_ip_keeps_the_reclaimed_rows_address(db: Pool<Postgres>) -> a
     assert_eq!(ip, UNKNOWN_IP);
     Ok(())
 }
+
+async fn main_loop_ping(
+    db: &Pool<Postgres>,
+    worker: &str,
+    last_job: Option<(Uuid, &str)>,
+) -> anyhow::Result<()> {
+    update_worker_ping_main_loop_query(
+        worker,
+        &[],
+        None,
+        None,
+        Some(1),
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        false,
+        None,
+        last_job,
+        db,
+    )
+    .await
+}
+
+/// The main-loop ping sends a job only once, so most pings carry none. Those must keep the last
+/// job the row holds, whether this ping or the job poller wrote it.
+#[sqlx::test]
+async fn main_loop_ping_without_a_job_keeps_the_last_one(db: Pool<Postgres>) -> anyhow::Result<()> {
+    insert_ping(&db, "wk-last-job", None).await?;
+    let job_id = Uuid::new_v4();
+    main_loop_ping(&db, "wk-last-job", Some((job_id, "admins"))).await?;
+    main_loop_ping(&db, "wk-last-job", None).await?;
+    let (current_job_id, workspace_id): (Option<Uuid>, Option<String>) = sqlx::query_as(
+        "SELECT current_job_id, current_job_workspace_id FROM worker_ping WHERE worker = $1",
+    )
+    .bind("wk-last-job")
+    .fetch_one(&db)
+    .await?;
+    assert_eq!(current_job_id, Some(job_id));
+    assert_eq!(workspace_id.as_deref(), Some("admins"));
+    Ok(())
+}

backend/windmill-api-flow-conversations/src/lib.rs

@@ -14,7 +14,7 @@ pub use windmill_common::flow_conversations::FlowConversation;
 use windmill_common::{
     db::{UserDB, DB},
     error::{JsonResult, Result},
-    flow_conversations::MessageType,
+    flow_conversations::{running_turns, MessageType, RunningTurn},
     utils::{not_found_if_none, paginate, truncate_with_ellipsis, Pagination},
 };
 
@@ -81,7 +81,7 @@ async fn list_conversations(
     Path(w_id): Path<String>,
     Query(pagination): Query<Pagination>,
     Query(query): Query<ListConversationsQuery>,
-) -> JsonResult<Vec<FlowConversation>> {
+) -> JsonResult<Vec<ListedConversation>> {
     let (per_page, offset) = paginate(pagination);
     let mut tx = user_db.clone().begin(&authed).await?;
 
@@ -123,9 +123,27 @@ async fn list_conversations(
     let conversations = sqlx::query_as::<Postgres, FlowConversation>(&sql)
         .fetch_all(&mut *tx)
         .await?;
+    let ids: Vec<Uuid> = conversations.iter().map(|c| c.id).collect();
+    let mut running = running_turns(&mut *tx, &ids).await?;
 
     tx.commit().await?;
-    Ok(Json(conversations))
+    Ok(Json(
+        conversations
+            .into_iter()
+            .map(|conversation| ListedConversation {
+                running_turn: running.remove(&conversation.id),
+                conversation,
+            })
+            .collect(),
+    ))
+}
+
+#[derive(Serialize)]
+pub struct ListedConversation {
+    #[serde(flatten)]
+    pub conversation: FlowConversation,
+    /// Lets a chat that opens after a turn started follow it, and tell which chats are busy.
+    pub running_turn: Option<RunningTurn>,
 }
 
 async fn delete_conversation(

backend/windmill-api-integration-tests/tests/datatable_roles.rs

@@ -391,7 +391,11 @@ async fn concurrent_role_creations_both_survive(db: Pool<Postgres>) -> anyhow::R
         assert_eq!(a.0, 200, "{}", a.1);
         assert_eq!(b.0, 200, "{}", b.1);
 
-        let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
+        let catalog = windmill_common::datatable_roles::read_role_catalog(
+            &db,
+            windmill_common::datatable_roles::DatatableRoleCluster::Instance,
+        )
+        .await?;
         let recorded: Vec<&str> = catalog.values().map(|r| r.name.as_str()).collect();
         for name in &names {
             assert!(
@@ -460,7 +464,11 @@ async fn a_role_delete_that_fails_part_way_leaves_the_role_disabled(
         let body = resp.text().await?;
         assert_eq!(status, 400, "{body}");
 
-        let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
+        let catalog = windmill_common::datatable_roles::read_role_catalog(
+            &db,
+            windmill_common::datatable_roles::DatatableRoleCluster::Instance,
+        )
+        .await?;
         let role = catalog
             .get(&id)
             .expect("a failed delete keeps the entry to retry");

backend/windmill-api-integration-tests/tests/fork_background.rs

@@ -0,0 +1,136 @@
+use serde_json::{json, Value};
+use sqlx::{Pool, Postgres};
+
+use windmill_test_utils::*;
+
+async fn wait_for_fork(client: &reqwest::Client, base_url: &str, creation_id: &str) -> Value {
+    for _ in 0..100 {
+        let status: Value = client
+            .get(format!("{base_url}/fork_creation_status/{creation_id}"))
+            .header("Authorization", "Bearer SECRET_TOKEN")
+            .send()
+            .await
+            .unwrap()
+            .json()
+            .await
+            .unwrap();
+        if status["status"] != "running" {
+            return status;
+        }
+        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
+    }
+    panic!("fork creation {creation_id} still running");
+}
+
+/// A fork created in the background is reported by `fork_creation_status`, including a failure
+/// that happens after the request has already returned.
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_fork_created_in_background(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let base_url = format!(
+        "http://localhost:{}/api/w/test-workspace/workspaces",
+        server.addr.port()
+    );
+    let client = reqwest::Client::new();
+    let create = |id: &str| {
+        client
+            .post(format!("{base_url}/create_fork?background=true"))
+            .header("Authorization", "Bearer SECRET_TOKEN")
+            .json(&json!({ "id": id, "name": id }))
+            .send()
+    };
+
+    let resp = create("wm-fork-bg-ok").await?;
+    assert_eq!(resp.status(), 200);
+    let ok_creation = resp.text().await?;
+    let status = wait_for_fork(&client, &base_url, &ok_creation).await;
+    assert_eq!(status, json!({ "status": "completed" }));
+    // Nobody but the user who started a creation reads it.
+    let resp = client
+        .get(format!("{base_url}/fork_creation_status/{ok_creation}"))
+        .header("Authorization", "Bearer SECRET_TOKEN_2")
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 404);
+    let exists: bool =
+        sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM workspace WHERE id = 'wm-fork-bg-ok')")
+            .fetch_one(&db)
+            .await?;
+    assert!(exists);
+    // Under CE's workspace cap.
+    sqlx::query("UPDATE workspace SET deleted = true WHERE id = 'wm-fork-bg-ok'")
+        .execute(&db)
+        .await?;
+
+    // Hold the fork's write lock so the request is validated and returns, then commit a workspace
+    // with the same id: the background write fails on it.
+    let mut blocker = db.begin().await?;
+    sqlx::query("SELECT pg_advisory_xact_lock(hashtext('fork:wm-fork-bg-err'))")
+        .execute(&mut *blocker)
+        .await?;
+    let resp = create("wm-fork-bg-err").await?;
+    assert_eq!(resp.status(), 200);
+    let err_creation = resp.text().await?;
+    let resp = create("wm-fork-bg-err").await?;
+    assert_eq!(resp.status(), 400, "a second creation while the first is in flight");
+    sqlx::query("INSERT INTO workspace (id, name, owner) VALUES ('wm-fork-bg-err', 'x', 'x')")
+        .execute(&mut *blocker)
+        .await?;
+    blocker.commit().await?;
+    let status = wait_for_fork(&client, &base_url, &err_creation).await;
+    assert_eq!(status["status"], "failed");
+    assert!(
+        status["error"].as_str().unwrap().contains("workspace_pkey"),
+        "{status}"
+    );
+
+    // A retry of the same id is a new attempt: the failed one does not read its outcome.
+    sqlx::query("DELETE FROM workspace WHERE id = 'wm-fork-bg-err'")
+        .execute(&db)
+        .await?;
+    let resp = create("wm-fork-bg-err").await?;
+    assert_eq!(resp.status(), 200);
+    let retry_creation = resp.text().await?;
+    assert_eq!(
+        wait_for_fork(&client, &base_url, &retry_creation).await,
+        json!({ "status": "completed" })
+    );
+    let resp = client
+        .get(format!("{base_url}/fork_creation_status/{err_creation}"))
+        .header("Authorization", "Bearer SECRET_TOKEN")
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 404);
+
+    // An attempt whose server stopped heartbeating before its fork committed reads as failed.
+    let abandoned = "5c3e1b1e-0000-4000-8000-000000000000";
+    sqlx::query(
+        "INSERT INTO workspace_fork_creation
+             (fork_workspace_id, parent_workspace_id, created_by, creation_id, heartbeat_at)
+         VALUES ('wm-fork-gone', 'test-workspace', 'test@windmill.dev', $1::uuid,
+                 now() - interval '2 minutes')",
+    )
+    .bind(abandoned)
+    .execute(&db)
+    .await?;
+    assert_eq!(
+        wait_for_fork(&client, &base_url, abandoned).await["status"],
+        "failed"
+    );
+
+    // Without the flag the fork is created before the answer, which clients asking for a
+    // background fork recognise a server without it by.
+    sqlx::query("UPDATE workspace SET deleted = true WHERE id = 'wm-fork-bg-err'")
+        .execute(&db)
+        .await?;
+    let resp = client
+        .post(format!("{base_url}/create_fork"))
+        .header("Authorization", "Bearer SECRET_TOKEN")
+        .json(&json!({ "id": "wm-fork-sync", "name": "wm-fork-sync" }))
+        .send()
+        .await?;
+    assert_eq!(resp.text().await?, "Created forked workspace wm-fork-sync");
+
+    Ok(())
+}

backend/windmill-api-integration-tests/tests/resources.rs

@@ -566,6 +566,113 @@ async fn test_resource_value_cache_is_identity_scoped(db: Pool<Postgres>) -> any
     Ok(())
 }
 
+/// A token scoped to one resource must not read, through the `$var:`/`$res:` references it
+/// writes into that resource, what its scopes would refuse to read directly.
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_interpolated_references_need_the_token_scopes(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    initialize_tracing().await;
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let base = format!("http://localhost:{port}/api/w/test-workspace");
+
+    let resp = authed(client().post(format!("{base}/variables/create")))
+        .json(&json!({"path": "u/test-user/secret", "value": "CANARY", "is_secret": true, "description": ""}))
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 201);
+    let resp = authed(client().post(format!("{base}/resources/create")))
+        .json(&json!({"path": "u/test-user/other", "value": {"pw": "OTHER"}, "resource_type": "object", "description": ""}))
+        .send()
+        .await?;
+    assert_eq!(resp.status(), 201);
+
+    let mint = |scopes: serde_json::Value| async move {
+        let resp =
+            authed(client().post(format!("http://localhost:{port}/api/users/tokens/create")))
+                .json(
+                    &json!({"label": "scoped", "scopes": scopes, "workspace_id": "test-workspace"}),
+                )
+                .send()
+                .await
+                .unwrap();
+        assert_eq!(resp.status(), 201);
+        resp.text().await.unwrap()
+    };
+    let probe_only = mint(json!(["resources:write:u/test-user/probe"])).await;
+    let with_var = mint(json!([
+        "resources:read:u/test-user/probe",
+        "variables:read:u/test-user/secret"
+    ]))
+    .await;
+
+    let read_probe = |value: serde_json::Value, token: String| {
+        let base = base.clone();
+        let probe_only = probe_only.clone();
+        async move {
+            let resp = client()
+                .post(format!("{base}/resources/create?update_if_exists=true"))
+                .bearer_auth(&probe_only)
+                .json(&json!({"path": "u/test-user/probe", "value": value, "resource_type": "object", "description": ""}))
+                .send()
+                .await
+                .unwrap();
+            assert_eq!(resp.status(), 201);
+            let resp = client()
+                .get(format!(
+                    "{base}/resources/get_value_interpolated/u/test-user/probe"
+                ))
+                .bearer_auth(&token)
+                .send()
+                .await
+                .unwrap();
+            (resp.status().as_u16(), resp.text().await.unwrap())
+        }
+    };
+
+    for value in [
+        json!({"v": "$var:u/test-user/secret"}),
+        json!({"v": "$jsonvar:u/test-user/secret"}),
+        json!({"v": ["$res:u/test-user/other"]}),
+    ] {
+        let (status, body) = read_probe(value.clone(), probe_only.clone()).await;
+        assert_eq!(status, 403, "{value}: {body}");
+        assert!(
+            !body.contains("CANARY") && !body.contains("OTHER"),
+            "{body}"
+        );
+    }
+
+    let (status, body) =
+        read_probe(json!({"v": "$var:u/test-user/secret"}), with_var.clone()).await;
+    assert_eq!((status, body.as_str()), (200, r#"{"v":"CANARY"}"#));
+
+    // The resource's own linked secrets (at its path, or `<path>_<field>`) are part of it.
+    for (path, value) in [
+        ("u/test-user/probe", "LINKED"),
+        ("u/test-user/probe_key", "KEY"),
+    ] {
+        let resp = authed(client().post(format!("{base}/variables/create")))
+            .json(&json!({"path": path, "value": value, "is_secret": true, "description": ""}))
+            .send()
+            .await?;
+        assert_eq!(resp.status(), 201);
+    }
+    let (status, body) = read_probe(
+        json!({"pw": "$var:u/test-user/probe", "key": "$var:u/test-user/probe_key"}),
+        probe_only.clone(),
+    )
+    .await;
+    assert_eq!(status, 200, "{body}");
+    assert_eq!(
+        serde_json::from_str::<serde_json::Value>(&body)?,
+        json!({"pw": "LINKED", "key": "KEY"})
+    );
+
+    Ok(())
+}
+
 /// A resource whose value contains a `$WM_*` contextual variable (e.g. `$WM_TOKEN`) is
 /// job-dependent and must NEVER be cached — even when first read WITHOUT a `job_id`, where the
 /// placeholder is left unresolved (caching that would serve a stale placeholder to a later job

backend/windmill-api-integration-tests/tests/workspace_dependencies.rs

@@ -0,0 +1,57 @@
+use serde_json::json;
+use sqlx::{Pool, Postgres};
+
+#[allow(unused_imports)]
+use windmill_test_utils::*;
+
+/// An admin of one workspace must not be able to write dependency files into another
+/// by naming it in the request body.
+#[sqlx::test(migrations = "../migrations", fixtures("base"))]
+async fn test_create_rejects_body_workspace_other_than_path(
+    db: Pool<Postgres>,
+) -> anyhow::Result<()> {
+    initialize_tracing().await;
+
+    // test-user-2 is a plain user of test-workspace and an admin of its own workspace.
+    sqlx::query("INSERT INTO workspace (id, name, owner) VALUES ('own-workspace', 'own-workspace', 'test-user-2')")
+        .execute(&db)
+        .await?;
+    sqlx::query("INSERT INTO usr (workspace_id, email, username, is_admin, role) VALUES ('own-workspace', 'test2@windmill.dev', 'test-user-2', true, 'Admin')")
+        .execute(&db)
+        .await?;
+
+    let (_client, port, _server) = init_client(db.clone()).await;
+
+    let resp = reqwest::Client::new()
+        .post(format!(
+            "http://localhost:{port}/api/w/own-workspace/workspace_dependencies/create"
+        ))
+        .header("Authorization", "Bearer SECRET_TOKEN_2")
+        .json(&json!({
+            "workspace_id": "test-workspace",
+            "language": "python3",
+            "content": "requests==2.28.0"
+        }))
+        .send()
+        .await?;
+    let status = resp.status().as_u16();
+    let body = resp.text().await?;
+    assert_eq!(
+        status, 400,
+        "expected the mismatched body to be rejected, got {body}"
+    );
+    assert!(
+        body.contains("does not match"),
+        "expected the workspace mismatch rejection, got {body}"
+    );
+
+    let written: i64 = sqlx::query_scalar("SELECT count(*) FROM workspace_dependencies")
+        .fetch_one(&db)
+        .await?;
+    assert_eq!(
+        written, 0,
+        "no dependency file may be written in either workspace"
+    );
+
+    Ok(())
+}

backend/windmill-api-jobs/src/execution.rs

@@ -95,6 +95,83 @@ pub async fn check_tag_as_written_available_for_workspace(
     }
 }
 
+/// Clears a `parent_job` / `root_job` the caller cannot claim. They identify the pushed job
+/// (`WM_FLOW_JOB_ID`, `WM_ROOT_FLOW_JOB_ID`, the OIDC token's flow path), so only a job's own
+/// `WM_TOKEN` may name that job or its ancestors, and a workspace admin any job of the workspace.
+/// Cleared rather than refused: the SDKs send them from inside a job whatever token they hold.
+pub async fn drop_unclaimable_run_lineage(
+    db: &DB,
+    w_id: &str,
+    run_query: &mut RunJobQuery,
+    authed: &ApiAuthed,
+) -> error::Result<()> {
+    let referenced: Vec<Uuid> = [run_query.parent_job, run_query.root_job]
+        .into_iter()
+        .flatten()
+        .collect();
+    let unclaimable = unclaimable_run_lineage(db, w_id, referenced, authed).await?;
+    for field in [&mut run_query.parent_job, &mut run_query.root_job] {
+        if field.is_some_and(|id| unclaimable.contains(&id)) {
+            tracing::warn!(
+                "ignoring parent_job/root_job {field:?} that {} cannot claim in {w_id}",
+                authed.username
+            );
+            *field = None;
+        }
+    }
+    Ok(())
+}
+
+/// The jobs of `referenced` that `authed` cannot claim as its own run lineage: anything but the
+/// token's own job and that job's `parent_job`, `root_job` and `flow_innermost_root_job`, or for
+/// a workspace admin anything outside the workspace.
+pub async fn unclaimable_run_lineage(
+    db: &DB,
+    w_id: &str,
+    mut referenced: Vec<Uuid>,
+    authed: &ApiAuthed,
+) -> error::Result<Vec<Uuid>> {
+    referenced.sort();
+    referenced.dedup();
+    if let Some(token_job) = authed.job_id {
+        referenced.retain(|id| *id != token_job);
+        if !referenced.is_empty() {
+            if let Some(lineage) = sqlx::query!(
+                "SELECT parent_job, root_job, flow_innermost_root_job FROM v2_job
+                WHERE id = $1 AND workspace_id = $2",
+                token_job,
+                w_id
+            )
+            .fetch_optional(db)
+            .await?
+            {
+                let ancestors = [
+                    lineage.parent_job,
+                    lineage.root_job,
+                    lineage.flow_innermost_root_job,
+                ];
+                referenced.retain(|id| !ancestors.contains(&Some(*id)));
+            }
+        }
+    }
+    if referenced.is_empty() {
+        return Ok(referenced);
+    }
+    let in_workspace = if authed.is_admin {
+        sqlx::query_scalar!(
+            "SELECT id FROM v2_job WHERE id = ANY($1) AND workspace_id = $2",
+            &referenced,
+            w_id,
+        )
+        .fetch_all(db)
+        .await?
+    } else {
+        vec![]
+    };
+    referenced.retain(|id| !in_workspace.contains(id));
+    Ok(referenced)
+}
+
 #[cfg(feature = "enterprise")]
 pub async fn check_license_key_valid() -> error::Result<()> {
     use windmill_common::ee_oss::LICENSE_KEY_VALID;
@@ -776,6 +853,8 @@ pub async fn run_flow<'c>(
     bool,
     Option<sqlx::Transaction<'c, sqlx::Postgres>>,
 )> {
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(db, w_id, &mut run_query, authed).await?;
     let on_behalf_of = flow_version_info.on_behalf_of(w_id, &db).await?;
     let FlowVersionInfo {
         version,
@@ -1017,6 +1096,8 @@ pub async fn push_script_job_by_path_into_queue<'c>(
 
     let script_path = script_path.to_path();
     check_scopes(&authed, || format!("jobs:run:scripts:{script_path}"))?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let userdb_authed = UserDbWithAuthed { db: user_db.clone(), authed: &authed.to_authed_ref() };
     let (job_payload, tag, delete_after_use, delete_after_secs, timeout, on_behalf_of) =

backend/windmill-api-schedule/src/lib.rs

@@ -409,7 +409,8 @@ async fn create_schedule(
             paused_until,
             cron_version,
             dynamic_skip,
-            labels
+            labels,
+            late_run_streak
         "#,
         w_id,
         ns.path,
@@ -621,7 +622,10 @@ async fn edit_schedule(
             edited_by               = $25,
             edited_at               = now(),
             permissioned_as         = $26,
-            labels                  = COALESCE($27, labels)
+            labels                  = COALESCE($27, labels),
+            late_run_streak            = 0,
+            missed_occurrences     = 0,
+            last_missed_at              = NULL
         WHERE path = $19 AND workspace_id = $20
         RETURNING
             workspace_id,
@@ -656,7 +660,8 @@ async fn edit_schedule(
             paused_until,
             cron_version,
             dynamic_skip,
-            labels
+            labels,
+            late_run_streak
         "#,
         es.schedule,
         es.timezone,
@@ -1004,6 +1009,9 @@ async fn list_schedule(
 pub struct ScheduleWJobs {
     pub path: String,
     pub jobs: Option<Vec<serde_json::Value>>,
+    pub late_run_streak: i32,
+    pub missed_occurrences: i32,
+    pub last_missed_at: Option<chrono::DateTime<chrono::Utc>>,
 }
 
 async fn list_schedule_with_jobs(
@@ -1020,7 +1028,7 @@ async fn list_schedule_with_jobs(
         // - use of the `ix_v2_job_root_by_path` index; hence the `parent_job IS NULL` clause.
         // - both `workspace_id = $1` checks are required to hit both indexes.
         "SELECT
-            schedule.path, t.jobs FROM schedule,
+            schedule.path, t.jobs, schedule.late_run_streak, schedule.missed_occurrences, schedule.last_missed_at FROM schedule,
             LATERAL(SELECT ARRAY(
                 SELECT json_build_object('id', id, 'success', status = 'success', 'duration_ms', duration_ms)
                 FROM v2_job_completed c JOIN v2_job j USING (id)
@@ -1157,7 +1165,10 @@ pub async fn set_enabled(
         r#"
         UPDATE schedule SET
             enabled = $1,
-            email = $2
+            email = $2,
+            late_run_streak = 0,
+            missed_occurrences = 0,
+            last_missed_at = NULL
         WHERE path = $3 AND workspace_id = $4
         RETURNING
             workspace_id,
@@ -1192,7 +1203,8 @@ pub async fn set_enabled(
             paused_until,
             cron_version,
             dynamic_skip,
-            labels
+            labels,
+            late_run_streak
         "#,
         payload.enabled,
         authed.email,

backend/windmill-api-settings/src/lib.rs

@@ -42,7 +42,6 @@ use axum::{
     routing::{get, post},
     Json, Router,
 };
-use serde_json::json;
 
 use serde::{Deserialize, Serialize};
 use windmill_ai::ai_cache::bump_instance_ai_config_revision;
@@ -61,6 +60,7 @@ use windmill_common::{
         ACCENT_COLOR_SETTING, AI_CONFIG_SETTING, APP_WORKSPACED_ROUTE_SETTING,
         AUTOMATE_USERNAME_CREATION_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING, CUSTOM_TAGS_SETTING,
         DEFAULT_TAGS_WORKSPACES_SETTING, DISABLE_HUB_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS,
+        EXTERNAL_INSTANCE_PG_SETTING,
         GITHUB_APP_WEBHOOK_BASE_URL_SETTING, HTTP_ROUTE_DEFAULT_ALLOWED_ORIGINS_SETTING,
         HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING,
         INSTANCE_BANNER_SETTING, MAX_RETENTION_OVERRIDE_WORKSPACES,
@@ -181,6 +181,22 @@ pub fn global_service() -> Router {
             "/refresh_custom_instance_user_pwd",
             post(refresh_custom_instance_user_pwd),
         )
+        .route(
+            "/external_instance_pg/status",
+            get(get_external_instance_pg_status),
+        )
+        .route(
+            "/external_instance_pg/setup",
+            post(setup_external_instance_pg),
+        )
+        .route(
+            "/external_instance_pg/databases",
+            get(list_external_instance_pg_databases),
+        )
+        .route(
+            "/external_instance_pg/databases/{name}",
+            post(create_external_instance_pg_database).delete(drop_external_instance_pg_database),
+        )
         .route(
             "/setup_custom_instance_pg_database/{name}",
             post(setup_custom_instance_pg_database),
@@ -303,7 +319,9 @@ pub async fn test_email(
 use windmill_object_store::ObjectSettings;
 
 #[cfg(feature = "parquet")]
-use windmill_object_store::build_object_store_from_settings;
+use windmill_object_store::{
+    build_object_store_from_settings, build_public_object_store_from_settings,
+};
 
 #[cfg(feature = "parquet")]
 pub async fn test_s3_bucket(
@@ -338,9 +356,15 @@ pub async fn test_s3_bucket(
             })?;
     }
 
-    let client = build_object_store_from_settings(test_s3_bucket, Some(&db))
-        .await?
-        .store;
+    // The restricted client re-judges every address it connects to: the checks above resolve the
+    // endpoint separately from the connect, which a rebinding name answers differently.
+    let client = if restrict {
+        build_public_object_store_from_settings(test_s3_bucket).await?
+    } else {
+        build_object_store_from_settings(test_s3_bucket, Some(&db))
+            .await?
+            .store
+    };
 
     let run = async {
         let mut list = client.list(Some(
@@ -546,7 +570,7 @@ async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> {
     // Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding
     // attempts (a name resolving to both a public and a private address).
     for addr in addrs {
-        if is_forbidden_ip(addr.ip()) {
+        if windmill_common::ssrf::is_private_ip(&addr.ip()) {
             // The resolved address stays out of the message: it is the server's resolver's
             // answer, and this message is only ever shown to the caller being constrained.
             return Err(error::Error::NotAuthorized(format!(
@@ -599,51 +623,6 @@ fn extract_host(endpoint: &str) -> Option<String> {
     }
 }
 
-#[cfg(feature = "parquet")]
-fn is_forbidden_ip(ip: std::net::IpAddr) -> bool {
-    use std::net::{IpAddr, Ipv4Addr};
-    match ip {
-        IpAddr::V4(v4) => {
-            v4.is_loopback()
-                || v4.is_private()
-                || v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint
-                || v4.is_unspecified()
-                || v4.is_broadcast()
-                || v4.is_documentation()
-                || v4.is_multicast()
-                || v4.octets()[0] == 0 // 0.0.0.0/8
-                || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT
-        }
-        IpAddr::V6(v6) => {
-            // Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible
-            // ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g.
-            // 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network.
-            let seg = v6.segments();
-            let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0];
-            let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0];
-            if let Some(v4) = v6.to_ipv4_mapped() {
-                return is_forbidden_ip(IpAddr::V4(v4));
-            }
-            if is_v4_compatible || is_nat64 {
-                let embedded = Ipv4Addr::new(
-                    (seg[6] >> 8) as u8,
-                    (seg[6] & 0xff) as u8,
-                    (seg[7] >> 8) as u8,
-                    (seg[7] & 0xff) as u8,
-                );
-                if is_forbidden_ip(IpAddr::V4(embedded)) {
-                    return true;
-                }
-            }
-            v6.is_loopback()
-                || v6.is_unspecified()
-                || v6.is_multicast()
-                || (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local
-                || (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local
-        }
-    }
-}
-
 #[cfg(feature = "parquet")]
 async fn get_object_storage_usage(
     Extension(db): Extension<DB>,
@@ -890,6 +869,14 @@ pub async fn set_global_setting_internal(
         )));
     }
 
+    if key == EXTERNAL_INSTANCE_PG_SETTING {
+        return windmill_common::external_instance_pg::write_external_instance_pg_setting(
+            db,
+            Some(&value),
+        )
+        .await;
+    }
+
     run_setting_pre_write_hook(db, &key, &value).await?;
 
     match value {
@@ -1285,7 +1272,7 @@ async fn set_instance_config(
     let desired_map = desired.global_settings.to_settings_map();
     if !desired_map.is_empty() {
         let current_map = current.global_settings.to_settings_map();
-        let settings_diff =
+        let mut settings_diff =
             instance_config::diff_global_settings(&current_map, &desired_map, ApplyMode::Merge);
         let ai_config_changed = settings_diff
             .upserts
@@ -1314,8 +1301,15 @@ async fn set_instance_config(
         }
 
         for (key, value) in &settings_diff.upserts {
-            run_setting_pre_write_hook(&db, key, value).await?;
+            if key != EXTERNAL_INSTANCE_PG_SETTING {
+                run_setting_pre_write_hook(&db, key, value).await?;
+            }
         }
+        windmill_common::external_instance_pg::write_external_instance_pg_from_diff(
+            &db,
+            &mut settings_diff,
+        )
+        .await?;
 
         instance_config::apply_settings_diff(&db, &settings_diff)
             .await
@@ -1680,6 +1674,8 @@ struct CustomInstanceDb {
     tag: Option<String>,
     #[serde(default, skip_serializing_if = "Vec::is_empty")]
     used_by_workspaces: Vec<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    workspace_id: Option<String>,
 }
 
 #[derive(Deserialize, Debug, Serialize, Default)]
@@ -1722,7 +1718,40 @@ async fn list_custom_instance_pg_databases(
             ))
         })?;
 
-    if windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
+    if !windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
+        // A fork copy's name gives away the workspace it was reserved for, so every pending fork on
+        // the instance would be listed. Kept for members of that workspace, and wherever the
+        // caller's workspaces use it, e.g. the fork it was finalized into.
+        let reserved_visible: BTreeSet<String> = sqlx::query_scalar(
+            r#"SELECT e.k FROM global_settings gs
+               CROSS JOIN LATERAL jsonb_each(gs.value->'databases') AS e(k, v)
+               WHERE gs.name = 'custom_instance_pg_databases' AND e.v->>'workspace_id' IS NOT NULL
+                 AND (EXISTS (SELECT 1 FROM usr WHERE usr.email = $1
+                                AND usr.workspace_id = e.v->>'workspace_id')
+                   OR EXISTS (SELECT 1 FROM usr JOIN workspace_settings ws
+                                ON ws.workspace_id = usr.workspace_id
+                              CROSS JOIN LATERAL jsonb_each(
+                                  CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
+                                      THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
+                              WHERE usr.email = $1
+                                AND dt.value->'database'->>'resource_type' = 'instance'
+                                AND dt.value->'database'->>'resource_path' = e.k))"#,
+        )
+        .bind(&authed.email)
+        .fetch_all(&db)
+        .await?
+        .into_iter()
+        .collect();
+        result.retain(|dbname, entry| {
+            entry.workspace_id.is_none() || reserved_visible.contains(dbname)
+        });
+        // Which workspace reserved a copy is still only for superadmins.
+        for entry in result.values_mut() {
+            entry.workspace_id = None;
+        }
+        return Ok(Json(result));
+    }
+    {
         // Enrich each database with the list of workspaces referencing it through
         // either a ducklake catalog or a datatable database whose resource_type is
         // 'instance'. Not stored in DB to avoid drift.
@@ -1778,6 +1807,139 @@ async fn refresh_custom_instance_user_pwd(
     Ok(Json(()))
 }
 
+async fn get_external_instance_pg_status(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+) -> JsonResult<windmill_common::external_instance_pg::ExternalInstancePgStatus> {
+    require_super_admin(&db, &authed).await?;
+    Ok(Json(
+        windmill_common::external_instance_pg::external_instance_pg_status(&db).await?,
+    ))
+}
+
+#[derive(Deserialize)]
+struct SetupExternalInstancePgBody {
+    #[serde(default)]
+    rotate_passwords: bool,
+}
+
+async fn setup_external_instance_pg(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+    Json(body): Json<SetupExternalInstancePgBody>,
+) -> JsonResult<windmill_common::external_instance_pg::ExternalInstancePgSetupReport> {
+    require_super_admin(&db, &authed).await?;
+    let report = windmill_common::external_instance_pg::setup_external_instance_pg_unchecked(
+        &db,
+        body.rotate_passwords,
+    )
+    .await?;
+    let rotated = body.rotate_passwords.to_string();
+    let success = report.success.to_string();
+    windmill_audit::audit_oss::audit_log(
+        &db,
+        &authed,
+        "settings.setup_external_instance_pg",
+        windmill_audit::ActionKind::Update,
+        "global",
+        Some(&authed.email),
+        Some(
+            [
+                ("rotate_passwords", rotated.as_str()),
+                ("success", success.as_str()),
+            ]
+            .into(),
+        ),
+    )
+    .await?;
+    Ok(Json(report))
+}
+
+#[derive(Serialize)]
+struct ExternalInstancePgDatabase {
+    #[serde(flatten)]
+    status: windmill_common::instance_config::CustomInstanceDb,
+    used_by_workspaces: Vec<String>,
+}
+
+async fn list_external_instance_pg_databases(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+) -> JsonResult<std::collections::BTreeMap<String, ExternalInstancePgDatabase>> {
+    require_super_admin(&db, &authed).await?;
+    let databases = windmill_common::external_instance_pg::external_instance_databases(&db).await?;
+    let mut usages =
+        windmill_common::external_instance_pg::external_instance_database_usages(&db).await?;
+    Ok(Json(
+        databases
+            .into_iter()
+            .map(|(name, status)| {
+                let used_by_workspaces = usages.remove(&name).unwrap_or_default();
+                (
+                    name,
+                    ExternalInstancePgDatabase {
+                        status,
+                        used_by_workspaces: used_by_workspaces.into_iter().collect(),
+                    },
+                )
+            })
+            .collect(),
+    ))
+}
+
+async fn create_external_instance_pg_database(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+    Path(dbname): Path<String>,
+    Json(body): Json<SetupCustomInstanceDbBody>,
+) -> JsonResult<()> {
+    require_super_admin(&db, &authed).await?;
+    let tag = body.tag.as_deref().unwrap_or("datatable");
+    let mut tx = db.begin().await?;
+    windmill_common::external_instance_pg::create_external_instance_database_unchecked(
+        &db, &mut tx, &dbname, tag, None,
+    )
+    .await?;
+    tx.commit().await?;
+    windmill_audit::audit_oss::audit_log(
+        &db,
+        &authed,
+        "settings.create_external_instance_pg_database",
+        windmill_audit::ActionKind::Create,
+        "global",
+        Some(&authed.email),
+        Some([("dbname", dbname.as_str()), ("tag", tag)].into()),
+    )
+    .await?;
+    Ok(Json(()))
+}
+
+async fn drop_external_instance_pg_database(
+    authed: ApiAuthed,
+    Extension(db): Extension<DB>,
+    Path(dbname): Path<String>,
+) -> JsonResult<()> {
+    require_super_admin(&db, &authed).await?;
+    // A data table naming a dropped database fails on every job, far from the drop that caused it.
+    let mut tx = db.begin().await?;
+    windmill_common::external_instance_pg::drop_external_instance_database_unchecked(
+        &mut tx, &dbname, None,
+    )
+    .await?;
+    tx.commit().await?;
+    windmill_audit::audit_oss::audit_log(
+        &db,
+        &authed,
+        "settings.drop_external_instance_pg_database",
+        windmill_audit::ActionKind::Delete,
+        "global",
+        Some(&authed.email),
+        Some([("dbname", dbname.as_str())].into()),
+    )
+    .await?;
+    Ok(Json(()))
+}
+
 #[derive(Deserialize)]
 struct SetupCustomInstanceDbBody {
     tag: Option<String>,
@@ -1789,18 +1951,46 @@ async fn setup_custom_instance_pg_database(
     Path(dbname): Path<String>,
     Json(body): Json<SetupCustomInstanceDbBody>,
 ) -> JsonResult<CustomInstanceDb> {
+    // Before anything is recorded: the status written below replaces the registry entry, and with it
+    // the workspace a fork copy is reserved for.
+    require_super_admin(&db, &authed).await?;
+    windmill_common::workspaces::ensure_instance_pg_available(&db).await?;
+    // Fork cleanup checks and drops the database and its entry under this lock. Held from before
+    // the setup creates the database to after its entry is written, neither lands on the other's
+    // half-done state: a dropped database with its entry written back, or the reverse.
+    let mut tx = db.begin().await?;
+    windmill_common::datatable_roles::lock_instance_databases_governance(&mut tx, [dbname.trim()])
+        .await?;
     let mut logs = CustomInstanceDbLogs::default();
     let result = setup_custom_instance_pg_database_inner(authed, &db, &dbname, &mut logs).await;
     let success = result.is_ok();
     let error = result.err().map(|e| e.to_string());
-    let status =
-        CustomInstanceDb { logs, success, error, tag: body.tag, used_by_workspaces: vec![] };
+    let status = CustomInstanceDb {
+        logs,
+        success,
+        error,
+        tag: body.tag,
+        used_by_workspaces: vec![],
+        workspace_id: None,
+    };
     let status_json = serde_json::to_value(&status).map_err(to_anyhow)?;
-    // Save that the database was setup successfully
-    sqlx::query!(
-        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
-        json!({ dbname: status_json })
-    ).execute(&db).await?;
+    // The fork reservation is carried over inside the write, from whatever the row holds then: a
+    // rename migrating it while the setup above ran would otherwise be overwritten with the value
+    // this request started from, stranding the copy under the archived workspace.
+    let saved = sqlx::query_scalar::<_, serde_json::Value>(
+        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}',
+               COALESCE(value->'databases', '{}'::jsonb)
+                   || jsonb_build_object($1::text, $2::jsonb || jsonb_build_object(
+                          'workspace_id', value->'databases'->$1::text->'workspace_id')))
+           WHERE name = 'custom_instance_pg_databases'
+           RETURNING value->'databases'->$1::text"#,
+    )
+    .bind(&dbname)
+    .bind(&status_json)
+    .fetch_one(&mut *tx)
+    .await?;
+    tx.commit().await?;
+    let status: CustomInstanceDb = serde_json::from_value(saved).map_err(to_anyhow)?;
 
     Ok(Json(status))
 }
@@ -2582,8 +2772,7 @@ mod tests {
 
 #[cfg(all(test, feature = "parquet"))]
 mod object_storage_test_hardening {
-    use super::{extract_host, is_forbidden_ip, validate_object_storage_test};
-    use std::net::IpAddr;
+    use super::{extract_host, validate_object_storage_test};
     use windmill_object_store::ObjectSettings;
 
     // IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host`
@@ -2644,40 +2833,6 @@ mod object_storage_test_hardening {
         }
     }
 
-    fn ip(s: &str) -> IpAddr {
-        s.parse().unwrap()
-    }
-
-    #[test]
-    fn forbids_internal_ips() {
-        for s in [
-            "127.0.0.1",                // loopback
-            "169.254.169.254",          // cloud metadata (link-local)
-            "10.0.0.5",                 // private
-            "172.16.3.4",               // private
-            "192.168.1.10",             // private
-            "0.0.0.0",                  // unspecified
-            "100.64.0.1",               // CGNAT
-            "::1",                      // IPv6 loopback
-            "fe80::1",                  // IPv6 link-local
-            "fc00::1",                  // IPv6 unique local
-            "::ffff:127.0.0.1",         // IPv4-mapped loopback
-            "::ffff:169.254.169.254",   // IPv4-mapped metadata
-            "::169.254.169.254",        // IPv4-compatible metadata
-            "64:ff9b::169.254.169.254", // NAT64-embedded metadata
-            "64:ff9b::a9fe:a9fe",       // NAT64-embedded metadata (hex form)
-        ] {
-            assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden");
-        }
-    }
-
-    #[test]
-    fn allows_public_ips() {
-        for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] {
-            assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed");
-        }
-    }
-
     #[test]
     fn extracts_host_from_endpoint() {
         let cases = [

backend/windmill-api-workspaces/Cargo.toml

@@ -50,6 +50,7 @@ sha2.workspace = true
 sqlx.workspace = true
 tokio.workspace = true
 tokio-postgres.workspace = true
+tokio-util.workspace = true
 tracing.workspace = true
 url.workspace = true
 uuid.workspace = true

backend/windmill-api-workspaces/src/datatable_acl.rs

@@ -6,7 +6,7 @@
  * LICENSE-AGPL for a copy of the license.
  */
 
-//! Ownership and grants on the objects of an instance data table.
+//! Ownership and grants on the objects of a data table on a cluster Windmill manages.
 //!
 //! [`datatable_permissions`](crate::datatable_permissions) decides who may connect as which role;
 //! this decides what each role may then touch. Every change is a real `GRANT`, `REVOKE`,
@@ -33,7 +33,7 @@ use windmill_audit::audit_oss::audit_log;
 use windmill_audit::ActionKind;
 use windmill_common::datatable_roles::{
     lock_role_catalog, quote_ident, read_role_catalog, read_role_catalog_tx, DatatableRoleCatalog,
-    ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
+    DatatableRoleCluster, ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
 };
 use windmill_common::error::{pg_error_message, Error, JsonResult, Result};
 use windmill_common::workspaces::{resolve_governing_datatable, DataTable, GoverningDatatable};
@@ -297,22 +297,22 @@ fn role_names(catalog: &DatatableRoleCatalog) -> Vec<String> {
     names
 }
 
-fn ensure_instance(governing: &GoverningDatatable) -> Result<()> {
-    if governing.is_instance() {
-        return Ok(());
-    }
-    Err(Error::BadRequest(format!(
-        "Data table '{}' is backed by a Postgres resource, so its access is managed on that \
-         server directly. Only a data table on the Windmill instance's own database has data \
-         table roles to grant to.",
-        governing.name
-    )))
+/// The cluster whose roles the data table's grants name.
+fn ensure_managed(governing: &GoverningDatatable) -> Result<DatatableRoleCluster> {
+    governing.role_cluster().ok_or_else(|| {
+        Error::BadRequest(format!(
+            "Data table '{}' is backed by a Postgres resource, so its access is managed on that \
+             server directly. Only a data table on a database Windmill manages has data table \
+             roles to grant to.",
+            governing.name
+        ))
+    })
 }
 
 /// The data table's `admin` connection, and the notices Postgres sends on it.
 ///
-/// Authorization: connects as `custom_instance_user` with the instance's own credentials and checks
-/// nothing. Callers MUST have authorized the request first — a request about to be refused must
+/// Authorization: connects as `custom_instance_user` with the cluster's stored credentials and
+/// checks nothing. Callers MUST have authorized the request first — a request about to be refused must
 /// not get as far as this connection.
 async fn connect_as_admin_unchecked(
     db: &DB,
@@ -322,21 +322,33 @@ async fn connect_as_admin_unchecked(
     mpsc::UnboundedReceiver<DbError>,
     String,
 )> {
-    ensure_instance(governing)?;
+    let cluster = ensure_managed(governing)?;
     // Built from the authorized entry, never by resolving the settings again: a save in between
     // could point the entry at a resource on another server and back, and this connection would
     // then alter a database the later checks of the entry never see.
-    let mut pg = PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
-    pg.dbname = governing
+    let dbname = governing
         .datatable
         .database
         .as_ref()
         .expect("a governing entry owns a database")
         .resource_path
         .clone();
-    pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
-    pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
-    let dbname = pg.dbname.clone();
+    let pg = match cluster {
+        DatatableRoleCluster::Instance => {
+            let mut pg =
+                PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
+            pg.dbname = dbname.clone();
+            pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
+            pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
+            pg
+        }
+        DatatableRoleCluster::ExternalInstance => {
+            windmill_common::external_instance_pg::external_instance_connection_unchecked(
+                db, &dbname, false,
+            )
+            .await?
+        }
+    };
     let (client, notices) = connect_with_notices(db, &pg).await?;
     Ok((client, notices, dbname))
 }
@@ -1085,12 +1097,12 @@ async fn get_datatable_acl(
     let target: AclTarget = query.try_into()?;
     let governing = resolve_governing_datatable(&db, &w_id, &datatable_name).await?;
     ensure_reaches_governing_datatable(&db, &w_id, &datatable_name, &governing, &authed).await?;
-    ensure_instance(&governing)?;
+    let cluster = ensure_managed(&governing)?;
     let editable = ensure_governs_datatable(&db, &authed, &w_id, &governing)
         .await
         .is_ok();
     let roles = if editable {
-        role_names(&read_role_catalog(&db).await?)
+        role_names(&read_role_catalog(&db, cluster).await?)
     } else {
         vec![]
     };
@@ -1376,7 +1388,7 @@ async fn authorize_acl_change(
 ) -> Result<GoverningDatatable> {
     let governing = resolve_governing_datatable(db, w_id, datatable_name).await?;
     ensure_governs_datatable(db, authed, w_id, &governing).await?;
-    ensure_instance(&governing)?;
+    ensure_managed(&governing)?;
     Ok(governing)
 }
 
@@ -1386,8 +1398,14 @@ static APPLY_SLOT: tokio::sync::Semaphore = tokio::sync::Semaphore::const_new(1)
 /// provisioned before data table roles gave `custom_instance_user` none. Adds that option to its
 /// database and `public` privileges, and nothing else: default privileges are left alone, since a
 /// schema's change of owner is planned against them. Best-effort, as a grant it fails to enable is
-/// refused when it runs.
-async fn ensure_grant_options(client: &tokio_postgres::Client, db: &DB, dbname: &str) {
+/// refused when it runs. An external instance database was created with the options, so one
+/// missing there is someone's deliberate revoke and is left alone.
+async fn ensure_grant_options(
+    client: &tokio_postgres::Client,
+    db: &DB,
+    cluster: DatatableRoleCluster,
+    dbname: &str,
+) {
     let held = client
         .query_one(
             "SELECT has_database_privilege(current_database(), 'CONNECT WITH GRANT OPTION')
@@ -1399,7 +1417,7 @@ async fn ensure_grant_options(client: &tokio_postgres::Client, db: &DB, dbname:
         )
         .await
         .is_ok_and(|row| row.get::<_, bool>(0));
-    if held {
+    if held || cluster != DatatableRoleCluster::Instance {
         return;
     }
     if let Err(e) = grant_options_as_server(db, dbname).await {
@@ -1625,7 +1643,8 @@ async fn plan_datatable_acl(
 ) -> JsonResult<AclPlan> {
     crate::datatable_acl_oss::ensure_datatable_acl_available()?;
     let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
-    let catalog = read_role_catalog(&db).await?;
+    let cluster = ensure_managed(&governing)?;
+    let catalog = read_role_catalog(&db, cluster).await?;
     let (client, _notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
     Ok(Json(
         build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?,
@@ -1649,6 +1668,7 @@ async fn apply_datatable_acl(
     // connection could wait forever on a pool that concurrent applies, queued on the same locks,
     // have exhausted.
     let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
+    let cluster = ensure_managed(&governing)?;
     // Applies queue on an instance-wide lock while each holds a direct connection to the instance's
     // Postgres; unbounded, the queue alone could exhaust its connection limit. One at a time per
     // server, and the ones waiting hold no connection at all.
@@ -1657,7 +1677,7 @@ async fn apply_datatable_acl(
         .await
         .map_err(|e| Error::internal_err(format!("ACL apply slot closed: {e}")))?;
     let (mut client, mut notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
-    ensure_grant_options(&client, &db, &dbname).await;
+    ensure_grant_options(&client, &db, cluster, &dbname).await;
 
     // Held until the change is committed: a role renamed or dropped meanwhile would change what
     // the plan names, and a settings save could move the entry onto another database. Taken in the
@@ -1672,7 +1692,7 @@ async fn apply_datatable_acl(
     .fetch_optional(&mut *tx)
     .await?
     .flatten();
-    let catalog = read_role_catalog_tx(&mut tx).await?;
+    let catalog = read_role_catalog_tx(&mut tx, cluster).await?;
 
     let plan = build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?;
     if !entry_unchanged(&governing, entry_now) || &plan.statements != confirmed {

backend/windmill-api-workspaces/src/datatable_clone.rs

@@ -11,8 +11,8 @@
 //! The fork request makes its copies before it writes the fork: each is created, filled and — for a
 //! data table under roles — given the source's owners and grants. What each copy was made from
 //! stays in the request ([`MadeCopy`]), so the fork's transaction checks it against the source as it
-//! is then, and a fork that fails drops the instance copies it made ([`drop_copies_after`]) and names
-//! the others, which live on servers of the workspace's own.
+//! is then, and a fork that fails drops the copies it made on a cluster Windmill manages
+//! ([`drop_copies_after`]) and names the others, which live on servers of the workspace's own.
 
 use std::collections::BTreeSet;
 
@@ -22,8 +22,8 @@ use windmill_common::error::{pg_error_message, Error, Result};
 use windmill_common::utils::require_admin;
 use windmill_common::worker::CLOUD_HOSTED;
 use windmill_common::workspaces::{
-    get_datatable_resource_from_db_unchecked, DataTableDatabase, DataTableForkBehavior,
-    GoverningDatatable,
+    get_datatable_resource_from_db_unchecked, DataTableCatalogResourceType, DataTableDatabase,
+    DataTableForkBehavior, GoverningDatatable,
 };
 use windmill_common::{PgDatabase, DB};
 
@@ -312,27 +312,38 @@ pub(crate) async fn drop_copies_after(db: &DB, copies: Vec<MadeCopy>, error: Err
 }
 
 async fn drop_copy(db: &DB, source_database: &DataTableDatabase, dbname: &str) -> Result<()> {
-    if source_database.resource_type
-        == windmill_common::workspaces::DataTableCatalogResourceType::Instance
-    {
-        match windmill_common::drop_unused_instance_database(db, dbname).await? {
-            windmill_common::Cleanup::Dropped => Ok(()),
-            windmill_common::Cleanup::InUse(users) => Err(Error::BadRequest(format!(
-                "kept, since workspaces {} now use it",
-                users.join(", ")
-            ))),
-            windmill_common::Cleanup::Waiter(pid) => Err(Error::BadRequest(format!(
-                "kept, since a request (pid {pid}) is still waiting to name it"
-            ))),
+    match source_database.resource_type {
+        DataTableCatalogResourceType::Instance => {
+            match windmill_common::drop_unused_instance_database(db, dbname).await? {
+                windmill_common::Cleanup::Dropped => Ok(()),
+                windmill_common::Cleanup::InUse(users) => Err(Error::BadRequest(format!(
+                    "kept, since workspaces {} now use it",
+                    users.join(", ")
+                ))),
+                windmill_common::Cleanup::Waiter(pid) => Err(Error::BadRequest(format!(
+                    "kept, since a request (pid {pid}) is still waiting to name it"
+                ))),
+            }
         }
-    } else {
+        // Refused, and the copy kept, when anything names it by then.
+        DataTableCatalogResourceType::ExternalInstance => {
+            let mut tx = db.begin().await?;
+            windmill_common::external_instance_pg::drop_external_instance_database_unchecked(
+                &mut tx, dbname, None,
+            )
+            .await?;
+            tx.commit().await?;
+            Ok(())
+        }
+        DataTableCatalogResourceType::Postgresql => {
         // On a server of the workspace's own, where a resource edited meanwhile can already name it
         // and nothing locks such an edit: dropping it could take someone's data.
-        Err(Error::BadRequest(
-            "kept on its PostgreSQL server, where it may already be in use; drop it there once it \
-             is not, before forking the same data table under this id again"
-                .to_string(),
-        ))
+            Err(Error::BadRequest(
+                "kept on its PostgreSQL server, where it may already be in use; drop it there once \
+                 it is not, before forking the same data table under this id again"
+                    .to_string(),
+            ))
+        }
     }
 }
 
@@ -349,10 +360,10 @@ async fn make_copy(
             request.name
         ))
     })?;
-    let is_instance = governing.is_instance();
+    let managed = source_database.resource_type.is_windmill_managed();
     // Resolved from the snapshot, not read again: the connection the copy is made on is then
     // exactly what these rows describe, which the fork's own clone of them is checked against.
-    let (server, connection): (PgDatabase, Option<ConnectionSnapshot>) = if is_instance {
+    let (server, connection): (PgDatabase, Option<ConnectionSnapshot>) = if managed {
         let server = serde_json::from_value(
             get_datatable_resource_from_db_unchecked(db, parent_w_id, request.name).await?,
         )
@@ -383,22 +394,44 @@ async fn make_copy(
 
     // Dumped before the database exists, so a source that cannot be read leaves nothing behind.
     // Ownership never carries over: the restore runs as the target's connection user. Grants do,
-    // except on the instance, where the replay below is what reproduces them.
+    // except on a cluster Windmill manages, which plants its own and where the replay below is what
+    // reproduces the roles'.
     let dump = pg_dump_database(
         &server,
         PgDumpOptions {
             schema_only: request.behavior == DataTableForkBehavior::SchemaOnly,
             no_owner: true,
-            no_acl: is_instance,
+            no_acl: managed,
             ..Default::default()
         },
     )
     .await?;
 
-    if is_instance {
-        windmill_common::create_custom_instance_database(db, request.dbname, "datatable").await?;
-    } else {
-        create_database_on_server(db, &server, request.dbname).await?;
+    match source_database.resource_type {
+        DataTableCatalogResourceType::Instance => {
+            windmill_common::create_custom_instance_database(
+                db,
+                request.dbname,
+                "datatable",
+                Some(parent_w_id),
+            )
+            .await?
+        }
+        DataTableCatalogResourceType::ExternalInstance => {
+            let mut tx = db.begin().await?;
+            windmill_common::external_instance_pg::create_external_instance_database_unchecked(
+                db,
+                &mut tx,
+                request.dbname,
+                "datatable",
+                Some(parent_w_id),
+            )
+            .await?;
+            tx.commit().await?;
+        }
+        DataTableCatalogResourceType::Postgresql => {
+            create_database_on_server(db, &server, request.dbname).await?
+        }
     }
 
     let target = PgDatabase { dbname: request.dbname.to_string(), ..server.clone() };
@@ -481,11 +514,16 @@ async fn fill(
     let replayed = now.datatable.permissions.is_some();
     if replayed {
         crate::datatable_replay_oss::ensure_replay()?;
-        let catalog = read_role_catalog_tx(&mut tx).await?;
+        // The copy is on the source's cluster, so its roles come from that cluster's catalog.
+        let cluster = governing
+            .role_cluster()
+            .ok_or_else(|| Error::internal_err("a replayed clone is on a managed database"))?;
+        let catalog = read_role_catalog_tx(&mut tx, cluster).await?;
         // The replay leaves `CONNECT` to the catalog, and creating the database only tried to set
         // it: a copy `PUBLIC` could still connect to would admit logins the source turns away.
         windmill_common::datatable_roles::converge_connect_grants_with(
             db,
+            cluster,
             &target.dbname,
             &catalog,
         )

backend/windmill-api-workspaces/src/datatable_migrations.rs

@@ -11,7 +11,7 @@
 //! to keep that file focused on core workspace configuration.
 
 use crate::workspaces::{
-    is_instance_datatable, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
+    managed_datatable_kind, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
     PgDumpOptions,
 };
 
@@ -1669,7 +1669,9 @@ async fn generate_initial_datatable_migration(
     // without what a replay elsewhere cannot run: the replaying user owns none of this
     // database's objects, and the grants Windmill plants in an instance database (`ALTER
     // DEFAULT PRIVILEGES FOR ROLE ...`) fail even replaying onto the same server.
-    let no_acl = is_instance_datatable(&db, &w_id, &datatable_name).await?;
+    let no_acl = managed_datatable_kind(&db, &w_id, &datatable_name)
+        .await?
+        .is_some();
     let dump_file = pg_dump_database(
         &pg_db,
         PgDumpOptions {

backend/windmill-api-workspaces/src/workspaces_extra.rs

@@ -56,6 +56,11 @@ pub(crate) async fn change_workspace_id(
 
     let mut tx = db.begin().await?;
 
+    // The settings copy below carries every data table entry to the new id, which fork cleanup of
+    // the old id cannot see until this commits: without the lock it could drop a copy the renamed
+    // workspace goes on using. Before the pairing lock, as forking takes the two in that order.
+    windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
+
     // A rename rewrites the workspace's dev flag and reparents its children, so it decides on the
     // same state the pairing handlers do: without this lock a concurrent create/attach could commit
     // an active dev workspace under the shell this rename is about to archive. Both ids, since the
@@ -869,6 +874,10 @@ pub(crate) async fn change_workspace_id(
         }
     }
 
+    // After every workspace_settings write above: fork cleanup locks a settings row before the
+    // registry, so taking the registry first here would deadlock with it.
+    migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
+
     // Audit log in the same transaction as the workspace changes
     audit_log(
         &mut *tx,
@@ -937,6 +946,14 @@ pub(crate) async fn change_workspace_id(
     let (_schedules_count, canceled_count, _deleted_tokens_count) =
         archive_workspace_impl(&db, &old_id, &authed.username, None).await?;
 
+    // The old id stays live between the commit above and the archive, and a fork copy created for
+    // it in that window registers under it. Creation checks the workspace is live under the fork
+    // lock, so once this has run under it, no copy can be reserved for the old id any more.
+    let mut tx = db.begin().await?;
+    windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
+    migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
+    tx.commit().await?;
+
     info!(
         "Workspace id change completed: moved {} to {}, archived old workspace",
         old_id, rw.new_id
@@ -948,6 +965,50 @@ pub(crate) async fn change_workspace_id(
     ))
 }
 
+/// A fork copy reserved for the old id would otherwise be unreachable: its creator cannot import
+/// into it or finish its fork under the new id, and nothing else would ever drop it.
+async fn migrate_fork_reservations(
+    tx: &mut Transaction<'_, Postgres>,
+    old_id: &str,
+    new_id: &str,
+) -> Result<()> {
+    const MIGRATE: &str = r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (
+               SELECT COALESCE(jsonb_object_agg(k, CASE WHEN v->>'workspace_id' = $1
+                   THEN jsonb_set(v, '{workspace_id}', to_jsonb($2::text)) ELSE v END), '{}'::jsonb)
+               FROM jsonb_each(COALESCE(value->'databases', '{}'::jsonb)) AS e(k, v)
+           ))
+           WHERE name = $3"#;
+    sqlx::query(MIGRATE)
+        .bind(old_id)
+        .bind(new_id)
+        .bind("custom_instance_pg_databases")
+        .execute(&mut **tx)
+        .await?;
+    // External copies are registered in the cluster state, which its writers rewrite whole under
+    // the lifecycle lock. Only taken when there is something to move, as setup can hold it for as
+    // long as the cluster takes to answer.
+    let state = windmill_common::global_settings::EXTERNAL_INSTANCE_PG_STATE_SETTING;
+    let reserved = sqlx::query_scalar::<_, bool>(
+        "SELECT EXISTS (SELECT 1 FROM global_settings, jsonb_each(value->'databases') AS e(k, v)
+          WHERE name = $1 AND jsonb_typeof(value->'databases') = 'object'
+            AND v->>'workspace_id' = $2)",
+    )
+    .bind(state)
+    .bind(old_id)
+    .fetch_one(&mut **tx)
+    .await?;
+    if reserved {
+        windmill_common::external_instance_pg::lock_external_instance_pg_state(tx).await?;
+        sqlx::query(MIGRATE)
+            .bind(old_id)
+            .bind(new_id)
+            .bind(state)
+            .execute(&mut **tx)
+            .await?;
+    }
+    Ok(())
+}
+
 #[derive(Deserialize)]
 pub(crate) struct DeleteWorkspaceQuery {
     pub(crate) only_delete_forks: Option<bool>,
@@ -1430,9 +1491,7 @@ pub async fn drop_forked_datatable_databases(
             _ => continue,
         };
 
-        if database.resource_type
-            == windmill_common::workspaces::DataTableCatalogResourceType::Instance
-        {
+        if database.resource_type.is_windmill_managed() {
             let db_to_drop = &database.resource_path;
             if !db_to_drop.starts_with("wm_fork_") {
                 errors.push(format!(
@@ -1441,7 +1500,112 @@ pub async fn drop_forked_datatable_databases(
                 ));
                 continue;
             }
-            if let Err(e) = windmill_common::drop_custom_instance_database(&db, db_to_drop).await {
+            // The fork's own entry is what is going away; anything else still reaching the copy,
+            // a child fork's pointer at this entry included, keeps it. The lock keeps a child fork
+            // from gaining such a pointer before the drop.
+            // A task of its own, so a client going away cannot stop it between dropping the
+            // database and committing the entry's removal.
+            let dropped = tokio::spawn({
+                let (db, w_id, dt_name, db_to_drop) = (
+                    db.clone(),
+                    w_id.clone(),
+                    dt_name.clone(),
+                    db_to_drop.clone(),
+                );
+                let resource_type = database.resource_type;
+                async move {
+                    let mut tx = db.begin().await?;
+                    // The three locks a settings save takes, in its order: this workspace's data
+                    // tables, its settings row, and the database itself. Without them a save could
+                    // rename this entry, or point another one here, either side of the check below.
+                    windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
+                    // The snapshot above was read unlocked: a save committing since could have
+                    // repointed this entry, and the entry is removed below whatever it names by then.
+                    let current = sqlx::query_scalar::<_, Option<serde_json::Value>>(
+                        "SELECT datatable->'datatables'->$2 FROM workspace_settings
+                     WHERE workspace_id = $1 FOR UPDATE",
+                    )
+                    .bind(&w_id)
+                    .bind(&dt_name)
+                    .fetch_optional(&mut *tx)
+                    .await?
+                    .flatten()
+                    .and_then(|v| serde_json::from_value::<DataTable>(v).ok());
+                    if !current.is_some_and(|dt| {
+                        dt.forked_from.is_some()
+                            && dt.database.is_some_and(|d| {
+                                d.resource_type == resource_type && d.resource_path == db_to_drop
+                            })
+                    }) {
+                        return Err(Error::BadRequest(
+                            "the data table changed while it was being cleaned up".to_string(),
+                        ));
+                    }
+                    let external = resource_type
+                        == windmill_common::workspaces::DataTableCatalogResourceType::ExternalInstance;
+                    if external {
+                        // Before the governance lock, in the order settings saves and fork
+                        // finalization take the two.
+                        windmill_common::external_instance_pg::lock_external_instance_pg_state(
+                            &mut tx,
+                        )
+                        .await?;
+                    }
+                    windmill_common::datatable_roles::lock_instance_databases_governance(
+                        &mut tx,
+                        [db_to_drop.as_str()],
+                    )
+                    .await?;
+                    // Before the entry is removed: child fork pointers are found through it.
+                    let uses = windmill_common::workspaces::managed_database_uses(
+                        &mut tx,
+                        resource_type,
+                        &db_to_drop,
+                        Some((w_id.as_str(), dt_name.as_str())),
+                    )
+                    .await?;
+                    if !uses.is_empty() {
+                        return Err(Error::BadRequest(format!(
+                            "it is still used by {}",
+                            uses.join(", ")
+                        )));
+                    }
+                    // The entry goes with the database: a fork this one is cloned into afterwards
+                    // must not inherit a pointer at a data table whose database is gone.
+                    sqlx::query(
+                        "UPDATE workspace_settings SET datatable = datatable #- ARRAY['datatables', $2]
+                         WHERE workspace_id = $1",
+                    )
+                    .bind(&w_id)
+                    .bind(&dt_name)
+                    .execute(&mut *tx)
+                    .await?;
+                    if external {
+                        // Checks the uses of the database itself, and unregisters it.
+                        windmill_common::external_instance_pg::drop_external_instance_database_unchecked(
+                            &mut tx,
+                            &db_to_drop,
+                            Some((w_id.as_str(), dt_name.as_str())),
+                        )
+                        .await?;
+                    } else {
+                        windmill_common::drop_custom_instance_database_keep_entry(&db, &db_to_drop)
+                            .await?;
+                        sqlx::query(
+                            "UPDATE global_settings SET value = value #- ARRAY['databases', $1]
+                             WHERE name = 'custom_instance_pg_databases'",
+                        )
+                        .bind(&db_to_drop)
+                        .execute(&mut *tx)
+                        .await?;
+                    }
+                    tx.commit().await?;
+                    Ok::<_, Error>(())
+                }
+            })
+            .await
+            .unwrap_or_else(|e| Err(Error::internal_err(format!("cleanup task failed: {e}"))));
+            if let Err(e) = dropped {
                 errors.push(format!(
                     "Could not drop instance database '{}' for datatable://{}: {}",
                     db_to_drop, dt_name, e
@@ -1808,7 +1972,17 @@ async fn resolve_fork_catalog_pg(
             "ducklake://{ducklake_name}: malformed registry catalog identity `{catalog}`"
         ))
     })?;
-    let catalog_resource = if resource_type == "instance" {
+    let catalog_resource = if resource_type == "external_instance" {
+        serde_json::to_value(
+            windmill_common::external_instance_pg::external_instance_connection_unchecked(
+                db,
+                resource_path,
+                false,
+            )
+            .await?,
+        )
+        .map_err(|e| Error::internal_err(format!("serializing pg creds: {e}")))?
+    } else if resource_type == "instance" {
         let mut pg_creds = windmill_common::PgDatabase::parse_uri(
             &windmill_common::get_database_url().await?.as_str().await,
         )?;

backend/windmill-api/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: "3.0.3"
 
 info:
-  version: 1.818.0
+  version: 1.819.0
   title: Windmill API
 
   contact:
@@ -1398,6 +1398,14 @@ paths:
         - workspace
       parameters:
         - $ref: "#/components/parameters/WorkspaceId"
+        - name: background
+          description: |
+            return once the request is validated and create the fork in the background, answering
+            with the creation id getForkCreationStatus reads
+          in: query
+          required: false
+          schema:
+            type: boolean
       requestBody:
         description: new forked workspace
         required: true
@@ -1407,12 +1415,45 @@ paths:
               $ref: "#/components/schemas/CreateWorkspaceFork"
       responses:
         "201":
-          description: forked workspace created
+          description: forked workspace created, or with background set, the id of its creation
           content:
             text/plain:
               schema:
                 type: string
 
+  /w/{workspace}/workspaces/fork_creation_status/{creation_id}:
+    get:
+      summary: get the status of a fork created in the background
+      operationId: getForkCreationStatus
+      tags:
+        - workspace
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: creation_id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      responses:
+        "200":
+          description: fork creation status
+          content:
+            application/json:
+              schema:
+                type: object
+                properties:
+                  status:
+                    type: string
+                    enum: [running, completed, failed]
+                  step:
+                    type: string
+                    description: the part of the copy a running fork is in
+                  error:
+                    type: string
+                required:
+                  - status
+
   /w/{workspace}/workspaces/attach_dev_workspace:
     post:
       summary: attach an existing workspace as the dev workspace of this (prod) workspace
@@ -1572,6 +1613,104 @@ paths:
               schema:
                 type: object
 
+  /settings/external_instance_pg/status:
+    get:
+      summary: Returns whether the external instance cluster is configured and how its last setup went
+      operationId: getExternalInstancePgStatus
+      tags:
+        - setting
+      responses:
+        "200":
+          description: external instance cluster status
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ExternalInstancePgStatus"
+
+  /settings/external_instance_pg/setup:
+    post:
+      summary: Sets up the external instance cluster with its saved admin login, optionally rotating the passwords Windmill manages on it (enterprise edition only)
+      operationId: setupExternalInstancePg
+      tags:
+        - setting
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                rotate_passwords:
+                  type: boolean
+      responses:
+        "200":
+          description: the setup report, also stored as the last setup
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ExternalInstancePgSetupReport"
+
+  /settings/external_instance_pg/databases:
+    get:
+      summary: Lists the databases Windmill created on the external instance cluster, with the workspaces whose data tables, Ducklake catalogs or pending fork cleanups use each
+      operationId: listExternalInstancePgDatabases
+      tags:
+        - setting
+      responses:
+        "200":
+          description: databases by name
+          content:
+            application/json:
+              schema:
+                type: object
+                additionalProperties:
+                  $ref: "#/components/schemas/CustomInstanceDb"
+
+  /settings/external_instance_pg/databases/{name}:
+    post:
+      summary: Creates a database on the external instance cluster (enterprise edition only)
+      operationId: createExternalInstancePgDatabase
+      tags:
+        - setting
+      parameters:
+        - name: name
+          in: path
+          required: true
+          schema:
+            type: string
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              properties:
+                tag:
+                  $ref: "#/components/schemas/CustomInstanceDbTag"
+      responses:
+        "200":
+          description: database created
+          content:
+            application/json:
+              schema: {}
+    delete:
+      summary: Drops a database Windmill created on the external instance cluster, refused while a data table, Ducklake catalog or pending fork cleanup uses it
+      operationId: dropExternalInstancePgDatabase
+      tags:
+        - setting
+      parameters:
+        - name: name
+          in: path
+          required: true
+          schema:
+            type: string
+      responses:
+        "200":
+          description: database dropped
+          content:
+            application/json:
+              schema: {}
+
   /settings/list_custom_instance_pg_databases:
     post:
       summary: Returns the set-up statuses of custom instance pg databases
@@ -1590,13 +1729,19 @@ paths:
 
   /settings/datatable_roles:
     get:
-      summary: list the instance's data table roles
+      summary: list the data table roles of one Windmill-managed Postgres cluster
       operationId: listInstanceDatatableRoles
       tags:
         - setting
+      parameters:
+        - in: query
+          name: cluster
+          required: false
+          schema:
+            $ref: "#/components/schemas/DatatableRoleCluster"
       responses:
         "200":
-          description: the instance role catalog
+          description: the cluster's role catalog
           content:
             application/json:
               schema:
@@ -1604,7 +1749,7 @@ paths:
                 items:
                   $ref: "#/components/schemas/InstanceDatatableRole"
     post:
-      summary: create a data table role on the instance's Postgres cluster
+      summary: create a data table role on a Windmill-managed Postgres cluster
       operationId: createInstanceDatatableRole
       tags:
         - setting
@@ -1618,6 +1763,8 @@ paths:
               properties:
                 name:
                   type: string
+                cluster:
+                  $ref: "#/components/schemas/DatatableRoleCluster"
       responses:
         "200":
           description: the created role
@@ -5351,7 +5498,7 @@ paths:
                       type: string
                     resource_type:
                       type: string
-                      enum: [postgres, instance]
+                      enum: [postgres, instance, external_instance]
                     resource_path:
                       type: string
                     governing_workspace_id:
@@ -10898,6 +11045,8 @@ paths:
             text/plain:
               schema:
                 type: string
+        "400":
+          description: the body's workspace_id does not match the workspace in the path
 
   /w/{workspace}/workspace_dependencies/archive/{language}:
     post:
@@ -11815,6 +11964,11 @@ paths:
           content:
             application/json:
               schema: {}
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run_wait_result/fv/{version}:
     post:
@@ -11855,6 +12009,11 @@ paths:
           content:
             application/json:
               schema: {}
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
     get:
       summary: run flow by version with GET and wait until completion
@@ -11887,6 +12046,11 @@ paths:
           content:
             application/json:
               schema: {}
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run_and_stream/f/{path}:
     post:
@@ -11928,6 +12092,11 @@ paths:
             text/event-stream:
               schema:
                 type: string
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
     get:
       summary: run flow by path with GET and stream updates via SSE
@@ -11961,6 +12130,11 @@ paths:
             text/event-stream:
               schema:
                 type: string
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run_and_stream/fv/{version}:
     post:
@@ -12008,6 +12182,11 @@ paths:
             text/event-stream:
               schema:
                 type: string
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
     get:
       summary: run flow by version with GET and stream updates via SSE
@@ -12047,6 +12226,11 @@ paths:
             text/event-stream:
               schema:
                 type: string
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run_and_stream/p/{path}:
     post:
@@ -13552,7 +13736,7 @@ paths:
         Returns the flat list of all item paths visible to the caller across
         scripts, flows, apps, raw apps, variables, and resources. Intended to
         feed an entirely client-side path autocomplete UI: the frontend fetches
-        once (server caches per workspace for 60s) and performs all prefix/segment
+        once (server caches per caller for 60s) and performs all prefix/segment
         computation locally. Capped at 20,000 paths (5,000 per table).
       operationId: listPathAutocompletePaths
       tags:
@@ -15456,6 +15640,11 @@ paths:
               schema:
                 type: string
                 format: uuid
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run/fv/{version}:
     post:
@@ -15513,6 +15702,11 @@ paths:
               schema:
                 type: string
                 format: uuid
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run/batch_rerun_jobs:
     post:
@@ -16006,6 +16200,11 @@ paths:
               schema:
                 type: string
                 format: uuid
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run_wait_result/preview_flow:
     post:
@@ -16035,6 +16234,11 @@ paths:
           content:
             application/json:
               schema: {}
+        "409":
+          description: >-
+            Chat-enabled flow only: the conversation named by `memory_id` is still answering
+            a message. The body is JSON: `{ "error": string, "running_turn": { "job_id",
+            "user_seq" } }`.
 
   /w/{workspace}/jobs/run/dynamic_select:
     post:
@@ -28926,6 +29130,23 @@ components:
         is_test:
           type: boolean
           description: Started from the flow editor's test panel rather than a deployed run
+        running_turn:
+          type: object
+          nullable: true
+          description: >-
+            The turn the conversation is still answering, set by the list endpoint: its
+            newest user message, while the flow run it started is queued or running. A
+            run into this conversation is refused with 409 until the turn ends.
+          required: [job_id, user_seq]
+          properties:
+            job_id:
+              type: string
+              format: uuid
+              description: The flow run of the turn
+            user_seq:
+              type: integer
+              format: int64
+              description: created_seq of the user message that started the turn
 
     FlowConversationMessage:
       type: object
@@ -29530,6 +29751,7 @@ components:
       properties:
         workspace_id:
           type: string
+          description: must equal the workspace in the request path
         language:
           $ref: "#/components/schemas/ScriptLang"
         name:
@@ -32058,6 +32280,16 @@ components:
                   - id
                   - success
                   - duration_ms
+            late_run_streak:
+              type: integer
+              description: runs in a row, up to the latest, that started or finished after the next occurrence was due, so that occurrence was missed
+            missed_occurrences:
+              type: integer
+              description: occurrences missed by the latest streak of late runs, kept after it ends; a lower bound once a single late run misses 1000
+            last_missed_at:
+              type: string
+              format: date-time
+              description: latest missed occurrence, kept after the schedule runs on time again; past 1000 misses in one late run, when the miss was detected, at most one period later
 
     ErrorHandler:
       type: string
@@ -33989,9 +34221,55 @@ components:
         - ducklake
         - datatable
 
+    ExternalInstancePgSetupStep:
+      type: object
+      required: [name, status, message]
+      properties:
+        name:
+          type: string
+        status:
+          type: string
+          enum: [ok, warning, error]
+        message:
+          type: string
+
+    ExternalInstancePgSetupReport:
+      type: object
+      required: [success, finished_at, steps]
+      properties:
+        success:
+          type: boolean
+          description: no step failed; warnings leave it true
+        finished_at:
+          type: string
+          format: date-time
+        steps:
+          type: array
+          items:
+            $ref: "#/components/schemas/ExternalInstancePgSetupStep"
+
+    ExternalInstancePgStatus:
+      type: object
+      required: [configured, database_count]
+      properties:
+        configured:
+          type: boolean
+        database_count:
+          type: integer
+        last_setup:
+          $ref: "#/components/schemas/ExternalInstancePgSetupReport"
+
+    DatatableRoleCluster:
+      type: string
+      description: >-
+        The Windmill-managed Postgres cluster a data table role is a login on: Windmill's own
+        (behind `instance` data tables) or the external instance cluster (behind
+        `external_instance` ones). Defaults to `instance`.
+      enum: [instance, external_instance]
+
     InstanceDatatableRole:
       type: object
-      required: [id, name, enabled]
+      required: [id, name, enabled, cluster]
       properties:
         id:
           type: string
@@ -33999,6 +34277,8 @@ components:
           type: string
         enabled:
           type: boolean
+        cluster:
+          $ref: "#/components/schemas/DatatableRoleCluster"
 
     DatatableRoleTenants:
       type: object
@@ -34020,8 +34300,10 @@ components:
         supported:
           type: boolean
           description: >-
-            Whether this data table can be put under roles at all. Only one backed by the
-            instance database can: a role is a login on that cluster.
+            Whether this data table can be put under roles at all. Only one on a database Windmill
+            manages can: a role is a login on that database's cluster.
+        cluster:
+          $ref: "#/components/schemas/DatatableRoleCluster"
         permissioned:
           type: boolean
         default_role:
@@ -34333,7 +34615,10 @@ components:
           type: array
           items:
             type: string
-          description: Workspaces that reference this database via a ducklake catalog or datatable database with resource_type 'instance'. Computed at request time, not persisted.
+          description: Workspaces that reference this database through a ducklake catalog or a datatable database of the kind being listed — 'instance' for the instance databases endpoint, 'external_instance' for the external cluster one. Computed at request time, not persisted, and only returned to superadmins.
+        workspace_id:
+          type: string
+          description: The workspace a member created this database for as a fork copy. Only that workspace can import into it or point a fork at it.
 
     NewSqsTrigger:
       type: object
@@ -36341,6 +36626,7 @@ components:
                       - postgresql
                       - mysql
                       - instance
+                      - external_instance
                   resource_path:
                     type: string
                 required:
@@ -36404,6 +36690,7 @@ components:
                     enum:
                       - postgresql
                       - instance
+                      - external_instance
                   resource_path:
                     type: string
                 required:

backend/windmill-api/src/apps.rs

@@ -4444,8 +4444,15 @@ async fn execute_component(
                 (JobPayload::Code(raw_code), tag, None)
             }
             // inline script: run mode (deployed app) with an entry in `app_script`.
-            (None, Some(RawCode { language, path, cache_ttl, tag, .. }), Some(id)) => (
-                JobPayload::AppScript { id: AppScriptId(id), cache_ttl, language, path },
+            // The path is derived like the legacy arm's: job identity (e.g. OIDC `sub`)
+            // reads it, so a caller must not choose it.
+            (None, Some(RawCode { language, cache_ttl, tag, .. }), Some(id)) => (
+                JobPayload::AppScript {
+                    id: AppScriptId(id),
+                    cache_ttl,
+                    language,
+                    path: Some(inline_run_path(path, &component)?),
+                },
                 resolved_inline_tag(tag),
                 None,
             ),
@@ -4805,8 +4812,9 @@ async fn upload_s3_file_from_app(
                 if let Some(ref s3_resource_path) = query.s3_resource_path {
                     if matched_input.allow_user_resources {
                         if let Some(authed) = opt_authed {
+                            let viewer = policy_granted_viewer(&authed);
                             let db_with_opt_authed = DbWithOptAuthed::from_authed(
-                                &authed,
+                                &viewer,
                                 db.clone(),
                                 Some(user_db.clone()),
                             );
@@ -5773,6 +5781,14 @@ async fn exists_app(
     Ok(Json(exists))
 }
 
+/// The viewer as whom a resource an app policy lets the viewer pick (`allow_user_resources`)
+/// is resolved. The policy, not the token, grants that resource, and app tokens are minted
+/// with a fixed scope set that never names variables, so the references inside it resolve on
+/// the viewer's RLS alone.
+fn policy_granted_viewer(authed: &ApiAuthed) -> ApiAuthed {
+    ApiAuthed { scopes: None, ..authed.clone() }
+}
+
 async fn build_args(
     policy: &Policy,
     PolicyTriggerableInputs {
@@ -5799,8 +5815,9 @@ async fn build_args(
                 key.and_then(|x| x.clone().strip_prefix("$res:").map(|x| x.to_string()))
             {
                 if let Some(authed) = authed {
+                    let viewer = policy_granted_viewer(authed);
                     let db_with_opt_authed =
-                        DbWithOptAuthed::from_authed(authed, db.clone(), Some(user_db.clone()));
+                        DbWithOptAuthed::from_authed(&viewer, db.clone(), Some(user_db.clone()));
                     let res = get_resource_value_interpolated_internal(
                         &db_with_opt_authed,
                         w_id,

backend/windmill-api/src/capture.rs

@@ -66,6 +66,7 @@ use crate::{
     args::RawWebhookArgs,
     db::{ApiAuthed, DB},
     users::fetch_api_authed,
+    utils::check_scopes,
 };
 
 use axum::{
@@ -319,6 +320,7 @@ async fn get_configs(
     Extension(user_db): Extension<UserDB>,
     Path((w_id, runnable_kind, path)): Path<(String, RunnableKind, StripPath)>,
 ) -> JsonResult<Vec<CaptureConfig>> {
+    check_scopes(&authed, || format!("capture:read:{}", path.to_path()))?;
     let mut tx = user_db.begin(&authed).await?;
 
     let configs = sqlx::query_as!(
@@ -557,6 +559,7 @@ async fn set_config(
     Path(w_id): Path<String>,
     Json(nc): Json<NewCaptureConfig>,
 ) -> JsonResult<Option<TriggerConfig>> {
+    check_scopes(&authed, || format!("capture:write:{}", nc.path))?;
     let nc = match nc.trigger_kind {
         TriggerKind::Postgres => {
             set_postgres_trigger_config(&w_id, authed.clone(), &db, user_db.clone(), nc).await?
@@ -616,6 +619,7 @@ async fn ping_config(
         StripPath,
     )>,
 ) -> Result<()> {
+    check_scopes(&authed, || format!("capture:write:{}", path.to_path()))?;
     let mut tx = user_db.begin(&authed).await?;
     if matches!(trigger_kind, TriggerKind::Postgres) {
         windmill_common::datatable_roles::lock_datatable_streams(&mut *tx, false).await?;
@@ -667,6 +671,7 @@ async fn list_captures(
     Path((w_id, runnable_kind, path)): Path<(String, RunnableKind, StripPath)>,
     Query(query): Query<ListCapturesQuery>,
 ) -> JsonResult<Vec<Capture>> {
+    check_scopes(&authed, || format!("capture:read:{}", path.to_path()))?;
     let mut tx = user_db.begin(&authed).await?;
 
     let (per_page, offset) = paginate(Pagination { page: query.page, per_page: query.per_page });
@@ -713,12 +718,44 @@ async fn list_captures(
     Ok(Json(captures))
 }
 
+/// A capture addressed by id carries no path in the route, so its path scope can only be
+/// checked against the row. Looked up under the caller's RLS so the check never reveals
+/// whether an id the caller cannot see exists, and the refusal omits the path, which the
+/// caller named only by id.
+async fn check_capture_id_scope(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    authed: &ApiAuthed,
+    w_id: &str,
+    id: i64,
+    action: &str,
+) -> Result<()> {
+    if authed.scopes.is_none() {
+        return Ok(());
+    }
+    let path = sqlx::query_scalar!(
+        "SELECT path FROM capture WHERE id = $1 AND workspace_id = $2",
+        id,
+        w_id,
+    )
+    .fetch_optional(&mut **tx)
+    .await?;
+    if let Some(path) = path {
+        check_scopes(authed, || format!("capture:{action}:{path}")).map_err(|_| {
+            windmill_common::error::Error::PermissionDenied(format!(
+                "This token's capture:{action} scope does not cover capture {id}"
+            ))
+        })?;
+    }
+    Ok(())
+}
+
 async fn get_capture(
     authed: ApiAuthed,
     Extension(user_db): Extension<UserDB>,
     Path((w_id, id)): Path<(String, i64)>,
 ) -> JsonResult<Capture> {
     let mut tx = user_db.begin(&authed).await?;
+    check_capture_id_scope(&mut tx, &authed, &w_id, id, "read").await?;
 
     let capture = sqlx::query_as!(
         Capture,
@@ -751,6 +788,7 @@ async fn delete_capture(
     Path((w_id, id)): Path<(String, i64)>,
 ) -> Result<()> {
     let mut tx = user_db.begin(&authed).await?;
+    check_capture_id_scope(&mut tx, &authed, &w_id, id, "write").await?;
     // capture RLS only keys on the path segment, so without workspace_id an id from
     // another workspace whose path collides with the caller's grants would be deleted.
     sqlx::query!(
@@ -781,8 +819,10 @@ async fn move_captures_and_configs(
     Path((w_id, runnable_kind, old_path)): Path<(String, RunnableKind, StripPath)>,
     Json(body): Json<MoveCapturesAndConfigsBody>,
 ) -> Result<()> {
-    let mut tx = user_db.begin(&authed).await?;
     let old_path = old_path.to_path();
+    check_scopes(&authed, || format!("capture:write:{}", old_path))?;
+    check_scopes(&authed, || format!("capture:write:{}", body.new_path))?;
+    let mut tx = user_db.begin(&authed).await?;
 
     sqlx::query!(
         r#"

backend/windmill-api/src/jobs.rs

@@ -4197,13 +4197,15 @@ async fn get_args(
 
 async fn get_started_at_by_ids(
     Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
     Json(mut ids): Json<Vec<Uuid>>,
 ) -> JsonResult<Vec<Option<chrono::DateTime<chrono::Utc>>>> {
     ids.truncate(100);
 
     let started_at = sqlx::query!(
-        "SELECT id, started_at FROM v2_job_queue WHERE id = ANY($1)",
-        ids.as_slice()
+        "SELECT id, started_at FROM v2_job_queue WHERE id = ANY($1) AND workspace_id = $2",
+        ids.as_slice(),
+        &w_id
     )
     .fetch_all(&db)
     .await?;
@@ -5411,7 +5413,7 @@ async fn resume_suspended_job_internal(
     verify_suspended_secret(&w_id, &db, job_id, resume_id, &approver, secret).await?;
 
     // Get flow info - works for step-level, flow-level, and WAC approval
-    let (flow_info, is_flow_level, is_wac) = get_flow_info_for_resume(job_id, &db).await?;
+    let (flow_info, is_flow_level, is_wac) = get_flow_info_for_resume(job_id, &w_id, &db).await?;
 
     // HMAC secret = full capability. Skip approval_conditions checks: possession of the full
     // resume URL is the authorization (it is only disclosed to intended approvers, e.g. when a
@@ -5643,7 +5645,11 @@ struct FlowInfo {
 /// Returns (FlowInfo, is_flow_level, is_wac) where:
 /// - is_flow_level: job_id was a flow job (pre-approval)
 /// - is_wac: job_id is a WAC workflow suspended for approval (target is itself)
-async fn get_flow_info_for_resume(job_id: Uuid, db: &DB) -> error::Result<(FlowInfo, bool, bool)> {
+async fn get_flow_info_for_resume(
+    job_id: Uuid,
+    w_id: &str,
+    db: &DB,
+) -> error::Result<(FlowInfo, bool, bool)> {
     // Single query that determines if job_id is a flow, step, or WAC job,
     // and fetches the appropriate suspended job info.
     // For WAC jobs (no parent, not a flow), the job itself is the suspended target.
@@ -5652,7 +5658,7 @@ async fn get_flow_info_for_resume(job_id: Uuid, db: &DB) -> error::Result<(FlowI
         WITH job_info AS (
             SELECT id, kind::text AS kind, parent_job
             FROM v2_job
-            WHERE id = $1
+            WHERE id = $1 AND workspace_id = $2
         )
         SELECT
             q.id AS "id!",
@@ -5672,10 +5678,15 @@ async fn get_flow_info_for_resume(job_id: Uuid, db: &DB) -> error::Result<(FlowI
         FOR UPDATE OF q
         "#,
         job_id,
+        w_id,
     )
     .fetch_optional(db)
     .await?
-    .ok_or_else(|| anyhow::anyhow!("job not found or parent flow not in queue: {}", job_id))?;
+    .ok_or_else(|| {
+        Error::NotFound(format!(
+            "job not found or parent flow not in queue: {job_id}"
+        ))
+    })?;
 
     let flow_info = FlowInfo {
         id: result.id,
@@ -5979,12 +5990,7 @@ pub async fn create_job_signature(
     Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>,
     Query(approver): Query<QueryApprover>,
 ) -> error::Result<String> {
-    // The HMAC is treated as full authority by the resume endpoints, so minting
-    // it requires run scope on the suspended job's flow — not merely any
-    // jobs:run scope. No-op for unscoped tokens (incl. the in-flow substep token
-    // used by wmill.get_resume_urls()).
-    let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?;
-    check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?;
+    require_resume_mint_authority(&db, &w_id, &authed, job_id).await?;
     let key = get_workspace_key(&w_id, &db).await?;
     create_signature(key, job_id, resume_id, approver.approver)
 }
@@ -6077,12 +6083,7 @@ pub async fn get_resume_urls(
     Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>,
     Query(approver): Query<QueryApprover>,
 ) -> error::JsonResult<ResumeUrls> {
-    // These URLs embed a resume signature (full resume capability), so a scoped
-    // token must hold run scope on the suspended job's flow. No-op for unscoped
-    // tokens (incl. the in-flow substep token). Trusted internal callers use
-    // get_resume_urls_internal directly and are unaffected.
-    let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?;
-    check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?;
+    require_resume_mint_authority(&db, &w_id, &authed, job_id).await?;
     get_resume_urls_internal(
         Extension(db),
         Path((w_id, job_id, resume_id)),
@@ -6107,23 +6108,9 @@ pub async fn get_wac_approval_urls(
             "step_key must be the key of a wait_for_approval step".to_string(),
         ));
     }
-    let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?;
-    check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?;
-
-    // This handler writes to the job's status row, so the job must actually be in
-    // the caller's workspace — `v2_job_status` is keyed by job id alone and would
-    // otherwise take a write aimed at another workspace's job.
-    let in_workspace = sqlx::query_scalar!(
-        "SELECT EXISTS (SELECT 1 FROM v2_job WHERE id = $1 AND workspace_id = $2)",
-        job_id,
-        w_id
-    )
-    .fetch_one(&db)
-    .await?
-    .unwrap_or(false);
-    if !in_workspace {
-        return Err(Error::NotFound(format!("job {job_id} not found")));
-    }
+    // Also the workspace check for the `v2_job_status` write below, which is keyed by job id
+    // alone and would otherwise take a write aimed at another workspace's job.
+    require_resume_mint_authority(&db, &w_id, &authed, job_id).await?;
 
     // The approval belongs to the WAC parent, but WM_JOB_ID is the child job when
     // this is called from inside a task() rather than a step(). Resolve up so the
@@ -6323,12 +6310,34 @@ pub async fn get_resume_urls_internal(
     Ok(Json(res))
 }
 
-/// Resolve the runnable path of the flow a (possibly step) job belongs to, used
+/// A resume signature is full approval authority: `jobs_u/resume|cancel` skip the step's
+/// approval_conditions and accept any resume_id. So only a workspace admin, or a job token
+/// whose run lineage claims the suspended job ([`unclaimable_run_lineage`]), may mint one, and
+/// a scoped token also needs run scope on the flow. The job must be in `w_id`, whose key signs
+/// the result.
+async fn require_resume_mint_authority(
+    db: &DB,
+    w_id: &str,
+    authed: &ApiAuthed,
+    job_id: Uuid,
+) -> error::Result<()> {
+    let flow_path = resume_target_flow_path(db, w_id, job_id).await?;
+    check_scopes(authed, || format!("jobs:run:flows:{}", flow_path))?;
+    if !unclaimable_run_lineage(db, w_id, vec![job_id], authed)
+        .await?
+        .is_empty()
+    {
+        return Err(Error::PermissionDenied(format!(
+            "only job {job_id}'s own run or a workspace admin can sign resume or approval urls for it"
+        )));
+    }
+    Ok(())
+}
+
+/// Resolve the runnable path of the flow a (possibly step) job of `w_id` belongs to, used
 /// to scope-check resume-signature minting against `jobs:run:flows:<path>`.
-/// Returns an empty string when the path can't be resolved (e.g. previews or an
-/// unknown job); an empty path only matters for path-restricted tokens, which
-/// would not be running such a flow. Never hard-fails, so it can't break resume
-/// for unscoped tokens (the in-flow `get_resume_urls()` path).
+/// Returns an empty string when the path can't be resolved (e.g. previews); an empty
+/// path only matters for path-restricted tokens, which would not be running such a flow.
 async fn resume_target_flow_path(db: &DB, w_id: &str, job_id: Uuid) -> error::Result<String> {
     let job = sqlx::query!(
         r#"SELECT kind::text as "kind!", parent_job, runnable_path
@@ -6337,10 +6346,8 @@ async fn resume_target_flow_path(db: &DB, w_id: &str, job_id: Uuid) -> error::Re
         w_id
     )
     .fetch_optional(db)
-    .await?;
-    let Some(job) = job else {
-        return Ok(String::new());
-    };
+    .await?
+    .ok_or_else(|| Error::NotFound(format!("job {job_id} not found")))?;
     // All flow kinds: the job itself is the flow whose path scopes the resume.
     if matches!(
         job.kind.as_str(),
@@ -6499,6 +6506,8 @@ struct BatchReRunQueryReturnType {
     scheduled_for: chrono::DateTime<Utc>,
     #[serde(skip_serializing_if = "Option::is_none")]
     schema: Option<serde_json::Value>,
+    #[serde(skip)]
+    created_by: String,
 }
 
 async fn batch_rerun_compute_js_expression(
@@ -6548,7 +6557,7 @@ fn batch_rerun_jobs_inner(
                 BatchReRunQueryReturnType,
                 r#"WITH norm AS (
                         SELECT
-                            j.id, j.workspace_id, j.runnable_path, j.runnable_id, j.kind, j.args,
+                            j.id, j.workspace_id, j.runnable_path, j.runnable_id, j.kind, j.args, j.created_by,
                             -- Project effective kind for dispatch: pass script/flow through;
                             -- for singlestepflow, read the wrapped runnable's type from
                             -- raw_flow.modules[id='a'].value.type (always 'script' or 'flow').
@@ -6587,6 +6596,7 @@ fn batch_rerun_jobs_inner(
                         COALESCE(s.hash, f.id, n.ssf_hash, 0::bigint) AS "script_hash!: _",
                         COALESCE(jc.started_at, jq.scheduled_for, make_date(1970, 1, 1)) AS "scheduled_for!: _",
                         n.args AS input,
+                        n.created_by,
                         -- Pinned schema for script/flow; latest-by-path fallback for
                         -- singlestepflow so input_transforms still resolve at rerun time.
                         COALESCE(
@@ -6643,6 +6653,10 @@ async fn batch_rerun_handle_job(
     w_id: &String,
     body: &BatchReRunJobsBodyArgs,
 ) -> error::Result<String> {
+    // The source jobs are selected on the root pool, and a rerun copies their args into a
+    // job the caller owns, so each one must pass the same read gate as `jobs_u/get`.
+    require_job_read_access(db, user_db, authed, w_id, &job.id, &job.created_by, None).await?;
+
     let options = if matches!(job.kind, JobKind::Script) {
         &body.script_options_by_path
     } else {
@@ -6941,7 +6955,8 @@ async fn resolve_nested_restart(
     parent_step_id: &str,
     parent_branch_or_iteration_n: Option<usize>,
     nested_path: Vec<NestedRestartStep>,
-    parent_flow_version: Option<i64>,
+    // (flow path of the restarted job, version to restart on)
+    parent_flow_version: Option<(&str, i64)>,
 ) -> error::Result<(
     Option<windmill_common::flow_status::BranchChosen>,
     Option<Box<RestartedFrom>>,
@@ -6969,8 +6984,8 @@ async fn resolve_nested_restart(
         )
     })?;
 
-    let flow_data = if let Some(version) = parent_flow_version {
-        cache::flow::fetch_version(db, version).await?
+    let flow_data = if let Some((flow_path, version)) = parent_flow_version {
+        windmill_queue::fetch_restart_flow_version(db, workspace_id, flow_path, version).await?
     } else {
         cache::job::fetch_flow(db, &row.job_kind, row.runnable_id)
             .or_else(|_| cache::job::fetch_preview_flow(db, &parent_job_id, row.raw_flow))
@@ -7147,6 +7162,8 @@ pub async fn restart_flow(
         .script_path
         .with_context(|| "No flow path set for completed flow job")?;
     check_scopes(&authed, || format!("jobs:run:flows:{flow_path}"))?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let ehm = HashMap::new();
     let push_args = completed_job
@@ -7164,7 +7181,7 @@ pub async fn restart_flow(
         &step_id,
         branch_or_iteration_n,
         nested_path,
-        flow_version,
+        flow_version.map(|v| (flow_path.as_str(), v)),
     )
     .await?;
 
@@ -7345,6 +7362,13 @@ pub async fn run_workflow_as_code(
     let args = PushArgs { args: &task.args.unwrap_or_else(HashMap::new), extra: Some(extra) };
     check_tag_available_for_workspace(&db, &w_id, &run_query.tag, &args, &authed).await?;
     check_scopes(&authed, || format!("jobs:run"))?;
+    // The task becomes a child of `job_id`, runs its code and writes into its flow status, so
+    // only that job itself (the SDK's `task` wrapper, on its `WM_TOKEN`) or an admin may push it.
+    if authed.job_id != Some(job_id) && !authed.is_admin {
+        return Err(error::Error::PermissionDenied(format!(
+            "only job {job_id}'s own WM_TOKEN can run its workflow tasks"
+        )));
+    }
 
     if !is_valid_entrypoint_name(&entrypoint) {
         return Err(error::Error::BadRequest(format!(
@@ -7761,6 +7785,8 @@ pub async fn run_wait_result_job_by_path_get(
     let tag = run_query.tag.clone().or(tag);
     let push_args = PushArgs { args: &args.args, extra: args.extra };
     check_tag_available_for_workspace(&db, &w_id, &tag, &push_args, &authed).await?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let (email, permissioned_as, push_authed, tx) =
         if let Some(on_behalf_of) = on_behalf_authed.as_ref() {
@@ -7907,6 +7933,8 @@ pub async fn run_wait_result_script_by_path_internal(
     let tag = run_query.tag.clone().or(tag);
     let push_args = PushArgs { args: &args.args, extra: args.extra };
     check_tag_available_for_workspace(&db, &w_id, &tag, &push_args, &authed).await?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let (email, permissioned_as, push_authed, tx) =
         if let Some(on_behalf_of) = on_behalf_of.as_ref() {
@@ -8021,6 +8049,8 @@ pub async fn run_wait_result_script_by_hash(
     let tag = run_query.tag.clone().or(tag);
     let push_args = PushArgs { args: &args.args, extra: args.extra };
     check_tag_available_for_workspace(&db, &w_id, &tag, &push_args, &authed).await?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let (email, permissioned_as, push_authed, tx) = if let Some(obo) = on_behalf_of.as_ref() {
         (
@@ -9625,6 +9655,13 @@ async fn run_preview_flow_job(
     // jobs:run scope so a narrowly-scoped token cannot escape its scope. See run_preview_script.
     check_scopes(&authed, || format!("jobs:run"))?;
     require_path_read_access_for_preview(&authed, &raw_flow.path)?;
+    // Restarting copies the source runs' step results into the new run, and the queue resolves
+    // them with the service pool, so every run the request names must be readable as the caller.
+    let mut level = raw_flow.restarted_from.as_ref();
+    while let Some(r) = level {
+        require_job_update_read_access(&db, &user_db, &authed, &w_id, &r.flow_job_id, None).await?;
+        level = r.nested.as_deref();
+    }
     let scheduled_for = run_query.get_scheduled_for(&db).await?;
     let tag = run_query.tag.clone().or(raw_flow.tag.clone());
     let tx = PushIsolationLevel::Isolated(user_db.clone(), authed.clone().into());
@@ -10040,6 +10077,8 @@ pub async fn run_job_by_hash_inner(
     let push_args = PushArgs { args: &args.args, extra: args.extra };
 
     check_tag_available_for_workspace(&db, &w_id, &tag, &push_args, &authed).await?;
+    let mut run_query = run_query;
+    drop_unclaimable_run_lineage(&db, &w_id, &mut run_query, &authed).await?;
 
     let (email, permissioned_as, push_authed, tx) = if let Some(obo) = on_behalf_of.as_ref() {
         (

backend/windmill-api/src/lib.rs

@@ -1279,6 +1279,15 @@ pub async fn run_server(
     });
 
     server.await?;
+    let forks = &windmill_api_workspaces::workspaces::BACKGROUND_FORKS;
+    forks.close();
+    if !forks.is_empty() {
+        tracing::info!(
+            "server off. waiting for {} fork(s) being created",
+            forks.len()
+        );
+        forks.wait().await;
+    }
     #[cfg(feature = "agent_worker_server")]
     for (i, bg_processor) in agent_workers_bg_processor.into_iter().enumerate() {
         tracing::info!("server off. shutting down agent worker bg processor {i}");

backend/windmill-api/src/path_autocomplete.rs

@@ -16,23 +16,55 @@ use axum::{
     routing::get,
     Json, Router,
 };
+use quick_cache::{sync::Cache, Weighter};
 use serde::{Deserialize, Serialize};
-use windmill_common::error::JsonResult;
+use windmill_common::{db::UserDB, error::JsonResult};
+use windmill_store::var_resource_cache::auth_identity;
 
-use crate::db::{ApiAuthed, DB};
+use crate::db::ApiAuthed;
 
 // Per-table row cap is inlined into the SQL as `LIMIT 5000`.
 // With 6 tables, the absolute ceiling is ~30k paths pre-dedup.
 /// Final cap applied after dedup/sort.
 const MAX_PATHS: usize = 20_000;
-/// TTL for the per-workspace path list cache.
+/// TTL for the path list cache.
 const CACHE_TTL: Duration = Duration::from_secs(60);
 
-/// Workspace-wide path list cache keyed by workspace_id only.
-/// One entry per workspace shared across all users — autocomplete is a
-/// navigation hint, not an access gate. Saves memory and warms faster.
-static PATHS_CACHE: LazyLock<quick_cache::sync::Cache<String, (Arc<Vec<String>>, Instant)>> =
-    LazyLock::new(|| quick_cache::sync::Cache::new(500));
+/// Total paths held across all cache entries.
+const CACHE_MAX_PATHS: u64 = 2_000_000;
+
+type CacheKey = (String, String);
+type CacheValue = (Arc<Vec<String>>, Instant);
+
+#[derive(Clone)]
+struct PathCountWeighter;
+
+impl Weighter<CacheKey, CacheValue> for PathCountWeighter {
+    fn weight(&self, _key: &CacheKey, (paths, _): &CacheValue) -> u64 {
+        (paths.len() as u64).max(1)
+    }
+}
+
+/// Keyed by (workspace_id, [`auth_identity`]): the list is read under the caller's RLS,
+/// so an entry must only ever be served back to the same authorization context.
+/// Weighted by path count because the key space grows with callers, not workspaces.
+/// Single-sharded: quick_cache splits the weight budget across shards and refuses an
+/// entry heavier than one shard's share, which would drop the largest workspaces.
+static PATHS_CACHE: LazyLock<Cache<CacheKey, CacheValue, PathCountWeighter>> =
+    LazyLock::new(|| {
+        let options = quick_cache::OptionsBuilder::new()
+            .shards(1)
+            .estimated_items_capacity(1000)
+            .weight_capacity(CACHE_MAX_PATHS)
+            .build()
+            .expect("every cache option is set");
+        Cache::with_options(
+            options,
+            PathCountWeighter,
+            Default::default(),
+            Default::default(),
+        )
+    });
 
 pub fn workspaced_service() -> Router {
     Router::new().route("/list_paths", get(list_paths))
@@ -53,20 +85,22 @@ struct ListPathsQuery {
 }
 
 async fn list_paths(
-    _authed: ApiAuthed,
-    Extension(db): Extension<DB>,
+    authed: ApiAuthed,
+    Extension(user_db): Extension<UserDB>,
     Path(w_id): Path<String>,
     Query(ListPathsQuery { force }): Query<ListPathsQuery>,
 ) -> JsonResult<ListPathsResponse> {
+    let cache_key = (w_id, auth_identity(&authed));
     if !force {
-        if let Some((cached, cached_at)) = PATHS_CACHE.get(&w_id) {
+        if let Some((cached, cached_at)) = PATHS_CACHE.get(&cache_key) {
             if cached_at.elapsed() < CACHE_TTL {
                 return Ok(Json(ListPathsResponse { paths: cached }));
             }
-            PATHS_CACHE.remove(&w_id);
+            PATHS_CACHE.remove(&cache_key);
         }
     }
 
+    let mut tx = user_db.begin(&authed).await?;
     let mut paths: Vec<String> = sqlx::query_scalar!(
         r#"
         SELECT path AS "path!" FROM (
@@ -83,16 +117,17 @@ async fn list_paths(
             (SELECT path FROM resource WHERE workspace_id = $1 LIMIT 5000)
         ) t
         "#,
-        &w_id,
+        &cache_key.0,
     )
-    .fetch_all(&db)
+    .fetch_all(&mut *tx)
     .await?;
+    tx.commit().await?;
 
     paths.sort_unstable();
     paths.truncate(MAX_PATHS);
     let paths = Arc::new(paths);
 
-    PATHS_CACHE.insert(w_id, (paths.clone(), Instant::now()));
+    PATHS_CACHE.insert(cache_key, (paths.clone(), Instant::now()));
 
     Ok(Json(ListPathsResponse { paths }))
 }

backend/windmill-api/src/static_assets.rs

@@ -51,30 +51,17 @@ const TWO_HUNDRED: &str = "200.html";
 /// Check if the original path requires cross-origin isolation headers.
 ///
 /// CANONICAL COEP RATIONALE (the dev-server mirror in `frontend/vite.config.js`
-/// and the navigation guards in `frontend/src/routes/(root)/(logged)/+layout.svelte`
-/// point here): the headers are needed for SharedArrayBuffer and TypeScript
-/// workers (raw app editor at `/apps_raw/edit|add`, in-browser bundler at
-/// `/ui_builder/`). The raw app *viewer* (`/apps_raw/get/`) must NOT get them:
-/// COEP `require-corp` blocks the viewed app's cross-origin subresources
-/// (external images, embeds) that lack CORP — and since headers stick to the
-/// document, apps would break on a page reload while working when reached via
-/// client-side navigation.
-///
-/// Public apps (`/public/` and custom paths `/a/`) opt in via the `wm_coep`
-/// query param: a public (raw) app must set COEP to be embeddable as an iframe
-/// inside a cross-origin-isolated page (which requires the embedded document to
-/// also set COEP). It is opt-in rather than always-on because cross-origin
-/// isolation also blocks subresources without CORP (e.g. external image URLs
-/// or embeds used by classic apps), so we only enable it when the embedder
-/// explicitly requests it.
+/// points here). Only public apps (`/public/` and custom paths `/a/`) ever get
+/// them, and only when they opt in via the `wm_coep` query param: a public (raw)
+/// app must set COEP to be embeddable as an iframe inside a cross-origin-isolated
+/// page (which requires the embedded document to also set COEP). It is opt-in
+/// rather than always-on because COEP `require-corp` blocks subresources without
+/// CORP (external image URLs, embeds), and since headers stick to the document,
+/// isolating an SPA route leaks into every page reached from it client-side.
 #[cfg(feature = "static_frontend")]
 fn needs_cross_origin_isolation(original_path: &str, query: Option<&str>) -> bool {
-    // no trailing slash on edit/add: matches the +layout.svelte guards
-    original_path.starts_with("/apps_raw/edit")
-        || original_path.starts_with("/apps_raw/add")
-        || original_path.starts_with("/ui_builder/")
-        || ((original_path.starts_with("/public/") || original_path.starts_with("/a/"))
-            && query_has_flag(query, "wm_coep"))
+    (original_path.starts_with("/public/") || original_path.starts_with("/a/"))
+        && query_has_flag(query, "wm_coep")
 }
 
 /// Returns true if `query` contains the given flag key (with or without a
@@ -98,8 +85,6 @@ fn serve_path(path: &str, original_path: &str, query: Option<&str>) -> Response<
                 .header(header::CONTENT_TYPE, mime.as_ref())
                 .header(header::ACCESS_CONTROL_ALLOW_ORIGIN, "*");
 
-            // Add cross-origin isolation headers only for paths that need them
-            // (apps_raw editor needs SharedArrayBuffer for TypeScript workers)
             if needs_cross_origin_isolation(original_path, query) {
                 res = res
                     .header("Cross-Origin-Opener-Policy", "same-origin")
@@ -167,12 +152,13 @@ mod tests {
 
     #[test]
     fn test_needs_cross_origin_isolation() {
-        // editor + bundler are always isolated, regardless of query
-        assert!(needs_cross_origin_isolation("/apps_raw/edit/foo", None));
-        assert!(needs_cross_origin_isolation("/apps_raw/add", None));
-        assert!(needs_cross_origin_isolation("/ui_builder/index.html", None));
-
-        // the raw app viewer must NOT be isolated
+        // the raw app editor, its UI builder frames and the viewer are never isolated
+        assert!(!needs_cross_origin_isolation("/apps_raw/edit/foo", None));
+        assert!(!needs_cross_origin_isolation("/apps_raw/add", None));
+        assert!(!needs_cross_origin_isolation(
+            "/ui_builder/index.html",
+            None
+        ));
         assert!(!needs_cross_origin_isolation(
             "/apps_raw/get/u/foo/bar",
             None

backend/windmill-api/src/tracing_init.rs

@@ -48,7 +48,9 @@ impl<B> OnResponse<B> for MyOnResponse {
             let status = response.status().as_u16();
             if response.status().is_success() || response.status().is_redirection() {
                 tracing::info!(latency = latency, status = status, "response")
-            } else if response.status().as_u16() == 404 {
+            } else if status == 404 || status == 409 {
+                // A refused turn is as expected as a miss: the flow chat takes the turn that
+                // refused it and sends its message after it.
                 tracing::warn!(latency = latency, status = status, "response")
             } else {
                 tracing::error!(latency = latency, status = status, "response")

backend/windmill-api/src/workspace_dependencies.rs

@@ -34,13 +34,20 @@ async fn create(
     authed: ApiAuthed,
     // Extension(user_db): Extension<UserDB>,
     Extension(db): Extension<DB>,
+    Path(w_id): Path<String>,
     Json(nwd): Json<NewWorkspaceDependencies>,
 ) -> error::Result<(StatusCode, String)> {
-    tracing::info!(workspace_id = %nwd.workspace_id, name = ?nwd.name, language = ?nwd.language, "create workspace dependencies");
+    tracing::info!(workspace_id = %w_id, name = ?nwd.name, language = ?nwd.language, "create workspace dependencies");
     require_admin(authed.is_admin, &authed.username)?;
+    // `require_admin` vouches for the path workspace only; the body must not target another.
+    if nwd.workspace_id != w_id {
+        return Err(error::Error::BadRequest(format!(
+            "workspace_id `{}` in the request body does not match the workspace `{}` in the path",
+            nwd.workspace_id, w_id
+        )));
+    }
 
     let dep_path = WorkspaceDependencies::to_path(&nwd.name, nwd.language)?;
-    let w_id = nwd.workspace_id.clone();
     let email = authed.email.clone();
     let username = authed.username.clone();
 

backend/windmill-common/Cargo.toml

@@ -76,6 +76,7 @@ bitflags.workspace = true
 once_cell.workspace = true
 phf.workspace = true
 tokio-postgres.workspace = true
+postgres-protocol.workspace = true
 postgres-native-tls.workspace = true
 native-tls.workspace = true
 

backend/windmill-common/src/datatable_roles.rs

@@ -6,22 +6,67 @@
  * LICENSE-AGPL for a copy of the license.
  */
 
-//! The instance's data table role catalog.
+//! The instance's data table role catalogs.
 //!
-//! A data table role is a real Postgres login role on the Windmill cluster, named exactly as the
-//! user named it, shared by every instance database. Windmill decides who may ask for a role (the
-//! per-data-table tenant lists in [`crate::workspaces`]); Postgres decides what the role may then
-//! touch. The catalog here is only the first half's vocabulary plus the cluster provisioning.
+//! A data table role is a real Postgres login role on one cluster — Windmill's own, or the external
+//! instance cluster — named exactly as the user named it, shared by every database Windmill manages
+//! on that cluster. Each cluster has its own catalog: a role exists where it was created and nowhere
+//! else. Windmill decides who may ask for a role (the per-data-table tenant lists in
+//! [`crate::workspaces`]); Postgres decides what the role may then touch. The catalog here is only
+//! the first half's vocabulary plus the cluster provisioning.
 //!
 //! Entries are keyed by a generated id so a rename moves nothing else: tenants name the id.
 
 use std::collections::BTreeMap;
 
+use serde::{Deserialize, Serialize};
+
 use crate::{
     error::{Error, Result},
+    workspaces::DataTableCatalogResourceType,
     DB,
 };
 
+/// The cluster a role catalog belongs to.
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(rename_all = "snake_case")]
+pub enum DatatableRoleCluster {
+    /// Windmill's own Postgres, behind `instance` data tables.
+    #[default]
+    Instance,
+    /// The external instance cluster, behind `external_instance` data tables.
+    ExternalInstance,
+}
+
+impl DatatableRoleCluster {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Instance => "instance",
+            Self::ExternalInstance => "external_instance",
+        }
+    }
+
+    pub fn parse(value: &str) -> Result<Self> {
+        match value {
+            "instance" => Ok(Self::Instance),
+            "external_instance" => Ok(Self::ExternalInstance),
+            other => Err(Error::BadRequest(format!(
+                "Unknown data table role cluster '{other}': expected instance or external_instance"
+            ))),
+        }
+    }
+
+    /// The cluster whose roles a data table on `kind` can use. `None` for a resource-backed one,
+    /// which is never under roles.
+    pub fn of(kind: DataTableCatalogResourceType) -> Option<Self> {
+        match kind {
+            DataTableCatalogResourceType::Instance => Some(Self::Instance),
+            DataTableCatalogResourceType::ExternalInstance => Some(Self::ExternalInstance),
+            DataTableCatalogResourceType::Postgresql => None,
+        }
+    }
+}
+
 /// The connection every data table resolved to before roles existed (`custom_instance_user`). It
 /// owns every pre-existing object, so it is a reserved name rather than a catalog entry: never
 /// created, renamed or dropped.
@@ -164,28 +209,42 @@ pub async fn lock_instance_databases_governance<'a>(
 /// need the names — but callers MUST NOT let `pwd` reach a response, a log line, an audit record
 /// or an export. Nothing about who may call it: the credential is the whole risk, and `Debug` is
 /// hand-written to redact it for the same reason.
-pub async fn read_role_catalog(db: &DB) -> Result<DatatableRoleCatalog> {
-    crate::datatable_roles_oss::read_role_catalog(db).await
+pub async fn read_role_catalog(
+    db: &DB,
+    cluster: DatatableRoleCluster,
+) -> Result<DatatableRoleCatalog> {
+    crate::datatable_roles_oss::read_role_catalog(db, cluster).await
 }
 
 /// As [`read_role_catalog`], reading inside the caller's transaction so the value is the one
 /// [`lock_role_catalog`] is protecting. Same disclosure contract.
 pub async fn read_role_catalog_tx(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    cluster: DatatableRoleCluster,
 ) -> Result<DatatableRoleCatalog> {
-    crate::datatable_roles_oss::read_role_catalog_tx(tx).await
+    crate::datatable_roles_oss::read_role_catalog_tx(tx, cluster).await
 }
 
-/// Record a role, in the caller's transaction so it commits with the `CREATE ROLE` it describes.
+/// The cluster a role belongs to, or `None` if no role has this id.
+pub async fn role_cluster(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    id: &str,
+) -> Result<Option<DatatableRoleCluster>> {
+    crate::datatable_roles_oss::role_cluster(tx, id).await
+}
+
+/// Record a role, in the caller's transaction. On Windmill's own cluster that commits it with the
+/// `CREATE ROLE` it describes; on the external cluster the role already exists by then.
 ///
 /// Authorization: writes a generated Postgres credential. Callers MUST restrict this to superadmin
 /// paths and MUST hold [`lock_role_catalog`] on `tx`.
 pub async fn insert_role_catalog_entry(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
     id: &str,
+    cluster: DatatableRoleCluster,
     role: &InstanceDatatableRole,
 ) -> Result<()> {
-    crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, role).await
+    crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, cluster, role).await
 }
 
 /// Update a role's recorded name, login flag and password. Same contract as
@@ -215,7 +274,7 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
         .find(|(_, role)| role.name == name)
         .ok_or_else(|| {
             Error::NotFound(format!(
-                "'{name}' is not a data table role of this instance. Defined roles: {}.",
+                "'{name}' is not a data table role of this database's cluster. Defined roles: {}.",
                 catalog
                     .values()
                     .map(|r| r.name.as_str())
@@ -231,70 +290,90 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
     Ok(entry.0.as_str())
 }
 
-/// Every instance database the registry knows about. Role provisioning has to reach all of them:
-/// a role that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
+/// Every database Windmill manages on `cluster`. Role provisioning has to reach all of them: a role
+/// that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
 ///
-/// Authorization: checks nothing, and names every instance database across all workspaces. Callers
+/// Authorization: checks nothing, and names every managed database across all workspaces. Callers
 /// MUST be superadmin-gated or keep the names server-side; never return them to a workspace caller.
-pub async fn registered_instance_databases(db: &DB) -> Result<Vec<String>> {
-    crate::datatable_roles_oss::registered_instance_databases(db).await
+pub async fn registered_instance_databases(
+    db: &DB,
+    cluster: DatatableRoleCluster,
+) -> Result<Vec<String>> {
+    crate::datatable_roles_oss::registered_instance_databases(db, cluster).await
 }
 
-/// `CONNECT` on `dbname` for every enabled role, and none for `PUBLIC`. Run at role creation, at
-/// database creation, and lazily whenever an instance data table is administered, so a database
-/// provisioned before a role existed is repaired rather than left silently unreachable.
+/// `CONNECT` on `dbname` for every enabled role of `cluster`, and none for `PUBLIC`. Run at role
+/// creation, at database creation, and lazily whenever a managed data table is administered, so a
+/// database provisioned before a role existed is repaired rather than left silently unreachable.
 ///
 /// Authorization: rewrites a database's ACL with the server's own credentials and checks nothing.
 /// Callers MUST have authorized administration of `dbname` — superadmin, or an admin of the
 /// workspace governing a data table on it.
-pub async fn converge_connect_grants(db: &DB, dbname: &str) -> Result<()> {
-    crate::datatable_roles_oss::converge_connect_grants(db, dbname).await
+pub async fn converge_connect_grants(
+    db: &DB,
+    cluster: DatatableRoleCluster,
+    dbname: &str,
+) -> Result<()> {
+    crate::datatable_roles_oss::converge_connect_grants(db, cluster, dbname).await
 }
 
-/// As [`converge_connect_grants`], with a catalog the caller already read. Same contract.
+/// As [`converge_connect_grants`], with the catalog of `cluster` the caller already read. Same
+/// contract.
 pub async fn converge_connect_grants_with(
     db: &DB,
+    cluster: DatatableRoleCluster,
     dbname: &str,
     catalog: &DatatableRoleCatalog,
 ) -> Result<()> {
-    crate::datatable_roles_oss::converge_connect_grants_with(db, dbname, catalog).await
+    crate::datatable_roles_oss::converge_connect_grants_with(db, cluster, dbname, catalog).await
 }
 
-/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user`, and `CONNECT` on
-/// every registered database. No privileges beyond that — an admin grants them through SQL or the
-/// ACL editor.
+/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user` on `cluster`. No
+/// privileges beyond that — an admin grants them through SQL or the ACL editor.
+///
+/// On Windmill's own cluster the DDL runs on `tx`, so it commits with the catalog row. The external
+/// cluster is another server: the role is created there before `tx` commits, and callers MUST drop
+/// it again ([`drop_datatable_role`]) if `tx` then fails to commit.
 ///
 /// Authorization: creates a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
-pub async fn create_instance_role(
+/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
+pub async fn create_datatable_role(
+    db: &DB,
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    cluster: DatatableRoleCluster,
     name: &str,
     password: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::create_instance_role(tx, name, password).await
+    crate::datatable_roles_oss::create_datatable_role(db, tx, cluster, name, password).await
 }
 
 /// Authorization: alters a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
-pub async fn set_instance_role_login(
+/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
+pub async fn set_datatable_role_login(
+    db: &DB,
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    cluster: DatatableRoleCluster,
     name: &str,
     enabled: bool,
 ) -> Result<()> {
-    crate::datatable_roles_oss::set_instance_role_login(tx, name, enabled).await
+    crate::datatable_roles_oss::set_datatable_role_login(db, tx, cluster, name, enabled).await
 }
 
-/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one.
+/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one. On the
+/// external cluster the rename lands before `tx` commits, and callers MUST rename it back if `tx`
+/// then fails to commit.
 ///
 /// Authorization: renames a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
-pub async fn rename_instance_role(
+/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
+pub async fn rename_datatable_role(
+    db: &DB,
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    cluster: DatatableRoleCluster,
     from: &str,
     to: &str,
     password: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::rename_instance_role(tx, from, to, password).await
+    crate::datatable_roles_oss::rename_datatable_role(db, tx, cluster, from, to, password).await
 }
 
 /// A role owning anything in any database blocks its own `DROP ROLE`, and both its objects and the
@@ -302,8 +381,9 @@ pub async fn rename_instance_role(
 /// registry. An unreachable database aborts the whole delete: dropping the role while one database
 /// still holds objects owned by it leaves those objects owned by a numeric OID nobody can name.
 ///
-/// Each pass runs as the instance's own Postgres user rather than `custom_instance_user`, which
-/// owns the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
+/// Each pass runs as the cluster's administrator rather than `custom_instance_user`: on Windmill's
+/// own cluster the instance's Postgres user, on the external one its configured admin login. Both
+/// own the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
 /// could only undo what it granted itself, so a privilege planted by an operator in psql — the
 /// ordinary way privileges reach a role — would survive and block the drop.
 ///
@@ -311,16 +391,19 @@ pub async fn rename_instance_role(
 /// MUST restrict this to superadmin paths, and MUST hold [`lock_role_catalog`] on `tx`.
 ///
 /// The per-database passes open their own connections and cannot join `tx`; the lock is what keeps
-/// a concurrent mutation out while they run. Only the final `DROP ROLE` is on `tx`, so it commits
-/// or rolls back with the catalog write that forgets the role. Those passes commit as they go, so
-/// callers MUST have disabled the role in an earlier committed transaction: a failure part-way
-/// then leaves a disabled role to retry, not an enabled one already stripped in some databases.
-pub async fn drop_instance_role(
+/// a concurrent mutation out while they run. On Windmill's own cluster only the final `DROP ROLE`
+/// is on `tx`, so it commits or rolls back with the catalog write that forgets the role; on the
+/// external cluster it runs there, and tolerates a role already gone so a retry after a failed
+/// commit can finish. The passes commit as they go, so callers MUST have disabled the role in an
+/// earlier committed transaction: a failure part-way then leaves a disabled role to retry, not an
+/// enabled one already stripped in some databases.
+pub async fn drop_datatable_role(
     db: &DB,
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    cluster: DatatableRoleCluster,
     name: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::drop_instance_role(db, tx, name).await
+    crate::datatable_roles_oss::drop_datatable_role(db, tx, cluster, name).await
 }
 
 #[cfg(test)]

backend/windmill-common/src/datatable_roles_oss.rs

@@ -24,12 +24,12 @@ pub fn datatable_roles_unavailable() -> Error {
 #[cfg(all(feature = "private", feature = "enterprise"))]
 pub(crate) use crate::datatable_roles_ee::{
     can_use_datatable_role, can_use_datatable_role_in_governing_workspace, converge_connect_grants,
-    converge_connect_grants_with, create_instance_role, delete_role_catalog_entry,
-    drop_instance_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
+    converge_connect_grants_with, create_datatable_role, delete_role_catalog_entry,
+    drop_datatable_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
     ensure_instance_db_grant_options_unchecked, forget_datatable_role_everywhere,
     insert_role_catalog_entry, read_role_catalog, read_role_catalog_tx,
-    registered_instance_databases, rename_instance_role, resolve_datatable_role_connection,
-    set_instance_role_login, update_role_catalog_entry,
+    registered_instance_databases, rename_datatable_role, resolve_datatable_role_connection,
+    role_cluster, set_datatable_role_login, update_role_catalog_entry,
 };
 
 #[cfg(not(all(feature = "private", feature = "enterprise")))]
@@ -39,7 +39,7 @@ pub(crate) use ce::*;
 mod ce {
     use super::datatable_roles_unavailable as unavailable;
     use crate::{
-        datatable_roles::{DatatableRoleCatalog, InstanceDatatableRole},
+        datatable_roles::{DatatableRoleCatalog, DatatableRoleCluster, InstanceDatatableRole},
         db::AuthedRef,
         error::Result,
         workspaces::{
@@ -50,17 +50,31 @@ mod ce {
 
     type Tx<'a> = sqlx::Transaction<'a, sqlx::Postgres>;
 
-    pub(crate) async fn read_role_catalog(_db: &DB) -> Result<DatatableRoleCatalog> {
+    pub(crate) async fn read_role_catalog(
+        _db: &DB,
+        _cluster: DatatableRoleCluster,
+    ) -> Result<DatatableRoleCatalog> {
         Err(unavailable())
     }
 
-    pub(crate) async fn read_role_catalog_tx(_tx: &mut Tx<'_>) -> Result<DatatableRoleCatalog> {
+    pub(crate) async fn read_role_catalog_tx(
+        _tx: &mut Tx<'_>,
+        _cluster: DatatableRoleCluster,
+    ) -> Result<DatatableRoleCatalog> {
+        Err(unavailable())
+    }
+
+    pub(crate) async fn role_cluster(
+        _tx: &mut Tx<'_>,
+        _id: &str,
+    ) -> Result<Option<DatatableRoleCluster>> {
         Err(unavailable())
     }
 
     pub(crate) async fn insert_role_catalog_entry(
         _tx: &mut Tx<'_>,
         _id: &str,
+        _cluster: DatatableRoleCluster,
         _role: &InstanceDatatableRole,
     ) -> Result<()> {
         Err(unavailable())
@@ -78,43 +92,57 @@ mod ce {
         Err(unavailable())
     }
 
-    pub(crate) async fn registered_instance_databases(_db: &DB) -> Result<Vec<String>> {
+    pub(crate) async fn registered_instance_databases(
+        _db: &DB,
+        _cluster: DatatableRoleCluster,
+    ) -> Result<Vec<String>> {
         Err(unavailable())
     }
 
-    /// Nothing to converge: with no roles to admit, an instance database keeps the `CONNECT`
-    /// grants it was created with, `PUBLIC`'s included, as it did before roles existed.
-    pub(crate) async fn converge_connect_grants(_db: &DB, _dbname: &str) -> Result<()> {
+    /// Nothing to converge: with no roles to admit, a managed database keeps the `CONNECT` grants
+    /// it was created with, as it did before roles existed.
+    pub(crate) async fn converge_connect_grants(
+        _db: &DB,
+        _cluster: DatatableRoleCluster,
+        _dbname: &str,
+    ) -> Result<()> {
         Ok(())
     }
 
     /// As [`converge_connect_grants`].
     pub(crate) async fn converge_connect_grants_with(
         _db: &DB,
+        _cluster: DatatableRoleCluster,
         _dbname: &str,
         _catalog: &DatatableRoleCatalog,
     ) -> Result<()> {
         Ok(())
     }
 
-    pub(crate) async fn create_instance_role(
+    pub(crate) async fn create_datatable_role(
+        _db: &DB,
         _tx: &mut Tx<'_>,
+        _cluster: DatatableRoleCluster,
         _name: &str,
         _password: &str,
     ) -> Result<()> {
         Err(unavailable())
     }
 
-    pub(crate) async fn set_instance_role_login(
+    pub(crate) async fn set_datatable_role_login(
+        _db: &DB,
         _tx: &mut Tx<'_>,
+        _cluster: DatatableRoleCluster,
         _name: &str,
         _enabled: bool,
     ) -> Result<()> {
         Err(unavailable())
     }
 
-    pub(crate) async fn rename_instance_role(
+    pub(crate) async fn rename_datatable_role(
+        _db: &DB,
         _tx: &mut Tx<'_>,
+        _cluster: DatatableRoleCluster,
         _from: &str,
         _to: &str,
         _password: &str,
@@ -122,12 +150,18 @@ mod ce {
         Err(unavailable())
     }
 
-    pub(crate) async fn drop_instance_role(_db: &DB, _tx: &mut Tx<'_>, _name: &str) -> Result<()> {
+    pub(crate) async fn drop_datatable_role(
+        _db: &DB,
+        _tx: &mut Tx<'_>,
+        _cluster: DatatableRoleCluster,
+        _name: &str,
+    ) -> Result<()> {
         Err(unavailable())
     }
 
     pub(crate) async fn ensure_instance_db_grant_options_unchecked(
         _db: &DB,
+        _cluster: DatatableRoleCluster,
         _dbname: &str,
     ) -> Result<()> {
         Err(unavailable())

backend/windmill-common/src/error.rs

@@ -303,7 +303,11 @@ impl IntoResponse for Error {
 
         let e = &self;
 
-        if matches!(status, axum::http::StatusCode::NOT_FOUND) {
+        // A refused turn is as expected as a miss: the chat follows the turn that refused it.
+        if matches!(
+            status,
+            axum::http::StatusCode::NOT_FOUND | axum::http::StatusCode::CONFLICT
+        ) {
             tracing::warn!(message = e.to_string());
         } else {
             tracing::error!(message = e.to_string(), error = ?e);

backend/windmill-common/src/external_instance_pg.rs

@@ -0,0 +1,456 @@
+/*
+ * Author: Ruben Fiszel
+ * Copyright: Windmill Labs, Inc 2022
+ * This file and its contents are licensed under the AGPLv3 License.
+ * Please see the included NOTICE for copyright information and
+ * LICENSE-AGPL for a copy of the license.
+ */
+
+//! The external Postgres cluster behind `external_instance` data tables and Ducklake catalogs.
+//!
+//! Windmill administers that cluster itself, logged in as the user in
+//! [`EXTERNAL_INSTANCE_PG_SETTING`]. It creates `custom_instance_user` and
+//! `custom_instance_replication_user` there, with passwords it generates and keeps in
+//! [`EXTERNAL_INSTANCE_PG_STATE_SETTING`]. They share their names with the roles on Windmill's own
+//! cluster, but they are different roles with different passwords.
+//!
+//! The cluster may hold data Windmill did not create. Two Windmill instances sharing one is not
+//! supported: each would keep resetting the passwords the other depends on.
+
+use std::collections::{BTreeMap, BTreeSet};
+
+use serde::{Deserialize, Serialize};
+
+use crate::{
+    error::{Error, Result},
+    global_settings::{EXTERNAL_INSTANCE_PG_SETTING, EXTERNAL_INSTANCE_PG_STATE_SETTING},
+    instance_config::{CustomInstanceDb, ExternalInstancePg},
+    DB,
+};
+
+/// What Windmill keeps about the external cluster. Server-managed and hidden: never part of the
+/// instance config, never readable by an agent worker. No `Debug`: it carries live passwords.
+#[derive(Serialize, Deserialize, Clone, Default)]
+pub struct ExternalInstancePgState {
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub user_pwd: Option<String>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub replication_pwd: Option<String>,
+    /// The databases Windmill created on the cluster. It only ever drops one of these.
+    #[serde(default)]
+    pub databases: BTreeMap<String, CustomInstanceDb>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub last_setup: Option<ExternalInstancePgSetupReport>,
+    /// The cluster ([`external_instance_pg_address`]) the last successful setup converged. Databases
+    /// are only created on a cluster setup succeeded on: the passwords above exist as soon as setup
+    /// first runs, whether or not the cluster accepted them.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub set_up_for: Option<String>,
+}
+
+/// What identifies the cluster a configuration points at. Other fields (admin login, sslmode) can
+/// change without it becoming another cluster.
+pub fn external_instance_pg_address(config: &ExternalInstancePg) -> String {
+    format!(
+        "{}:{}",
+        config.host.trim().to_lowercase(),
+        config.port.unwrap_or(5432)
+    )
+}
+
+#[derive(Serialize, Deserialize, Clone, Debug)]
+pub struct ExternalInstancePgSetupReport {
+    /// No step failed. Warnings leave it true.
+    pub success: bool,
+    pub finished_at: chrono::DateTime<chrono::Utc>,
+    pub steps: Vec<ExternalInstancePgSetupStep>,
+}
+
+#[derive(Serialize, Deserialize, Clone, Debug)]
+pub struct ExternalInstancePgSetupStep {
+    pub name: String,
+    pub status: SetupStepStatus,
+    pub message: String,
+}
+
+#[derive(Serialize, Deserialize, Clone, Copy, Debug, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum SetupStepStatus {
+    Ok,
+    Warning,
+    Error,
+}
+
+/// The status the settings page shows without running anything.
+#[derive(Serialize, Debug)]
+pub struct ExternalInstancePgStatus {
+    pub configured: bool,
+    pub database_count: usize,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub last_setup: Option<ExternalInstancePgSetupReport>,
+}
+
+/// Authorization: returns the cluster's admin password and checks nothing. Callers MUST be
+/// superadmin or an internal server path.
+pub(crate) async fn read_external_instance_pg_config<'c>(
+    executor: impl sqlx::PgExecutor<'c>,
+) -> Result<Option<ExternalInstancePg>> {
+    let value = sqlx::query_scalar!(
+        "SELECT value FROM global_settings WHERE name = $1",
+        EXTERNAL_INSTANCE_PG_SETTING
+    )
+    .fetch_optional(executor)
+    .await?;
+    value
+        .map(|v| {
+            serde_json::from_value(v).map_err(|e| {
+                Error::internal_err(format!("reading {EXTERNAL_INSTANCE_PG_SETTING}: {e}"))
+            })
+        })
+        .transpose()
+}
+
+/// Authorization: returns the passwords Windmill generated on the cluster and checks nothing.
+/// Callers MUST be superadmin or an internal server path.
+pub(crate) async fn read_external_instance_pg_state<'c>(
+    executor: impl sqlx::PgExecutor<'c>,
+) -> Result<ExternalInstancePgState> {
+    let value = sqlx::query_scalar!(
+        "SELECT value FROM global_settings WHERE name = $1",
+        EXTERNAL_INSTANCE_PG_STATE_SETTING
+    )
+    .fetch_optional(executor)
+    .await?;
+    match value {
+        None => Ok(ExternalInstancePgState::default()),
+        Some(v) => serde_json::from_value(v).map_err(|e| {
+            Error::internal_err(format!("reading {EXTERNAL_INSTANCE_PG_STATE_SETTING}: {e}"))
+        }),
+    }
+}
+
+/// Authorization: reads the hidden cluster state and checks nothing. Callers MUST be superadmin.
+pub async fn external_instance_pg_status(db: &DB) -> Result<ExternalInstancePgStatus> {
+    let configured = read_external_instance_pg_config(db).await?.is_some();
+    let state = read_external_instance_pg_state(db).await?;
+    Ok(ExternalInstancePgStatus {
+        configured,
+        database_count: state.databases.len(),
+        last_setup: state.last_setup,
+    })
+}
+
+/// The databases Windmill created on the external cluster, without the passwords kept beside them.
+///
+/// Authorization: names every database across all workspaces, and the workspace each fork copy is
+/// reserved for, and checks nothing. Callers MUST be superadmin or an internal authorization or
+/// lifecycle path that does not return the names to a workspace caller.
+pub async fn external_instance_databases(db: &DB) -> Result<BTreeMap<String, CustomInstanceDb>> {
+    Ok(read_external_instance_pg_state(db).await?.databases)
+}
+
+/// The workspaces whose data tables or Ducklake catalogs name each database on the external cluster,
+/// and the forks whose Ducklake metadata schemas there are still waiting to be dropped: those rows
+/// outlive a settings change, and cleanup cannot drop a schema in a database that is gone. A row
+/// whose schema is already dropped only waits on object storage, which needs no database.
+///
+/// Authorization: reads every workspace's settings and checks nothing. Callers MUST be superadmin
+/// or an internal lifecycle path.
+pub async fn external_instance_database_usages<'c>(
+    db: impl sqlx::PgExecutor<'c>,
+) -> Result<BTreeMap<String, BTreeSet<String>>> {
+    let rows = sqlx::query_as::<_, (String, String)>(
+        "SELECT ws.workspace_id, entry->'database'->>'resource_path'
+         FROM workspace_settings ws
+         CROSS JOIN LATERAL jsonb_each(
+             CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
+                 THEN ws.datatable->'datatables'
+                 ELSE '{}'::jsonb END
+         ) AS dt(k, entry)
+         WHERE entry->'database'->>'resource_type' = 'external_instance'
+           AND entry->'database'->>'resource_path' IS NOT NULL
+         UNION ALL
+         SELECT ws.workspace_id, entry->'catalog'->>'resource_path'
+         FROM workspace_settings ws
+         CROSS JOIN LATERAL jsonb_each(
+             CASE WHEN jsonb_typeof(ws.ducklake->'ducklakes') = 'object'
+                 THEN ws.ducklake->'ducklakes'
+                 ELSE '{}'::jsonb END
+         ) AS dl(k, entry)
+         WHERE entry->'catalog'->>'resource_type' = 'external_instance'
+           AND entry->'catalog'->>'resource_path' IS NOT NULL
+         UNION ALL
+         SELECT workspace_id, substring(catalog FROM length('external_instance:') + 1)
+         FROM fork_ducklake_namespace
+         WHERE catalog LIKE 'external\\_instance:%' AND NOT schema_dropped",
+    )
+    .fetch_all(db)
+    .await?;
+    let mut usages: BTreeMap<String, BTreeSet<String>> = BTreeMap::new();
+    for (workspace_id, dbname) in rows {
+        usages.entry(dbname).or_default().insert(workspace_id);
+    }
+    Ok(usages)
+}
+
+/// Refuse to unset the cluster while Windmill still has databases or data table roles on it, or a
+/// workspace still points at one: every data table there would stop resolving, and every role
+/// would be a login nothing can drop any more. Allowed on every edition, so a
+/// downgraded instance can still clear a setting it no longer uses.
+async fn ensure_external_instance_pg_removable(conn: &mut sqlx::PgConnection) -> Result<()> {
+    ensure_external_instance_pg_unused(conn, &format!("removing {EXTERNAL_INSTANCE_PG_SETTING}")).await
+}
+
+/// Refuse while Windmill has databases or data table roles on the cluster, or a workspace points
+/// at one of its databases. `before` finishes the sentence saying what to do first.
+async fn ensure_external_instance_pg_unused(
+    conn: &mut sqlx::PgConnection,
+    before: &str,
+) -> Result<()> {
+    let state = read_external_instance_pg_state(&mut *conn).await?;
+    let usages = external_instance_database_usages(&mut *conn).await?;
+    let roles = sqlx::query_scalar::<_, String>(
+        "SELECT name FROM datatable_role WHERE cluster = 'external_instance' ORDER BY name",
+    )
+    .fetch_all(&mut *conn)
+    .await?;
+    if state.databases.is_empty() && usages.is_empty() && roles.is_empty() {
+        return Ok(());
+    }
+    let mut held = vec![];
+    if !(state.databases.is_empty() && usages.is_empty()) {
+        let names = state
+            .databases
+            .keys()
+            .chain(usages.keys())
+            .collect::<BTreeSet<_>>()
+            .into_iter()
+            .cloned()
+            .collect::<Vec<_>>()
+            .join(", ");
+        held.push(format!("databases in use ({names})"));
+    }
+    if !roles.is_empty() {
+        held.push(format!("data table roles ({})", roles.join(", ")));
+    }
+    Err(Error::BadRequest(format!(
+        "The external instance cluster still holds {}. Drop them and repoint the data tables and \
+         Ducklake catalogs using them before {before}.",
+        held.join(" and ")
+    )))
+}
+
+/// Refuse a workspace setting that newly names an `external_instance` database on an edition
+/// without them.
+pub fn ensure_external_instance_available() -> Result<()> {
+    crate::external_instance_pg_oss::ensure_external_instance_available()
+}
+
+/// The connection an `external_instance` database resolves to: `custom_instance_user`, or the
+/// replication user, on the external cluster.
+///
+/// Authorization: returns live credentials and checks nothing. Callers MUST have authorized access
+/// to the data table that names `dbname`.
+pub async fn external_instance_connection_unchecked(
+    db: &DB,
+    dbname: &str,
+    replication: bool,
+) -> Result<crate::PgDatabase> {
+    crate::external_instance_pg_oss::external_instance_connection_unchecked(db, dbname, replication)
+        .await
+}
+
+/// Create `dbname` on the external cluster and register it. Refuses a name already taken there,
+/// whoever took it.
+///
+/// Runs on `tx`, which it takes [`lock_external_instance_pg_state`] on: the registration lands when
+/// the caller commits. Take that lock before any database governance lock, as saves do.
+///
+/// Authorization: checks nothing. Callers MUST be superadmin, or be cloning a data table they may
+/// fork into a `wm_fork_` database.
+pub async fn create_external_instance_database_unchecked(
+    db: &DB,
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    dbname: &str,
+    tag: &str,
+    for_workspace: Option<&str>,
+) -> Result<()> {
+    crate::external_instance_pg_oss::create_external_instance_database_unchecked(
+        db,
+        tx,
+        dbname,
+        tag,
+        for_workspace,
+    )
+    .await
+}
+
+/// Drop `dbname` from the external cluster: only a database Windmill registered creating, and still
+/// carries the mark it set there. Refused while anything uses it
+/// ([`crate::workspaces::managed_database_uses`]), except the `exempt` data table entry: the fork
+/// copy being cleaned up.
+///
+/// Runs on `tx`, like [`create_external_instance_database_unchecked`]: the database is gone at
+/// once, its registry entry when the caller commits.
+///
+/// Authorization: checks nothing. Callers MUST be superadmin, or be deleting the fork that owns
+/// this `wm_fork_` database.
+pub async fn drop_external_instance_database_unchecked(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    dbname: &str,
+    exempt: Option<(&str, &str)>,
+) -> Result<()> {
+    crate::external_instance_pg_oss::drop_external_instance_database_unchecked(tx, dbname, exempt)
+        .await
+}
+
+/// Serializes everything that changes which databases exist on the external cluster, or which data
+/// tables name them: setup, creates, drops, and data table saves. Held until `tx` ends.
+pub async fn lock_external_instance_pg_state(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+) -> Result<()> {
+    sqlx::query("SELECT pg_advisory_xact_lock(hashtext($1))")
+        .bind(EXTERNAL_INSTANCE_PG_STATE_SETTING)
+        .execute(&mut **tx)
+        .await?;
+    Ok(())
+}
+
+/// Refuse a data table naming `dbname` unless Windmill created it on the external cluster. Takes
+/// the lock drops take, so none can remove the database before `tx`, which saves the data table,
+/// commits.
+///
+/// Authorization: its refusal says whether Windmill created a database of that name, which is
+/// instance-wide knowledge. Callers MUST have authorized the caller as superadmin first.
+pub async fn ensure_external_instance_database_registered(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    dbname: &str,
+) -> Result<()> {
+    lock_external_instance_pg_state(tx).await?;
+    if read_external_instance_pg_state(&mut **tx)
+        .await?
+        .databases
+        .contains_key(dbname)
+    {
+        return Ok(());
+    }
+    Err(Error::BadRequest(format!(
+        "Windmill did not create a database named '{dbname}' on the external instance cluster. \
+         Create it from the instance settings first."
+    )))
+}
+
+/// Write [`EXTERNAL_INSTANCE_PG_SETTING`]: `None`, null or an empty string unsets it. Every writer
+/// of global settings goes through this for that key — the per-key and bulk endpoints as well as
+/// the declarative sync — instead of writing the row itself.
+///
+/// The checks and the write share one transaction holding [`lock_external_instance_pg_state`]. A
+/// check taken outside it could pass while a database create still reads the old cluster, which
+/// would then register a database there after the setting names another one.
+///
+/// Authorization: checks nothing. Callers MUST be superadmin, or the declarative instance config
+/// sync, which applies what the operator deployed.
+pub async fn write_external_instance_pg_setting(
+    db: &DB,
+    value: Option<&serde_json::Value>,
+) -> Result<()> {
+    let value = match value {
+        None | Some(serde_json::Value::Null) => None,
+        Some(serde_json::Value::String(s)) if s.trim().is_empty() => None,
+        Some(value) => Some(value),
+    };
+    let mut tx = db.begin().await?;
+    lock_external_instance_pg_state(&mut tx).await?;
+    // Every check runs on this transaction's own connection: it holds the advisory lock, and
+    // taking a second connection from the pool while other writers queue on that lock is how a
+    // small pool deadlocks.
+    match value {
+        None => {
+            ensure_external_instance_pg_removable(&mut tx).await?;
+            sqlx::query("DELETE FROM global_settings WHERE name = $1")
+                .bind(EXTERNAL_INSTANCE_PG_SETTING)
+                .execute(&mut *tx)
+                .await?;
+        }
+        Some(value) => {
+            crate::external_instance_pg_oss::validate_external_instance_pg_setting(value)?;
+            ensure_external_instance_pg_not_repointed(&mut tx, value).await?;
+            sqlx::query(
+                "INSERT INTO global_settings (name, value) VALUES ($1, $2)
+                 ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value, updated_at = now()",
+            )
+            .bind(EXTERNAL_INSTANCE_PG_SETTING)
+            .bind(value)
+            .execute(&mut *tx)
+            .await?;
+        }
+    }
+    tx.commit().await?;
+    tracing::info!(
+        "{} global setting {EXTERNAL_INSTANCE_PG_SETTING}",
+        if value.is_some() { "Set" } else { "Unset" }
+    );
+    Ok(())
+}
+
+/// [`write_external_instance_pg_setting`] for a settings diff: writes the key if the diff touches
+/// it, and takes it out of the diff so the generic apply does not write it again.
+///
+/// Authorization: checks nothing. Callers MUST be superadmin, or the declarative instance config
+/// sync, which applies what the operator deployed.
+pub async fn write_external_instance_pg_from_diff(
+    db: &DB,
+    diff: &mut crate::instance_config::SettingsDiff,
+) -> Result<()> {
+    if let Some(value) = diff.upserts.remove(EXTERNAL_INSTANCE_PG_SETTING) {
+        write_external_instance_pg_setting(db, Some(&value)).await?;
+    }
+    if let Some(i) = diff
+        .deletes
+        .iter()
+        .position(|k| k == EXTERNAL_INSTANCE_PG_SETTING)
+    {
+        diff.deletes.remove(i);
+        write_external_instance_pg_setting(db, None).await?;
+    }
+    Ok(())
+}
+
+/// Refuse pointing the setting at another host or port while databases or data table roles live on
+/// the current one. Data tables name databases, and the role catalog names logins, not clusters, so
+/// both would silently resolve to whatever the new cluster holds under the same names. Other fields
+/// (admin login, sslmode) may change freely.
+async fn ensure_external_instance_pg_not_repointed(
+    conn: &mut sqlx::PgConnection,
+    value: &serde_json::Value,
+) -> Result<()> {
+    let Some(current) = read_external_instance_pg_config(&mut *conn).await? else {
+        return Ok(());
+    };
+    let Ok(desired) = serde_json::from_value::<ExternalInstancePg>(value.clone()) else {
+        return Ok(());
+    };
+    if external_instance_pg_address(&current) == external_instance_pg_address(&desired) {
+        return Ok(());
+    }
+    ensure_external_instance_pg_unused(
+        conn,
+        &format!("pointing {EXTERNAL_INSTANCE_PG_SETTING} at another cluster"),
+    )
+    .await
+}
+
+/// Converge the external cluster on the configured login: check what it can do, create or update
+/// Windmill's two roles with the stored passwords, and report anything that would get in the way.
+/// With `rotate_passwords`, generate new passwords first. Safe to run again; running it again is
+/// how a failed rotation is repaired.
+///
+/// Authorization: administers the external cluster with its admin credentials and checks nothing.
+/// Callers MUST be superadmin.
+pub async fn setup_external_instance_pg_unchecked(
+    db: &DB,
+    rotate_passwords: bool,
+) -> Result<ExternalInstancePgSetupReport> {
+    crate::external_instance_pg_oss::setup_external_instance_pg_unchecked(db, rotate_passwords)
+        .await
+}

backend/windmill-common/src/external_instance_pg_oss.rs

@@ -0,0 +1,82 @@
+/*
+ * Author: Ruben Fiszel
+ * Copyright: Windmill Labs, Inc 2022
+ * This file and its contents are licensed under the AGPLv3 License.
+ * Please see the included NOTICE for copyright information and
+ * LICENSE-AGPL for a copy of the license.
+ */
+
+//! Where the external instance cluster comes from: the enterprise implementation, or a refusal.
+//! `private` alone is not that edition: community builds carry it.
+
+use crate::error::Error;
+
+pub fn external_instance_pg_unavailable() -> Error {
+    Error::BadRequest(
+        "External instance databases are a Windmill Enterprise Edition feature".to_string(),
+    )
+}
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+pub(crate) use crate::external_instance_pg_ee::{
+    create_external_instance_database_unchecked, drop_external_instance_database_unchecked,
+    external_instance_connection_unchecked, setup_external_instance_pg_unchecked,
+    validate_external_instance_pg_setting,
+};
+
+#[cfg(all(feature = "private", feature = "enterprise"))]
+pub(crate) fn ensure_external_instance_available() -> crate::error::Result<()> {
+    Ok(())
+}
+
+#[cfg(not(all(feature = "private", feature = "enterprise")))]
+pub(crate) use ce::*;
+
+#[cfg(not(all(feature = "private", feature = "enterprise")))]
+mod ce {
+    use super::external_instance_pg_unavailable as unavailable;
+    use crate::{
+        error::Result, external_instance_pg::ExternalInstancePgSetupReport, PgDatabase, DB,
+    };
+
+    pub(crate) fn validate_external_instance_pg_setting(_value: &serde_json::Value) -> Result<()> {
+        Err(unavailable())
+    }
+
+    pub(crate) fn ensure_external_instance_available() -> Result<()> {
+        Err(unavailable())
+    }
+
+    pub(crate) async fn setup_external_instance_pg_unchecked(
+        _db: &DB,
+        _rotate_passwords: bool,
+    ) -> Result<ExternalInstancePgSetupReport> {
+        Err(unavailable())
+    }
+
+    pub(crate) async fn external_instance_connection_unchecked(
+        _db: &DB,
+        _dbname: &str,
+        _replication: bool,
+    ) -> Result<PgDatabase> {
+        Err(unavailable())
+    }
+
+    pub(crate) async fn create_external_instance_database_unchecked(
+        _db: &DB,
+        _tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+        _dbname: &str,
+        _tag: &str,
+        _for_workspace: Option<&str>,
+    ) -> Result<()> {
+        Err(unavailable())
+    }
+
+    pub(crate) async fn drop_external_instance_database_unchecked(
+        _tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+        _dbname: &str,
+        _exempt: Option<(&str, &str)>,
+    ) -> Result<()> {
+        Err(unavailable())
+    }
+}

backend/windmill-common/src/flow_conversations.rs

@@ -69,7 +69,9 @@ pub async fn get_or_create_conversation_with_id(
     is_test: bool,
 ) -> Result<FlowConversation> {
     if let Some(existing) = lock_conversation(tx, w_id, conversation_id).await? {
-        return same_kind(existing, is_test);
+        let existing = same_kind(existing, is_test)?;
+        refuse_running_turn(tx, conversation_id).await?;
+        return Ok(existing);
     }
 
     // Truncate title to 25 characters max
@@ -104,7 +106,74 @@ pub async fn get_or_create_conversation_with_id(
                 "conversation {conversation_id} belongs to another workspace"
             ))
         })?;
-    same_kind(existing, is_test)
+    let existing = same_kind(existing, is_test)?;
+    refuse_running_turn(tx, conversation_id).await?;
+    Ok(existing)
+}
+
+/// The turn a conversation is still answering: its newest user message, while the flow run
+/// that message started is still queued or running.
+#[derive(Serialize, Debug, Clone, Copy)]
+pub struct RunningTurn {
+    pub job_id: Uuid,
+    /// `created_seq` of the user message that started the turn.
+    pub user_seq: i64,
+}
+
+/// One running turn per conversation holds its agent memory; a second run would write the
+/// same memory concurrently. Checked under the conversation's row lock, so two runs sent at
+/// once cannot both pass: the second waits, then sees the first's message and queued job.
+async fn refuse_running_turn(
+    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
+    conversation_id: Uuid,
+) -> Result<()> {
+    let Some(turn) = running_turns(&mut **tx, &[conversation_id])
+        .await?
+        .remove(&conversation_id)
+    else {
+        return Ok(());
+    };
+    // A JSON body, so a chat client can follow the running turn instead of failing.
+    Err(crate::error::Error::Generic(
+        axum::http::StatusCode::CONFLICT,
+        serde_json::json!({
+            "error": "this conversation is still answering a message; wait for it to finish or stop it before sending another",
+            "running_turn": turn,
+        })
+        .to_string(),
+    ))
+}
+
+/// The running turn of each of `conversation_ids` that has one.
+///
+/// It answers for whatever ids it is given and checks no permission of its own, so the
+/// executor must be one the caller is entitled to read those conversations through: a
+/// `user_db` transaction under RLS, or a transaction holding ids the caller has already
+/// authorized. Handed a raw pool and ids from a request, it would report other users' jobs.
+pub async fn running_turns<'e, E: sqlx::PgExecutor<'e>>(
+    executor: E,
+    conversation_ids: &[Uuid],
+) -> Result<std::collections::HashMap<Uuid, RunningTurn>> {
+    let rows = sqlx::query!(
+        r#"SELECT c.id AS "id!", u.job_id AS "job_id!", u.created_seq AS "user_seq!"
+         FROM unnest($1::uuid[]) AS c(id)
+         CROSS JOIN LATERAL (
+             SELECT job_id, created_seq
+             FROM flow_conversation_message
+             WHERE conversation_id = c.id AND message_type = 'user'
+             ORDER BY created_seq DESC
+             LIMIT 1
+         ) u
+         WHERE u.job_id IS NOT NULL
+           AND EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = u.job_id)"#,
+        conversation_ids
+    )
+    .fetch_all(executor)
+    .await?;
+    Ok(rows
+        .into_iter()
+        .map(|r| (r.id, RunningTurn { job_id: r.job_id, user_seq: r.user_seq }))
+        .collect())
 }
 
 /// `memory_id` is the caller's to choose, so a preview run could name a deployed

backend/windmill-common/src/global_settings.rs

@@ -57,6 +57,11 @@ pub const SAML_METADATA_SETTING: &str = "saml_metadata";
 pub const SMTP_SETTING: &str = "smtp_settings";
 pub const TEAMS_SETTING: &str = "teams";
 pub const INDEXER_SETTING: &str = "indexer_settings";
+pub const EXTERNAL_INSTANCE_PG_SETTING: &str = "external_instance_pg";
+/// Turns off Windmill's own Postgres as a data table and Ducklake substrate. Absent means on,
+/// which is what every instance that predates the setting expects.
+pub const INSTANCE_PG_DISABLED_SETTING: &str = "instance_pg_disabled";
+pub const EXTERNAL_INSTANCE_PG_STATE_SETTING: &str = "external_instance_pg_state";
 pub const TIMEOUT_WAIT_RESULT_SETTING: &str = "timeout_wait_result";
 
 pub const UNIQUE_ID_SETTING: &str = "uid";
@@ -448,6 +453,9 @@ pub const AGENT_WORKER_BLOCKED_SETTINGS: &[&str] = &[
     // resolve datatable connections through the dedicated datatable endpoints, never these.
     "custom_instance_pg_databases",
     "custom_instance_replication_pwd",
+    // The external cluster's admin login, and the passwords Windmill generated on it.
+    EXTERNAL_INSTANCE_PG_SETTING,
+    EXTERNAL_INSTANCE_PG_STATE_SETTING,
 ];
 
 /// Whether an agent worker may read the given global setting over HTTP.

backend/windmill-common/src/instance_config.rs

@@ -354,6 +354,8 @@ pub struct GlobalSettings {
     pub ducklake_settings: Option<DucklakeSettings>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub custom_instance_pg_databases: Option<CustomInstancePgDatabases>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub external_instance_pg: Option<ExternalInstancePg>,
 
     // Opaque settings (EE-private structs or no clear schema)
     #[serde(skip_serializing_if = "Option::is_none")]
@@ -477,6 +479,8 @@ pub struct SmtpSettings {
     pub smtp_tls_implicit: Option<bool>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub smtp_disable_tls: Option<bool>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub smtp_clicktracking_off: Option<bool>,
 }
 
 // ---------------------------------------------------------------------------
@@ -813,6 +817,9 @@ pub struct CustomInstanceDb {
     pub error: Option<String>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub tag: Option<String>,
+    /// The workspace a member created this fork copy for. Absent when a superadmin created it.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub workspace_id: Option<String>,
 }
 
 /// Setup log entries for a custom instance database.
@@ -839,6 +846,36 @@ pub struct CustomInstanceDbLogs {
     pub user_connect: String,
 }
 
+// ---------------------------------------------------------------------------
+// External instance PG cluster
+// ---------------------------------------------------------------------------
+
+/// The external Postgres cluster Windmill manages for `external_instance` data tables and Ducklake
+/// catalogs. `user` logs in as the cluster's administrator: it needs `CREATEDB` and `CREATEROLE`.
+/// `dbname` is only where that login connects to run cluster-wide statements.
+///
+/// Every field defaults rather than being required: this deserializes as part of the whole
+/// instance config, and one malformed row must not make every other setting unreadable. The
+/// write path and every use reject an incomplete value instead.
+#[derive(Deserialize, Serialize, Clone, Debug, Default)]
+#[cfg_attr(feature = "instance_config_schema", derive(schemars::JsonSchema))]
+pub struct ExternalInstancePg {
+    #[serde(default)]
+    pub host: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub port: Option<u16>,
+    #[serde(default)]
+    pub user: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub password: Option<StringOrSecretRef>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub dbname: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub sslmode: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub root_certificate_pem: Option<String>,
+}
+
 // ---------------------------------------------------------------------------
 // Autoscaling (worker config)
 // ---------------------------------------------------------------------------
@@ -975,6 +1012,7 @@ pub const PROTECTED_SETTINGS: &[&str] = &[
     "ducklake_settings",
     "custom_instance_pg_databases",
     "custom_instance_replication_pwd",
+    "external_instance_pg_state",
     "uid",
     "rsa_keys",
     "jwt_secret",
@@ -1000,6 +1038,8 @@ pub const HIDDEN_SETTINGS: &[&str] = &[
     // Server-only (written by setup/refresh via direct SQL), never operator-authored —
     // hidden so the config machinery can't read, rewrite, or drop it.
     "custom_instance_replication_pwd",
+    // Same for the passwords and database registry Windmill keeps for the external cluster.
+    "external_instance_pg_state",
 ];
 
 /// Top-level settings whose entire value is sensitive and must be fully redacted in logs.
@@ -1011,6 +1051,7 @@ const SENSITIVE_SETTINGS: &[&str] = &[
     "license_key",
     "ducklake_user_pg_pwd",
     "custom_instance_replication_pwd",
+    "external_instance_pg_state",
     "pip_index_url",
     "pip_extra_index_url",
     "npm_config_registry",
@@ -1036,6 +1077,7 @@ const NESTED_SENSITIVE_FIELDS: &[(&str, &[&str])] = &[
         &["secret_key", "serviceAccountKey", "accessKey"],
     ),
     ("custom_instance_pg_databases", &["user_pwd"]),
+    ("external_instance_pg", &["password"]),
     ("github_enterprise_app", &["private_key"]),
 ];
 
@@ -1377,7 +1419,8 @@ pub async fn sync_global_settings_declarative(
     crate::global_settings::parse_max_token_expiration_days(desired.get(max_expiration_key))
         .map_err(|e| anyhow::anyhow!("{max_expiration_key}: {e}"))?;
 
-    let diff = diff_global_settings(current, desired, ApplyMode::Replace);
+    let mut diff = diff_global_settings(current, desired, ApplyMode::Replace);
+    crate::external_instance_pg::write_external_instance_pg_from_diff(db, &mut diff).await?;
     apply_settings_diff(db, &diff).await?;
 
     Ok(())
@@ -1510,6 +1553,10 @@ pub fn resolve_env_refs(settings: &mut GlobalSettings) -> Result<(), String> {
         resolve_env_option(&mut pg.user_pwd)?;
     }
 
+    if let Some(pg) = &mut settings.external_instance_pg {
+        resolve_env_option(&mut pg.password)?;
+    }
+
     Ok(())
 }
 
@@ -2282,6 +2329,20 @@ mod tests {
         assert_eq!(original.custom_tags, reconstructed.custom_tags);
     }
 
+    #[test]
+    fn smtp_settings_keep_clicktracking_off() {
+        let smtp = serde_json::json!({
+            "smtp_host": "smtp.example.com",
+            "smtp_port": 587,
+            "smtp_tls_implicit": false,
+            "smtp_disable_tls": false,
+            "smtp_clicktracking_off": true,
+        });
+        let settings: GlobalSettings =
+            serde_json::from_value(serde_json::json!({ "smtp_settings": smtp })).unwrap();
+        assert_eq!(settings.to_settings_map()["smtp_settings"], smtp);
+    }
+
     // -----------------------------------------------------------------------
     // Serialization edge cases
     // -----------------------------------------------------------------------
@@ -2479,39 +2540,33 @@ mod tests {
     }
 
     #[test]
-    fn custom_instance_replication_pwd_is_isolated_from_config() {
-        // The replication-role password is server-only: written by setup/refresh via direct
-        // SQL, never operator-authored. It must stay out of the declarative config surface
-        // (hidden on read) and be undeletable, so config sync can't read, rewrite, or drop it.
-        assert!(HIDDEN_SETTINGS.contains(&"custom_instance_replication_pwd"));
-        assert!(PROTECTED_SETTINGS.contains(&"custom_instance_replication_pwd"));
-        assert!(SENSITIVE_SETTINGS.contains(&"custom_instance_replication_pwd"));
+    fn server_generated_db_passwords_are_isolated_from_config() {
+        // These hold passwords the server generates: written by setup/refresh via direct SQL,
+        // never operator-authored. They must stay out of the declarative config surface
+        // (hidden on read) and be undeletable, so config sync can't read, rewrite, or drop them.
+        for key in [
+            "custom_instance_replication_pwd",
+            "external_instance_pg_state",
+        ] {
+            assert!(HIDDEN_SETTINGS.contains(&key), "{key}");
+            assert!(PROTECTED_SETTINGS.contains(&key), "{key}");
+            assert!(SENSITIVE_SETTINGS.contains(&key), "{key}");
 
-        // A stray desired value (e.g. flattened into `extra`) is ignored, not upserted.
-        let mut desired = BTreeMap::new();
-        desired.insert(
-            "custom_instance_replication_pwd".to_string(),
-            serde_json::json!("attacker-set"),
-        );
-        let diff = diff_global_settings(&BTreeMap::new(), &desired, ApplyMode::Merge);
-        assert!(
-            diff.upserts.is_empty(),
-            "hidden setting must not be upserted"
-        );
+            // A stray desired value (e.g. flattened into `extra`) is ignored, not upserted.
+            let mut desired = BTreeMap::new();
+            desired.insert(key.to_string(), serde_json::json!("attacker-set"));
+            let diff = diff_global_settings(&BTreeMap::new(), &desired, ApplyMode::Merge);
+            assert!(diff.upserts.is_empty(), "{key} must not be upserted");
 
-        // A current value is never deleted by a Replace that omits it.
-        let mut current = BTreeMap::new();
-        current.insert(
-            "custom_instance_replication_pwd".to_string(),
-            serde_json::json!("live"),
-        );
-        let diff = diff_global_settings(&current, &BTreeMap::new(), ApplyMode::Replace);
-        assert!(
-            !diff
-                .deletes
-                .contains(&"custom_instance_replication_pwd".to_string()),
-            "hidden setting must not be deleted"
-        );
+            // A current value is never deleted by a Replace that omits it.
+            let mut current = BTreeMap::new();
+            current.insert(key.to_string(), serde_json::json!("live"));
+            let diff = diff_global_settings(&current, &BTreeMap::new(), ApplyMode::Replace);
+            assert!(
+                !diff.deletes.contains(&key.to_string()),
+                "{key} must not be deleted"
+            );
+        }
     }
 
     #[test]

backend/windmill-common/src/job_provenance.rs

@@ -0,0 +1,104 @@
+use uuid::Uuid;
+
+use crate::{db::DB, error::Result, jobs::JobKind};
+
+/// Where a job's code comes from, as far as its chain of parents can prove it.
+pub struct JobProvenance {
+    /// True only when the job and every ancestor run code at a path taken from a
+    /// deployed version, never from the request that created the job.
+    pub deployed: bool,
+    pub parent_path: Option<String>,
+    pub root_id: Uuid,
+    pub root_path: Option<String>,
+    pub root_kind: JobKind,
+    pub root_trigger_kind: Option<String>,
+}
+
+struct LineageJob {
+    id: Uuid,
+    parent_job: Option<Uuid>,
+    kind: JobKind,
+    runnable_path: Option<String>,
+    trigger_kind: Option<String>,
+    origin_verified: bool,
+}
+
+/// Reads any job of `w_id` regardless of the caller: authorize access to `job_id` first.
+pub async fn job_provenance(db: &DB, job_id: &Uuid, w_id: &str) -> Result<Option<JobProvenance>> {
+    let lineage = sqlx::query_as!(
+        LineageJob,
+        r#"WITH RECURSIVE lineage AS (
+            SELECT id, parent_job, kind, runnable_path, runnable_id, trigger_kind, trigger,
+                0 AS depth
+            FROM v2_job WHERE id = $1 AND workspace_id = $2
+          UNION ALL
+            SELECT p.id, p.parent_job, p.kind, p.runnable_path, p.runnable_id, p.trigger_kind,
+                p.trigger, l.depth + 1
+            FROM v2_job p JOIN lineage l ON p.id = l.parent_job
+            WHERE p.workspace_id = $2 AND l.depth < 100
+        )
+        SELECT id AS "id!", parent_job, kind AS "kind!: JobKind", runnable_path,
+            trigger_kind::text AS trigger_kind,
+            -- A restart takes its flow version from the request, so a trusted flow path
+            -- can carry another flow's code: the version must belong to that path.
+            CASE kind
+                WHEN 'flow' THEN EXISTS (SELECT 1 FROM flow_version fv
+                    WHERE fv.id = runnable_id AND fv.path = runnable_path AND fv.workspace_id = $2)
+                WHEN 'script' THEN EXISTS (SELECT 1 FROM script s
+                    WHERE s.hash = runnable_id AND s.path = runnable_path AND s.workspace_id = $2)
+                -- An app editor preview also runs an app script, at an app path it does not
+                -- have to own; only deployed-app runs are stamped with their app.
+                WHEN 'appscript' THEN COALESCE(trigger_kind = 'app'
+                    AND starts_with(runnable_path, trigger || '/'), false)
+                ELSE true
+            END AS "origin_verified!"
+        FROM lineage ORDER BY depth"#,
+        job_id,
+        w_id
+    )
+    .fetch_all(db)
+    .await?;
+
+    let Some(root) = lineage.last() else {
+        return Ok(None);
+    };
+    // An ancestor missing from v2_job (other workspace, depth cap) leaves the chain unproven.
+    let deployed = root.parent_job.is_none()
+        && lineage
+            .iter()
+            .all(|j| runs_stored_code(j.kind) && j.origin_verified);
+    Ok(Some(JobProvenance {
+        deployed,
+        parent_path: lineage.get(1).and_then(|j| j.runnable_path.clone()),
+        root_id: root.id,
+        root_path: root.runnable_path.clone(),
+        root_kind: root.kind,
+        root_trigger_kind: root.trigger_kind.clone(),
+    }))
+}
+
+/// Whether the job's code and its path are both fixed by what was deployed (or, for a
+/// hub script, published), rather than supplied by the request that created the job.
+fn runs_stored_code(kind: JobKind) -> bool {
+    match kind {
+        JobKind::Script
+        | JobKind::Script_Hub
+        | JobKind::Flow
+        | JobKind::SingleStepFlow
+        | JobKind::FlowScript
+        | JobKind::FlowNode
+        | JobKind::AIAgent
+        | JobKind::AppScript => true,
+        JobKind::Preview
+        | JobKind::FlowPreview
+        | JobKind::Dependencies
+        | JobKind::FlowDependencies
+        | JobKind::AppDependencies
+        | JobKind::Identity
+        | JobKind::Noop
+        | JobKind::DeploymentCallback
+        | JobKind::UnassignedScript
+        | JobKind::UnassignedFlow
+        | JobKind::UnassignedSinglestepFlow => false,
+    }
+}

backend/windmill-common/src/lib.rs

@@ -58,6 +58,10 @@ pub mod ee_oss;
 pub mod email_ee;
 pub mod email_oss;
 pub mod error;
+pub mod external_instance_pg;
+#[cfg(all(feature = "private", feature = "enterprise"))]
+mod external_instance_pg_ee;
+pub mod external_instance_pg_oss;
 pub mod external_ip;
 #[cfg(feature = "private")]
 pub mod feature_usage_ee;
@@ -86,6 +90,7 @@ pub mod workspace_dependencies;
 #[cfg(feature = "private")]
 pub mod git_sync_ee;
 pub mod git_sync_oss;
+pub mod job_provenance;
 pub mod jobs;
 pub mod jwt;
 pub mod login_rate_limit;
@@ -1114,7 +1119,13 @@ impl PgDatabase {
                 if err_str.contains("password authentication failed for user")
                     && err_str.contains("custom_instance_user")
                 {
-                    if let Some(db) = main_db {
+                    // The external instance cluster has a `custom_instance_user` of its own, whose
+                    // password setup manages. Rotating the local one would break every instance
+                    // data table and fix nothing.
+                    let local = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
+                    let on_local_cluster = local.host == self.host
+                        && local.port.unwrap_or(5432) == self.port.unwrap_or(5432);
+                    if let Some(db) = main_db.filter(|_| on_local_cluster) {
                         tracing::warn!(
                             "custom_instance_user password auth failed, refreshing and retrying..."
                         );
@@ -1664,13 +1675,41 @@ pub async fn instance_database_users(
 }
 
 /// Drop a custom instance database: validate, terminate connections, DROP DATABASE, remove from global_settings.
+///
+/// Authorization: drops any instance database but Windmill's own and checks nothing. Callers MUST
+/// be superadmin, or have established the caller may drop this one — a fork's owner cleaning up
+/// its own copy that nothing else uses.
 pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Result<()> {
     drop_custom_instance_database_on(&mut *db.acquire().await?, dbname).await
 }
 
+/// [`drop_custom_instance_database`] leaving its registry entry, for a caller holding row locks in
+/// a transaction: the registry write has to go through that transaction, as waiting on another
+/// connection for a lock the transaction's own peers hold is a deadlock Postgres cannot see. Same
+/// authorization contract.
+pub async fn drop_custom_instance_database_keep_entry(db: &DB, dbname: &str) -> error::Result<()> {
+    drop_instance_database_keep_entry_on(&mut *db.acquire().await?, dbname).await
+}
+
 async fn drop_custom_instance_database_on(
     conn: &mut sqlx::PgConnection,
     dbname: &str,
+) -> error::Result<()> {
+    let dbname = dbname.trim();
+    drop_instance_database_keep_entry_on(&mut *conn, dbname).await?;
+    // Always remove from global_settings
+    sqlx::query!(
+        r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
+        dbname
+    )
+    .execute(&mut *conn)
+    .await?;
+    Ok(())
+}
+
+async fn drop_instance_database_keep_entry_on(
+    conn: &mut sqlx::PgConnection,
+    dbname: &str,
 ) -> error::Result<()> {
     let dbname = dbname.trim();
     validate_dbname(dbname)?;
@@ -1717,14 +1756,6 @@ async fn drop_custom_instance_database_on(
         tracing::info!("Database '{}' does not exist, skipping drop", dbname);
     }
 
-    // Always remove from global_settings
-    sqlx::query!(
-        r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
-        dbname
-    )
-    .execute(&mut *conn)
-    .await?;
-
     Ok(())
 }
 
@@ -1749,26 +1780,30 @@ pub(crate) fn instance_db_grants(dbname: &str) -> String {
     )
 }
 
-/// Re-apply [`instance_db_grants`] to an instance database provisioned before data table roles
-/// existed, whose grants carry no grant option. Connects as the instance's own Postgres user —
-/// the database and `public` schema owner — since only it can hand out an option it holds.
+/// Re-apply [`instance_db_grants`] to a managed database provisioned before data table roles
+/// existed, whose grants carry no grant option. Connects as the cluster's administrator — the
+/// database and `public` schema owner — since only it can hand out an option it holds.
 ///
-/// Authorization: reaches an instance database with the server's own credentials and checks
+/// Authorization: reaches a managed database with the server's own credentials and checks
 /// nothing. Callers MUST have authorized administration of `dbname` — superadmin, or an admin of
 /// the workspace governing a data table on it.
 pub async fn ensure_instance_db_grant_options_unchecked(
     db: &DB,
+    cluster: crate::datatable_roles::DatatableRoleCluster,
     dbname: &str,
 ) -> error::Result<()> {
-    crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, dbname).await
+    crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, cluster, dbname)
+        .await
 }
 
 /// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
-/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake").
+/// The `tag` is stored in global_settings metadata (e.g. "datatable" or "ducklake"). `for_workspace`
+/// is the workspace a member creates a fork copy for; see [`ensure_fork_database_available_to`].
 pub async fn create_custom_instance_database(
     db: &DB,
     dbname: &str,
     tag: &str,
+    for_workspace: Option<&str>,
 ) -> error::Result<()> {
     let dbname = dbname.trim();
     validate_dbname(dbname)?;
@@ -1798,7 +1833,7 @@ pub async fn create_custom_instance_database(
 
     // Nothing names a database that failed past this point, and its name blocks the retry: drop it
     // rather than leave it behind.
-    if let Err(e) = finish_custom_instance_database(db, dbname, tag).await {
+    if let Err(e) = finish_custom_instance_database(db, dbname, tag, for_workspace).await {
         match drop_unused_instance_database(db, dbname).await {
             Ok(Cleanup::InUse(users)) => tracing::warn!(
                 "Kept '{dbname}' after failing to set it up: workspaces {} use it",
@@ -1819,7 +1854,13 @@ pub async fn create_custom_instance_database(
     // A data table role can only reach a database it may CONNECT to, and PUBLIC's default CONNECT
     // would otherwise let every role in regardless of what this instance defines. Best-effort: a
     // failure here leaves the database usable as `admin`, and the next role change repairs it.
-    if let Err(e) = crate::datatable_roles::converge_connect_grants(db, dbname).await {
+    if let Err(e) = crate::datatable_roles::converge_connect_grants(
+        db,
+        crate::datatable_roles::DatatableRoleCluster::Instance,
+        dbname,
+    )
+    .await
+    {
         tracing::warn!("Could not set CONNECT grants on instance database '{dbname}': {e}");
     }
 
@@ -1828,7 +1869,12 @@ pub async fn create_custom_instance_database(
 }
 
 /// Grant `custom_instance_user` its privileges on a database just created, and register it.
-async fn finish_custom_instance_database(db: &DB, dbname: &str, tag: &str) -> error::Result<()> {
+async fn finish_custom_instance_database(
+    db: &DB,
+    dbname: &str,
+    tag: &str,
+    for_workspace: Option<&str>,
+) -> error::Result<()> {
     // Grant permissions to custom_instance_user
     let wmill_pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
     let new_pg_creds = PgDatabase { dbname: dbname.to_string(), ..wmill_pg_creds };
@@ -1855,7 +1901,8 @@ async fn finish_custom_instance_database(db: &DB, dbname: &str, tag: &str) -> er
         },
         "success": true,
         "error": null,
-        "tag": tag
+        "tag": tag,
+        "workspace_id": for_workspace,
     });
     sqlx::query!(
         r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
@@ -1863,6 +1910,75 @@ async fn finish_custom_instance_database(db: &DB, dbname: &str, tag: &str) -> er
     )
     .execute(db)
     .await?;
+
+    Ok(())
+}
+
+/// The system's CA bundle file, for libpq clients that cannot take `sslrootcert=system`: that value
+/// needs libpq 16, and verify-full only.
+pub fn system_ca_bundle() -> Option<std::path::PathBuf> {
+    std::env::var_os("SSL_CERT_FILE")
+        .map(std::path::PathBuf::from)
+        .into_iter()
+        .chain(
+            [
+                "/etc/ssl/certs/ca-certificates.crt",
+                "/etc/pki/tls/certs/ca-bundle.crt",
+                "/etc/ssl/cert.pem",
+                "/etc/ssl/ca-bundle.pem",
+            ]
+            .map(std::path::PathBuf::from),
+        )
+        .find(|path| path.is_file())
+}
+
+/// Refuse a workspace member writing a fork copy into, or pointing a fork at, the managed database
+/// `dbname` of `kind`, unless `w_id` created it for that ([`create_custom_instance_database`], or
+/// its external instance counterpart) and nothing uses it yet. The `wm_fork_` prefix is no
+/// authorization: every database of a cluster answers to the same `custom_instance_user`, so a name
+/// is all it takes to reach another workspace's copy.
+///
+/// Runs on `conn`: its callers hold a transaction with the fork lock while they check, and a
+/// second connection taken from the pool under it is how a small pool deadlocks.
+///
+/// Authorization: reads the global registries and every workspace's settings, and names other
+/// workspaces in its refusal. Callers MUST have authorized `w_id` for the caller first — a member
+/// of it forking or importing there — and MUST NOT call it on a workspace the caller is not in.
+pub async fn ensure_fork_database_available_to(
+    conn: &mut sqlx::PgConnection,
+    kind: workspaces::DataTableCatalogResourceType,
+    dbname: &str,
+    w_id: &str,
+) -> error::Result<()> {
+    let created_for = match kind {
+        workspaces::DataTableCatalogResourceType::ExternalInstance => {
+            external_instance_pg::read_external_instance_pg_state(&mut *conn)
+                .await?
+                .databases
+                .remove(dbname)
+                .and_then(|entry| entry.workspace_id)
+        }
+        _ => sqlx::query_scalar::<_, Option<String>>(
+            "SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
+             WHERE name = 'custom_instance_pg_databases'",
+        )
+        .bind(dbname)
+        .fetch_optional(&mut *conn)
+        .await?
+        .flatten(),
+    };
+    if created_for.as_deref() != Some(w_id) {
+        return Err(Error::BadRequest(format!(
+            "Database '{dbname}' was not created for a fork of workspace '{w_id}'"
+        )));
+    }
+    let uses = workspaces::managed_database_uses(conn, kind, dbname, None).await?;
+    if !uses.is_empty() {
+        return Err(Error::BadRequest(format!(
+            "Database '{dbname}' is already in use: {}",
+            uses.join(", ")
+        )));
+    }
     Ok(())
 }
 

backend/windmill-common/src/oidc_oss.rs

@@ -64,6 +64,12 @@ pub struct JobClaim {
     pub fork_parent_workspace: Option<String>,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub end_user_email: Option<String>,
+    pub job_kind: String,
+    pub deployed: bool,
+    pub root_job_id: String,
+    pub root_path: Option<String>,
+    pub root_job_kind: String,
+    pub root_trigger_kind: Option<String>,
 }
 
 #[cfg(not(feature = "private"))]

backend/windmill-common/src/ssrf.rs

@@ -396,7 +396,10 @@ pub fn webhook_ssrf_error_message(e: &SsrfValidationError) -> String {
     }
 }
 
-fn is_private_ip(ip: &IpAddr) -> bool {
+/// Whether `ip` is loopback, private, link-local, or otherwise not a public
+/// internet destination. The one predicate every server-side outbound-URL guard
+/// (SSRF checks, git remotes, object storage tests) judges addresses with.
+pub fn is_private_ip(ip: &IpAddr) -> bool {
     match ip {
         IpAddr::V4(ipv4) => is_private_ipv4(ipv4),
         IpAddr::V6(ipv6) => is_private_ipv6(ipv6),
@@ -408,6 +411,7 @@ fn is_private_ipv4(ip: &Ipv4Addr) -> bool {
         || ip.is_private()        // 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
         || ip.is_link_local()     // 169.254.0.0/16 (AWS IMDS lives here)
         || ip.is_broadcast()      // 255.255.255.255
+        || ip.is_multicast()      // 224.0.0.0/4
         // RFC 1122 "this network": the whole /8, not just the unspecified
         // address `is_unspecified()` matches — stacks that map 0.x.y.z onto the
         // local host make `0.0.0.1` a bypass.
@@ -418,17 +422,60 @@ fn is_private_ipv4(ip: &Ipv4Addr) -> bool {
 }
 
 fn is_private_ipv6(ip: &Ipv6Addr) -> bool {
+    let seg = ip.segments();
     ip.is_loopback()           // ::1
         || ip.is_unspecified() // ::
+        || ip.is_multicast()   // ff00::/8
         // Unique local addresses (fc00::/7)
-        || (ip.segments()[0] & 0xfe00) == 0xfc00
-        // Link-local (fe80::/10)
-        || (ip.segments()[0] & 0xffc0) == 0xfe80
-        // IPv4-mapped addresses — check the embedded IPv4
-        || match ip.to_ipv4_mapped() {
-            Some(ipv4) => is_private_ipv4(&ipv4),
-            None => false,
-        }
+        || (seg[0] & 0xfe00) == 0xfc00
+        // Link-local (fe80::/10) and deprecated site-local (fec0::/10)
+        || (seg[0] & 0xffc0) == 0xfe80
+        || (seg[0] & 0xffc0) == 0xfec0
+        // NAT64 local-use prefix (64:ff9b:1::/48, RFC 8215): a site's own
+        // translator, whose IPv4 placement depends on the prefix length it uses.
+        || (seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2] == 0x0001)
+        || embedded_ipv4(ip).is_some_and(|v4| is_private_ipv4(&v4))
+}
+
+/// The IPv4 address an IPv6 address carries through a transition mechanism,
+/// which a dual-stack host or a translator on the path will deliver to. Guards
+/// must judge that address too, else e.g. `64:ff9b::a9fe:a9fe` reaches
+/// 169.254.169.254 through a NAT64 gateway.
+fn embedded_ipv4(ip: &Ipv6Addr) -> Option<Ipv4Addr> {
+    let seg = ip.segments();
+    let low32 = || {
+        Ipv4Addr::new(
+            (seg[6] >> 8) as u8,
+            seg[6] as u8,
+            (seg[7] >> 8) as u8,
+            seg[7] as u8,
+        )
+    };
+    match seg {
+        // IPv4-mapped ::ffff:0:0/96
+        [0, 0, 0, 0, 0, 0xffff, _, _] => Some(low32()),
+        // IPv4-translated ::ffff:0:0:0/96 (SIIT)
+        [0, 0, 0, 0, 0xffff, 0, _, _] => Some(low32()),
+        // IPv4-compatible ::/96 (deprecated, still routed by some stacks)
+        [0, 0, 0, 0, 0, 0, _, _] => Some(low32()),
+        // NAT64 well-known prefix 64:ff9b::/96
+        [0x0064, 0xff9b, 0, 0, 0, 0, _, _] => Some(low32()),
+        // 6to4 2002::/16: the site router's IPv4 sits right after the prefix
+        [0x2002, hi, lo, ..] => Some(Ipv4Addr::new(
+            (hi >> 8) as u8,
+            hi as u8,
+            (lo >> 8) as u8,
+            lo as u8,
+        )),
+        // Teredo 2001::/32: the client's IPv4 is stored bit-inverted
+        [0x2001, 0, ..] => Some(Ipv4Addr::new(
+            !(seg[6] >> 8) as u8,
+            !seg[6] as u8,
+            !(seg[7] >> 8) as u8,
+            !seg[7] as u8,
+        )),
+        _ => None,
+    }
 }
 
 #[cfg(test)]
@@ -486,31 +533,50 @@ mod tests {
     }
 
     #[test]
-    fn test_private_ipv4() {
-        assert!(is_private_ipv4(&"127.0.0.1".parse().unwrap()));
-        assert!(is_private_ipv4(&"10.0.0.1".parse().unwrap()));
-        assert!(is_private_ipv4(&"172.16.0.1".parse().unwrap()));
-        assert!(is_private_ipv4(&"192.168.1.1".parse().unwrap()));
-        assert!(is_private_ipv4(&"169.254.169.254".parse().unwrap()));
-        assert!(is_private_ipv4(&"0.0.0.0".parse().unwrap()));
-        // The whole 0.0.0.0/8 block, not just the unspecified address.
-        assert!(is_private_ipv4(&"0.0.0.1".parse().unwrap()));
-        assert!(is_private_ipv4(&"0.255.255.255".parse().unwrap()));
-        assert!(is_private_ipv4(&"100.64.0.1".parse().unwrap()));
-        assert!(!is_private_ipv4(&"8.8.8.8".parse().unwrap()));
-        assert!(!is_private_ipv4(&"1.1.1.1".parse().unwrap()));
-    }
-
-    #[test]
-    fn test_private_ipv6() {
-        assert!(is_private_ipv6(&"::1".parse().unwrap()));
-        assert!(is_private_ipv6(&"::".parse().unwrap()));
-        assert!(is_private_ipv6(&"fc00::1".parse().unwrap()));
-        assert!(is_private_ipv6(&"fd00::1".parse().unwrap()));
-        assert!(is_private_ipv6(&"fe80::1".parse().unwrap()));
-        // IPv4-mapped loopback
-        assert!(is_private_ipv6(&"::ffff:127.0.0.1".parse().unwrap()));
-        assert!(!is_private_ipv6(&"2001:4860:4860::8888".parse().unwrap()));
+    fn test_private_ip() {
+        for s in [
+            "127.0.0.1",
+            "10.0.0.1",
+            "172.16.0.1",
+            "192.168.1.1",
+            "169.254.169.254",
+            "100.64.0.1",
+            // The whole 0.0.0.0/8 block, not just the unspecified address.
+            "0.0.0.0",
+            "0.0.0.1",
+            "0.255.255.255",
+            "::1",
+            "::",
+            "fc00::1",
+            "fd00::1",
+            "fe80::1",
+            // IPv4 carried inside IPv6 by a transition mechanism.
+            "::ffff:127.0.0.1",         // IPv4-mapped
+            "::169.254.169.254",        // IPv4-compatible
+            "::ffff:0:a9fe:a9fe",       // IPv4-translated (SIIT)
+            "64:ff9b::a9fe:a9fe",       // NAT64 well-known prefix
+            "64:ff9b:1::a9fe:a9fe",     // NAT64 local-use prefix
+            "64:ff9b:1:ffff::1",        // any address in the local-use prefix
+            "2002:a9fe:a9fe::1",        // 6to4
+            "2002:7f00:1::",            // 6to4
+            "2001:0:1:2:0:0:80ff:fffe", // Teredo, client 127.0.0.1 bit-inverted
+            "2001:0:1:2::5601:5601",    // Teredo, client 169.254.169.254 bit-inverted
+        ] {
+            let ip: IpAddr = s.parse().unwrap();
+            assert!(is_private_ip(&ip), "{s} should be private");
+        }
+        for s in [
+            "8.8.8.8",
+            "1.1.1.1",
+            "2001:4860:4860::8888",
+            "2606:4700:4700::1111",
+            "64:ff9b::808:808",      // NAT64 of 8.8.8.8
+            "2002:808:808::1",       // 6to4 of 8.8.8.8
+            "2001:0:1:2::f7f7:f7f7", // Teredo, client 8.8.8.8
+        ] {
+            let ip: IpAddr = s.parse().unwrap();
+            assert!(!is_private_ip(&ip), "{s} should be public");
+        }
     }
 
     #[tokio::test]

backend/windmill-common/src/worker.rs

@@ -1939,6 +1939,9 @@ pub async fn update_ping_http(
                 insert_ping.occupancy_rate_30m,
                 insert_ping.native_mode.unwrap_or(false),
                 insert_ping.ip.as_deref(),
+                insert_ping
+                    .last_job_executed
+                    .zip(insert_ping.last_job_workspace_id.as_deref()),
                 db,
             )
             .await?
@@ -2220,12 +2223,15 @@ pub async fn update_worker_ping_main_loop_query(
     occupancy_rate_30m: Option<f32>,
     native_mode: bool,
     ip: Option<&str>,
+    last_job: Option<(Uuid, &str)>,
     db: &DB,
 ) -> anyhow::Result<()> {
+    let (last_job_id, last_job_workspace_id) = last_job.unzip();
     timeout(Duration::from_secs(10), sqlx::query!(
         "UPDATE worker_ping SET ping_at = now(), jobs_executed = $1, custom_tags = $2,
          occupancy_rate = $3, memory_usage = $4, wm_memory_usage = $5, vcpus = COALESCE($7, vcpus),
-         memory = COALESCE($8, memory), occupancy_rate_15s = $9, occupancy_rate_5m = $10, occupancy_rate_30m = $11, native_mode = $12, ip = COALESCE($13, ip) WHERE worker = $6",
+         memory = COALESCE($8, memory), occupancy_rate_15s = $9, occupancy_rate_5m = $10, occupancy_rate_30m = $11, native_mode = $12, ip = COALESCE($13, ip),
+         current_job_id = COALESCE($14, current_job_id), current_job_workspace_id = COALESCE($15, current_job_workspace_id) WHERE worker = $6",
         jobs_executed,
         tags,
         occupancy_rate,
@@ -2239,6 +2245,8 @@ pub async fn update_worker_ping_main_loop_query(
         occupancy_rate_30m,
         native_mode,
         ip,
+        last_job_id,
+        last_job_workspace_id,
     )
         .execute(db))
     .await??;

backend/windmill-common/src/workspaces.rs

@@ -1542,6 +1542,47 @@ pub enum DataTableCatalogResourceType {
     #[strum(serialize = "postgres")]
     Postgresql,
     Instance,
+    /// On the external instance cluster ([`crate::external_instance_pg`]). Enterprise Edition.
+    #[serde(rename = "external_instance")]
+    #[strum(serialize = "external_instance")]
+    ExternalInstance,
+}
+
+impl DataTableCatalogResourceType {
+    /// A database Windmill created and administers, on its own cluster or the external one, as
+    /// opposed to one a user brought as a resource.
+    pub fn is_windmill_managed(self) -> bool {
+        matches!(self, Self::Instance | Self::ExternalInstance)
+    }
+}
+
+/// Refuse a new use of Windmill's own Postgres as a data table or Ducklake substrate where an
+/// operator turned it off, and on the managed cloud, which never had it. Entries already on it
+/// keep resolving: this gates what a save may newly name, not what runs.
+pub async fn ensure_instance_pg_available<'c>(executor: impl sqlx::PgExecutor<'c>) -> Result<()> {
+    if *crate::worker::CLOUD_HOSTED {
+        return Err(Error::BadRequest(
+            "Windmill's own database cannot back a data table or Ducklake catalog on Windmill Cloud"
+                .to_string(),
+        ));
+    }
+    let disabled = sqlx::query_scalar::<_, Option<serde_json::Value>>(
+        "SELECT value FROM global_settings WHERE name = $1",
+    )
+    .bind(crate::global_settings::INSTANCE_PG_DISABLED_SETTING)
+    .fetch_optional(executor)
+    .await?
+    .flatten()
+    .is_some_and(|v| v.as_bool().unwrap_or(false));
+    if disabled {
+        return Err(Error::BadRequest(
+            "Windmill's own database is disabled as a data table and Ducklake substrate on this \
+             instance. Use the external instance cluster, or turn it back on in the instance \
+             settings."
+                .to_string(),
+        ));
+    }
+    Ok(())
 }
 
 /// Build a self-teaching error for an unresolved `datatable://<name>` reference.
@@ -1621,14 +1662,82 @@ pub struct GoverningDatatable {
     pub governor: Option<DataTableReference>,
 }
 
+/// Everything still using the Windmill-managed database `dbname`, one description per use: data
+/// table entries naming it, fork entries pointing at those, Ducklake catalogs on it, and fork
+/// Ducklake metadata schemas there that cleanup has not dropped yet. `exempt` is the one data table
+/// entry, `(workspace_id, name)`, the caller is about to stop using it through; pointers at that
+/// entry still count, since dropping the database would leave them resolving to nothing.
+///
+/// Authorization: reads every workspace's settings and checks nothing. Callers MUST only turn the
+/// answer into a refusal for someone allowed to administer `dbname`.
+pub async fn managed_database_uses(
+    conn: &mut sqlx::PgConnection,
+    kind: DataTableCatalogResourceType,
+    dbname: &str,
+    exempt: Option<(&str, &str)>,
+) -> Result<Vec<String>> {
+    let (exempt_workspace, exempt_name) = exempt.unzip();
+    Ok(sqlx::query_scalar::<_, String>(
+        "WITH entries AS (
+             SELECT ws.workspace_id::text AS workspace_id, dt.key AS name, dt.value
+             FROM workspace_settings ws
+             CROSS JOIN LATERAL jsonb_each(
+                 CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
+                     THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
+         ), naming AS (
+             SELECT workspace_id, name FROM entries
+             WHERE value->'database'->>'resource_type' = $1
+               AND value->'database'->>'resource_path' = $2
+         )
+         SELECT format('data table ''%s'' in workspace ''%s''', name, workspace_id) FROM naming
+         WHERE $3::text IS NULL OR NOT (workspace_id = $3 AND name = $4)
+         UNION ALL
+         SELECT format('data table ''%s'' in workspace ''%s'', which points at the one in ''%s''',
+                       e.name, e.workspace_id, n.workspace_id)
+         FROM entries e JOIN naming n
+           ON e.value->'reference'->>'workspace_id' = n.workspace_id
+          AND e.value->'reference'->>'datatable' = n.name
+         UNION ALL
+         SELECT format('Ducklake ''%s'' in workspace ''%s''', dl.key, ws.workspace_id)
+         FROM workspace_settings ws
+         CROSS JOIN LATERAL jsonb_each(
+             CASE WHEN jsonb_typeof(ws.ducklake->'ducklakes') = 'object'
+                 THEN ws.ducklake->'ducklakes' ELSE '{}'::jsonb END) dl
+         WHERE dl.value->'catalog'->>'resource_type' = $1
+           AND dl.value->'catalog'->>'resource_path' = $2
+         UNION ALL
+         SELECT format('the Ducklake namespace of fork ''%s'', not cleaned up yet', workspace_id)
+         FROM fork_ducklake_namespace
+         WHERE catalog = $1 || ':' || $2 AND NOT schema_dropped
+         ORDER BY 1",
+    )
+    .bind(kind.as_ref())
+    .bind(dbname)
+    .bind(exempt_workspace)
+    .bind(exempt_name)
+    .fetch_all(&mut *conn)
+    .await?)
+}
+
+/// Held by fork cleanup of `w_id`'s data tables and by forking `w_id`, which can hand the new fork
+/// pointers at them, so a pointer cannot appear between cleanup's check and its drop.
+pub async fn lock_fork_datatables(conn: &mut sqlx::PgConnection, w_id: &str) -> Result<()> {
+    sqlx::query("SELECT pg_advisory_xact_lock(hashtext('fork_datatables:' || $1))")
+        .bind(w_id)
+        .execute(&mut *conn)
+        .await?;
+    Ok(())
+}
+
 impl GoverningDatatable {
-    /// Backed by the Windmill instance's own Postgres, which is the only substrate data table
-    /// roles apply to.
-    pub fn is_instance(&self) -> bool {
+    /// The Windmill-managed cluster whose data table roles this entry can use. `None` for a
+    /// resource-backed one: roles are logins Windmill creates, and it creates none on a host a
+    /// workspace admin chose.
+    pub fn role_cluster(&self) -> Option<crate::datatable_roles::DatatableRoleCluster> {
         self.datatable
             .database
             .as_ref()
-            .is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
+            .and_then(|d| crate::datatable_roles::DatatableRoleCluster::of(d.resource_type))
     }
 
     /// The workspace whose admins administer the data table and whose members its tenants are.
@@ -1776,20 +1885,19 @@ pub async fn resolve_workspace_governing_datatables(
                     clone,
                 )),
                 (None, Some(governed_by)) => {
-                    let (governor_ws, governor_name) =
-                        (governed_by.workspace_id.clone(), governed_by.datatable.clone());
+                    let (governor_ws, governor_name) = (
+                        governed_by.workspace_id.clone(),
+                        governed_by.datatable.clone(),
+                    );
                     let clone = clone.or(Some((ws, name, datatable)));
                     next.push((i, governor_ws, governor_name, clone));
                 }
                 (None, None) => resolved.push((
                     i,
                     match clone {
-                        None => GoverningDatatable {
-                            workspace_id: ws,
-                            name,
-                            datatable,
-                            governor: None,
-                        },
+                        None => {
+                            GoverningDatatable { workspace_id: ws, name, datatable, governor: None }
+                        }
                         Some((clone_ws, clone_name, mut clone_datatable)) => {
                             clone_datatable.permissions = datatable.permissions;
                             GoverningDatatable {
@@ -1830,7 +1938,8 @@ pub async fn resolve_workspace_governing_datatables(
 }
 
 /// Build the `admin` connection for a governing entry: `custom_instance_user` for an instance
-/// database, the user's own resource for a BYO-postgres one.
+/// database, on Windmill's cluster or the external one; the user's own resource for a BYO-postgres
+/// one.
 async fn resolve_datatable_connection_unchecked(
     db: &DB,
     governing: &GoverningDatatable,
@@ -1841,7 +1950,16 @@ async fn resolve_datatable_connection_unchecked(
         .database
         .as_ref()
         .expect("a governing entry owns a database");
-    if database.resource_type == DataTableCatalogResourceType::Instance {
+    if database.resource_type == DataTableCatalogResourceType::ExternalInstance {
+        let pg_creds = crate::external_instance_pg::external_instance_connection_unchecked(
+            db,
+            &database.resource_path,
+            replication,
+        )
+        .await?;
+        serde_json::to_value(&pg_creds)
+            .map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))
+    } else if database.resource_type == DataTableCatalogResourceType::Instance {
         let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
         pg_creds.dbname = database.resource_path.clone();
         if replication {
@@ -1879,8 +1997,31 @@ pub async fn get_datatable_resource_from_db_unchecked(
     w_id: &str,
     name: &str,
 ) -> Result<serde_json::Value> {
+    Ok(get_datatable_connection_and_kind_unchecked(db, w_id, name)
+        .await?
+        .0)
+}
+
+/// As [`get_datatable_resource_from_db_unchecked`], also reporting the kind of database Windmill
+/// manages behind it, `None` for a user resource. One resolution answers both: a caller reading the
+/// kind separately can be handed one kind's connection and the other kind's checks by a save
+/// landing between the two, and the entry a pointer lands on is another workspace's to change.
+///
+/// Same authorization contract: the connection reaches every role.
+pub async fn get_datatable_connection_and_kind_unchecked(
+    db: &DB,
+    w_id: &str,
+    name: &str,
+) -> Result<(serde_json::Value, Option<DataTableCatalogResourceType>)> {
     let governing = resolve_governing_datatable(db, w_id, name).await?;
-    resolve_datatable_connection_unchecked(db, &governing, false).await
+    let kind = governing
+        .datatable
+        .database
+        .as_ref()
+        .map(|d| d.resource_type)
+        .filter(|kind| kind.is_windmill_managed());
+    let connection = resolve_datatable_connection_unchecked(db, &governing, false).await?;
+    Ok((connection, kind))
 }
 
 /// Same as [`get_datatable_resource_from_db_unchecked`] but for postgres trigger
@@ -2385,6 +2526,10 @@ pub enum DucklakeCatalogResourceType {
     Postgresql,
     Mysql,
     Instance,
+    /// On the external instance cluster ([`crate::external_instance_pg`]). Enterprise Edition.
+    #[serde(rename = "external_instance")]
+    #[strum(serialize = "external_instance")]
+    ExternalInstance,
 }
 
 #[derive(Deserialize, Serialize)]
@@ -2912,7 +3057,16 @@ async fn ducklake_conn_data(
     let ducklake = serde_json::from_value::<Ducklake>(ducklake)?;
 
     let catalog_resource =
-        if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
+        if ducklake.catalog.resource_type == DucklakeCatalogResourceType::ExternalInstance {
+            let pg_creds = crate::external_instance_pg::external_instance_connection_unchecked(
+                db,
+                &ducklake.catalog.resource_path,
+                false,
+            )
+            .await?;
+            serde_json::to_value(&pg_creds)
+                .map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))?
+        } else if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
             let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
             pg_creds.dbname = ducklake.catalog.resource_path.clone();
             pg_creds.user = Some("custom_instance_user".to_string());
@@ -3293,6 +3447,14 @@ async fn register_fork_ducklake_namespace(
     {
         return Ok(());
     }
+    let mut tx = db.begin().await?;
+    // A row naming an external database counts as a use of it. Written under the lock a drop takes,
+    // and only while the database is still registered, so a drop cannot slip in between the
+    // settings this attach resolved and the row that protects the database.
+    if let Some(dbname) = catalog.strip_prefix("external_instance:") {
+        crate::external_instance_pg::ensure_external_instance_database_registered(&mut tx, dbname)
+            .await?;
+    }
     sqlx::query!(
         "INSERT INTO fork_ducklake_namespace
            (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)
@@ -3307,9 +3469,10 @@ async fn register_fork_ducklake_namespace(
         &storage_ref,
         data_path,
     )
-    .execute(db)
+    .execute(&mut *tx)
     .await
     .map_err(|e| Error::internal_err(format!("registering fork ducklake namespace: {e:#}")))?;
+    tx.commit().await?;
     let mut locations = FORK_DUCKLAKE_REGISTERED
         .get(w_id)
         .filter(|(_, exp)| *exp > now)

backend/windmill-dep-map/src/workspace_dependencies.rs

@@ -11,6 +11,8 @@ use crate::{
 
 #[derive(sqlx::FromRow, Clone, Serialize, Deserialize, Hash, Debug)]
 pub struct NewWorkspaceDependencies {
+    /// Trusted as-is by `create`: a caller deserializing this from a request must first check it
+    /// against the workspace the caller is authorized in.
     pub workspace_id: String,
     pub language: ScriptLang,
     pub name: Option<String>,

backend/windmill-object-store/Cargo.toml

@@ -11,6 +11,7 @@ enterprise = []
 parquet = [
     "windmill-common/parquet",
     "dep:object_store",
+    "dep:reqwest_object_store",
     "dep:aws-sdk-sts",
     "dep:aws-smithy-types-convert",
     "dep:datafusion",
@@ -48,6 +49,7 @@ globset.workspace = true
 axum.workspace = true
 
 object_store = { workspace = true, optional = true }
+reqwest_object_store = { workspace = true, optional = true }
 aws-sdk-sts = { workspace = true, optional = true }
 aws-smithy-types-convert = { workspace = true, optional = true }
 datafusion = { workspace = true, optional = true }

backend/windmill-object-store/src/lib.rs

@@ -583,8 +583,175 @@ pub async fn ambient_aws_credentials(
     ambient_aws_credentials_provider(region).await.get().await
 }
 
+/// An object-store HTTP connector whose client cannot reach a non-public address.
+///
+/// Validating a caller-supplied endpoint up front does not bind the connection: object_store
+/// resolves the host again when it connects, so a name can answer a public address to the check
+/// and an internal one to the connect. This client resolves every host through
+/// [`PublicOnlyResolver`], which judges the very addresses it hands to the connect, refuses IP-literal
+/// hosts that are not public (they never reach a resolver), and follows no redirect.
+///
+/// The environment's egress proxy (`HTTP(S)_PROXY`/`ALL_PROXY`) is still used, and its own host is
+/// exempt from the resolver check: it is operator configuration and typically internal. A request
+/// whose target is the proxy host itself is refused, so the exemption only ever serves the proxy
+/// hop. A proxied request's target is resolved by the proxy, out of this client's reach (see
+/// `ssrf::ValidatedTarget`).
+#[cfg(feature = "parquet")]
+#[derive(Debug, Clone, Copy)]
+pub struct PublicOnlyConnector;
+
+#[cfg(feature = "parquet")]
+impl object_store::client::HttpConnector for PublicOnlyConnector {
+    fn connect(
+        &self,
+        options: &ClientOptions,
+    ) -> object_store::Result<object_store::client::HttpClient> {
+        let allow_http = options
+            .get_config_value(&object_store::ClientConfigKey::AllowHttp)
+            .is_some_and(|v| v == "true");
+        let proxy_hosts: Arc<[String]> = system_proxy_hosts().into();
+        let client = reqwest_object_store::Client::builder()
+            .dns_resolver(Arc::new(PublicOnlyResolver {
+                proxy_hosts: proxy_hosts.clone(),
+            }))
+            .redirect(reqwest_object_store::redirect::Policy::none())
+            .connect_timeout(std::time::Duration::from_secs(5))
+            .default_headers(HeaderMap::from_iter([(
+                reqwest::header::ACCEPT_ENCODING,
+                reqwest::header::HeaderValue::from_static(""),
+            )]))
+            .no_gzip()
+            .no_brotli()
+            .no_deflate()
+            .no_zstd()
+            .https_only(!allow_http)
+            .build()
+            .map_err(|e| object_store::Error::Generic {
+                store: "HTTP client",
+                source: Box::new(e),
+            })?;
+        Ok(object_store::client::HttpClient::new(PublicOnlyClient {
+            client,
+            proxy_hosts,
+        }))
+    }
+}
+
+#[cfg(feature = "parquet")]
+#[derive(Debug)]
+struct PublicOnlyClient {
+    client: reqwest_object_store::Client,
+    proxy_hosts: Arc<[String]>,
+}
+
+#[cfg(feature = "parquet")]
+#[async_trait]
+impl object_store::client::HttpService for PublicOnlyClient {
+    async fn call(
+        &self,
+        req: object_store::client::HttpRequest,
+    ) -> Result<object_store::client::HttpResponse, object_store::client::HttpError> {
+        let host = req.uri().host().unwrap_or_default();
+        let literal = host
+            .trim_start_matches('[')
+            .trim_end_matches(']')
+            .parse::<std::net::IpAddr>()
+            .ok();
+        let refused = if literal.is_some_and(|ip| windmill_common::ssrf::is_private_ip(&ip)) {
+            Some("a private, loopback, or link-local address")
+        } else if self
+            .proxy_hosts
+            .iter()
+            .any(|p| p.eq_ignore_ascii_case(host))
+        {
+            Some("the egress proxy")
+        } else {
+            None
+        };
+        if let Some(what) = refused {
+            return Err(object_store::client::HttpError::new(
+                object_store::client::HttpErrorKind::Unknown,
+                std::io::Error::other(format!("'{host}' is {what}")),
+            ));
+        }
+        object_store::client::HttpService::call(&self.client, req).await
+    }
+}
+
+#[cfg(feature = "parquet")]
+struct PublicOnlyResolver {
+    proxy_hosts: Arc<[String]>,
+}
+
+/// The hosts of the proxies reqwest picks up from the environment.
+#[cfg(feature = "parquet")]
+fn system_proxy_hosts() -> Vec<String> {
+    [
+        "HTTP_PROXY",
+        "http_proxy",
+        "HTTPS_PROXY",
+        "https_proxy",
+        "ALL_PROXY",
+        "all_proxy",
+    ]
+    .into_iter()
+    .filter_map(|k| std::env::var(k).ok())
+    .filter_map(|v| {
+        let v = v.trim();
+        let url = if v.contains("://") {
+            v.to_string()
+        } else {
+            format!("http://{v}")
+        };
+        Some(
+            reqwest::Url::parse(&url)
+                .ok()?
+                .host_str()?
+                .to_ascii_lowercase(),
+        )
+    })
+    .collect()
+}
+
+#[cfg(feature = "parquet")]
+impl reqwest_object_store::dns::Resolve for PublicOnlyResolver {
+    fn resolve(
+        &self,
+        name: reqwest_object_store::dns::Name,
+    ) -> reqwest_object_store::dns::Resolving {
+        let is_proxy = self
+            .proxy_hosts
+            .iter()
+            .any(|p| p.eq_ignore_ascii_case(name.as_str()));
+        Box::pin(async move {
+            let host = name.as_str();
+            let addrs: Vec<std::net::SocketAddr> =
+                tokio::net::lookup_host((host, 0)).await?.collect();
+            if !is_proxy
+                && addrs
+                    .iter()
+                    .any(|a| windmill_common::ssrf::is_private_ip(&a.ip()))
+            {
+                return Err(format!(
+                    "'{host}' resolves to a private, loopback, or link-local address"
+                )
+                .into());
+            }
+            Ok(Box::new(addrs.into_iter()) as reqwest_object_store::dns::Addrs)
+        })
+    }
+}
+
 #[cfg(feature = "parquet")]
 pub async fn build_s3_client(s3_resource_ref: &S3Resource) -> error::Result<Arc<dyn ObjectStore>> {
+    build_s3_client_with(s3_resource_ref, None).await
+}
+
+#[cfg(feature = "parquet")]
+async fn build_s3_client_with(
+    s3_resource_ref: &S3Resource,
+    connector: Option<PublicOnlyConnector>,
+) -> error::Result<Arc<dyn ObjectStore>> {
     let static_creds = s3_resource_has_static_credentials(s3_resource_ref);
 
     let credentials_provider = if !static_creds {
@@ -623,6 +790,9 @@ pub async fn build_s3_client(s3_resource_ref: &S3Resource) -> error::Result<Arc<
     if !s3_resource.use_ssl {
         store_builder = store_builder.with_allow_http(true)
     }
+    if let Some(connector) = connector {
+        store_builder = store_builder.with_http_connector(connector);
+    }
 
     if let Some(key) = s3_resource.access_key {
         if key != "" {
@@ -655,6 +825,14 @@ pub async fn build_s3_client(s3_resource_ref: &S3Resource) -> error::Result<Arc<
 #[cfg(feature = "parquet")]
 fn build_azure_blob_client(
     azure_blob_resource_ref: &AzureBlobResource,
+) -> error::Result<Arc<dyn ObjectStore>> {
+    build_azure_blob_client_with(azure_blob_resource_ref, None)
+}
+
+#[cfg(feature = "parquet")]
+fn build_azure_blob_client_with(
+    azure_blob_resource_ref: &AzureBlobResource,
+    connector: Option<PublicOnlyConnector>,
 ) -> error::Result<Arc<dyn ObjectStore>> {
     let blob_resource = azure_blob_resource_ref.clone();
 
@@ -701,6 +879,9 @@ fn build_azure_blob_client(
     if !blob_resource.use_ssl.unwrap_or(false) {
         store_builder = store_builder.with_allow_http(true)
     }
+    if let Some(connector) = connector {
+        store_builder = store_builder.with_http_connector(connector);
+    }
 
     if let Some(key) = blob_resource.access_key {
         if key != "" {
@@ -735,6 +916,14 @@ pub fn gcs_service_account_key_is_blank(service_account_key: &str) -> bool {
 
 #[cfg(feature = "parquet")]
 async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result<Arc<dyn ObjectStore>> {
+    build_gcs_client_with(gcs_resource_ref, None).await
+}
+
+#[cfg(feature = "parquet")]
+async fn build_gcs_client_with(
+    gcs_resource_ref: &GcsResource,
+    connector: Option<PublicOnlyConnector>,
+) -> error::Result<Arc<dyn ObjectStore>> {
     let gcs_resource = gcs_resource_ref.clone();
 
     let mut store_builder = GoogleCloudStorageBuilder::new()
@@ -754,6 +943,9 @@ async fn build_gcs_client(gcs_resource_ref: &GcsResource) -> error::Result<Arc<d
     if !gcs_service_account_key_is_blank(&gcs_resource.service_account_key) {
         store_builder = store_builder.with_service_account_key(gcs_resource.service_account_key);
     }
+    if let Some(connector) = connector {
+        store_builder = store_builder.with_http_connector(connector);
+    }
 
     let store = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| store_builder.build()))
         .map_err(|panic_info| {
@@ -936,6 +1128,37 @@ pub fn object_store_location(resource: &ObjectStoreResource) -> String {
 pub async fn build_object_store_from_settings(
     settings: ObjectSettings,
     init_private_key: Option<&windmill_common::DB>,
+) -> error::Result<ExpirableObjectStore> {
+    build_object_store_from_settings_with(settings, init_private_key, None).await
+}
+
+/// [`build_object_store_from_settings`] for a caller-supplied endpoint: the client refuses to
+/// connect to any non-public address (see [`PublicOnlyConnector`]). Filesystem and OIDC settings
+/// are refused; whether S3, Azure or GCS settings carry explicit credentials (rather than falling
+/// back to the server's ambient ones) is still the caller's to check.
+#[cfg(feature = "parquet")]
+pub async fn build_public_object_store_from_settings(
+    settings: ObjectSettings,
+) -> error::Result<Arc<dyn ObjectStore>> {
+    match settings {
+        ObjectSettings::S3(_) | ObjectSettings::Azure(_) | ObjectSettings::Gcs(_) => {
+            build_object_store_from_settings_with(settings, None, Some(PublicOnlyConnector))
+                .await
+                .map(|x| x.store)
+        }
+        ObjectSettings::AwsOidc(_) | ObjectSettings::Filesystem(_) => {
+            Err(error::Error::BadRequest(
+                "This object storage backend cannot be restricted to public endpoints".to_string(),
+            ))
+        }
+    }
+}
+
+#[cfg(feature = "parquet")]
+async fn build_object_store_from_settings_with(
+    settings: ObjectSettings,
+    init_private_key: Option<&windmill_common::DB>,
+    connector: Option<PublicOnlyConnector>,
 ) -> error::Result<ExpirableObjectStore> {
     let located =
         |store: Arc<dyn ObjectStore>, resource: ObjectStoreResource| ExpirableObjectStore {
@@ -946,12 +1169,14 @@ pub async fn build_object_store_from_settings(
     match settings {
         ObjectSettings::S3(s3_settings) => {
             let s3_resource = s3_resource_from_settings(s3_settings);
-            build_s3_client(&s3_resource)
+            build_s3_client_with(&s3_resource, connector)
                 .await
                 .map(|x| located(x, ObjectStoreResource::S3(s3_resource)))
         }
-        ObjectSettings::Azure(azure_settings) => build_azure_blob_client(&azure_settings)
-            .map(|x| located(x, ObjectStoreResource::Azure(azure_settings))),
+        ObjectSettings::Azure(azure_settings) => {
+            build_azure_blob_client_with(&azure_settings, connector)
+                .map(|x| located(x, ObjectStoreResource::Azure(azure_settings)))
+        }
         ObjectSettings::AwsOidc(ref s3_aws_oidc_settings) => {
             let token_generator = crate::job_s3_helpers_oss::TokenGenerator::AsServerInstance();
             let res = crate::job_s3_helpers_oss::generate_s3_aws_oidc_resource(
@@ -969,7 +1194,7 @@ pub async fn build_object_store_from_settings(
                     location: Some(object_store_location(&res)),
                 })
         }
-        ObjectSettings::Gcs(gcs_settings) => build_gcs_client(&gcs_settings)
+        ObjectSettings::Gcs(gcs_settings) => build_gcs_client_with(&gcs_settings, connector)
             .await
             .map(|x| located(x, ObjectStoreResource::Gcs(gcs_settings))),
         ObjectSettings::Filesystem(fs) => build_filesystem_client(&fs.root_path)
@@ -2701,6 +2926,44 @@ mod tests {
         assert_ne!(a.location, b.location);
     }
 
+    /// The restricted store judges the addresses it connects to, not only what an earlier check
+    /// resolved: neither a hostname that resolves to loopback nor a loopback literal gets a
+    /// connection.
+    #[cfg(feature = "parquet")]
+    #[tokio::test]
+    async fn test_public_store_refuses_to_connect_to_private_address() {
+        use futures::StreamExt;
+
+        let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+        for endpoint in ["localhost", "127.0.0.1"] {
+            let settings = ObjectSettings::S3(S3Settings {
+                bucket: Some("windmill".to_string()),
+                region: Some("us-east-1".to_string()),
+                access_key: Some("key".to_string()),
+                secret_key: Some("secret".to_string()),
+                endpoint: Some(endpoint.to_string()),
+                allow_http: Some(true),
+                path_style: Some(true),
+                store_logs: None,
+                port: Some(listener.local_addr().unwrap().port()),
+            });
+            let store = build_public_object_store_from_settings(settings)
+                .await
+                .unwrap();
+            // The listener never answers, so a store that connects waits forever: bound it.
+            let err =
+                tokio::time::timeout(std::time::Duration::from_secs(30), store.list(None).next())
+                    .await
+                    .unwrap_or_else(|_| panic!("the store connected to {endpoint}"))
+                    .unwrap()
+                    .unwrap_err();
+            assert!(
+                format!("{err:?}").contains("private, loopback, or link-local"),
+                "{endpoint}: {err:?}"
+            );
+        }
+    }
+
     // --- get_logs_from_store test ---
 
     #[cfg(feature = "parquet")]