fix(datatables): refuse moving the external cluster to another host or port while it holds databases

Every settings writer goes through the same check as removal: the login, TLS and maintenance
database may still change, the cluster may not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "1.813.0"
+  ".": "1.814.0"
 }

AGENTS.md

@@ -29,10 +29,6 @@ Open-source platform for internal tools, workflows, API integrations, background
 - **Agent workers**: `docs/agent-worker-e2e.md` — building and running one locally. An agent
   reaches the DB only through the API, so `Connection::Http` paths are never taken by a plain
   `cargo run`; a normal build cannot start one at all.
-- **External instance data tables**: `docs/external-instance-datatables.md` — the cluster Windmill
-  administers behind `external_instance` data tables and Ducklake catalogs: its invariants (one
-  lifecycle lock, managed-object markers, the setup gate, per-cluster roles, fork copy ownership)
-  and how to run one locally
 - **Enterprise**: `docs/enterprise.md` — EE file conventions and PR workflow
 - **Auth surface**: `docs/auth-surface.md` — credential precedence, session/cache invalidation
   scope, which token labels email their owner at expiry, how OAuth login matches `login_type`, and

CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## [1.814.0](https://github.com/windmill-labs/windmill/compare/v1.813.0...v1.814.0) (2026-09-17)
+
+
+### Features
+
+* **ai-chat:** add list_workers and list_data_metrics global tools ([#11143](https://github.com/windmill-labs/windmill/issues/11143)) ([e954d33](https://github.com/windmill-labs/windmill/commit/e954d33613e4ff5027667eb8f646615d9bbd499d))
+* **ai-chat:** merge get_job_logs and get_flow_run_details into get_run ([#11172](https://github.com/windmill-labs/windmill/issues/11172)) ([5bb37ca](https://github.com/windmill-labs/windmill/commit/5bb37ca3388666fba72c55534e37f37bb3e9299e))
+* allow git sync auto-pull, promotion and PRs on Pro licenses ([#11173](https://github.com/windmill-labs/windmill/issues/11173)) ([02e47de](https://github.com/windmill-labs/windmill/commit/02e47de8b4c4f3f54753aabf8c67bc8e71ffb957))
+* badge chat-input flows on the home list ([#11164](https://github.com/windmill-labs/windmill/issues/11164)) ([3d08197](https://github.com/windmill-labs/windmill/commit/3d0819718221f885b61e73d02b43dcc853c7d02a))
+* collect flow conversations and agent memory once their last message goes ([#11178](https://github.com/windmill-labs/windmill/issues/11178)) ([23c24a9](https://github.com/windmill-labs/windmill/commit/23c24a9688d4c8c462f53221334d538280f16bca))
+* flow chat model picker on a shared model-settings component ([#11187](https://github.com/windmill-labs/windmill/issues/11187)) ([189793c](https://github.com/windmill-labs/windmill/commit/189793c2e4db7f1c853695ebcc895c1ec82ed19f))
+* keep flow inputs and seed the agent when chat mode is enabled ([#11177](https://github.com/windmill-labs/windmill/issues/11177)) ([68f2248](https://github.com/windmill-labs/windmill/commit/68f2248018fc218a090bf939e1eb22ff97d5bc22))
+* let plan mode search and read connected mcp servers ([#11205](https://github.com/windmill-labs/windmill/issues/11205)) ([5371519](https://github.com/windmill-labs/windmill/commit/5371519f0f5ce7750982dcdb374dca72115902e7))
+* let test_run_flow name the conversation of a chat-mode test run ([#11198](https://github.com/windmill-labs/windmill/issues/11198)) ([6e1ef93](https://github.com/windmill-labs/windmill/commit/6e1ef93f329cb396ffc3df3304d592e8fa0e0e71))
+* managed memory with an inherited or custom memory id per step ([#11118](https://github.com/windmill-labs/windmill/issues/11118)) ([c297ed0](https://github.com/windmill-labs/windmill/commit/c297ed0052d998fb8f063faa2a36c6eb03e327be))
+* render the flow chat through the shared session chat components ([#11175](https://github.com/windmill-labs/windmill/issues/11175)) ([a9ec0ae](https://github.com/windmill-labs/windmill/commit/a9ec0aec3ac0c6b0f7919d0eb2168816923826d7))
+* show flow step detail inside the graph tab on narrow detail layouts ([#11168](https://github.com/windmill-labs/windmill/issues/11168)) ([64dffe6](https://github.com/windmill-labs/windmill/commit/64dffe6106ad6a55b61a423c855a4b5b0cef533e))
+* store mcp tool call, result and reasoning on flow conversation rows ([#11176](https://github.com/windmill-labs/windmill/issues/11176)) ([a571117](https://github.com/windmill-labs/windmill/commit/a571117f3fd2cef14c920770645c60ee358fdfdd))
+* tell test flow conversations from deployed ones and rename a chat ([#11179](https://github.com/windmill-labs/windmill/issues/11179)) ([4eab995](https://github.com/windmill-labs/windmill/commit/4eab995cf7cf091a5e4640da4cb77e0921bb7fdf))
+
+
+### Bug Fixes
+
+* disable a schedule whose cron has no run left instead of panicking ([#11195](https://github.com/windmill-labs/windmill/issues/11195)) ([381d447](https://github.com/windmill-labs/windmill/commit/381d4470ef699ea82283742132e56556b95d2bd2))
+* skip expiry notifications for app embed and SDK tokens ([#11169](https://github.com/windmill-labs/windmill/issues/11169)) ([9d348f8](https://github.com/windmill-labs/windmill/commit/9d348f84c7830f36b6153472556fd70e3d84cd24))
+
 ## [1.813.0](https://github.com/windmill-labs/windmill/compare/v1.812.0...v1.813.0) (2026-09-16)
 
 

ai_evals/README.md

@@ -175,6 +175,10 @@ the decrypted value, exactly as against a real backend. The chat's read path pas
 Seed a recognizable secret (the existing fixture uses `sk_live_do_not_leak_me`) and
 assert it via `valueExcludes` to catch a leak.
 
+`toolExpect.toolCallArgs` entries support `sharedByAtLeast: <n>`: at least `n` recorded
+calls to that tool must carry the same non-blank string in the field. Use it for calls that
+have to share an identifier, like two test runs of one chat conversation.
+
 `toolExpect.toolCallArgs` entries additionally support `fieldMustBeAbsent: true`: no
 recorded call to that tool may pass the field at all (an explicit `null` counts as
 passing it). Use it for partial-update tools, where supplying a field the model could

ai_evals/adapters/frontend/backendPreview.ts

@@ -1,5 +1,5 @@
-import { randomUUID } from 'node:crypto'
 import type { BackendValidationSettings } from '../../core/backendValidation'
+import { buildWorkspaceId } from './workspaceId'
 
 interface CompletedJobResultMaybe {
 	completed: boolean
@@ -24,7 +24,6 @@ export interface CompletedPreviewJob {
 const tokenCache = new Map<string, Promise<string>>()
 const sharedWorkspaceQueue = new Map<string, Promise<void>>()
 const managedSharedWorkspacePrefixes = ['f/evals/']
-const DEFAULT_WORKSPACE_PREFIX = 'ai-evals'
 
 export class BackendPreviewClient {
 	constructor(private readonly settings: BackendValidationSettings) {}
@@ -441,16 +440,6 @@ async function withSharedWorkspaceLock<T>(workspaceId: string, body: () => Promi
 	}
 }
 
-function buildWorkspaceId(caseId: string, attempt: number): string {
-	const caseSlug = caseId
-		.toLowerCase()
-		.replace(/[^a-z0-9-]+/g, '-')
-		.replace(/^-+|-+$/g, '')
-		.slice(0, 30)
-	const suffix = randomUUID().slice(0, 8)
-	return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || 'case'}-a${attempt}-${suffix}`
-}
-
 function extractFolderName(path: string): string | null {
 	if (!path.startsWith('f/')) {
 		return null

ai_evals/adapters/frontend/mockBackend.ts

@@ -11,6 +11,7 @@ import type {
 	Script
 } from '../../../frontend/src/lib/gen'
 import type {
+	DataMetric,
 	DataTableTables,
 	DataTableTableSchema,
 	EndpointTool,
@@ -112,6 +113,9 @@ export interface BenchmarkWorkspaceRunnables {
 	aiProviders?: BenchmarkWorkspaceAiProvider[]
 	resources?: BenchmarkWorkspaceResource[]
 	datatables?: BenchmarkDatatableSeed[]
+	/** DuckLake catalog names, as `list_ducklakes` reports them. */
+	ducklakes?: string[]
+	dataMetrics?: DataMetric[]
 	jobs?: BenchmarkWorkspaceJob[]
 }
 
@@ -673,6 +677,27 @@ export function listBenchmarkDatatables(workspace: string): DataTableTables[] |
 	}))
 }
 
+// ============= DuckLake catalogs and declared metrics =============
+
+/** Seeded DuckLake names, or `null` for a non-benchmark workspace. */
+export function listBenchmarkDucklakes(workspace: string): string[] | null {
+	const runnables = benchmarkWorkspaceRunnables.get(workspace)
+	return runnables ? (runnables.ducklakes ?? []) : null
+}
+
+/**
+ * Seeded metric declarations, or `null` for a non-benchmark workspace.
+ *
+ * The `table` / `path_prefix` filters are ignored: which rows a filter selects is
+ * `canonical_table_path`'s business and is pinned by `ducklakeTools.test.ts`.
+ * Re-deriving it here would give the eval its own copy of that spec to drift from,
+ * and the case this serves measures whether the model reaches for the tool at all.
+ */
+export function listBenchmarkDataMetrics(workspace: string): DataMetric[] | null {
+	const runnables = benchmarkWorkspaceRunnables.get(workspace)
+	return runnables ? (runnables.dataMetrics ?? []) : null
+}
+
 export function getBenchmarkDatatableSchema(input: {
 	workspace: string
 	datatableName: string
@@ -840,6 +865,29 @@ export function runBenchmarkFlowByPath(input: {
 	})
 }
 
+/**
+ * Mirror `JobService.runFlowPreview` for benchmark workspaces, including the server's
+ * refusal of a chat-enabled flow run that names no conversation (`memory_id`).
+ */
+export function runBenchmarkFlowPreview(input: {
+	workspace: string
+	memoryId?: string
+	requestBody?: { path?: string; value?: { chat_input_enabled?: boolean }; args?: unknown }
+}): string {
+	if (input.requestBody?.value?.chat_input_enabled && !input.memoryId) {
+		throw new Error('Bad request: memory_id is required for chat-enabled flows')
+	}
+	const args = (input.requestBody?.args ?? {}) as Record<string, unknown>
+	return createBenchmarkCompletedJob({
+		workspace: input.workspace,
+		jobKind: 'flowpreview',
+		success: true,
+		args,
+		result: { path: input.requestBody?.path, args, mocked: true },
+		logs: 'Mock benchmark flow preview completed successfully.'
+	})
+}
+
 export function previewBenchmarkSchedule(input: {
 	requestBody?: Record<string, unknown>
 }): Record<string, unknown> {

ai_evals/adapters/frontend/vitestAdapter.test.ts

@@ -76,7 +76,9 @@ vi.mock('$lib/gen', async () => {
 		listBenchmarkPlainResources,
 		listBenchmarkApps,
 		listBenchmarkDatatables,
+		listBenchmarkDataMetrics,
 		listBenchmarkDrafts,
+		listBenchmarkDucklakes,
 		listBenchmarkFlows,
 		listBenchmarkJobs,
 		listBenchmarkScripts,
@@ -87,6 +89,7 @@ vi.mock('$lib/gen', async () => {
 		previewBenchmarkSchedule,
 		runBenchmarkDatatableSql,
 		runBenchmarkFlowByPath,
+		runBenchmarkFlowPreview,
 		runBenchmarkScriptByPath,
 		runBenchmarkScriptPreview,
 		updateBenchmarkDraft,
@@ -293,6 +296,14 @@ vi.mock('$lib/gen', async () => {
 							args: data.requestBody
 						})
 					: actual.JobService.runScriptByPath(data),
+			runFlowPreview: async (data: {
+				workspace: string
+				memoryId?: string
+				requestBody?: { path?: string; value?: { chat_input_enabled?: boolean }; args?: unknown }
+			}) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? runBenchmarkFlowPreview(data)
+					: actual.JobService.runFlowPreview(data as any),
 			runFlowByPath: async (data: {
 				workspace: string
 				path: string
@@ -341,6 +352,10 @@ vi.mock('$lib/gen', async () => {
 				hasBenchmarkWorkspace(data.workspace)
 					? (listBenchmarkDatatables(data.workspace) ?? [])
 					: actual.WorkspaceService.listDataTableTables(data),
+			listDucklakes: async (data: { workspace: string }) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? (listBenchmarkDucklakes(data.workspace) ?? [])
+					: actual.WorkspaceService.listDucklakes(data),
 			getDataTableTableSchema: async (data: {
 				workspace: string
 				datatableName: string
@@ -356,6 +371,12 @@ vi.mock('$lib/gen', async () => {
 						})
 					: actual.WorkspaceService.getDataTableTableSchema(data)
 		}),
+		DataMetricService: wrapService(actual.DataMetricService, {
+			listDataMetrics: async (data: { workspace: string }) =>
+				hasBenchmarkWorkspace(data.workspace)
+					? { metrics: listBenchmarkDataMetrics(data.workspace) ?? [] }
+					: actual.DataMetricService.listDataMetrics(data)
+		}),
 		ScheduleService: wrapService(actual.ScheduleService, {
 			existsSchedule: async (data: { workspace: string; path: string }) =>
 				hasBenchmarkWorkspace(data.workspace) ? false : actual.ScheduleService.existsSchedule(data),

ai_evals/adapters/frontend/windmillBackend.ts

@@ -1,9 +1,8 @@
-import { randomUUID } from "node:crypto";
 import type { WindmillBackendSettings } from "../../core/windmillBackendSettings";
+import { buildWorkspaceId } from "./workspaceId";
 
 const tokenCache = new Map<string, Promise<string>>();
 const sharedWorkspaceQueue = new Map<string, Promise<void>>();
-const DEFAULT_WORKSPACE_PREFIX = "ai-evals";
 
 export class WindmillBackendClient {
   constructor(private readonly settings: WindmillBackendSettings) {}
@@ -179,16 +178,6 @@ async function withSharedWorkspaceLock<T>(
   }
 }
 
-function buildWorkspaceId(caseId: string, attempt: number): string {
-  const caseSlug = caseId
-    .toLowerCase()
-    .replace(/[^a-z0-9-]+/g, "-")
-    .replace(/^-+|-+$/g, "")
-    .slice(0, 30);
-  const suffix = randomUUID().slice(0, 8);
-  return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || "case"}-a${attempt}-${suffix}`;
-}
-
 async function expectOk(response: Response, context: string): Promise<void> {
   if (response.ok) {
     return;

ai_evals/adapters/frontend/workspaceId.test.ts

@@ -0,0 +1,21 @@
+import { describe, expect, it } from "bun:test";
+import { buildWorkspaceId } from "./workspaceId";
+
+describe("buildWorkspaceId", () => {
+  // `workspace.proper_id` rejects `--`, which a case id can carry itself and
+  // which truncating a slug on a hyphen produces once the suffix adds its own.
+  // One id per shape: cut landing on a hyphen, cut landing mid-word, no cut, and
+  // a doubled hyphen no cut ever reaches.
+  it("stays within the id length cap and the proper_id format", () => {
+    for (const caseId of [
+      "global-test6-secret-variable-draft",
+      "global-test23-datatable-query-select",
+      "short",
+      "global--test-foo",
+    ]) {
+      const id = buildWorkspaceId(caseId, 1);
+      expect(id.length).toBeLessThanOrEqual(50);
+      expect(id).toMatch(/^\w+(-\w+)*$/);
+    }
+  });
+});

ai_evals/adapters/frontend/workspaceId.ts

@@ -0,0 +1,22 @@
+import { randomUUID } from "node:crypto";
+
+const DEFAULT_WORKSPACE_PREFIX = "ai-evals";
+
+// A workspace id must be at most 50 characters AND match `^\w+(-\w+)*$`
+// (`workspace.proper_id`), so the case slug yields to the random suffix that
+// makes the id unique, and no hyphen may end up doubled — neither one already in
+// the case id nor one a truncation leaves for the suffix to follow.
+const MAX_WORKSPACE_ID_LENGTH = 50;
+
+export function buildWorkspaceId(caseId: string, attempt: number): string {
+  const caseSlug = caseId
+    .toLowerCase()
+    .replace(/[^a-z0-9-]+/g, "-")
+    .replace(/-{2,}/g, "-")
+    .replace(/^-+|-+$/g, "");
+  const suffix = `-a${attempt}-${randomUUID().slice(0, 8)}`;
+  const head = `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || "case"}`;
+  return `${head
+    .slice(0, MAX_WORKSPACE_ID_LENGTH - suffix.length)
+    .replace(/-+$/, "")}${suffix}`;
+}

ai_evals/cases/global.yaml

@@ -1919,10 +1919,11 @@
     - when the lookup fails, tells the user instead of inventing table names
     - does not write scripts or resources to answer a read-only question
 
-# --- API catalog (search_api_endpoints / call_api_get / call_api_endpoint) ---
-# The harness serves the catalog and the executed calls itself (mock
-# listMcpTools + benchmark fetch handlers in adapters/frontend), so these cases
-# do not require an mcp-enabled eval backend.
+# --- Dedicated tools preferred over the API catalog ---
+# The harness serves worker/queue reads itself (benchmark fetch handlers in
+# adapters/frontend), so these cases do not require an mcp-enabled eval backend.
+# The stale `api-catalog` in the id below is kept so results stay comparable
+# across benchmark runs.
 
 - id: global-test30-api-catalog-workers
   prompt: |-
@@ -1934,23 +1935,42 @@
     draftCountExactly: 0
   toolExpect:
     requiredToolsUsed:
-      - search_api_endpoints
-      - call_api_get
+      - list_workers
     forbiddenToolsUsed:
+      - call_api_get
       - call_api_endpoint
       - write_script
       - deploy_workspace_item
-    toolCallArgs:
-      - tool: call_api_get
-        field: name
-        stringIncludesAnyOf:
-          - listWorkers
   # Read-only workspace inspection produces no draft; validate via tool use.
   skipJudge: true
   judgeChecklist:
-    - discovers the workers endpoint through the API catalog instead of guessing or fabricating
+    - reads worker state through list_workers instead of guessing or fabricating
     - reports worker status from the returned data
 
+- id: global-test37-ducklake-declared-measure
+  prompt: |-
+    We track orders in the main ducklake. Write me a duckdb script that reports total
+    revenue by month. Keep it as a draft, don't deploy it.
+  initial: ai_evals/fixtures/frontend/global/initial/ducklake_orders_metrics.json
+  runtime:
+    maxTurns: 8
+  validate:
+    draftCountExactly: 1
+  toolExpect:
+    requiredToolsUsed:
+      - list_data_metrics
+    forbiddenToolsUsed:
+      - deploy_workspace_item
+      - delete_workspace_item
+  # The judge runs: the point is not that the tool was called but that the number it
+  # describes is the declared one. `revenue` excludes test rows, so an aggregate that
+  # reproduces it without the filter is plausible, runnable and wrong.
+  judgeChecklist:
+    - totals revenue with the declared sum over the amount column rather than an invented aggregate over a guessed column
+    - excludes test orders from the total, as the declared revenue measure does
+    - groups by month using the declared order_month expression over order_date
+    - does not introduce column names absent from the declarations
+
 - id: global-test31-draft-test-run-not-deployed
   prompt: |-
     Update `f/evals/global/format_greeting` so the provided name is uppercased in the greeting, then run it with name "ada" to check it works.
@@ -2140,6 +2160,32 @@
     - creates an AI draft of f/evals/global/process_invoice applying 8% tax
     - does not deploy or save the draft
 
+- id: global-test38-chat-flow-follow-up-same-conversation
+  prompt: |-
+    I want to check that my support chat flow `f/evals/global/support_chat` remembers what was said.
+    Test it: first send "My name is Ada", then send "What is my name?" as a follow-up in the same chat.
+  initial: ai_evals/fixtures/frontend/global/initial/support_chat_flow.json
+  runtime:
+    maxTurns: 8
+  validate:
+    draftCountExactly: 0
+  toolExpect:
+    requiredToolsUsed:
+      - test_run_flow
+    # A chat flow's memory lives in its conversation, so a follow-up only reaches the first
+    # turn's history when both test runs name the same conversation.
+    toolCallArgs:
+      - tool: test_run_flow
+        field: memory_id
+        sharedByAtLeast: 2
+    forbiddenToolsUsed:
+      - run_flow
+      - deploy_workspace_item
+  # The judge cannot observe runs; what this case guards is the conversation the runs share.
+  skipJudge: true
+  judgeChecklist:
+    - test-runs the chat flow twice, the second message as a follow-up in the first run's conversation
+
 - id: global-undo-created-draft
   prompt: |-
     Create a draft Postgres resource at `u/admin/scratch_db` for host db.example.com port 5432, database `orders`, user `app`, and tell me what fields it ended up with.

ai_evals/core/types.ts

@@ -182,6 +182,13 @@ export interface ToolCallArgumentRule {
    * the point is that the model filled it in at all rather than what it said.
    */
   nonEmpty?: boolean;
+  /**
+   * Existential over calls: at least this many recorded calls to `tool` carry the
+   * same non-blank string in `field`. Use when calls have to share an identifier —
+   * e.g. test runs that continue one conversation — while a retry with a rejected
+   * value in between is still acceptable.
+   */
+  sharedByAtLeast?: number;
   /**
    * Universal over calls: no recorded call to `tool` may pass `field` at all.
    * For partial-update tools, where supplying a field the model could not have

ai_evals/core/validators.test.ts

@@ -396,6 +396,32 @@ describe("validateToolExpectations", () => {
     expect(nonEmptyCheck?.details).toContain("blank on 1 of 2");
   });
 
+  it("requires sharedByAtLeast calls to carry one value, not merely a value each", () => {
+    const run = (ids: (string | undefined)[]) =>
+      validateToolExpectations({
+        run: {
+          success: true,
+          actual: {},
+          assistantMessageCount: 1,
+          toolCallCount: ids.length,
+          toolsUsed: ["test_run_flow"],
+          toolCallDetails: ids.map((memory_id) => ({
+            name: "test_run_flow",
+            arguments: { path: "f/chat", memory_id },
+          })),
+          skillsInvoked: [],
+        },
+        toolExpect: {
+          toolCallArgs: [{ tool: "test_run_flow", field: "memory_id", sharedByAtLeast: 2 }],
+        },
+      }).find((c) => c.name.includes("is shared by at least 2 calls"))?.passed;
+
+    expect(run(["a", "b"])).toBe(false);
+    expect(run(["a"])).toBe(false);
+    expect(run([undefined, undefined])).toBe(false);
+    expect(run(["rejected", "a", "a"])).toBe(true);
+  });
+
   it("passes nonEmpty when every call filled the field", () => {
     const checks = validateToolExpectations({
       run: {

ai_evals/core/validators.ts

@@ -320,6 +320,23 @@ export function validateToolExpectations(input: {
       );
     }
 
+    if (rule.sharedByAtLeast !== undefined) {
+      const counts = new Map<string, number>();
+      for (const value of values) {
+        if (typeof value === "string" && value.trim().length > 0) {
+          counts.set(value, (counts.get(value) ?? 0) + 1);
+        }
+      }
+      const mostShared = Math.max(0, ...counts.values());
+      checks.push(
+        check(
+          `${rule.tool}.${rule.field} is shared by at least ${rule.sharedByAtLeast} calls`,
+          mostShared >= rule.sharedByAtLeast,
+          `most calls sharing one value: ${mostShared}; values: ${summarizeToolValues(values)}`
+        )
+      );
+    }
+
     if (rule.fieldMustBeAbsent) {
       // Anything other than `undefined` was supplied — an explicit `null` is the
       // model passing the field, not omitting it.

ai_evals/fixtures/frontend/global/initial/ducklake_orders_metrics.json

@@ -0,0 +1,36 @@
+{
+  "workspace": {
+    "ducklakes": ["main"],
+    "dataMetrics": [
+      {
+        "script_path": "f/analytics/orders_pipeline",
+        "table_path": "main/main.orders",
+        "kind": "measure",
+        "name": "revenue",
+        "expr": "sum(amount)",
+        "filter": "not is_test"
+      },
+      {
+        "script_path": "f/analytics/orders_pipeline",
+        "table_path": "main/main.orders",
+        "kind": "measure",
+        "name": "order_count",
+        "expr": "count(*)"
+      },
+      {
+        "script_path": "f/analytics/orders_pipeline",
+        "table_path": "main/main.orders",
+        "kind": "dimension",
+        "name": "order_month",
+        "expr": "date_trunc('month', order_date)"
+      },
+      {
+        "script_path": "f/analytics/orders_pipeline",
+        "table_path": "main/main.orders",
+        "kind": "dimension",
+        "name": "region",
+        "expr": "region"
+      }
+    ]
+  }
+}

ai_evals/fixtures/frontend/global/initial/support_chat_flow.json

@@ -0,0 +1,59 @@
+{
+  "workspace": {
+    "flows": [
+      {
+        "path": "f/evals/global/support_chat",
+        "summary": "Support chat",
+        "description": "Answers customer questions in a chat, remembering earlier messages.",
+        "schema": {
+          "$schema": "https://json-schema.org/draft/2020-12/schema",
+          "type": "object",
+          "properties": {
+            "user_message": {
+              "type": "string",
+              "description": "Message from user"
+            }
+          },
+          "required": ["user_message"]
+        },
+        "value": {
+          "chat_input_enabled": true,
+          "modules": [
+            {
+              "id": "assistant",
+              "summary": "Support assistant",
+              "value": {
+                "type": "aiagent",
+                "tools": [],
+                "input_transforms": {
+                  "provider": {
+                    "type": "static",
+                    "value": {
+                      "kind": "anthropic",
+                      "model": "claude-haiku-4-5-20251001",
+                      "resource": "$res:f/evals/ai/anthropic"
+                    }
+                  },
+                  "user_message": {
+                    "type": "javascript",
+                    "expr": "flow_input.user_message"
+                  },
+                  "system_prompt": {
+                    "type": "static",
+                    "value": "You are a friendly support assistant. Keep answers short."
+                  },
+                  "memory": {
+                    "type": "static",
+                    "value": { "kind": "auto", "context_length": 10 }
+                  },
+                  "streaming": { "type": "static", "value": true },
+                  "output_type": { "type": "static", "value": "text" }
+                }
+              }
+            }
+          ]
+        }
+      }
+    ]
+  }
+}

backend/.sqlx/query-12329c3359a7944ab5fa3aa27ddca1b26f340ccf574b9fa07641fe88b2d2987c.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success)\n         VALUES ($1, $2, $3, $4, $5, $6)",
+  "query": "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success, tool_arguments, tool_result, reasoning, attachments)\n         VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)",
   "describe": {
     "columns": [],
     "parameters": {
@@ -21,10 +21,14 @@
         "Text",
         "Uuid",
         "Varchar",
-        "Bool"
+        "Bool",
+        "Text",
+        "Text",
+        "Text",
+        "Jsonb"
       ]
     },
     "nullable": []
   },
-  "hash": "b1a9a433e577133869c067b2ce383fc6ce4e9df307feb5fd3edc0d1276d61ff1"
+  "hash": "12329c3359a7944ab5fa3aa27ddca1b26f340ccf574b9fa07641fe88b2d2987c"
 }

backend/.sqlx/query-48c8522a4fed219c5011f4ba63c81cfe028a8b2a32bd790840cef65c452a8c31.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by\n         FROM flow_conversation\n         WHERE id = $1 AND workspace_id = $2\n         FOR UPDATE",
+  "query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test\n         FROM flow_conversation\n         WHERE id = $1 AND workspace_id = $2",
   "describe": {
     "columns": [
       {
@@ -37,6 +37,11 @@
         "ordinal": 6,
         "name": "created_by",
         "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "is_test",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -52,8 +57,9 @@
       true,
       false,
       false,
+      false,
       false
     ]
   },
-  "hash": "6f32c1feed096ff706ae359ad6a3ca33b3f82ca38289dfa4a69aa95041027d57"
+  "hash": "48c8522a4fed219c5011f4ba63c81cfe028a8b2a32bd790840cef65c452a8c31"
 }

backend/.sqlx/query-5b9c9eb64051f291fed4be9bc0b0cc0aef2e7bde732899976eddac36a2da7658.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "UPDATE flow_conversation SET title = $1, updated_at = updated_at\n         WHERE id = $2 AND workspace_id = $3\n         RETURNING id",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Uuid"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Uuid",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false
+    ]
+  },
+  "hash": "5b9c9eb64051f291fed4be9bc0b0cc0aef2e7bde732899976eddac36a2da7658"
+}

backend/.sqlx/query-6d259b8cce5da5fecefe4ce322789b6b2cc43b51f2056c677d58f39c31fb26cb.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title)\n         VALUES ($1, $2, $3, $4, $5)\n         ON CONFLICT (id) DO NOTHING\n         RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by",
+  "query": "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title, is_test)\n         VALUES ($1, $2, $3, $4, $5, $6)\n         ON CONFLICT (id) DO NOTHING\n         RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test",
   "describe": {
     "columns": [
       {
@@ -37,6 +37,11 @@
         "ordinal": 6,
         "name": "created_by",
         "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "is_test",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -45,7 +50,8 @@
         "Varchar",
         "Varchar",
         "Varchar",
-        "Varchar"
+        "Varchar",
+        "Bool"
       ]
     },
     "nullable": [
@@ -55,8 +61,9 @@
       true,
       false,
       false,
+      false,
       false
     ]
   },
-  "hash": "c1e3ed3ecc3bcb98f60ba8196d33fee4a74f61b061e5025ecb75882208b3ba8f"
+  "hash": "6d259b8cce5da5fecefe4ce322789b6b2cc43b51f2056c677d58f39c31fb26cb"
 }

backend/.sqlx/query-71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0.json

@@ -0,0 +1,38 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT id, name, enabled, pwd FROM datatable_role",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "id",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "name",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 2,
+        "name": "enabled",
+        "type_info": "Bool"
+      },
+      {
+        "ordinal": 3,
+        "name": "pwd",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      false,
+      false,
+      false,
+      true
+    ]
+  },
+  "hash": "71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0"
+}

backend/.sqlx/query-79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629.json

@@ -0,0 +1,29 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "\n        SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n        FROM workspace_settings ws\n        JOIN workspace w ON w.id = ws.workspace_id AND w.deleted = false\n        CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n        WHERE ws.workspace_id <> $1\n          AND dt.value->'database'->>'resource_type' = 'instance'\n          AND dt.value->'database'->>'resource_path' = $2\n        ORDER BY ws.workspace_id, dt.key\n        ",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "workspace_id!",
+        "type_info": "Varchar"
+      },
+      {
+        "ordinal": 1,
+        "name": "datatable!",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      false,
+      null
+    ]
+  },
+  "hash": "79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629"
+}

backend/.sqlx/query-86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba.json

@@ -0,0 +1,17 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "INSERT INTO datatable_role (id, name, enabled, pwd) VALUES ($1, $2, $3, $4)",
+  "describe": {
+    "columns": [],
+    "parameters": {
+      "Left": [
+        "Varchar",
+        "Varchar",
+        "Bool",
+        "Text"
+      ]
+    },
+    "nullable": []
+  },
+  "hash": "86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba"
+}

backend/.sqlx/query-9008f9abb70a9a07e38acb20bea6a710d0efd77dac4aedeb88d72240e816530b.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "\n        SELECT\n            j.args as \"args: Json<HashMap<String, Box<RawValue>>>\",\n            js.flow_status as \"flow_status: Json<windmill_common::flow_status::FlowStatus>\"\n        FROM v2_job_status js\n        INNER JOIN v2_job j ON j.id = js.id\n        WHERE js.id = $1\n        ",
+  "query": "\n        SELECT\n            j.args as \"args: Json<HashMap<String, Box<RawValue>>>\",\n            js.flow_status as \"flow_status: Json<windmill_common::flow_status::FlowStatus>\",\n            j.runnable_path\n        FROM v2_job_status js\n        INNER JOIN v2_job j ON j.id = js.id\n        WHERE js.id = $1\n        ",
   "describe": {
     "columns": [
       {
@@ -12,6 +12,11 @@
         "ordinal": 1,
         "name": "flow_status: Json<windmill_common::flow_status::FlowStatus>",
         "type_info": "Jsonb"
+      },
+      {
+        "ordinal": 2,
+        "name": "runnable_path",
+        "type_info": "Varchar"
       }
     ],
     "parameters": {
@@ -20,9 +25,10 @@
       ]
     },
     "nullable": [
+      true,
       true,
       true
     ]
   },
-  "hash": "dd89d652154748d6d7e625e31778f6885d0ee62d29a4b8894a4b459dd215a103"
+  "hash": "9008f9abb70a9a07e38acb20bea6a710d0efd77dac4aedeb88d72240e816530b"
 }

backend/.sqlx/query-a4a823f70b3dbe6aaf4a61c98345e94c5042fd5e6351fea139a66ecb1fb812ab.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success\n             FROM flow_conversation_message\n             WHERE conversation_id = $1\n               AND created_seq > $2\n             ORDER BY created_seq ASC\n             LIMIT $3\n             ",
+  "query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n             FROM (\n                SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n                FROM flow_conversation_message\n                WHERE conversation_id = $1\n                ORDER BY created_seq DESC\n                LIMIT $2 OFFSET $3\n             ) AS messages\n             ORDER BY created_seq ASC\n             ",
   "describe": {
     "columns": [
       {
@@ -58,6 +58,26 @@
         "ordinal": 8,
         "name": "success",
         "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "tool_arguments",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 10,
+        "name": "tool_result",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 11,
+        "name": "reasoning",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 12,
+        "name": "attachments",
+        "type_info": "Jsonb"
       }
     ],
     "parameters": {
@@ -76,8 +96,12 @@
       false,
       false,
       true,
-      false
+      false,
+      true,
+      true,
+      true,
+      true
     ]
   },
-  "hash": "e8802be9203c1e88a06e337260ccca029380139f89a01a89033e36a6ed9ac082"
+  "hash": "a4a823f70b3dbe6aaf4a61c98345e94c5042fd5e6351fea139a66ecb1fb812ab"
 }

backend/.sqlx/query-b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6.json

@@ -0,0 +1,20 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT jsonb_object_keys(value->'databases') FROM global_settings\n         WHERE name = 'custom_instance_pg_databases'",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "jsonb_object_keys",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": []
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6"
+}

backend/.sqlx/query-d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40.json

@@ -0,0 +1,24 @@
+{
+  "db_name": "PostgreSQL",
+  "query": "SELECT dt.key AS \"datatable!\"\n           FROM workspace_settings ws\n           CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n           WHERE ws.workspace_id = $1\n             AND dt.key <> $2\n             AND NOT dt.value ? 'permissions'\n             AND dt.value->'database'->>'resource_type' = 'instance'\n             AND dt.value->'database'->>'resource_path' = $3\n           ORDER BY dt.key",
+  "describe": {
+    "columns": [
+      {
+        "ordinal": 0,
+        "name": "datatable!",
+        "type_info": "Text"
+      }
+    ],
+    "parameters": {
+      "Left": [
+        "Text",
+        "Text",
+        "Text"
+      ]
+    },
+    "nullable": [
+      null
+    ]
+  },
+  "hash": "d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40"
+}

backend/.sqlx/query-d6fa78c43b6c5f8040d7bccb29ad8627be1dac6fbe0097735a52f47c173f51c9.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success\n             FROM (\n                SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success\n                FROM flow_conversation_message\n                WHERE conversation_id = $1\n                ORDER BY created_seq DESC\n                LIMIT $2 OFFSET $3\n             ) AS messages\n             ORDER BY created_seq ASC\n             ",
+  "query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n             FROM flow_conversation_message\n             WHERE conversation_id = $1\n               AND created_seq > $2\n             ORDER BY created_seq ASC\n             LIMIT $3\n             ",
   "describe": {
     "columns": [
       {
@@ -58,6 +58,26 @@
         "ordinal": 8,
         "name": "success",
         "type_info": "Bool"
+      },
+      {
+        "ordinal": 9,
+        "name": "tool_arguments",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 10,
+        "name": "tool_result",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 11,
+        "name": "reasoning",
+        "type_info": "Text"
+      },
+      {
+        "ordinal": 12,
+        "name": "attachments",
+        "type_info": "Jsonb"
       }
     ],
     "parameters": {
@@ -76,8 +96,12 @@
       false,
       false,
       true,
-      false
+      false,
+      true,
+      true,
+      true,
+      true
     ]
   },
-  "hash": "1c3473a0f9f6b6148b2c975f9f05bdefedf8a51c4e6ddf0eca367b9cc778d051"
+  "hash": "d6fa78c43b6c5f8040d7bccb29ad8627be1dac6fbe0097735a52f47c173f51c9"
 }

backend/.sqlx/query-dd84f9dfb238d18cb74f9e43228345427131bf8008eba6920021d6a449791534.json

@@ -1,6 +1,6 @@
 {
   "db_name": "PostgreSQL",
-  "query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by\n         FROM flow_conversation\n         WHERE id = $1 AND workspace_id = $2",
+  "query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test\n         FROM flow_conversation\n         WHERE id = $1 AND workspace_id = $2\n         FOR UPDATE",
   "describe": {
     "columns": [
       {
@@ -37,6 +37,11 @@
         "ordinal": 6,
         "name": "created_by",
         "type_info": "Varchar"
+      },
+      {
+        "ordinal": 7,
+        "name": "is_test",
+        "type_info": "Bool"
       }
     ],
     "parameters": {
@@ -52,8 +57,9 @@
       true,
       false,
       false,
+      false,
       false
     ]
   },
-  "hash": "c383cc023714b361d10c10e8fef1fc148ab1da942951ee9ffdddaecee76a6be9"
+  "hash": "dd84f9dfb238d18cb74f9e43228345427131bf8008eba6920021d6a449791534"
 }

backend/Cargo.lock

@@ -728,7 +728,7 @@ dependencies = [
  "futures-lite 2.6.1",
  "parking",
  "polling 3.11.0",
- "rustix 1.1.4",
+ "rustix 1.1.5",
  "slab",
  "windows-sys 0.61.2",
 ]
@@ -873,7 +873,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -1976,7 +1976,7 @@ dependencies = [
  "prettyplease 0.3.0",
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -2000,7 +2000,7 @@ dependencies = [
  "proc-macro-crate",
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -2144,7 +2144,7 @@ checksum = "6a1f896587b6f2c069c73d2f0913e2d590c3990285cd2f0b6aa02b786b4c679c"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -2164,9 +2164,9 @@ dependencies = [
 
 [[package]]
 name = "bytes-str"
-version = "0.2.8"
+version = "0.2.9"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "577d2bf5650f8554d5a372af5ac93535110a0fc75b3e702bb853369febf227c2"
+checksum = "4dde6d05e75a31ec9610eb6446a6f0a10dd30ff5100d720fee4c7c7a9008b5ba"
 dependencies = [
  "bytes",
  "serde",
@@ -2338,9 +2338,9 @@ dependencies = [
 
 [[package]]
 name = "cfg-if"
-version = "1.0.4"
+version = "1.0.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
+checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
 
 [[package]]
 name = "cfg_aliases"
@@ -2454,7 +2454,7 @@ dependencies = [
  "heck",
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -3047,7 +3047,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "strsim 0.11.1",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -3102,7 +3102,7 @@ checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785"
 dependencies = [
  "darling_core 0.24.1",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -4580,7 +4580,7 @@ checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
 dependencies = [
  "libc",
  "option-ext",
- "redox_users 0.5.2",
+ "redox_users 0.5.3",
  "windows-sys 0.61.2",
 ]
 
@@ -4603,7 +4603,7 @@ checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -4850,7 +4850,7 @@ checksum = "a65863d15a4ce2888bd2f0f543cc963d3879c3a022c8ee43f6141d479a3ac815"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -5198,7 +5198,7 @@ version = "0.13.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8640e34b88f7652208ce9e88b1a37a2ae95227d84abec377ccd3c5cfeb141ed4"
 dependencies = [
- "rustix 1.1.4",
+ "rustix 1.1.5",
  "windows-sys 0.59.0",
 ]
 
@@ -5319,7 +5319,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -9438,7 +9438,7 @@ dependencies = [
  "concurrent-queue",
  "hermit-abi 0.5.3",
  "pin-project-lite",
- "rustix 1.1.4",
+ "rustix 1.1.5",
  "windows-sys 0.61.2",
 ]
 
@@ -9574,7 +9574,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2bfe0f4c752e450fc2faf62654f1c134747922825d5b04ca717b8874f41a40c0"
 dependencies = [
  "proc-macro2",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -10235,11 +10235,10 @@ dependencies = [
 
 [[package]]
 name = "redox_users"
-version = "0.5.2"
+version = "0.5.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
+checksum = "60dc65c0ff1a7ae1294b0c67b9f14baf70b644404010370171787bfac1038fc0"
 dependencies = [
- "getrandom 0.2.17",
  "libredox",
  "thiserror 2.0.20",
 ]
@@ -10261,7 +10260,7 @@ checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -10580,7 +10579,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "serde_json",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -10798,9 +10797,9 @@ dependencies = [
 
 [[package]]
 name = "rustix"
-version = "1.1.4"
+version = "1.1.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
+checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
 dependencies = [
  "bitflags 2.13.2",
  "errno",
@@ -11217,7 +11216,7 @@ dependencies = [
  "proc-macro2",
  "quote",
  "serde_derive_internals 0.30.0",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -11416,7 +11415,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -11438,7 +11437,7 @@ checksum = "f852137cce035d6a4df67ccce505ff6b3e9fd3a10e3e52b24dc71e650bb1a9bd"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -11492,7 +11491,7 @@ checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -11560,7 +11559,7 @@ dependencies = [
  "darling 0.24.1",
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -12708,9 +12707,9 @@ dependencies = [
 
 [[package]]
 name = "syn"
-version = "3.0.5"
+version = "3.0.6"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
+checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -12745,7 +12744,7 @@ checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -13036,7 +13035,7 @@ dependencies = [
  "fastrand 2.5.0",
  "getrandom 0.4.3",
  "once_cell",
- "rustix 1.1.4",
+ "rustix 1.1.5",
  "windows-sys 0.61.2",
 ]
 
@@ -13055,7 +13054,7 @@ version = "0.4.4"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874"
 dependencies = [
- "rustix 1.1.4",
+ "rustix 1.1.5",
  "windows-sys 0.61.2",
 ]
 
@@ -13115,7 +13114,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -14075,7 +14074,7 @@ checksum = "f153acc4e99a5f2a5aefa09fb078be54e26271b2813f6041200b224c098d8328"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -14179,9 +14178,9 @@ checksum = "81b79ad29b5e19de4260020f8919b443b2ef0277d242ce532ec7b7a2cc8b6007"
 
 [[package]]
 name = "unicode-ident"
-version = "1.0.24"
+version = "1.0.26"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
+checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
 
 [[package]]
 name = "unicode-normalization"
@@ -14546,7 +14545,7 @@ dependencies = [
  "bumpalo",
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
  "wasm-bindgen-shared",
 ]
 
@@ -14589,7 +14588,7 @@ checksum = "8c89dcab8b516b6b603baca9d550b7282d68fcc7f367e3956cff7ebf406a3f12"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]
@@ -14794,7 +14793,7 @@ dependencies = [
 
 [[package]]
 name = "windmill"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -14882,7 +14881,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-ai"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "async-stream",
  "async-trait",
@@ -14896,6 +14895,7 @@ dependencies = [
  "eventsource-stream",
  "futures",
  "http 1.5.0",
+ "indexmap 2.14.2",
  "lazy_static",
  "mime_guess",
  "reqwest 0.13.5",
@@ -14915,7 +14915,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-alerting"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -14928,7 +14928,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "argon2",
@@ -15068,7 +15068,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-agent-workers"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15091,7 +15091,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-assets"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15108,7 +15108,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-auth"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15134,7 +15134,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-client"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "reqwest 0.12.28",
  "serde",
@@ -15144,7 +15144,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-configs"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15161,7 +15161,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-debug"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "base64 0.22.1",
@@ -15183,7 +15183,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-embeddings"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15206,7 +15206,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-flow-conversations"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15222,7 +15222,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-flows"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15244,7 +15244,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-groups"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15266,7 +15266,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-inputs"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15280,7 +15280,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-integration-tests"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -15315,7 +15315,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-jobs"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15340,7 +15340,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-npm-proxy"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15368,7 +15368,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-openapi"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15390,7 +15390,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-schedule"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15410,7 +15410,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-scripts"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15448,7 +15448,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-settings"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15477,7 +15477,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-sse"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "lazy_static",
  "serde",
@@ -15489,7 +15489,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-users"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "argon2",
  "axum 0.8.9",
@@ -15513,7 +15513,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-workers"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15527,7 +15527,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-api-workspaces"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "axum 0.8.9",
  "chrono",
@@ -15562,7 +15562,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-audit"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "chrono",
  "lazy_static",
@@ -15576,7 +15576,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-autoscaling"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "axum 0.8.9",
@@ -15595,7 +15595,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-common"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "aes-gcm",
  "aho-corasick",
@@ -15667,6 +15667,7 @@ dependencies = [
  "serde",
  "serde_json",
  "serde_yml",
+ "sha1",
  "sha2 0.10.9",
  "size",
  "spki",
@@ -15702,7 +15703,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-dep-map"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "chrono",
  "futures",
@@ -15722,7 +15723,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-git-sync"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "regex",
  "serde",
@@ -15739,7 +15740,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-indexer"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "astral-tokio-tar",
@@ -15766,7 +15767,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-jseval"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "futures",
@@ -15783,7 +15784,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-macros"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "itertools 0.14.0",
  "lazy_static",
@@ -15799,7 +15800,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-mcp"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -15820,7 +15821,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-native-triggers"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -15851,7 +15852,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-oauth"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "arc-swap",
@@ -15876,7 +15877,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-object-store"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-stream",
@@ -15911,7 +15912,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-operator"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "futures",
@@ -15929,7 +15930,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "convert_case 0.6.0",
  "serde",
@@ -15938,7 +15939,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-bash"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -15950,7 +15951,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-csharp"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -15962,7 +15963,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-go"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "gosyn",
@@ -15974,7 +15975,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-graphql"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -15986,7 +15987,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-java"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -15998,7 +15999,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-nu"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "nu-parser",
@@ -16009,7 +16010,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-php"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -16020,7 +16021,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -16032,7 +16033,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -16043,7 +16044,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16065,7 +16066,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-r"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -16077,7 +16078,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ruby"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16091,7 +16092,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-rust"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "convert_case 0.6.0",
@@ -16108,7 +16109,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16121,7 +16122,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde",
@@ -16133,7 +16134,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16151,7 +16152,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde-wasm-bindgen",
@@ -16167,7 +16168,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wac"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -16183,7 +16184,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-yaml"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -16197,7 +16198,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-queue"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16236,7 +16237,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-runtime-nativets"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "const_format",
@@ -16276,7 +16277,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-sql-datatype-parser-wasm"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "getrandom 0.3.4",
  "wasm-bindgen",
@@ -16287,7 +16288,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-store"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -16322,7 +16323,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-test-utils"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16346,7 +16347,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16379,7 +16380,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-amqp"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16406,7 +16407,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-azure"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16439,7 +16440,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-email"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16459,7 +16460,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-gcp"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16493,7 +16494,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-http"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16529,7 +16530,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-kafka"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16552,7 +16553,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-mqtt"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16576,7 +16577,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-nats"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-nats",
@@ -16600,7 +16601,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-postgres"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16635,7 +16636,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-sqs"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16663,7 +16664,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-trigger-websocket"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-trait",
@@ -16688,7 +16689,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-types"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "bitflags 2.13.2",
@@ -16707,7 +16708,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-worker"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-once-cell",
@@ -16825,7 +16826,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-worker-volumes"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "bytes",
  "futures",
@@ -17458,7 +17459,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156"
 dependencies = [
  "libc",
- "rustix 1.1.4",
+ "rustix 1.1.5",
 ]
 
 [[package]]
@@ -17519,7 +17520,7 @@ checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
  "synstructure 0.14.0",
 ]
 
@@ -17560,7 +17561,7 @@ checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
  "synstructure 0.14.0",
 ]
 
@@ -17616,7 +17617,7 @@ checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da"
 dependencies = [
  "proc-macro2",
  "quote",
- "syn 3.0.5",
+ "syn 3.0.6",
 ]
 
 [[package]]

backend/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "windmill"
-version = "1.813.0"
+version = "1.814.0"
 authors.workspace = true
 edition.workspace = true
 
@@ -88,7 +88,7 @@ members = [
 exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
 
 [workspace.package]
-version = "1.813.0"
+version = "1.814.0"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 edition = "2021"
 

backend/ee-repo-ref.txt

@@ -1 +1 @@
-151be033ea2bf2f768f381a8cd3b888bdd366cf9
+23d12f73e44a24bb91fa54d79dfc4ae1436e0227

backend/migrations/20260916144653_flow_conversation_message_tool_call.down.sql

@@ -0,0 +1,3 @@
+ALTER TABLE flow_conversation_message DROP COLUMN tool_arguments;
+ALTER TABLE flow_conversation_message DROP COLUMN tool_result;
+ALTER TABLE flow_conversation_message DROP COLUMN reasoning;

backend/migrations/20260916144653_flow_conversation_message_tool_call.up.sql

@@ -0,0 +1,12 @@
+-- A chat is rebuilt from its rows without reading jobs, so every tool row carries its call:
+-- the arguments the model wrote and the text the model got back, or what the call failed
+-- with. A script or flow tool's job holds the args its input transforms produced, not the
+-- model's; an MCP tool runs inside the agent's job, whose result lists every call of the
+-- turn with nothing tying one to a row. A provider-native web search carries only its
+-- citations, the provider never returning the query.
+ALTER TABLE flow_conversation_message ADD COLUMN tool_arguments TEXT;
+ALTER TABLE flow_conversation_message ADD COLUMN tool_result TEXT;
+
+-- The thinking behind this row. The agent job keeps the turn's thinking as one string;
+-- the rows keep it per iteration, next to the answer or tool call it led to.
+ALTER TABLE flow_conversation_message ADD COLUMN reasoning TEXT;

backend/migrations/20260916155738_flow_conversation_is_test.down.sql

@@ -0,0 +1 @@
+ALTER TABLE flow_conversation DROP COLUMN is_test;

backend/migrations/20260916155738_flow_conversation_is_test.up.sql

@@ -0,0 +1,26 @@
+-- A chat run from the flow editor's test panel is stored exactly like one from the
+-- deployed flow, so the two were indistinguishable once written. Marking them lets the
+-- lists tell a trial apart from a real conversation.
+ALTER TABLE flow_conversation ADD COLUMN is_test BOOLEAN NOT NULL DEFAULT false;
+
+-- Existing rows: a conversation whose messages came from a flowpreview run was a test.
+-- Derived once here because the job is purged on retention, after which the origin of an
+-- old conversation is unknowable.
+--
+-- Walked to the root job rather than matched directly: an existing message row never holds
+-- the flow job itself. The rows point at the step that produced them — the AI agent's job
+-- for an answer, the tool's own job for a tool call — whose kind is never 'flowpreview'.
+--
+-- `root_job` first, matching `get_root_job_id` (windmill-worker/src/common.rs): only it
+-- reaches the top of the run. `flow_innermost_root_job` stops at the closest flow scope by
+-- design, so an agent inside a subflow would land on that subflow's 'flow' row and the
+-- conversation would read as deployed.
+UPDATE flow_conversation c
+SET is_test = true
+WHERE EXISTS (
+    SELECT 1 FROM flow_conversation_message m
+    JOIN v2_job j ON j.id = m.job_id
+    JOIN v2_job root
+      ON root.id = coalesce(j.root_job, j.flow_innermost_root_job, j.parent_job, j.id)
+    WHERE m.conversation_id = c.id AND root.kind = 'flowpreview'
+);

backend/migrations/20260917074040_flow_conversation_message_attachments.down.sql

@@ -0,0 +1 @@
+ALTER TABLE flow_conversation_message DROP COLUMN attachments;

backend/migrations/20260917074040_flow_conversation_message_attachments.up.sql

@@ -0,0 +1,4 @@
+-- The files a user message carried, as object-storage references: `[{input, s3, storage?,
+-- filename?}]`. Only references, never file bytes and never a presigned URL, so a
+-- transcript can show a message's files without reading its run's args.
+ALTER TABLE flow_conversation_message ADD COLUMN attachments JSONB;

backend/migrations/20260917133824_datatable_role_cluster.down.sql

@@ -1,22 +0,0 @@
--- Roles on the external cluster are live logins there; dropping the column would forget them.
-LOCK TABLE datatable_role;
-DO $$
-BEGIN
-    IF EXISTS (SELECT 1 FROM datatable_role WHERE cluster <> 'instance') THEN
-        RAISE EXCEPTION 'datatable_role holds roles on the external instance cluster. Delete them in instance settings first.';
-    END IF;
-    -- Before this, only data tables on Windmill's own cluster could be under roles, and a role
-    -- block left with just `admin` survives deleting every external role.
-    IF EXISTS (
-        SELECT 1 FROM workspace_settings ws,
-            jsonb_each(CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
-                THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
-        WHERE dt.value->'database'->>'resource_type' = 'external_instance'
-          AND dt.value ? 'permissions'
-    ) THEN
-        RAISE EXCEPTION 'external instance data tables are still under roles. Turn their roles off first.';
-    END IF;
-END $$;
-ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_cluster_name_key;
-ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_name_key UNIQUE (name);
-ALTER TABLE datatable_role DROP COLUMN cluster;

backend/migrations/20260917133824_datatable_role_cluster.up.sql

@@ -1,8 +0,0 @@
--- A data table role is a Postgres login on one cluster: Windmill's own ('instance'), or the external
--- instance cluster ('external_instance'). Role names are the cluster's own key, so they are unique
--- per cluster rather than across the instance.
-ALTER TABLE datatable_role
-    ADD COLUMN cluster VARCHAR(20) NOT NULL DEFAULT 'instance'
-        CHECK (cluster IN ('instance', 'external_instance'));
-ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_name_key;
-ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_cluster_name_key UNIQUE (cluster, name);

backend/parsers/windmill-parser-wasm/Cargo.lock

@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
 
 [[package]]
 name = "windmill-common"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "aho-corasick",
  "anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-macros"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "proc-macro2",
  "quote",
@@ -6286,7 +6286,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "convert_case",
  "serde",
@@ -6295,7 +6295,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-bash"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-csharp"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-go"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-graphql"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-java"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-nu"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-php"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-py-imports"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-r"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ruby"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-rust"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-sql-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde",
@@ -6490,7 +6490,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-ts-asset"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wac"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-wasm"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-parser-yaml"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
 
 [[package]]
 name = "windmill-types"
-version = "1.813.0"
+version = "1.814.0"
 dependencies = [
  "anyhow",
  "bitflags",

backend/parsers/windmill-parser-wasm/Cargo.toml

@@ -12,7 +12,7 @@ resolver = "2"
 members = ["."]
 
 [workspace.package]
-version = "1.813.0"
+version = "1.814.0"
 edition = "2021"
 authors = ["Ruben Fiszel <ruben@windmill.dev>"]
 

backend/summarized_schema.txt

@@ -99,9 +99,9 @@ email_trigger: path(char), local_part(char), workspaced_local_part(bool), script
 favorite: usr(char), workspace_id(char), path(char), favorite_kind(favorite_kind)
 flow: workspace_id(char), path(char), summary(text), description(text), value(jsonb), edited_by(char), edited_at(ts), archived(bool), schema(json), extra_perms(jsonb), dependency_job(uuid), draft_only(bool), tag(char), ws_error_handler_muted(bool), dedicated_worker(bool), timeout(int), visible_to_runner_only(bool), concurrency_key(char), versions(bigint[]), on_behalf_of(varchar), on_behalf_of_email(text), lock_error_logs(text), labels(text[])
   FK: (workspace_id) -> workspace(id)
-flow_conversation: id(uuid), workspace_id(char), flow_path(char), title(char), created_at(ts), updated_at(ts), created_by(char)
+flow_conversation: id(uuid), workspace_id(char), flow_path(char), title(char), created_at(ts), updated_at(ts), created_by(char), is_test(bool)
   FK: (workspace_id) -> workspace(id)
-flow_conversation_message: id(uuid), conversation_id(uuid), message_type(message_type), content(text), job_id(uuid), created_at(ts), created_seq(int8), step_name(char), success(bool)
+flow_conversation_message: id(uuid), conversation_id(uuid), message_type(message_type), content(text), job_id(uuid), created_at(ts), created_seq(int8), step_name(char), success(bool), tool_arguments(text), tool_result(text), reasoning(text), attachments(jsonb)
   FK: (conversation_id) -> flow_conversation(id) | (job_id) -> v2_job(id)
 flow_iterator_data: job_id(uuid), itered(jsonb)
 flow_node: id(bigint), workspace_id(char), hash(bigint), path(char), lock(text), code(text), flow(jsonb), hash_v2(char(64))

backend/tests/instance_config.rs

@@ -1583,3 +1583,49 @@ async fn declarative_sync_rejects_an_unusable_default_allowed_origins(db: Pool<P
     .await
     .expect("a valid origin list must sync");
 }
+
+/// While Windmill has databases on the external cluster, a sync may change its login but not
+/// point it at another cluster: the databases would be stranded there, and their registry and
+/// role passwords applied to a cluster that has neither.
+#[cfg(all(feature = "enterprise", feature = "private"))]
+#[sqlx::test(fixtures("base"))]
+async fn declarative_sync_keeps_the_external_cluster_while_it_holds_databases(db: Pool<Postgres>) {
+    clear_settings_and_configs(&db).await;
+    let cluster = |host: &str, password: &str| serde_json::json!({ "host": host, "port": 5432, "user": "wm_admin", "password": password });
+    sqlx::query(
+        "INSERT INTO global_settings (name, value) VALUES
+             ('external_instance_pg', $1),
+             ('external_instance_pg_state', '{\"databases\": {\"dt_a\": {\"success\": true}}}')",
+    )
+    .bind(cluster("pg-a.internal", "one"))
+    .execute(&db)
+    .await
+    .unwrap();
+    let current = BTreeMap::from([(
+        "external_instance_pg".to_string(),
+        cluster("pg-a.internal", "one"),
+    )]);
+    let sync = |value: serde_json::Value| {
+        let desired = BTreeMap::from([("external_instance_pg".to_string(), value)]);
+        let (db, current) = (db.clone(), current.clone());
+        async move {
+            windmill_common::instance_config::sync_global_settings_declarative(
+                &db, &current, &desired,
+            )
+            .await
+        }
+    };
+
+    let err = sync(cluster("pg-b.internal", "one"))
+        .await
+        .expect_err("another host must be refused while dt_a is registered");
+    assert!(err.to_string().contains("dt_a"), "got: {err}");
+    assert_eq!(
+        get_global_setting(&db, "external_instance_pg").await,
+        Some(cluster("pg-a.internal", "one"))
+    );
+
+    sync(cluster("PG-A.internal ", "two"))
+        .await
+        .expect("a new login on the same cluster must sync");
+}

backend/tests/list_jobs.rs

@@ -1199,3 +1199,63 @@ async fn test_wm_labels_from_result_merged_with_static_labels(
 
     Ok(())
 }
+
+/// `tag` lives only on `v2_job`, which count_jobs joins only when `tags` is set.
+#[sqlx::test(fixtures("base"))]
+async fn test_count_completed_jobs_tags_filter(db: Pool<Postgres>) -> anyhow::Result<()> {
+    initialize_tracing().await;
+
+    let server = ApiServer::start(db.clone()).await?;
+    let port = server.addr.port();
+    let client = windmill_api_client::create_client(
+        &format!("http://localhost:{port}"),
+        "SECRET_TOKEN".to_string(),
+    );
+
+    for (ws, tag, status) in [
+        ("test-workspace", "deno", "success"),
+        ("test-workspace", "deno", "failure"),
+        ("test-workspace", "python3", "success"),
+        ("other-workspace", "deno", "success"),
+    ] {
+        let id = uuid::Uuid::new_v4();
+        sqlx::query("INSERT INTO v2_job (id, workspace_id, tag) VALUES ($1, $2, $3)")
+            .bind(id)
+            .bind(ws)
+            .bind(tag)
+            .execute(&db)
+            .await?;
+        sqlx::query(
+            "INSERT INTO v2_job_completed (id, workspace_id, status, duration_ms) VALUES ($1, $2, $3::job_status, 0)",
+        )
+        .bind(id)
+        .bind(ws)
+        .bind(status)
+        .execute(&db)
+        .await?;
+    }
+
+    for (query, expected) in [
+        ("", 3),
+        ("tags=deno", 2),
+        ("tags=deno&success=true", 1),
+        ("tags=deno,python3&completed_after_s_ago=3600", 3),
+    ] {
+        let response = client
+            .client()
+            .get(format!(
+                "{}/w/test-workspace/jobs/completed/count_jobs?{query}",
+                client.baseurl()
+            ))
+            .send()
+            .await?;
+        assert!(
+            response.status().is_success(),
+            "{query}: {}",
+            response.text().await?
+        );
+        assert_eq!(response.json::<i64>().await?, expected, "{query}");
+    }
+
+    Ok(())
+}

backend/tests/v2_job_delete_orphans.rs

@@ -258,6 +258,7 @@ async fn test_new_turns_wait_for_conversation_cleanup_and_recreate(
                     "test-user",
                     "hi again",
                     conv_id,
+                    false,
                 )
                 .await?;
                 windmill_common::flow_conversations::add_message_to_conversation_tx(
@@ -268,6 +269,7 @@ async fn test_new_turns_wait_for_conversation_cleanup_and_recreate(
                     windmill_common::flow_conversations::MessageType::User,
                     None,
                     true,
+                    None,
                 )
                 .await?;
                 tx.commit().await?;

backend/windmill-ai/Cargo.toml

@@ -23,6 +23,7 @@ async-trait.workspace = true
 async-stream.workspace = true
 base64.workspace = true
 bytes.workspace = true
+indexmap.workspace = true
 eventsource-stream.workspace = true
 futures.workspace = true
 http.workspace = true

backend/windmill-ai/src/providers/bedrock.rs

@@ -1074,7 +1074,8 @@ impl BedrockQueryBuilder {
 
         let mut accumulated_text = String::new();
         let mut events_str = String::new();
-        let mut accumulated_tool_calls: HashMap<String, StreamingToolCall> = HashMap::new();
+        let mut accumulated_tool_calls: indexmap::IndexMap<String, StreamingToolCall> =
+            indexmap::IndexMap::new();
         let mut current_tool_use_id: Option<String> = None;
         let mut usage: Option<TokenUsage> = None;
         // Claude reasoning block for the turn (only populated when thinking is on),
@@ -1263,7 +1264,10 @@ mod tests {
         // recovers the uncached share by subtracting the details back out.
         assert_eq!(usage["usage"]["prompt_tokens"], 1010);
         assert_eq!(usage["usage"]["completion_tokens"], 7);
-        assert_eq!(usage["usage"]["prompt_tokens_details"]["cached_tokens"], 900);
+        assert_eq!(
+            usage["usage"]["prompt_tokens_details"]["cached_tokens"],
+            900
+        );
         assert_eq!(
             usage["usage"]["prompt_tokens_details"]["cache_write_tokens"],
             100

backend/windmill-ai/src/sse.rs

@@ -1,6 +1,7 @@
 use std::collections::HashMap;
 
 use eventsource_stream::Eventsource;
+use indexmap::IndexMap;
 use reqwest::Response;
 use serde::Deserialize;
 use tokio_stream::StreamExt;
@@ -137,7 +138,9 @@ pub struct OpenAISSEParser {
     pub accumulated_content: String,
     /// The thinking streamed before the answer, kept so it can be stored with it.
     pub accumulated_reasoning: String,
-    pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
+    // Insertion-ordered in every parser: tool calls run and are persisted in the order the
+    // stream showed them, and a chat attaches a round's thinking to its first call.
+    pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
     pub events_str: String,
     pub stream_event_processor: Box<dyn StreamEventSink>,
     /// Token usage from final chunk (when stream_options.include_usage is true)
@@ -149,7 +152,7 @@ impl OpenAISSEParser {
         Self {
             accumulated_content: String::new(),
             accumulated_reasoning: String::new(),
-            accumulated_tool_calls: HashMap::new(),
+            accumulated_tool_calls: IndexMap::new(),
             events_str: String::new(),
             stream_event_processor,
             usage: None,
@@ -359,7 +362,7 @@ pub struct AnthropicSSEParser {
     pub accumulated_content: String,
     /// The thinking streamed before the answer, kept so it can be stored with it.
     pub accumulated_reasoning: String,
-    pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
+    pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
     pub events_str: String,
     pub stream_event_processor: Box<dyn StreamEventSink>,
     /// Track content block types by index
@@ -382,7 +385,7 @@ impl AnthropicSSEParser {
         Self {
             accumulated_content: String::new(),
             accumulated_reasoning: String::new(),
-            accumulated_tool_calls: HashMap::new(),
+            accumulated_tool_calls: IndexMap::new(),
             events_str: String::new(),
             stream_event_processor,
             content_blocks: HashMap::new(),
@@ -601,7 +604,7 @@ pub struct GeminiSSEParser {
     pub accumulated_content: String,
     /// The thinking streamed before the answer, kept so it can be stored with it.
     pub accumulated_reasoning: String,
-    pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
+    pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
     pub events_str: String,
     pub stream_event_processor: Box<dyn StreamEventSink>,
     tool_call_index: i64,
@@ -615,7 +618,7 @@ impl GeminiSSEParser {
         Self {
             accumulated_content: String::new(),
             accumulated_reasoning: String::new(),
-            accumulated_tool_calls: HashMap::new(),
+            accumulated_tool_calls: IndexMap::new(),
             events_str: String::new(),
             stream_event_processor,
             tool_call_index: 0,
@@ -833,7 +836,7 @@ pub struct OpenAIResponsesSSEParser {
     pub accumulated_content: String,
     /// The reasoning summary streamed before the answer, kept so it can be stored with it.
     pub accumulated_reasoning: String,
-    pub accumulated_tool_calls: HashMap<String, OpenAIToolCall>,
+    pub accumulated_tool_calls: IndexMap<String, OpenAIToolCall>,
     /// Maps item_id -> (name, call_id) for function calls
     tool_call_metadata: HashMap<String, (String, String)>,
     /// Maps item_id -> accumulated arguments
@@ -855,7 +858,7 @@ impl OpenAIResponsesSSEParser {
         Self {
             accumulated_content: String::new(),
             accumulated_reasoning: String::new(),
-            accumulated_tool_calls: HashMap::new(),
+            accumulated_tool_calls: IndexMap::new(),
             tool_call_metadata: HashMap::new(),
             tool_call_arguments: HashMap::new(),
             events_str: String::new(),

backend/windmill-ai/src/types.rs

@@ -78,17 +78,50 @@ impl Default for OutputType {
 #[serde(tag = "kind", rename_all = "lowercase")]
 pub enum Memory {
     Off,
-    Auto {
-        #[serde(default)]
+    Window {
+        #[serde(default, deserialize_with = "deserialize_null_as_zero")]
         context_length: usize,
-        #[serde(default)]
+    },
+    /// Written before `window`. Its `memory_id` stays a fallback behind the run's memory id.
+    Auto {
+        #[serde(default, deserialize_with = "deserialize_null_as_zero")]
+        context_length: usize,
+        #[serde(default, deserialize_with = "deserialize_blank_as_none")]
         memory_id: Option<Uuid>,
     },
+    /// Written before a step had history inputs of its own, and read on its own where it remains.
     Manual {
         messages: Vec<OpenAIMessage>,
     },
 }
 
+// An editor form can leave `""` in a legacy baked id it never filled; it means no id rather than
+// failing every run of the step.
+fn deserialize_blank_as_none<'de, D: serde::Deserializer<'de>>(
+    deserializer: D,
+) -> Result<Option<Uuid>, D::Error> {
+    match <Option<String> as serde::Deserialize>::deserialize(deserializer)? {
+        Some(id) if !id.trim().is_empty() => Uuid::parse_str(id.trim())
+            .map(Some)
+            .map_err(serde::de::Error::custom),
+        _ => Ok(None),
+    }
+}
+
+// A count the editor's number field was cleared of is stored as `null`, which `default` does not
+// cover; it reads as 0, memory off, rather than failing every run of the step.
+fn deserialize_null_as_zero<'de, D: serde::Deserializer<'de>>(
+    deserializer: D,
+) -> Result<usize, D::Error> {
+    <Option<usize> as serde::Deserialize>::deserialize(deserializer).map(Option::unwrap_or_default)
+}
+
+fn deserialize_present<'de, D: serde::Deserializer<'de>>(
+    deserializer: D,
+) -> Result<Option<serde_json::Value>, D::Error> {
+    <serde_json::Value as serde::Deserialize>::deserialize(deserializer).map(Some)
+}
+
 #[derive(Deserialize)]
 struct AIAgentArgsRaw {
     provider: ProviderWithResource,
@@ -103,6 +136,12 @@ struct AIAgentArgsRaw {
     streaming: Option<bool>,
     max_iterations: Option<usize>,
     memory: Option<Memory>,
+    // A null must stay distinguishable from an absent key: a step whose own memory id evaluates to
+    // nothing runs stateless instead of falling back to the run's memory id.
+    #[serde(default, deserialize_with = "deserialize_present")]
+    memory_id: Option<serde_json::Value>,
+    #[serde(default)]
+    previous_messages: Option<Vec<OpenAIMessage>>,
     enabled_tools: Option<Vec<String>>,
     // Legacy field for backward compatibility
     messages_context_length: Option<usize>,
@@ -124,6 +163,10 @@ pub struct AIAgentArgs {
     pub streaming: Option<bool>,
     pub max_iterations: Option<usize>,
     pub memory: Option<Memory>,
+    /// Memory id set on the step, overriding the run's. Empty when its expression produced none.
+    pub memory_id: Option<String>,
+    /// History supplied by the flow, replayed without reading or writing memory.
+    pub previous_messages: Option<Vec<OpenAIMessage>>,
     /// Which of the agent's tools this run may call; `narrow_roster` holds what the names are and
     /// what `None` means.
     pub enabled_tools: Option<Vec<String>>,
@@ -139,12 +182,17 @@ impl From<AIAgentArgsRaw> for AIAgentArgs {
         });
 
         // Backward compatibility: if context_length is 0, use off mode
-        let memory = memory.map(|memory| {
-            if let Memory::Auto { context_length: 0, .. } = memory {
+        let memory = memory.map(|memory| match memory {
+            Memory::Auto { context_length: 0, .. } | Memory::Window { context_length: 0 } => {
                 Memory::Off
-            } else {
-                memory
             }
+            memory => memory,
+        });
+
+        let memory_id = raw.memory_id.map(|value| match value {
+            serde_json::Value::Null => String::new(),
+            serde_json::Value::String(s) => s.trim().to_string(),
+            value => value.to_string(),
         });
 
         AIAgentArgs {
@@ -159,6 +207,8 @@ impl From<AIAgentArgsRaw> for AIAgentArgs {
             streaming: raw.streaming,
             max_iterations: raw.max_iterations,
             memory,
+            memory_id,
+            previous_messages: raw.previous_messages,
             enabled_tools: raw.enabled_tools,
             credentials_check: raw.credentials_check.unwrap_or(false),
         }

backend/windmill-api-flow-conversations/src/lib.rs

@@ -1,6 +1,6 @@
 use axum::{
     extract::{Path, Query},
-    routing::{delete, get},
+    routing::{delete, get, post},
     Extension, Json, Router,
 };
 use chrono::{DateTime, Utc};
@@ -15,13 +15,14 @@ use windmill_common::{
     db::{UserDB, DB},
     error::{JsonResult, Result},
     flow_conversations::MessageType,
-    utils::{not_found_if_none, paginate, Pagination},
+    utils::{not_found_if_none, paginate, truncate_with_ellipsis, Pagination},
 };
 
 pub fn workspaced_service() -> Router {
     Router::new()
         .route("/list", get(list_conversations))
         .route("/delete/{conversation_id}", delete(delete_conversation))
+        .route("/update/{conversation_id}", post(update_conversation))
         .route("/{conversation_id}/messages", get(list_messages))
 }
 
@@ -36,11 +37,37 @@ pub struct FlowConversationMessage {
     pub created_seq: i64,
     pub step_name: Option<String>,
     pub success: bool,
+    /// On a tool row, the arguments the model wrote. For a Windmill tool these exclude the
+    /// inputs its step wires in. Null for a web search, whose query the provider does not
+    /// return.
+    pub tool_arguments: Option<String>,
+    /// On a tool row, the text the model got back, or what the call failed with; a web
+    /// search's citations.
+    pub tool_result: Option<String>,
+    /// On an answer, the thinking that produced it; on a tool row, the thinking that led to
+    /// the call. The agent job keeps the turn's thinking as one string.
+    pub reasoning: Option<String>,
+    /// The files a user message carried, as object-storage references
+    /// (`[{input, s3, storage?, filename?}]`).
+    pub attachments: Option<sqlx::types::JsonValue>,
+}
+
+/// Which conversations a listing holds. A test chat was started from the editor's test
+/// panel; a deployed one from the flow itself.
+#[derive(Deserialize, Default, Clone, Copy)]
+#[serde(rename_all = "lowercase")]
+pub enum ConversationKind {
+    Test,
+    /// The default: a deployed flow's chat should not surface someone's trial runs.
+    #[default]
+    Deployed,
+    All,
 }
 
 #[derive(Deserialize)]
 pub struct ListConversationsQuery {
     pub flow_path: Option<String>,
+    pub kind: Option<ConversationKind>,
 }
 
 #[derive(Deserialize)]
@@ -67,6 +94,7 @@ async fn list_conversations(
         "created_at",
         "updated_at",
         "created_by",
+        "is_test",
     ])
     .and_where_eq("workspace_id", "?".bind(&w_id));
 
@@ -74,6 +102,16 @@ async fn list_conversations(
         sqlb.and_where_eq("flow_path", "?".bind(flow_path));
     }
 
+    match query.kind.unwrap_or_default() {
+        ConversationKind::Test => {
+            sqlb.and_where_eq("is_test", "true");
+        }
+        ConversationKind::Deployed => {
+            sqlb.and_where_eq("is_test", "false");
+        }
+        ConversationKind::All => {}
+    }
+
     sqlb.order_by("updated_at", true)
         .limit(per_page as i64)
         .offset(offset as i64);
@@ -101,7 +139,7 @@ async fn delete_conversation(
     // Verify the conversation exists and belongs to the user
     let conversation = sqlx::query_as!(
         FlowConversation,
-        "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by
+        "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test
          FROM flow_conversation
          WHERE id = $1 AND workspace_id = $2",
         conversation_id,
@@ -148,6 +186,50 @@ async fn delete_conversation(
     Ok(format!("Conversation {} deleted", conversation_id))
 }
 
+#[derive(Deserialize)]
+pub struct UpdateConversation {
+    pub title: String,
+}
+
+async fn update_conversation(
+    authed: ApiAuthed,
+    Extension(user_db): Extension<UserDB>,
+    Path((w_id, conversation_id)): Path<(String, Uuid)>,
+    Json(update): Json<UpdateConversation>,
+) -> Result<String> {
+    // Postgres refuses a NUL in a text column, so it must not reach the query as a 500.
+    if update.title.contains('\0') {
+        return Err(windmill_common::error::Error::BadRequest(
+            "title cannot contain a NUL character".to_string(),
+        ));
+    }
+    // The column is VARCHAR(255) and the helper appends an ellipsis to what it cuts, so the
+    // bound it takes is three short of the column's. A longer title would otherwise reach
+    // Postgres as a 22001 and come back a 500.
+    let title = truncate_with_ellipsis(update.title.trim(), 252);
+
+    let mut tx = user_db.clone().begin(&authed).await?;
+
+    // `updated_at` is kept: the list is ordered by it, and a rename must not move the
+    // chat to the top the way a new turn does.
+    let updated = sqlx::query_scalar!(
+        "UPDATE flow_conversation SET title = $1, updated_at = updated_at
+         WHERE id = $2 AND workspace_id = $3
+         RETURNING id",
+        title,
+        conversation_id,
+        &w_id
+    )
+    .fetch_optional(&mut *tx)
+    .await?;
+
+    not_found_if_none(updated, "Conversation", conversation_id.to_string())?;
+
+    tx.commit().await?;
+
+    Ok(format!("Conversation {} updated", conversation_id))
+}
+
 async fn list_messages(
     authed: ApiAuthed,
     Extension(user_db): Extension<UserDB>,
@@ -178,7 +260,7 @@ async fn list_messages(
     let messages = if let Some(after_seq) = query.after_seq {
         sqlx::query_as!(
             FlowConversationMessage,
-            r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success
+            r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
              FROM flow_conversation_message
              WHERE conversation_id = $1
                AND created_seq > $2
@@ -195,9 +277,9 @@ async fn list_messages(
         // Fetch messages for this conversation, oldest first, but reverse the order of the messages for easy rendering on the frontend
         sqlx::query_as!(
             FlowConversationMessage,
-            r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success
+            r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
              FROM (
-                SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success
+                SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
                 FROM flow_conversation_message
                 WHERE conversation_id = $1
                 ORDER BY created_seq DESC

backend/windmill-api-integration-tests/tests/datatable_roles.rs

@@ -391,11 +391,7 @@ async fn concurrent_role_creations_both_survive(db: Pool<Postgres>) -> anyhow::R
         assert_eq!(a.0, 200, "{}", a.1);
         assert_eq!(b.0, 200, "{}", b.1);
 
-        let catalog = windmill_common::datatable_roles::read_role_catalog(
-            &db,
-            windmill_common::datatable_roles::DatatableRoleCluster::Instance,
-        )
-        .await?;
+        let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
         let recorded: Vec<&str> = catalog.values().map(|r| r.name.as_str()).collect();
         for name in &names {
             assert!(
@@ -464,11 +460,7 @@ async fn a_role_delete_that_fails_part_way_leaves_the_role_disabled(
         let body = resp.text().await?;
         assert_eq!(status, 400, "{body}");
 
-        let catalog = windmill_common::datatable_roles::read_role_catalog(
-            &db,
-            windmill_common::datatable_roles::DatatableRoleCluster::Instance,
-        )
-        .await?;
+        let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
         let role = catalog
             .get(&id)
             .expect("a failed delete keeps the entry to retry");

backend/windmill-api-jobs/src/execution.rs

@@ -26,7 +26,9 @@ use windmill_api_auth::{check_scopes, get_scope_tags, ApiAuthed};
 use windmill_common::{
     db::{UserDB, UserDbWithAuthed},
     error::{self, Error},
-    flow_conversations::{add_message_to_conversation_tx, MessageType},
+    flow_conversations::{
+        add_message_to_conversation_tx, message_attachments, MessageExtras, MessageType,
+    },
     get_latest_flow_version_info_for_path,
     jobs::{
         check_tag_available_for_workspace_internal, format_result, script_path_to_payload,
@@ -653,9 +655,11 @@ pub async fn set_flow_memory_id(
 pub async fn process_flow_run_query_params(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
     job_id: Uuid,
+    w_id: &str,
+    flow_path: &str,
     run_query: &RunJobQuery,
 ) -> error::Result<()> {
-    if let Some(memory_id) = run_query.memory_id {
+    if let Some(memory_id) = run_query.memory_key(w_id, flow_path) {
         set_flow_memory_id(tx, job_id, memory_id).await?;
     }
     Ok(())
@@ -669,10 +673,13 @@ pub async fn handle_chat_conversation_messages(
     run_query: &RunJobQuery,
     user_message_raw: Option<&Box<serde_json::value::RawValue>>,
     job_id: Uuid,
+    is_test: bool,
+    // The run's args, for the files the message carried.
+    args: &HashMap<String, Box<serde_json::value::RawValue>>,
 ) -> error::Result<()> {
     // Names the query parameter rather than the field: it is not a flow argument, and
     // supplying it as one is the first thing tried on reading `memory_id is required`.
-    let memory_id = run_query.memory_id.ok_or_else(|| {
+    let memory_id = run_query.memory_key(w_id, flow_path).ok_or_else(|| {
         windmill_common::error::Error::BadRequest(
             "memory_id is required for chat-enabled flows. Pass it as the `memory_id` query \
              parameter, not as a flow argument: it names the conversation the turn belongs to, \
@@ -701,11 +708,12 @@ pub async fn handle_chat_conversation_messages(
         &authed.username,
         &user_message,
         memory_id,
+        is_test,
     )
     .await?;
 
-    // The run this message started. Its args are the only record of what the message
-    // carried besides its text — attachments and every other flow input — and nothing
+    // The run this message started. The row keeps the files the message carried as
+    // references; its args are the only record of every other flow input, and nothing
     // written later points at them: an assistant row holds the AI agent step's job.
     add_message_to_conversation_tx(
         tx,
@@ -715,6 +723,7 @@ pub async fn handle_chat_conversation_messages(
         MessageType::User,
         None,
         true,
+        Some(&MessageExtras { attachments: message_attachments(args), ..Default::default() }),
     )
     .await?;
 
@@ -822,7 +831,7 @@ pub async fn run_flow<'c>(
     .await?;
 
     // Set memory_id if provided (for agent memory)
-    if let Some(memory_id) = run_query.memory_id {
+    if let Some(memory_id) = run_query.memory_key(w_id, flow_path) {
         set_flow_memory_id(&mut tx, uuid, memory_id).await?;
     }
 
@@ -836,6 +845,8 @@ pub async fn run_flow<'c>(
             &run_query,
             args.args.get("user_message"),
             uuid,
+            false,
+            &args.args,
         )
         .await?;
     }

backend/windmill-api-jobs/src/types.rs

@@ -47,13 +47,25 @@ pub struct RunJobQuery {
     pub cache_ignore_s3_path: Option<bool>,
     pub skip_preprocessor: Option<bool>,
     pub poll_delay_ms: Option<u64>,
-    pub memory_id: Option<Uuid>,
+    /// Any string; see [`RunJobQuery::memory_key`].
+    pub memory_id: Option<String>,
     pub trigger_external_id: Option<String>,
     pub service_name: Option<String>,
     pub suspended_mode: Option<bool>,
 }
 
 impl RunJobQuery {
+    /// The memory id as stored in `flow_status.memory_id`: a uuid is kept, any other string hashed
+    /// within the workspace and the flow being run.
+    pub fn memory_key(&self, workspace_id: &str, flow_path: &str) -> Option<Uuid> {
+        self.memory_id
+            .as_deref()
+            .filter(|memory_id| !memory_id.trim().is_empty())
+            .map(|memory_id| {
+                windmill_common::flow_conversations::memory_key(workspace_id, flow_path, memory_id)
+            })
+    }
+
     pub async fn get_scheduled_for(
         &self,
         db: &DB,

backend/windmill-api-settings/src/lib.rs

@@ -42,7 +42,6 @@ use axum::{
     routing::{get, post},
     Json, Router,
 };
-use serde_json::json;
 
 use serde::{Deserialize, Serialize};
 use windmill_ai::ai_cache::bump_instance_ai_config_revision;
@@ -176,14 +175,6 @@ pub fn global_service() -> Router {
             "/external_instance_pg/setup",
             post(setup_external_instance_pg),
         )
-        .route(
-            "/external_instance_pg/databases",
-            get(list_external_instance_pg_databases),
-        )
-        .route(
-            "/external_instance_pg/databases/{name}",
-            post(create_external_instance_pg_database).delete(drop_external_instance_pg_database),
-        )
         .route(
             "/setup_custom_instance_pg_database/{name}",
             post(setup_custom_instance_pg_database),
@@ -893,14 +884,6 @@ pub async fn set_global_setting_internal(
         )));
     }
 
-    if key == EXTERNAL_INSTANCE_PG_SETTING {
-        return windmill_common::external_instance_pg::write_external_instance_pg_setting(
-            db,
-            Some(&value),
-        )
-        .await;
-    }
-
     run_setting_pre_write_hook(db, &key, &value).await?;
 
     match value {
@@ -962,6 +945,13 @@ async fn run_setting_pre_write_hook(
     value: &serde_json::Value,
 ) -> error::Result<()> {
     match key {
+        EXTERNAL_INSTANCE_PG_SETTING => {
+            windmill_common::external_instance_pg::check_external_instance_pg_write(
+                db,
+                Some(value),
+            )
+            .await?;
+        }
         // The instance AI config is written as an untyped blob through this generic
         // endpoint, so it never passes the typed check the workspace handler applies.
         // Rates that reach a cost total unbounded would make it negative or infinite.
@@ -1282,7 +1272,7 @@ async fn set_instance_config(
     let desired_map = desired.global_settings.to_settings_map();
     if !desired_map.is_empty() {
         let current_map = current.global_settings.to_settings_map();
-        let mut settings_diff =
+        let settings_diff =
             instance_config::diff_global_settings(&current_map, &desired_map, ApplyMode::Merge);
         let ai_config_changed = settings_diff
             .upserts
@@ -1311,15 +1301,16 @@ async fn set_instance_config(
         }
 
         for (key, value) in &settings_diff.upserts {
-            if key != EXTERNAL_INSTANCE_PG_SETTING {
-                run_setting_pre_write_hook(&db, key, value).await?;
-            }
+            run_setting_pre_write_hook(&db, key, value).await?;
+        }
+        if settings_diff
+            .deletes
+            .iter()
+            .any(|k| k == EXTERNAL_INSTANCE_PG_SETTING)
+        {
+            windmill_common::external_instance_pg::check_external_instance_pg_write(&db, None)
+                .await?;
         }
-        windmill_common::external_instance_pg::write_external_instance_pg_from_diff(
-            &db,
-            &mut settings_diff,
-        )
-        .await?;
 
         instance_config::apply_settings_diff(&db, &settings_diff)
             .await
@@ -1720,7 +1711,40 @@ async fn list_custom_instance_pg_databases(
             ))
         })?;
 
-    if windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
+    if !windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
+        // A fork copy's name gives away the workspace it was reserved for, so every pending fork on
+        // the instance would be listed. Kept for members of that workspace, and wherever the
+        // caller's workspaces use it, e.g. the fork it was finalized into.
+        let reserved_visible: BTreeSet<String> = sqlx::query_scalar(
+            r#"SELECT e.k FROM global_settings gs
+               CROSS JOIN LATERAL jsonb_each(gs.value->'databases') AS e(k, v)
+               WHERE gs.name = 'custom_instance_pg_databases' AND e.v->>'workspace_id' IS NOT NULL
+                 AND (EXISTS (SELECT 1 FROM usr WHERE usr.email = $1
+                                AND usr.workspace_id = e.v->>'workspace_id')
+                   OR EXISTS (SELECT 1 FROM usr JOIN workspace_settings ws
+                                ON ws.workspace_id = usr.workspace_id
+                              CROSS JOIN LATERAL jsonb_each(
+                                  CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
+                                      THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
+                              WHERE usr.email = $1
+                                AND dt.value->'database'->>'resource_type' = 'instance'
+                                AND dt.value->'database'->>'resource_path' = e.k))"#,
+        )
+        .bind(&authed.email)
+        .fetch_all(&db)
+        .await?
+        .into_iter()
+        .collect();
+        result.retain(|dbname, entry| {
+            entry.workspace_id.is_none() || reserved_visible.contains(dbname)
+        });
+        // Which workspace reserved a copy is still only for superadmins.
+        for entry in result.values_mut() {
+            entry.workspace_id = None;
+        }
+        return Ok(Json(result));
+    }
+    {
         // Enrich each database with the list of workspaces referencing it through
         // either a ducklake catalog or a datatable database whose resource_type is
         // 'instance'. Not stored in DB to avoid drift.
@@ -1824,87 +1848,6 @@ async fn setup_external_instance_pg(
     Ok(Json(report))
 }
 
-#[derive(Serialize)]
-struct ExternalInstancePgDatabase {
-    #[serde(flatten)]
-    status: windmill_common::instance_config::CustomInstanceDb,
-    used_by_workspaces: Vec<String>,
-}
-
-async fn list_external_instance_pg_databases(
-    authed: ApiAuthed,
-    Extension(db): Extension<DB>,
-) -> JsonResult<std::collections::BTreeMap<String, ExternalInstancePgDatabase>> {
-    require_super_admin(&db, &authed).await?;
-    let databases = windmill_common::external_instance_pg::external_instance_databases(&db).await?;
-    let mut usages =
-        windmill_common::external_instance_pg::external_instance_database_usages(&db).await?;
-    Ok(Json(
-        databases
-            .into_iter()
-            .map(|(name, status)| {
-                let used_by_workspaces = usages.remove(&name).unwrap_or_default();
-                (
-                    name,
-                    ExternalInstancePgDatabase {
-                        status,
-                        used_by_workspaces: used_by_workspaces.into_iter().collect(),
-                    },
-                )
-            })
-            .collect(),
-    ))
-}
-
-async fn create_external_instance_pg_database(
-    authed: ApiAuthed,
-    Extension(db): Extension<DB>,
-    Path(dbname): Path<String>,
-    Json(body): Json<SetupCustomInstanceDbBody>,
-) -> JsonResult<()> {
-    require_super_admin(&db, &authed).await?;
-    let tag = body.tag.as_deref().unwrap_or("datatable");
-    windmill_common::external_instance_pg::create_external_instance_database_unchecked(
-        &db, &dbname, tag, None,
-    )
-    .await?;
-    windmill_audit::audit_oss::audit_log(
-        &db,
-        &authed,
-        "settings.create_external_instance_pg_database",
-        windmill_audit::ActionKind::Create,
-        "global",
-        Some(&authed.email),
-        Some([("dbname", dbname.as_str()), ("tag", tag)].into()),
-    )
-    .await?;
-    Ok(Json(()))
-}
-
-async fn drop_external_instance_pg_database(
-    authed: ApiAuthed,
-    Extension(db): Extension<DB>,
-    Path(dbname): Path<String>,
-) -> JsonResult<()> {
-    require_super_admin(&db, &authed).await?;
-    // A data table naming a dropped database fails on every job, far from the drop that caused it.
-    windmill_common::external_instance_pg::drop_external_instance_database_unchecked(
-        &db, &dbname, None,
-    )
-    .await?;
-    windmill_audit::audit_oss::audit_log(
-        &db,
-        &authed,
-        "settings.drop_external_instance_pg_database",
-        windmill_audit::ActionKind::Delete,
-        "global",
-        Some(&authed.email),
-        Some([("dbname", dbname.as_str())].into()),
-    )
-    .await?;
-    Ok(Json(()))
-}
-
 #[derive(Deserialize)]
 struct SetupCustomInstanceDbBody {
     tag: Option<String>,
@@ -1919,16 +1862,15 @@ async fn setup_custom_instance_pg_database(
     // Before anything is recorded: the status written below replaces the registry entry, and with it
     // the workspace a fork copy is reserved for.
     require_super_admin(&db, &authed).await?;
-    // A re-run keeps the fork reservation: without it, the workspace the copy was made for could no
-    // longer import into it or finish its fork.
-    let workspace_id = sqlx::query_scalar::<_, Option<String>>(
-        "SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
-         WHERE name = 'custom_instance_pg_databases'",
+    // Fork cleanup checks and drops the database and its entry under this lock. Held from before
+    // the setup creates the database to after its entry is written, neither lands on the other's
+    // half-done state: a dropped database with its entry written back, or the reverse.
+    let mut tx = db.begin().await?;
+    windmill_common::datatable_roles::lock_instance_databases_governance(
+        &mut tx,
+        [dbname.trim()],
     )
-    .bind(&dbname)
-    .fetch_optional(&db)
-    .await?
-    .flatten();
+    .await?;
     let mut logs = CustomInstanceDbLogs::default();
     let result = setup_custom_instance_pg_database_inner(authed, &db, &dbname, &mut logs).await;
     let success = result.is_ok();
@@ -1939,14 +1881,26 @@ async fn setup_custom_instance_pg_database(
         error,
         tag: body.tag,
         used_by_workspaces: vec![],
-        workspace_id,
+        workspace_id: None,
     };
     let status_json = serde_json::to_value(&status).map_err(to_anyhow)?;
-    // Save that the database was setup successfully
-    sqlx::query!(
-        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
-        json!({ dbname: status_json })
-    ).execute(&db).await?;
+    // The fork reservation is carried over inside the write, from whatever the row holds then: a
+    // rename migrating it while the setup above ran would otherwise be overwritten with the value
+    // this request started from, stranding the copy under the archived workspace.
+    let saved = sqlx::query_scalar::<_, serde_json::Value>(
+        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}',
+               COALESCE(value->'databases', '{}'::jsonb)
+                   || jsonb_build_object($1::text, $2::jsonb || jsonb_build_object(
+                          'workspace_id', value->'databases'->$1::text->'workspace_id')))
+           WHERE name = 'custom_instance_pg_databases'
+           RETURNING value->'databases'->$1::text"#,
+    )
+    .bind(&dbname)
+    .bind(&status_json)
+    .fetch_one(&mut *tx)
+    .await?;
+    tx.commit().await?;
+    let status: CustomInstanceDb = serde_json::from_value(saved).map_err(to_anyhow)?;
 
     Ok(Json(status))
 }

backend/windmill-api-workspaces/src/datatable_acl.rs

@@ -6,7 +6,7 @@
  * LICENSE-AGPL for a copy of the license.
  */
 
-//! Ownership and grants on the objects of a data table on a cluster Windmill manages.
+//! Ownership and grants on the objects of an instance data table.
 //!
 //! [`datatable_permissions`](crate::datatable_permissions) decides who may connect as which role;
 //! this decides what each role may then touch. Every change is a real `GRANT`, `REVOKE`,
@@ -33,7 +33,7 @@ use windmill_audit::audit_oss::audit_log;
 use windmill_audit::ActionKind;
 use windmill_common::datatable_roles::{
     lock_role_catalog, quote_ident, read_role_catalog, read_role_catalog_tx, DatatableRoleCatalog,
-    DatatableRoleCluster, ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
+    ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
 };
 use windmill_common::error::{pg_error_message, Error, JsonResult, Result};
 use windmill_common::workspaces::{resolve_governing_datatable, DataTable, GoverningDatatable};
@@ -295,22 +295,22 @@ fn role_names(catalog: &DatatableRoleCatalog) -> Vec<String> {
     names
 }
 
-/// The cluster whose roles the data table's grants name.
-fn ensure_managed(governing: &GoverningDatatable) -> Result<DatatableRoleCluster> {
-    governing.role_cluster().ok_or_else(|| {
-        Error::BadRequest(format!(
-            "Data table '{}' is backed by a Postgres resource, so its access is managed on that \
-             server directly. Only a data table on a database Windmill manages has data table \
-             roles to grant to.",
-            governing.name
-        ))
-    })
+fn ensure_instance(governing: &GoverningDatatable) -> Result<()> {
+    if governing.is_instance() {
+        return Ok(());
+    }
+    Err(Error::BadRequest(format!(
+        "Data table '{}' is backed by a Postgres resource, so its access is managed on that \
+         server directly. Only a data table on the Windmill instance's own database has data \
+         table roles to grant to.",
+        governing.name
+    )))
 }
 
 /// The data table's `admin` connection, and the notices Postgres sends on it.
 ///
-/// Authorization: connects as `custom_instance_user` with the cluster's stored credentials and
-/// checks nothing. Callers MUST have authorized the request first — a request about to be refused must
+/// Authorization: connects as `custom_instance_user` with the instance's own credentials and checks
+/// nothing. Callers MUST have authorized the request first — a request about to be refused must
 /// not get as far as this connection.
 async fn connect_as_admin_unchecked(
     db: &DB,
@@ -320,33 +320,21 @@ async fn connect_as_admin_unchecked(
     mpsc::UnboundedReceiver<DbError>,
     String,
 )> {
-    let cluster = ensure_managed(governing)?;
+    ensure_instance(governing)?;
     // Built from the authorized entry, never by resolving the settings again: a save in between
     // could point the entry at a resource on another server and back, and this connection would
     // then alter a database the later checks of the entry never see.
-    let dbname = governing
+    let mut pg = PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
+    pg.dbname = governing
         .datatable
         .database
         .as_ref()
         .expect("a governing entry owns a database")
         .resource_path
         .clone();
-    let pg = match cluster {
-        DatatableRoleCluster::Instance => {
-            let mut pg =
-                PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
-            pg.dbname = dbname.clone();
-            pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
-            pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
-            pg
-        }
-        DatatableRoleCluster::ExternalInstance => {
-            windmill_common::external_instance_pg::external_instance_connection_unchecked(
-                db, &dbname, false,
-            )
-            .await?
-        }
-    };
+    pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
+    pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
+    let dbname = pg.dbname.clone();
     let (client, mut connection) = pg.connect(Some(db)).await?;
     // Unbounded: the driver must never wait on the receiver, which only drains once the statement
     // the driver is carrying has completed.
@@ -1032,12 +1020,12 @@ async fn get_datatable_acl(
     let target: AclTarget = query.try_into()?;
     let governing = resolve_governing_datatable(&db, &w_id, &datatable_name).await?;
     ensure_reaches_governing_datatable(&db, &w_id, &datatable_name, &governing, &authed).await?;
-    let cluster = ensure_managed(&governing)?;
+    ensure_instance(&governing)?;
     let editable = ensure_governs_datatable(&db, &authed, &w_id, &governing)
         .await
         .is_ok();
     let roles = if editable {
-        role_names(&read_role_catalog(&db, cluster).await?)
+        role_names(&read_role_catalog(&db).await?)
     } else {
         vec![]
     };
@@ -1322,7 +1310,7 @@ async fn authorize_acl_change(
 ) -> Result<GoverningDatatable> {
     let governing = resolve_governing_datatable(db, w_id, datatable_name).await?;
     ensure_governs_datatable(db, authed, w_id, &governing).await?;
-    ensure_managed(&governing)?;
+    ensure_instance(&governing)?;
     Ok(governing)
 }
 
@@ -1332,14 +1320,8 @@ static APPLY_SLOT: tokio::sync::Semaphore = tokio::sync::Semaphore::const_new(1)
 /// provisioned before data table roles gave `custom_instance_user` none. Adds that option to its
 /// database and `public` privileges, and nothing else: default privileges are left alone, since a
 /// schema's change of owner is planned against them. Best-effort, as a grant it fails to enable is
-/// refused when it runs. An external instance database was created with the options, so one
-/// missing there is someone's deliberate revoke and is left alone.
-async fn ensure_grant_options(
-    client: &tokio_postgres::Client,
-    db: &DB,
-    cluster: DatatableRoleCluster,
-    dbname: &str,
-) {
+/// refused when it runs.
+async fn ensure_grant_options(client: &tokio_postgres::Client, db: &DB, dbname: &str) {
     let held = client
         .query_one(
             "SELECT has_database_privilege(current_database(), 'CONNECT WITH GRANT OPTION')
@@ -1351,7 +1333,7 @@ async fn ensure_grant_options(
         )
         .await
         .is_ok_and(|row| row.get::<_, bool>(0));
-    if held || cluster != DatatableRoleCluster::Instance {
+    if held {
         return;
     }
     if let Err(e) = grant_options_as_server(db, dbname).await {
@@ -1577,8 +1559,7 @@ async fn plan_datatable_acl(
 ) -> JsonResult<AclPlan> {
     crate::datatable_acl_oss::ensure_datatable_acl_available()?;
     let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
-    let cluster = ensure_managed(&governing)?;
-    let catalog = read_role_catalog(&db, cluster).await?;
+    let catalog = read_role_catalog(&db).await?;
     let (client, _notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
     Ok(Json(
         build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?,
@@ -1602,7 +1583,6 @@ async fn apply_datatable_acl(
     // connection could wait forever on a pool that concurrent applies, queued on the same locks,
     // have exhausted.
     let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
-    let cluster = ensure_managed(&governing)?;
     // Applies queue on an instance-wide lock while each holds a direct connection to the instance's
     // Postgres; unbounded, the queue alone could exhaust its connection limit. One at a time per
     // server, and the ones waiting hold no connection at all.
@@ -1611,7 +1591,7 @@ async fn apply_datatable_acl(
         .await
         .map_err(|e| Error::internal_err(format!("ACL apply slot closed: {e}")))?;
     let (mut client, mut notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
-    ensure_grant_options(&client, &db, cluster, &dbname).await;
+    ensure_grant_options(&client, &db, &dbname).await;
 
     // Held until the change is committed: a role renamed or dropped meanwhile would change what
     // the plan names, and a settings save could move the entry onto another database. Taken in the
@@ -1626,7 +1606,7 @@ async fn apply_datatable_acl(
     .fetch_optional(&mut *tx)
     .await?
     .flatten();
-    let catalog = read_role_catalog_tx(&mut tx, cluster).await?;
+    let catalog = read_role_catalog_tx(&mut tx).await?;
 
     let plan = build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?;
     if !entry_unchanged(&governing, entry_now) || &plan.statements != confirmed {

backend/windmill-api-workspaces/src/datatable_migrations.rs

@@ -11,7 +11,7 @@
 //! to keep that file focused on core workspace configuration.
 
 use crate::workspaces::{
-    managed_datatable_kind, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
+    is_instance_datatable, pg_dump_database, strip_unreplayable_dump_lines, ItemComparison,
     PgDumpOptions,
 };
 
@@ -1556,9 +1556,7 @@ async fn generate_initial_datatable_migration(
     // without what a replay elsewhere cannot run: the replaying user owns none of this
     // database's objects, and the grants Windmill plants in an instance database (`ALTER
     // DEFAULT PRIVILEGES FOR ROLE ...`) fail even replaying onto the same server.
-    let no_acl = managed_datatable_kind(&db, &w_id, &datatable_name)
-        .await?
-        .is_some();
+    let no_acl = is_instance_datatable(&db, &w_id, &datatable_name).await?;
     let dump_file = pg_dump_database(
         &pg_db,
         PgDumpOptions {

backend/windmill-api-workspaces/src/workspaces.rs

@@ -2263,8 +2263,7 @@ struct DataTableTables {
     schemas: TableListMap,
     #[serde(skip_serializing_if = "Option::is_none")]
     error: Option<String>,
-    /// On a database Windmill manages, on its own cluster or the external one: the only kinds that
-    /// can be under roles or have their access edited.
+    /// On the instance database: the only kind that can be under roles or have its access edited.
     instance: bool,
     permissioned: bool,
     /// The roles this caller may connect as, by name; empty when not under roles.
@@ -2521,7 +2520,7 @@ async fn list_one_datatable_tables(
     };
     let result: Result<()> = async {
         let governing = resolve_governing_datatable(db, w_id, &entry.datatable_name).await?;
-        entry.instance = governing.role_cluster().is_some();
+        entry.instance = governing.is_instance();
         let usable =
             crate::datatable_permissions_oss::usable_datatable_roles(db, authed, w_id, &governing)
                 .await?;
@@ -3079,32 +3078,23 @@ pub(crate) async fn resolve_pg_source_checked(
         .map_err(|e| Error::internal_err(format!("Failed to parse database credentials: {}", e)))
 }
 
-/// The kind of the database backing the data table `name` when Windmill manages it (on its own
-/// cluster or the external one), `None` when it is a user resource.
-pub(crate) async fn managed_datatable_kind(
-    db: &DB,
-    w_id: &str,
-    name: &str,
-) -> Result<Option<DataTableCatalogResourceType>> {
+/// Whether the data table `name` is backed by the Windmill instance's own PostgreSQL
+/// rather than a user resource.
+pub(crate) async fn is_instance_datatable(db: &DB, w_id: &str, name: &str) -> Result<bool> {
     // Resolved rather than read: a pointer entry owns no database of its own, so only the entry it
-    // lands on can answer. A name that resolves to nothing keeps the historical `None`.
+    // lands on can answer. A name that resolves to nothing keeps the historical `false`.
     Ok(resolve_governing_datatable(db, w_id, name)
         .await
         .ok()
         .and_then(|g| g.datatable.database)
-        .map(|d| d.resource_type)
-        .filter(|kind| kind.is_windmill_managed()))
+        .is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance))
 }
 
 /// Same, for the `datatable://<name>` / `$res:<path>` form the import endpoints take.
-async fn managed_datatable_source_kind(
-    db: &DB,
-    w_id: &str,
-    source: &str,
-) -> Result<Option<DataTableCatalogResourceType>> {
+async fn is_instance_datatable_source(db: &DB, w_id: &str, source: &str) -> Result<bool> {
     match source.strip_prefix("datatable://") {
-        Some(name) => managed_datatable_kind(db, w_id, name).await,
-        None => Ok(None),
+        Some(name) => is_instance_datatable(db, w_id, name).await,
+        None => Ok(false),
     }
 }
 
@@ -3203,7 +3193,10 @@ pub(crate) async fn pg_dump_database(
     if let Some(ref password) = pg_db.password {
         cmd.env("PGPASSWORD", password);
     }
-    let _root_cert = apply_pg_tls_env(&mut cmd, pg_db)?;
+
+    if let Some(ref sslmode) = pg_db.sslmode {
+        cmd.env("PGSSLMODE", sslmode);
+    }
 
     let output = cmd
         .output()
@@ -3321,7 +3314,7 @@ async fn comment_out_unsupported_settings(
 
 /// A psql invocation against `pg_db`, carrying the connection settings the CLI reads
 /// from the environment.
-fn psql_command(pg_db: &PgDatabase) -> Result<(tokio::process::Command, Option<DumpFile>)> {
+fn psql_command(pg_db: &PgDatabase) -> tokio::process::Command {
     let mut cmd = tokio::process::Command::new("psql");
     cmd.arg("--host")
         .arg(&pg_db.host)
@@ -3338,88 +3331,10 @@ fn psql_command(pg_db: &PgDatabase) -> Result<(tokio::process::Command, Option<D
     if let Some(ref password) = pg_db.password {
         cmd.env("PGPASSWORD", password);
     }
-    let root_cert = apply_pg_tls_env(&mut cmd, pg_db)?;
-    Ok((cmd, root_cert))
-}
-
-/// Give libpq the TLS settings `PgDatabase::connect` applies. The returned file holds the root
-/// certificate `PGSSLROOTCERT` names, so it must outlive the command.
-fn apply_pg_tls_env(
-    cmd: &mut tokio::process::Command,
-    pg_db: &PgDatabase,
-) -> Result<Option<DumpFile>> {
     if let Some(ref sslmode) = pg_db.sslmode {
         cmd.env("PGSSLMODE", sslmode);
     }
-    if let Some(pem) = pg_db
-        .root_certificate_pem
-        .as_deref()
-        .filter(|p| !p.is_empty())
-    {
-        let file = DumpFile::new()?;
-        std::fs::write(&file.path, pem)
-            .map_err(|e| Error::internal_err(format!("Failed to write root certificate: {e}")))?;
-        cmd.env("PGSSLROOTCERT", &file.path);
-        return Ok(Some(file));
-    }
-    // Only a connection that asked to be verified against the system trust store. Without a file,
-    // libpq's own default would look for `~/.postgresql/root.crt` and refuse a verify-* mode. libpq
-    // takes the special `system` value with verify-full only, so verify-ca needs the bundle itself.
-    if pg_db.accept_invalid_certs == Some(false) {
-        match pg_db.sslmode.as_deref() {
-            Some("verify-full") => {
-                cmd.env("PGSSLROOTCERT", "system");
-            }
-            Some("verify-ca") => {
-                if let Some(bundle) = windmill_common::system_ca_bundle() {
-                    cmd.env("PGSSLROOTCERT", bundle);
-                }
-            }
-            _ => {}
-        }
-    }
-    Ok(None)
-}
-
-
-#[cfg(test)]
-mod pg_tls_env_tests {
-    use super::apply_pg_tls_env;
-    use windmill_common::PgDatabase;
-
-    fn root_cert_env(sslmode: &str) -> Option<std::ffi::OsString> {
-        let pg_db = PgDatabase {
-            host: "db".to_string(),
-            user: None,
-            password: None,
-            port: None,
-            sslmode: Some(sslmode.to_string()),
-            dbname: "d".to_string(),
-            root_certificate_pem: None,
-            accept_invalid_certs: Some(false),
-            use_iam_auth: None,
-            region: None,
-        };
-        let mut cmd = tokio::process::Command::new("psql");
-        apply_pg_tls_env(&mut cmd, &pg_db).unwrap();
-        cmd.as_std()
-            .get_envs()
-            .find(|(k, _)| *k == "PGSSLROOTCERT")
-            .and_then(|(_, v)| v.map(|v| v.to_os_string()))
-    }
-
-    #[test]
-    fn system_roots_only_through_verify_full() {
-        assert_eq!(
-            root_cert_env("verify-full").as_deref(),
-            Some("system".as_ref())
-        );
-        // libpq refuses `sslrootcert=system` with verify-ca, which would fail every dump and restore.
-        assert_ne!(
-            root_cert_env("verify-ca").as_deref(),
-            Some("system".as_ref())
-        );
-    }
+    cmd
 }
 
 /// GUC names the server backing `pg_db` knows about.
@@ -3429,8 +3344,7 @@ mod pg_tls_env_tests {
 /// and an unset mode, where `PgDatabase::connect` would hand a TLS-only server a
 /// plaintext socket and fail before the import ever starts.
 async fn server_setting_names(pg_db: &PgDatabase) -> Result<HashSet<String>> {
-    let (mut cmd, _root_cert) = psql_command(pg_db)?;
-    let output = cmd
+    let output = psql_command(pg_db)
         .arg("--tuples-only")
         .arg("--no-align")
         .arg("--command")
@@ -3464,8 +3378,7 @@ async fn pg_import_dump(target_db: &PgDatabase, dump_file: &DumpFile) -> Result<
     let supported_settings = server_setting_names(target_db).await?;
     comment_out_unsupported_settings(dump_file, &supported_settings).await?;
 
-    let (mut cmd, _root_cert) = psql_command(target_db)?;
-    let output = cmd
+    let output = psql_command(target_db)
         .arg("--set")
         .arg("ON_ERROR_STOP=1")
         .arg("--single-transaction")
@@ -3523,16 +3436,20 @@ async fn create_pg_database(
         }
     }
 
-    let source_kind = managed_datatable_source_kind(&db, &w_id, &req.source).await?;
-    if source_kind == Some(DataTableCatalogResourceType::ExternalInstance) {
-        windmill_common::external_instance_pg::create_external_instance_database_unchecked(
-            &db,
-            &req.target_dbname,
-            "datatable",
-            Some(&w_id),
-        )
-        .await?;
-    } else if source_kind == Some(DataTableCatalogResourceType::Instance) {
+    if is_instance_datatable_source(&db, &w_id, &req.source).await? {
+        // Held until the copy is registered, as a rename migrates reservations to the new id under
+        // it once the old one is archived: a copy registered after that would be reserved for an
+        // id nothing answers on.
+        let mut tx = db.begin().await?;
+        windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
+        let live = sqlx::query_scalar::<_, bool>("SELECT NOT deleted FROM workspace WHERE id = $1")
+            .bind(&w_id)
+            .fetch_optional(&mut *tx)
+            .await?
+            .unwrap_or(false);
+        if !live {
+            return Err(Error::BadRequest(format!("Workspace '{w_id}' is archived")));
+        }
         windmill_common::create_custom_instance_database(
             &db,
             &req.target_dbname,
@@ -3540,6 +3457,7 @@ async fn create_pg_database(
             Some(&w_id),
         )
         .await?;
+        tx.commit().await?;
     } else {
         let source_pg =
             resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
@@ -3640,12 +3558,12 @@ async fn ensure_datatable_is_clonable(
     }
     // The copy has to name a database of its own. A resource-backed entry reached through a
     // pointer names one this workspace does not own, so there is nothing here to repoint.
-    let is_managed = governing
+    let is_instance = governing
         .datatable
         .database
         .as_ref()
-        .is_some_and(|d| d.resource_type.is_windmill_managed());
-    if governing.workspace_id != w_id && !is_managed {
+        .is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance);
+    if governing.workspace_id != w_id && !is_instance {
         return Err(Error::BadRequest(format!(
             "Data table '{name}' points at a resource-backed data table in another workspace \
              and cannot be copied; fork it from the workspace that owns it."
@@ -3681,6 +3599,7 @@ async fn import_pg_database(
     }
 
     let schema_only = req.fork_behavior == DataTableForkBehavior::SchemaOnly;
+    let mut fork_lock: Option<Transaction<'_, Postgres>> = None;
     let source_pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
     let mut target_pg =
         resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.target).await?;
@@ -3693,14 +3612,20 @@ async fn import_pg_database(
                         .to_string(),
                 ));
             }
-            if let Some(kind) = managed_datatable_source_kind(&db, &w_id, &req.target).await? {
-                windmill_common::ensure_fork_database_available_to(
-                    &db,
-                    kind,
-                    override_dbname,
-                    &w_id,
+            if is_instance_datatable_source(&db, &w_id, &req.target).await? {
+                // Held until the restore is done: fork finalization takes the first, and every
+                // save newly naming a database, in any workspace, the second. Nothing may start
+                // using this database while `psql` is still filling it.
+                let mut tx = db.begin().await?;
+                windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
+                windmill_common::datatable_roles::lock_instance_databases_governance(
+                    &mut tx,
+                    [override_dbname.as_str()],
                 )
                 .await?;
+                windmill_common::ensure_fork_database_available_to(&db, override_dbname, &w_id)
+                    .await?;
+                fork_lock = Some(tx);
             }
         }
         target_pg.dbname = override_dbname.clone();
@@ -3711,12 +3636,8 @@ async fn import_pg_database(
     // what it creates it owns. Grants do, except around an instance data table — Windmill
     // plants `custom_instance_user` grants in one, which nothing else can replay. Elsewhere
     // the ACLs are user intent (`REVOKE ... FROM PUBLIC`) and dropping them widens access.
-    let no_acl = managed_datatable_source_kind(&db, &w_id, &req.target)
-        .await?
-        .is_some()
-        || managed_datatable_source_kind(&db, &w_id, &req.source)
-            .await?
-            .is_some();
+    let no_acl = is_instance_datatable_source(&db, &w_id, &req.target).await?
+        || is_instance_datatable_source(&db, &w_id, &req.source).await?;
 
     let dump_file = pg_dump_database(
         &source_pg,
@@ -3724,6 +3645,9 @@ async fn import_pg_database(
     )
     .await?;
     pg_import_dump(&target_pg, &dump_file).await?;
+    if let Some(tx) = fork_lock {
+        tx.commit().await?;
+    }
 
     Ok(format!(
         "Imported from '{}' into '{}'",
@@ -3815,51 +3739,54 @@ async fn edit_ducklake_config(
     )
     .await?;
 
-    let old_ducklakes = sqlx::query_scalar!(
-        r#"
-            SELECT ws.ducklake->'ducklakes' AS ducklake_name
-            FROM workspace_settings ws
-            WHERE ws.workspace_id = $1
-        "#,
-        &w_id
+    // Under the row lock the save writes with, taken before the database locks below as fork
+    // cleanup takes the two.
+    let old_ducklakes = sqlx::query_scalar::<_, Option<serde_json::Value>>(
+        "SELECT ws.ducklake->'ducklakes' FROM workspace_settings ws
+         WHERE ws.workspace_id = $1 FOR UPDATE",
     )
+    .bind(&w_id)
     .fetch_one(&mut *tx)
     .await?
     .unwrap_or(serde_json::Value::Null);
     let old_ducklakes: HashMap<String, Ducklake> =
         serde_json::from_value(old_ducklakes).unwrap_or_default();
 
-    // Check that non-superadmins are not abusing Instance databases. An unchanged catalog is left
-    // alone either way, so a downgraded instance can still save lakes that already name an
-    // external instance database.
-    for (name, dl) in new_config.settings.ducklakes.iter() {
-        let kind = &dl.catalog.resource_type;
-        if !matches!(
-            kind,
-            DucklakeCatalogResourceType::Instance | DucklakeCatalogResourceType::ExternalInstance
-        ) {
-            continue;
-        }
-        let unchanged = old_ducklakes.get(name).is_some_and(|old| {
-            &old.catalog.resource_type == kind
-                && old.catalog.resource_path == dl.catalog.resource_path
-        });
-        if unchanged {
-            continue;
-        }
-        if *kind == DucklakeCatalogResourceType::ExternalInstance {
-            windmill_common::external_instance_pg::ensure_external_instance_available()?;
-            windmill_common::external_instance_pg::ensure_external_instance_database_registered(
-                &mut tx,
-                &dl.catalog.resource_path,
-            )
-            .await?;
-        }
-        if !is_superadmin {
-            return Err(Error::BadRequest(
-                "Only superadmins can create or modify ducklakes with Instance databases"
-                    .to_string(),
-            ));
+    // Fork cleanup decides nothing uses an instance database under this lock, so a catalog newly
+    // put on one must not commit between its check and its drop.
+    windmill_common::datatable_roles::lock_instance_databases_governance(
+        &mut *tx,
+        new_config
+            .settings
+            .ducklakes
+            .iter()
+            .filter(|(name, dl)| {
+                dl.catalog.resource_type == DucklakeCatalogResourceType::Instance
+                    && old_ducklakes.get(name.as_str()).is_none_or(|old| {
+                        old.catalog.resource_type != DucklakeCatalogResourceType::Instance
+                            || old.catalog.resource_path != dl.catalog.resource_path
+                    })
+            })
+            .map(|(_, dl)| dl.catalog.resource_path.as_str()),
+    )
+    .await?;
+
+    // Check that non-superadmins are not abusing Instance databases
+    if !is_superadmin {
+        for (name, dl) in new_config.settings.ducklakes.iter() {
+            if dl.catalog.resource_type == DucklakeCatalogResourceType::Instance {
+                let old_dl = old_ducklakes.get(name);
+                if old_dl.is_none()
+                    || old_dl.unwrap().catalog.resource_type
+                        != DucklakeCatalogResourceType::Instance
+                    || old_dl.unwrap().catalog.resource_path != dl.catalog.resource_path
+                {
+                    return Err(Error::BadRequest(
+                        "Only superadmins can create or modify ducklakes with Instance databases"
+                            .to_string(),
+                    ));
+                }
+            }
         }
     }
 
@@ -3918,6 +3845,8 @@ async fn edit_datatable_config(
     let is_superadmin = require_super_admin(&db, &authed).await.is_ok();
 
     let mut tx = db.begin().await?;
+    // Ahead of the settings row, as fork cleanup of this workspace takes the two.
+    windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
 
     // Read under the row lock this transaction will write with. `permissions`, `reference` and
     // `forked_from` are carried across from what this read returns, so a permissions save
@@ -4031,7 +3960,6 @@ async fn edit_datatable_config(
                 // so these line up with the `datatable_configured` adoption counts.
                 created_substrates.push(match dt.database.as_ref().map(|d| d.resource_type) {
                     Some(DataTableCatalogResourceType::Instance) => "instance",
-                    Some(DataTableCatalogResourceType::ExternalInstance) => "external_instance",
                     Some(DataTableCatalogResourceType::Postgresql) => "postgresql",
                     None => "reference",
                 });
@@ -4051,20 +3979,18 @@ async fn edit_datatable_config(
             None => None,
         };
         // Carrying the block onto a resource-backed entry would produce a data table the chokepoint
-        // refuses on every job — a save that succeeds and breaks everything afterwards — and onto
-        // the other managed cluster, one whose role ids name nothing in that cluster's catalog.
-        // Refuse it instead: turning roles off first is one step, and it keeps discarding an access
-        // decision something somebody chose rather than a side effect of moving a database.
-        let old_kind = old.and_then(|old| old.database.as_ref()).map(|d| d.resource_type);
+        // refuses on every job — a save that succeeds and breaks everything afterwards. Refuse it
+        // instead: turning roles off first is one step, and it keeps discarding an access decision
+        // something somebody chose rather than a side effect of moving a database.
         if dt.permissions.is_some()
             && dt
                 .database
                 .as_ref()
-                .is_some_and(|d| Some(d.resource_type) != old_kind)
+                .is_some_and(|d| d.resource_type != DataTableCatalogResourceType::Instance)
         {
             return Err(Error::BadRequest(format!(
-                "Data table '{name}' is under roles, which belong to the cluster its database is \
-                 on. Turn its roles off before moving it to another kind of database."
+                "Data table '{name}' is under roles, which only a data table on the instance \
+                 database can be. Turn its roles off before moving it to a PostgreSQL resource."
             )));
         }
         // A pointer names no database of its own, so the form's empty `database` is correct there.
@@ -4089,39 +4015,26 @@ async fn edit_datatable_config(
     // Check that non-superadmins are not abusing Instance databases, which reach a database this
     // workspace does not own. Pointing an entry at another workspace's data table is not checked
     // here because it cannot be requested at all: `reference` is overwritten from the stored entry
-    // above, for every caller. An unchanged entry is left alone either way, so a downgraded
-    // instance can still save settings that already name an external instance database.
-    for (name, dt) in new_config.settings.datatables.iter() {
-        let Some(database) = dt
-            .database
-            .as_ref()
-            .filter(|d| d.resource_type.is_windmill_managed())
-        else {
-            continue;
-        };
-        let unchanged = old_datatables
-            .get(name)
-            .and_then(|o| o.database.as_ref())
-            .is_some_and(|o| {
-                o.resource_type == database.resource_type
-                    && o.resource_path == database.resource_path
-            });
-        if unchanged {
-            continue;
-        }
-        if database.resource_type == DataTableCatalogResourceType::ExternalInstance {
-            windmill_common::external_instance_pg::ensure_external_instance_available()?;
-            windmill_common::external_instance_pg::ensure_external_instance_database_registered(
-                &mut tx,
-                &database.resource_path,
-            )
-            .await?;
-        }
-        if !is_superadmin {
-            return Err(Error::BadRequest(
-                "Only superadmins can create or modify data tables with Instance databases"
-                    .to_string(),
-            ));
+    // above, for every caller.
+    if !is_superadmin {
+        for (name, dt) in new_config.settings.datatables.iter() {
+            let old_dt = old_datatables.get(name);
+            if dt
+                .database
+                .as_ref()
+                .is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
+            {
+                let unchanged = old_dt.and_then(|o| o.database.as_ref()).is_some_and(|o| {
+                    o.resource_type == DataTableCatalogResourceType::Instance
+                        && Some(&o.resource_path) == dt.database.as_ref().map(|d| &d.resource_path)
+                });
+                if !unchanged {
+                    return Err(Error::BadRequest(
+                        "Only superadmins can create or modify data tables with Instance databases"
+                            .to_string(),
+                    ));
+                }
+            }
         }
     }
 
@@ -4141,7 +4054,7 @@ async fn edit_datatable_config(
     // entry through a declared rename alone, and a settings sync never declares one, so an entry
     // without roles that newly points at such a database — a name added, or an existing one
     // repointed — would answer everyone there as `admin`. That holds whichever workspace governs it.
-    let newly_pointed: Vec<(&String, DataTableCatalogResourceType, &str)> = new_config
+    let newly_pointed: Vec<(&String, &str)> = new_config
         .settings
         .datatables
         .iter()
@@ -4150,7 +4063,7 @@ async fn edit_datatable_config(
             let db = dt
                 .database
                 .as_ref()
-                .filter(|d| d.resource_type.is_windmill_managed())?;
+                .filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
             let lookup = rename_src
                 .get(name.as_str())
                 .copied()
@@ -4162,44 +4075,66 @@ async fn edit_datatable_config(
                     old_db.resource_type != db.resource_type
                         || old_db.resource_path != db.resource_path
                 });
-            repointed.then_some((name, db.resource_type, db.resource_path.as_str()))
+            repointed.then_some((name, db.resource_path.as_str()))
         })
         .collect();
     // Another workspace turning roles on for the same database holds only its own settings row, so
-    // without this the scan below could read past its uncommitted write.
+    // without this the scan below could read past its uncommitted write. Every managed database
+    // this save newly names is locked, not just the ones the scan is about: fork cleanup takes the
+    // same lock to decide nothing uses the database it is dropping.
+    let newly_named: std::collections::BTreeSet<&str> = new_config
+        .settings
+        .datatables
+        .iter()
+        .filter_map(|(name, dt)| {
+            let db = dt
+                .database
+                .as_ref()
+                .filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
+            let lookup = rename_src
+                .get(name.as_str())
+                .copied()
+                .unwrap_or(name.as_str());
+            old_datatables
+                .get(lookup)
+                .and_then(|old| old.database.as_ref())
+                .is_none_or(|old_db| {
+                    old_db.resource_type != db.resource_type
+                        || old_db.resource_path != db.resource_path
+                })
+                .then_some(db.resource_path.as_str())
+        })
+        .collect();
     windmill_common::datatable_roles::lock_instance_databases_governance(
         &mut *tx,
-        newly_pointed.iter().map(|(_, _, dbname)| *dbname),
+        newly_pointed
+            .iter()
+            .map(|(_, dbname)| *dbname)
+            .chain(newly_named.iter().copied()),
     )
     .await?;
-    let governed_elsewhere: Vec<(String, String)> = if newly_pointed.is_empty() {
+    let governed_elsewhere: Vec<String> = if newly_pointed.is_empty() {
         vec![]
     } else {
-        sqlx::query_as(
-            "SELECT DISTINCT dt.value->'database'->>'resource_type',
-                    dt.value->'database'->>'resource_path'
-             FROM workspace_settings ws
+        sqlx::query_scalar(
+            "SELECT DISTINCT dt.value->'database'->>'resource_path' FROM workspace_settings ws
              CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
              WHERE ws.workspace_id <> $1 AND dt.value ? 'permissions'
-               AND dt.value->'database'->>'resource_type' IN ('instance', 'external_instance')",
+               AND dt.value->'database'->>'resource_type' = 'instance'",
         )
         .bind(&w_id)
         .fetch_all(&mut *tx)
         .await?
     };
-    for (name, kind, dbname) in newly_pointed {
+    for (name, dbname) in newly_pointed {
         let governed_here = old_datatables.values().any(|old| {
             old.permissions.is_some()
-                && old
-                    .database
-                    .as_ref()
-                    .is_some_and(|d| d.resource_type == kind && d.resource_path == dbname)
+                && old.database.as_ref().is_some_and(|d| {
+                    d.resource_type == DataTableCatalogResourceType::Instance
+                        && d.resource_path == dbname
+                })
         });
-        if governed_here
-            || governed_elsewhere
-                .iter()
-                .any(|(k, p)| k == kind.as_ref() && p == dbname)
-        {
+        if governed_here || governed_elsewhere.iter().any(|g| g == dbname) {
             return Err(Error::BadRequest(format!(
                 "Data table '{name}' would point at database '{dbname}', which a data table under \
                  roles uses, without carrying those roles: everyone reaching '{name}' would connect \
@@ -8213,14 +8148,13 @@ async fn point_kept_datatables_at_parent(
         if dt.reference.is_some() {
             continue;
         }
-        // Only instance databases, on either cluster. A resource-backed data table names a
-        // resource, and the settings clone gave the fork its own copy of that resource in its own
-        // workspace — pointing at the parent's entry would silently move the fork onto the
-        // parent's resource instead.
+        // Only instance databases. A resource-backed data table names a resource, and the settings
+        // clone gave the fork its own copy of that resource in its own workspace — pointing at the
+        // parent's entry would silently move the fork onto the parent's resource instead.
         if dt
             .database
             .as_ref()
-            .is_none_or(|d| !d.resource_type.is_windmill_managed())
+            .is_none_or(|d| d.resource_type != DataTableCatalogResourceType::Instance)
         {
             continue;
         }
@@ -8350,30 +8284,26 @@ async fn apply_forked_datatable(
         })?,
     };
 
-    if database.resource_type == DataTableCatalogResourceType::ExternalInstance {
-        windmill_common::external_instance_pg::ensure_external_instance_database_registered(
-            tx,
-            &fdt.new_dbname,
+    if database.resource_type == DataTableCatalogResourceType::Instance {
+        // Held until the fork commits, as every save newly naming a database takes it: none may
+        // claim the copy between the check below and this fork's entry landing on it.
+        windmill_common::datatable_roles::lock_instance_databases_governance(
+            &mut **tx,
+            [fdt.new_dbname.as_str()],
         )
         .await?;
     }
-    if database.resource_type.is_windmill_managed()
+    if database.resource_type == DataTableCatalogResourceType::Instance
         && !windmill_api_auth::is_super_admin_authed(db, authed).await?
     {
-        windmill_common::ensure_fork_database_available_to(
-            db,
-            database.resource_type,
-            &fdt.new_dbname,
-            parent_w_id,
-        )
-        .await?;
+        windmill_common::ensure_fork_database_available_to(db, &fdt.new_dbname, parent_w_id)
+            .await?;
     }
-    if database.resource_type.is_windmill_managed() {
+    if database.resource_type == DataTableCatalogResourceType::Instance {
         // The whole `database` object, not just its `resource_path`: a pointer entry has none to
-        // patch. `reference` goes with it — exactly one of the two may be set. The copy was created
-        // on the same cluster as its source, so it keeps the source's kind.
+        // patch. `reference` goes with it — exactly one of the two may be set.
         let new_database = serde_json::json!({
-            "resource_type": database.resource_type,
+            "resource_type": "instance",
             "resource_path": &fdt.new_dbname,
         });
         sqlx::query!(
@@ -8764,8 +8694,6 @@ async fn create_workspace_fork(
     let mut tx: Transaction<'_, Postgres> = db.begin().await?;
     // Before the settings clone reads the parent's data tables: a pointer this fork ends up with
     // must not be written after cleanup of the parent decided that nothing points at its copies.
-    // Also before the external cluster's lifecycle lock, which finalizing an external copy takes:
-    // fork cleanup takes the two in this order.
     windmill_common::workspaces::lock_fork_datatables(&mut tx, &parent_workspace_id).await?;
 
     if nw.is_dev_workspace {

backend/windmill-api-workspaces/src/workspaces_extra.rs

@@ -56,6 +56,11 @@ pub(crate) async fn change_workspace_id(
 
     let mut tx = db.begin().await?;
 
+    // The settings copy below carries every data table entry to the new id, which fork cleanup of
+    // the old id cannot see until this commits: without the lock it could drop a copy the renamed
+    // workspace goes on using. Before the pairing lock, as forking takes the two in that order.
+    windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
+
     // A rename rewrites the workspace's dev flag and reparents its children, so it decides on the
     // same state the pairing handlers do: without this lock a concurrent create/attach could commit
     // an active dev workspace under the shell this rename is about to archive. Both ids, since the
@@ -110,21 +115,6 @@ pub(crate) async fn change_workspace_id(
     .execute(&mut *tx)
     .await?;
 
-    // A fork copy reserved for the old id would otherwise be unreachable: its creator cannot
-    // import into it or finish its fork under the new id, and nothing else would ever drop it.
-    sqlx::query(
-        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (
-               SELECT COALESCE(jsonb_object_agg(k, CASE WHEN v->>'workspace_id' = $1
-                   THEN jsonb_set(v, '{workspace_id}', to_jsonb($2::text)) ELSE v END), '{}'::jsonb)
-               FROM jsonb_each(COALESCE(value->'databases', '{}'::jsonb)) AS e(k, v)
-           ))
-           WHERE name = 'custom_instance_pg_databases'"#,
-    )
-    .bind(&old_id)
-    .bind(&rw.new_id)
-    .execute(&mut *tx)
-    .await?;
-
     // Duplicate workspace settings (keep copy in old workspace for reference)
     info!("Duplicating workspace_settings table");
     sqlx::query!(
@@ -865,6 +855,10 @@ pub(crate) async fn change_workspace_id(
         }
     }
 
+    // After every workspace_settings write above: fork cleanup locks a settings row before the
+    // registry, so taking the registry first here would deadlock with it.
+    migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
+
     // Audit log in the same transaction as the workspace changes
     audit_log(
         &mut *tx,
@@ -933,6 +927,14 @@ pub(crate) async fn change_workspace_id(
     let (_schedules_count, canceled_count, _deleted_tokens_count) =
         archive_workspace_impl(&db, &old_id, &authed.username, None).await?;
 
+    // The old id stays live between the commit above and the archive, and a fork copy created for
+    // it in that window registers under it. Creation checks the workspace is live under the fork
+    // lock, so once this has run under it, no copy can be reserved for the old id any more.
+    let mut tx = db.begin().await?;
+    windmill_common::workspaces::lock_fork_datatables(&mut tx, &old_id).await?;
+    migrate_fork_reservations(&mut tx, &old_id, &rw.new_id).await?;
+    tx.commit().await?;
+
     info!(
         "Workspace id change completed: moved {} to {}, archived old workspace",
         old_id, rw.new_id
@@ -944,6 +946,28 @@ pub(crate) async fn change_workspace_id(
     ))
 }
 
+/// A fork copy reserved for the old id would otherwise be unreachable: its creator cannot import
+/// into it or finish its fork under the new id, and nothing else would ever drop it.
+async fn migrate_fork_reservations(
+    tx: &mut Transaction<'_, Postgres>,
+    old_id: &str,
+    new_id: &str,
+) -> Result<()> {
+    sqlx::query(
+        r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (
+               SELECT COALESCE(jsonb_object_agg(k, CASE WHEN v->>'workspace_id' = $1
+                   THEN jsonb_set(v, '{workspace_id}', to_jsonb($2::text)) ELSE v END), '{}'::jsonb)
+               FROM jsonb_each(COALESCE(value->'databases', '{}'::jsonb)) AS e(k, v)
+           ))
+           WHERE name = 'custom_instance_pg_databases'"#,
+    )
+    .bind(old_id)
+    .bind(new_id)
+    .execute(&mut **tx)
+    .await?;
+    Ok(())
+}
+
 #[derive(Deserialize)]
 pub(crate) struct DeleteWorkspaceQuery {
     pub(crate) only_delete_forks: Option<bool>,
@@ -1424,7 +1448,9 @@ pub async fn drop_forked_datatable_databases(
             _ => continue,
         };
 
-        if database.resource_type.is_windmill_managed() {
+        if database.resource_type
+            == windmill_common::workspaces::DataTableCatalogResourceType::Instance
+        {
             let db_to_drop = &database.resource_path;
             if !db_to_drop.starts_with("wm_fork_") {
                 errors.push(format!(
@@ -1436,24 +1462,54 @@ pub async fn drop_forked_datatable_databases(
             // The fork's own entry is what is going away; anything else still reaching the copy,
             // a child fork's pointer at this entry included, keeps it. The lock keeps a child fork
             // from gaining such a pointer before the drop.
-            let dropped = async {
-                let mut tx = db.begin().await?;
-                windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
-                if database.resource_type
-                    == windmill_common::workspaces::DataTableCatalogResourceType::ExternalInstance
-                {
-                    windmill_common::external_instance_pg::drop_external_instance_database_unchecked(
-                        &db,
-                        db_to_drop,
-                        Some((&w_id, dt_name)),
+            // A task of its own, so a client going away cannot stop it between dropping the
+            // database and committing the entry's removal.
+            let dropped = tokio::spawn({
+                let (db, w_id, dt_name, db_to_drop) = (
+                    db.clone(),
+                    w_id.clone(),
+                    dt_name.clone(),
+                    db_to_drop.clone(),
+                );
+                let resource_type = database.resource_type;
+                async move {
+                    let mut tx = db.begin().await?;
+                    // The three locks a settings save takes, in its order: this workspace's data
+                    // tables, its settings row, and the database itself. Without them a save could
+                    // rename this entry, or point another one here, either side of the check below.
+                    windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
+                    // The snapshot above was read unlocked: a save committing since could have
+                    // repointed this entry, and the entry is removed below whatever it names by then.
+                    let current = sqlx::query_scalar::<_, Option<serde_json::Value>>(
+                        "SELECT datatable->'datatables'->$2 FROM workspace_settings
+                     WHERE workspace_id = $1 FOR UPDATE",
+                    )
+                    .bind(&w_id)
+                    .bind(&dt_name)
+                    .fetch_optional(&mut *tx)
+                    .await?
+                    .flatten()
+                    .and_then(|v| serde_json::from_value::<DataTable>(v).ok());
+                    if !current.is_some_and(|dt| {
+                        dt.forked_from.is_some()
+                            && dt.database.is_some_and(|d| {
+                                d.resource_type == resource_type && d.resource_path == db_to_drop
+                            })
+                    }) {
+                        return Err(Error::BadRequest(
+                            "the data table changed while it was being cleaned up".to_string(),
+                        ));
+                    }
+                    windmill_common::datatable_roles::lock_instance_databases_governance(
+                        &mut tx,
+                        [db_to_drop.as_str()],
                     )
                     .await?;
-                } else {
                     let uses = windmill_common::workspaces::managed_database_uses(
                         &mut tx,
                         windmill_common::workspaces::DataTableCatalogResourceType::Instance,
-                        db_to_drop,
-                        Some((&w_id, dt_name)),
+                        &db_to_drop,
+                        Some((w_id.as_str(), dt_name.as_str())),
                     )
                     .await?;
                     if !uses.is_empty() {
@@ -1462,12 +1518,31 @@ pub async fn drop_forked_datatable_databases(
                             uses.join(", ")
                         )));
                     }
-                    windmill_common::drop_custom_instance_database(&db, db_to_drop).await?;
+                    // The entry goes with the database: a fork this one is cloned into afterwards must
+                    // not inherit a pointer at a data table whose database is gone.
+                    sqlx::query(
+                    "UPDATE workspace_settings SET datatable = datatable #- ARRAY['datatables', $2]
+                     WHERE workspace_id = $1",
+                )
+                .bind(&w_id)
+                .bind(&dt_name)
+                .execute(&mut *tx)
+                .await?;
+                    windmill_common::drop_custom_instance_database_keep_entry(&db, &db_to_drop)
+                        .await?;
+                    sqlx::query(
+                        "UPDATE global_settings SET value = value #- ARRAY['databases', $1]
+                     WHERE name = 'custom_instance_pg_databases'",
+                    )
+                    .bind(&db_to_drop)
+                    .execute(&mut *tx)
+                    .await?;
+                    tx.commit().await?;
+                    Ok::<_, Error>(())
                 }
-                tx.commit().await?;
-                Ok::<_, Error>(())
-            }
-            .await;
+            })
+            .await
+            .unwrap_or_else(|e| Err(Error::internal_err(format!("cleanup task failed: {e}"))));
             if let Err(e) = dropped {
                 errors.push(format!(
                     "Could not drop instance database '{}' for datatable://{}: {}",
@@ -1835,17 +1910,7 @@ async fn resolve_fork_catalog_pg(
             "ducklake://{ducklake_name}: malformed registry catalog identity `{catalog}`"
         ))
     })?;
-    let catalog_resource = if resource_type == "external_instance" {
-        serde_json::to_value(
-            windmill_common::external_instance_pg::external_instance_connection_unchecked(
-                db,
-                resource_path,
-                false,
-            )
-            .await?,
-        )
-        .map_err(|e| Error::internal_err(format!("serializing pg creds: {e}")))?
-    } else if resource_type == "instance" {
+    let catalog_resource = if resource_type == "instance" {
         let mut pg_creds = windmill_common::PgDatabase::parse_uri(
             &windmill_common::get_database_url().await?.as_str().await,
         )?;

backend/windmill-api/openapi.yaml

@@ -1,7 +1,7 @@
 openapi: "3.0.3"
 
 info:
-  version: 1.813.0
+  version: 1.814.0
   title: Windmill API
 
   contact:
@@ -1609,67 +1609,6 @@ paths:
               schema:
                 $ref: "#/components/schemas/ExternalInstancePgSetupReport"
 
-  /settings/external_instance_pg/databases:
-    get:
-      summary: Lists the databases Windmill created on the external instance cluster, with the workspaces whose data tables, Ducklake catalogs or pending fork cleanups use each
-      operationId: listExternalInstancePgDatabases
-      tags:
-        - setting
-      responses:
-        "200":
-          description: databases by name
-          content:
-            application/json:
-              schema:
-                type: object
-                additionalProperties:
-                  $ref: "#/components/schemas/CustomInstanceDb"
-
-  /settings/external_instance_pg/databases/{name}:
-    post:
-      summary: Creates a database on the external instance cluster (enterprise edition only)
-      operationId: createExternalInstancePgDatabase
-      tags:
-        - setting
-      parameters:
-        - name: name
-          in: path
-          required: true
-          schema:
-            type: string
-      requestBody:
-        required: true
-        content:
-          application/json:
-            schema:
-              type: object
-              properties:
-                tag:
-                  $ref: "#/components/schemas/CustomInstanceDbTag"
-      responses:
-        "200":
-          description: database created
-          content:
-            application/json:
-              schema: {}
-    delete:
-      summary: Drops a database Windmill created on the external instance cluster, refused while a data table, Ducklake catalog or pending fork cleanup uses it
-      operationId: dropExternalInstancePgDatabase
-      tags:
-        - setting
-      parameters:
-        - name: name
-          in: path
-          required: true
-          schema:
-            type: string
-      responses:
-        "200":
-          description: database dropped
-          content:
-            application/json:
-              schema: {}
-
   /settings/list_custom_instance_pg_databases:
     post:
       summary: Returns the set-up statuses of custom instance pg databases
@@ -1688,19 +1627,13 @@ paths:
 
   /settings/datatable_roles:
     get:
-      summary: list the data table roles of one Windmill-managed Postgres cluster
+      summary: list the instance's data table roles
       operationId: listInstanceDatatableRoles
       tags:
         - setting
-      parameters:
-        - in: query
-          name: cluster
-          required: false
-          schema:
-            $ref: "#/components/schemas/DatatableRoleCluster"
       responses:
         "200":
-          description: the cluster's role catalog
+          description: the instance role catalog
           content:
             application/json:
               schema:
@@ -1708,7 +1641,7 @@ paths:
                 items:
                   $ref: "#/components/schemas/InstanceDatatableRole"
     post:
-      summary: create a data table role on a Windmill-managed Postgres cluster
+      summary: create a data table role on the instance's Postgres cluster
       operationId: createInstanceDatatableRole
       tags:
         - setting
@@ -1722,8 +1655,6 @@ paths:
               properties:
                 name:
                   type: string
-                cluster:
-                  $ref: "#/components/schemas/DatatableRoleCluster"
       responses:
         "200":
           description: the created role
@@ -5336,7 +5267,7 @@ paths:
                       type: string
                     resource_type:
                       type: string
-                      enum: [postgres, instance, external_instance]
+                      enum: [postgres, instance]
                     resource_path:
                       type: string
                     governing_workspace_id:
@@ -11673,11 +11604,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
 
       requestBody:
         description: script args
@@ -11714,11 +11644,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
 
       requestBody:
         description: script args
@@ -11755,11 +11684,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
 
       responses:
         "200":
@@ -11782,11 +11710,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
         - name: poll_delay_ms
           description: delay between polling for job updates in milliseconds
           in: query
@@ -11824,11 +11751,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
         - name: poll_delay_ms
           description: delay between polling for job updates in milliseconds
           in: query
@@ -11864,11 +11790,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
         - name: poll_delay_ms
           description: delay between polling for job updates in milliseconds
           in: query
@@ -11912,11 +11837,10 @@ paths:
         - $ref: "#/components/parameters/NewJobId"
         - $ref: "#/components/parameters/SkipPreprocessor"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
         - name: poll_delay_ms
           description: delay between polling for job updates in milliseconds
           in: query
@@ -12870,6 +12794,15 @@ paths:
           in: query
           schema:
             type: string
+        - name: kind
+          description: which conversations to list - the flow editor's test chats, the deployed flow's own (the default), or both
+          in: query
+          schema:
+            type: string
+            enum:
+              - test
+              - deployed
+              - all
       responses:
         "200":
           description: flow conversations list
@@ -12880,6 +12813,40 @@ paths:
                 items:
                   $ref: "#/components/schemas/FlowConversation"
 
+  /w/{workspace}/flow_conversations/update/{conversation_id}:
+    post:
+      summary: rename flow conversation
+      operationId: updateFlowConversation
+      tags:
+        - flow_conversations
+      parameters:
+        - $ref: "#/components/parameters/WorkspaceId"
+        - name: conversation_id
+          description: conversation id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              required: [title]
+              properties:
+                title:
+                  type: string
+                  description: the chat's name
+      responses:
+        "200":
+          description: flow conversation updated
+          content:
+            text/plain:
+              schema:
+                type: string
+
   /w/{workspace}/flow_conversations/delete/{conversation_id}:
     delete:
       summary: delete flow conversation
@@ -15274,11 +15241,10 @@ paths:
           schema:
             type: boolean
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
       requestBody:
         description: flow args
         required: true
@@ -15332,11 +15298,10 @@ paths:
           schema:
             type: boolean
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
       requestBody:
         description: flow args
         required: true
@@ -15824,11 +15789,10 @@ paths:
             type: boolean
         - $ref: "#/components/parameters/NewJobId"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
 
       requestBody:
         description: preview
@@ -15856,11 +15820,10 @@ paths:
       parameters:
         - $ref: "#/components/parameters/WorkspaceId"
         - name: memory_id
-          description: memory ID for chat-enabled flows
+          description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
           in: query
           schema:
             type: string
-            format: uuid
 
       requestBody:
         description: preview
@@ -28707,7 +28670,7 @@ components:
     FlowConversation:
       type: object
       required:
-        [id, workspace_id, flow_path, created_at, updated_at, created_by]
+        [id, workspace_id, flow_path, created_at, updated_at, created_by, is_test]
       properties:
         id:
           type: string
@@ -28734,6 +28697,9 @@ components:
         created_by:
           type: string
           description: Username who created the conversation
+        is_test:
+          type: boolean
+          description: Started from the flow editor's test panel rather than a deployed run
 
     FlowConversationMessage:
       type: object
@@ -28773,6 +28739,50 @@ components:
         success:
           type: boolean
           description: Whether the message is a success
+        tool_arguments:
+          type: string
+          nullable: true
+          description: >-
+            On a tool row, the arguments the model wrote for the call. For a script, flow or
+            AI agent tool these exclude the inputs its step wires in, which only the tool's
+            job holds. Null for a provider-native web search, whose query the provider does
+            not return.
+        tool_result:
+          type: string
+          nullable: true
+          description: >-
+            On a tool row, the text the model got back from the call, or what the call
+            failed with — the row's own text names the tool rather than the reason. For a
+            provider-native web search, its citations.
+        reasoning:
+          type: string
+          nullable: true
+          description: >-
+            On an answer, the thinking that produced it; on a tool row, the thinking that
+            led to the call. Each round's thinking is on one row. The agent job's result
+            keeps the turn's thinking as a single string.
+        attachments:
+          type: array
+          nullable: true
+          description: >-
+            The files a user message carried, as object-storage references: every flow
+            input other than user_message that held one or a list of them, at most 20. Never
+            file bytes or a presigned URL.
+          items:
+            type: object
+            required: [input, s3]
+            properties:
+              input:
+                type: string
+                description: The flow input that held the file
+              s3:
+                type: string
+                description: The file's key in object storage
+              storage:
+                type: string
+                description: The secondary storage holding the file, absent for the primary one
+              filename:
+                type: string
 
     EndpointTool:
       type: object
@@ -33765,17 +33775,9 @@ components:
         last_setup:
           $ref: "#/components/schemas/ExternalInstancePgSetupReport"
 
-    DatatableRoleCluster:
-      type: string
-      description: >-
-        The Windmill-managed Postgres cluster a data table role is a login on: Windmill's own
-        (behind `instance` data tables) or the external instance cluster (behind
-        `external_instance` ones). Defaults to `instance`.
-      enum: [instance, external_instance]
-
     InstanceDatatableRole:
       type: object
-      required: [id, name, enabled, cluster]
+      required: [id, name, enabled]
       properties:
         id:
           type: string
@@ -33783,8 +33785,6 @@ components:
           type: string
         enabled:
           type: boolean
-        cluster:
-          $ref: "#/components/schemas/DatatableRoleCluster"
 
     DatatableRoleTenants:
       type: object
@@ -33806,10 +33806,8 @@ components:
         supported:
           type: boolean
           description: >-
-            Whether this data table can be put under roles at all. Only one on a database Windmill
-            manages can: a role is a login on that database's cluster.
-        cluster:
-          $ref: "#/components/schemas/DatatableRoleCluster"
+            Whether this data table can be put under roles at all. Only one backed by the
+            instance database can: a role is a login on that cluster.
         permissioned:
           type: boolean
         default_role:
@@ -36086,7 +36084,6 @@ components:
                       - postgresql
                       - mysql
                       - instance
-                      - external_instance
                   resource_path:
                     type: string
                 required:
@@ -36150,7 +36147,6 @@ components:
                     enum:
                       - postgresql
                       - instance
-                      - external_instance
                   resource_path:
                     type: string
                 required:

backend/windmill-api/src/apps.rs

@@ -4310,11 +4310,11 @@ async fn execute_component(
         }
     }
 
-    let is_flow = payload
+    let flow_path = payload
         .path
-        .as_ref()
-        .map(|p| p.starts_with("flow/"))
-        .unwrap_or(false);
+        .as_deref()
+        .and_then(|path| path.strip_prefix("flow/"))
+        .map(str::to_string);
 
     // Tag for inline-script jobs is read from the deployed policy in run mode;
     // only preview mode (editor) honors the client-supplied tag. This applies to
@@ -4444,8 +4444,9 @@ async fn execute_component(
 
     // Apply runnable query parameters if provided
     if let Some(ref run_query) = payload.run_query_params {
-        if is_flow {
-            crate::jobs::process_flow_run_query_params(&mut tx, uuid, run_query).await?;
+        if let Some(flow_path) = flow_path.as_deref() {
+            crate::jobs::process_flow_run_query_params(&mut tx, uuid, &w_id, flow_path, run_query)
+                .await?;
         }
     }
 

backend/windmill-api/src/jobs.rs

@@ -4369,12 +4369,12 @@ async fn count_completed_jobs_detail(
     Query(query): Query<CountCompletedJobsQuery>,
 ) -> error::JsonResult<i64> {
     let mut sqlb = SqlBuilder::select_from("v2_job_completed");
-    //FOR RLS
-    sqlb.join("v2_job USING (id)");
     sqlb.field("COUNT(*) as count");
 
+    // Filtering on v2_job.workspace_id instead would keep the planner off
+    // ix_job_workspace_id_completed_at_all and scan the whole retention window.
     if !(w_id == "admins" && query.all_workspaces.unwrap_or(false)) {
-        sqlb.and_where_eq("v2_job.workspace_id", "?".bind(&w_id));
+        sqlb.and_where_eq("v2_job_completed.workspace_id", "?".bind(&w_id));
     }
 
     if let Some(after_s_ago) = query.completed_after_s_ago {
@@ -4393,6 +4393,7 @@ async fn count_completed_jobs_detail(
     }
 
     if let Some(tags) = query.tags {
+        sqlb.join("v2_job USING (id)");
         sqlb.and_where_in(
             "v2_job.tag",
             &tags.split(",").map(|t| quote(t)).collect::<Vec<_>>(),
@@ -4400,7 +4401,19 @@ async fn count_completed_jobs_detail(
     }
 
     let sql = sqlb.sql()?;
-    let stats = sqlx::query_scalar::<_, i64>(&sql).fetch_one(&db).await?;
+    let mut tx = db.begin().await?;
+    set_list_jobs_statement_timeout(&mut tx).await?;
+    let stats = sqlx::query_scalar::<_, i64>(&sql)
+        .fetch_one(&mut *tx)
+        .await
+        .map_err(|e| {
+            list_jobs_timeout_error(
+                e,
+                "Counting completed jobs",
+                "Lower completed_after_s_ago or narrow the filters.",
+            )
+        })?;
+    tx.commit().await?;
 
     Ok(Json(stats))
 }
@@ -4429,6 +4442,33 @@ lazy_static::lazy_static! {
             .unwrap_or(30);
 }
 
+/// A client that gives up does not cancel its query, so without this bound every retry of a
+/// slow filter stacks another scan running until the connection-wide 5min timeout.
+async fn set_list_jobs_statement_timeout(tx: &mut Transaction<'_, Postgres>) -> error::Result<()> {
+    let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
+    if timeout_secs > 0 {
+        sqlx::query(&format!("SET LOCAL statement_timeout = '{timeout_secs}s'"))
+            .execute(&mut **tx)
+            .await?;
+    }
+    Ok(())
+}
+
+fn list_jobs_timeout_error(e: sqlx::Error, action: &str, hint: &str) -> Error {
+    let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
+    match e {
+        sqlx::Error::Database(ref db_err)
+            if timeout_secs > 0 && db_err.code().as_deref() == Some("57014") =>
+        {
+            Error::Generic(
+                StatusCode::BAD_REQUEST,
+                format!("{action} took more than {timeout_secs}s and was stopped. {hint}"),
+            )
+        }
+        e => e.into(),
+    }
+}
+
 async fn list_jobs(
     authed: ApiAuthed,
     Extension(user_db): Extension<UserDB>,
@@ -4545,32 +4585,14 @@ async fn list_jobs(
     };
     // tracing::info!("sql: {}", &sql);
     let mut tx: Transaction<'_, Postgres> = user_db.begin(&authed).await?;
-
-    // A client that gives up does not cancel its query, so without this bound every retry of a
-    // slow filter stacks another scan running until the connection-wide 5min timeout.
-    let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
-    if timeout_secs > 0 {
-        sqlx::query(&format!("SET LOCAL statement_timeout = '{timeout_secs}s'"))
-            .execute(&mut *tx)
-            .await?;
-    }
+    set_list_jobs_statement_timeout(&mut tx).await?;
 
     let jobs: Vec<UnifiedJob> = sqlx::query_as(&sql)
         .fetch_all(&mut *tx)
         .warn_after_seconds_with_sql(5, format!("list_jobs: {}", sql))
         .await
-        .map_err(|e| match e {
-            sqlx::Error::Database(ref db_err)
-                if timeout_secs > 0 && db_err.code().as_deref() == Some("57014") =>
-            {
-                Error::Generic(
-                    StatusCode::BAD_REQUEST,
-                    format!(
-                        "Listing jobs took more than {timeout_secs}s and was stopped. Set a start date or narrow the filters."
-                    ),
-                )
-            }
-            e => e.into(),
+        .map_err(|e| {
+            list_jobs_timeout_error(e, "Listing jobs", "Set a start date or narrow the filters.")
         })?;
     tx.commit().await?;
 
@@ -9546,7 +9568,7 @@ async fn run_preview_flow_job(
     .await?;
 
     // Set memory_id if provided (for agent memory)
-    if let Some(memory_id) = run_query.memory_id {
+    if let Some(memory_id) = run_query.memory_key(&w_id, &flow_path) {
         set_flow_memory_id(&mut tx, uuid, memory_id).await?;
     }
 
@@ -9560,6 +9582,9 @@ async fn run_preview_flow_job(
             &run_query,
             user_message.as_ref(),
             uuid,
+            // Run from the editor's test panel: a trial, not a real conversation.
+            true,
+            &flow_args,
         )
         .await?;
     }

backend/windmill-common/Cargo.toml

@@ -33,6 +33,7 @@ path = "src/lib.rs"
 tar.workspace = true
 hmac.workspace = true
 sha2.workspace = true
+sha1.workspace = true
 thiserror.workspace = true
 anyhow.workspace = true
 serde.workspace = true

backend/windmill-common/src/datatable_roles.rs

@@ -6,67 +6,22 @@
  * LICENSE-AGPL for a copy of the license.
  */
 
-//! The instance's data table role catalogs.
+//! The instance's data table role catalog.
 //!
-//! A data table role is a real Postgres login role on one cluster — Windmill's own, or the external
-//! instance cluster — named exactly as the user named it, shared by every database Windmill manages
-//! on that cluster. Each cluster has its own catalog: a role exists where it was created and nowhere
-//! else. Windmill decides who may ask for a role (the per-data-table tenant lists in
-//! [`crate::workspaces`]); Postgres decides what the role may then touch. The catalog here is only
-//! the first half's vocabulary plus the cluster provisioning.
+//! A data table role is a real Postgres login role on the Windmill cluster, named exactly as the
+//! user named it, shared by every instance database. Windmill decides who may ask for a role (the
+//! per-data-table tenant lists in [`crate::workspaces`]); Postgres decides what the role may then
+//! touch. The catalog here is only the first half's vocabulary plus the cluster provisioning.
 //!
 //! Entries are keyed by a generated id so a rename moves nothing else: tenants name the id.
 
 use std::collections::BTreeMap;
 
-use serde::{Deserialize, Serialize};
-
 use crate::{
     error::{Error, Result},
-    workspaces::DataTableCatalogResourceType,
     DB,
 };
 
-/// The cluster a role catalog belongs to.
-#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
-#[serde(rename_all = "snake_case")]
-pub enum DatatableRoleCluster {
-    /// Windmill's own Postgres, behind `instance` data tables.
-    #[default]
-    Instance,
-    /// The external instance cluster, behind `external_instance` data tables.
-    ExternalInstance,
-}
-
-impl DatatableRoleCluster {
-    pub fn as_str(self) -> &'static str {
-        match self {
-            Self::Instance => "instance",
-            Self::ExternalInstance => "external_instance",
-        }
-    }
-
-    pub fn parse(value: &str) -> Result<Self> {
-        match value {
-            "instance" => Ok(Self::Instance),
-            "external_instance" => Ok(Self::ExternalInstance),
-            other => Err(Error::BadRequest(format!(
-                "Unknown data table role cluster '{other}': expected instance or external_instance"
-            ))),
-        }
-    }
-
-    /// The cluster whose roles a data table on `kind` can use. `None` for a resource-backed one,
-    /// which is never under roles.
-    pub fn of(kind: DataTableCatalogResourceType) -> Option<Self> {
-        match kind {
-            DataTableCatalogResourceType::Instance => Some(Self::Instance),
-            DataTableCatalogResourceType::ExternalInstance => Some(Self::ExternalInstance),
-            DataTableCatalogResourceType::Postgresql => None,
-        }
-    }
-}
-
 /// The connection every data table resolved to before roles existed (`custom_instance_user`). It
 /// owns every pre-existing object, so it is a reserved name rather than a catalog entry: never
 /// created, renamed or dropped.
@@ -209,42 +164,28 @@ pub async fn lock_instance_databases_governance<'a>(
 /// need the names — but callers MUST NOT let `pwd` reach a response, a log line, an audit record
 /// or an export. Nothing about who may call it: the credential is the whole risk, and `Debug` is
 /// hand-written to redact it for the same reason.
-pub async fn read_role_catalog(
-    db: &DB,
-    cluster: DatatableRoleCluster,
-) -> Result<DatatableRoleCatalog> {
-    crate::datatable_roles_oss::read_role_catalog(db, cluster).await
+pub async fn read_role_catalog(db: &DB) -> Result<DatatableRoleCatalog> {
+    crate::datatable_roles_oss::read_role_catalog(db).await
 }
 
 /// As [`read_role_catalog`], reading inside the caller's transaction so the value is the one
 /// [`lock_role_catalog`] is protecting. Same disclosure contract.
 pub async fn read_role_catalog_tx(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    cluster: DatatableRoleCluster,
 ) -> Result<DatatableRoleCatalog> {
-    crate::datatable_roles_oss::read_role_catalog_tx(tx, cluster).await
+    crate::datatable_roles_oss::read_role_catalog_tx(tx).await
 }
 
-/// The cluster a role belongs to, or `None` if no role has this id.
-pub async fn role_cluster(
-    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    id: &str,
-) -> Result<Option<DatatableRoleCluster>> {
-    crate::datatable_roles_oss::role_cluster(tx, id).await
-}
-
-/// Record a role, in the caller's transaction. On Windmill's own cluster that commits it with the
-/// `CREATE ROLE` it describes; on the external cluster the role already exists by then.
+/// Record a role, in the caller's transaction so it commits with the `CREATE ROLE` it describes.
 ///
 /// Authorization: writes a generated Postgres credential. Callers MUST restrict this to superadmin
 /// paths and MUST hold [`lock_role_catalog`] on `tx`.
 pub async fn insert_role_catalog_entry(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
     id: &str,
-    cluster: DatatableRoleCluster,
     role: &InstanceDatatableRole,
 ) -> Result<()> {
-    crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, cluster, role).await
+    crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, role).await
 }
 
 /// Update a role's recorded name, login flag and password. Same contract as
@@ -274,7 +215,7 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
         .find(|(_, role)| role.name == name)
         .ok_or_else(|| {
             Error::NotFound(format!(
-                "'{name}' is not a data table role of this database's cluster. Defined roles: {}.",
+                "'{name}' is not a data table role of this instance. Defined roles: {}.",
                 catalog
                     .values()
                     .map(|r| r.name.as_str())
@@ -290,90 +231,70 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
     Ok(entry.0.as_str())
 }
 
-/// Every database Windmill manages on `cluster`. Role provisioning has to reach all of them: a role
-/// that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
+/// Every instance database the registry knows about. Role provisioning has to reach all of them:
+/// a role that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
 ///
-/// Authorization: checks nothing, and names every managed database across all workspaces. Callers
+/// Authorization: checks nothing, and names every instance database across all workspaces. Callers
 /// MUST be superadmin-gated or keep the names server-side; never return them to a workspace caller.
-pub async fn registered_instance_databases(
-    db: &DB,
-    cluster: DatatableRoleCluster,
-) -> Result<Vec<String>> {
-    crate::datatable_roles_oss::registered_instance_databases(db, cluster).await
+pub async fn registered_instance_databases(db: &DB) -> Result<Vec<String>> {
+    crate::datatable_roles_oss::registered_instance_databases(db).await
 }
 
-/// `CONNECT` on `dbname` for every enabled role of `cluster`, and none for `PUBLIC`. Run at role
-/// creation, at database creation, and lazily whenever a managed data table is administered, so a
-/// database provisioned before a role existed is repaired rather than left silently unreachable.
+/// `CONNECT` on `dbname` for every enabled role, and none for `PUBLIC`. Run at role creation, at
+/// database creation, and lazily whenever an instance data table is administered, so a database
+/// provisioned before a role existed is repaired rather than left silently unreachable.
 ///
 /// Authorization: rewrites a database's ACL with the server's own credentials and checks nothing.
 /// Callers MUST have authorized administration of `dbname` — superadmin, or an admin of the
 /// workspace governing a data table on it.
-pub async fn converge_connect_grants(
-    db: &DB,
-    cluster: DatatableRoleCluster,
-    dbname: &str,
-) -> Result<()> {
-    crate::datatable_roles_oss::converge_connect_grants(db, cluster, dbname).await
+pub async fn converge_connect_grants(db: &DB, dbname: &str) -> Result<()> {
+    crate::datatable_roles_oss::converge_connect_grants(db, dbname).await
 }
 
-/// As [`converge_connect_grants`], with the catalog of `cluster` the caller already read. Same
-/// contract.
+/// As [`converge_connect_grants`], with a catalog the caller already read. Same contract.
 pub async fn converge_connect_grants_with(
     db: &DB,
-    cluster: DatatableRoleCluster,
     dbname: &str,
     catalog: &DatatableRoleCatalog,
 ) -> Result<()> {
-    crate::datatable_roles_oss::converge_connect_grants_with(db, cluster, dbname, catalog).await
+    crate::datatable_roles_oss::converge_connect_grants_with(db, dbname, catalog).await
 }
 
-/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user` on `cluster`. No
-/// privileges beyond that — an admin grants them through SQL or the ACL editor.
-///
-/// On Windmill's own cluster the DDL runs on `tx`, so it commits with the catalog row. The external
-/// cluster is another server: the role is created there before `tx` commits, and callers MUST drop
-/// it again ([`drop_datatable_role`]) if `tx` then fails to commit.
+/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user`, and `CONNECT` on
+/// every registered database. No privileges beyond that — an admin grants them through SQL or the
+/// ACL editor.
 ///
 /// Authorization: creates a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
-pub async fn create_datatable_role(
-    db: &DB,
+/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
+pub async fn create_instance_role(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    cluster: DatatableRoleCluster,
     name: &str,
     password: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::create_datatable_role(db, tx, cluster, name, password).await
+    crate::datatable_roles_oss::create_instance_role(tx, name, password).await
 }
 
 /// Authorization: alters a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
-pub async fn set_datatable_role_login(
-    db: &DB,
+/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
+pub async fn set_instance_role_login(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    cluster: DatatableRoleCluster,
     name: &str,
     enabled: bool,
 ) -> Result<()> {
-    crate::datatable_roles_oss::set_datatable_role_login(db, tx, cluster, name, enabled).await
+    crate::datatable_roles_oss::set_instance_role_login(tx, name, enabled).await
 }
 
-/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one. On the
-/// external cluster the rename lands before `tx` commits, and callers MUST rename it back if `tx`
-/// then fails to commit.
+/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one.
 ///
 /// Authorization: renames a cluster-wide Postgres login. Callers MUST restrict this to superadmin
-/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
-pub async fn rename_datatable_role(
-    db: &DB,
+/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
+pub async fn rename_instance_role(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    cluster: DatatableRoleCluster,
     from: &str,
     to: &str,
     password: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::rename_datatable_role(db, tx, cluster, from, to, password).await
+    crate::datatable_roles_oss::rename_instance_role(tx, from, to, password).await
 }
 
 /// A role owning anything in any database blocks its own `DROP ROLE`, and both its objects and the
@@ -381,9 +302,8 @@ pub async fn rename_datatable_role(
 /// registry. An unreachable database aborts the whole delete: dropping the role while one database
 /// still holds objects owned by it leaves those objects owned by a numeric OID nobody can name.
 ///
-/// Each pass runs as the cluster's administrator rather than `custom_instance_user`: on Windmill's
-/// own cluster the instance's Postgres user, on the external one its configured admin login. Both
-/// own the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
+/// Each pass runs as the instance's own Postgres user rather than `custom_instance_user`, which
+/// owns the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
 /// could only undo what it granted itself, so a privilege planted by an operator in psql — the
 /// ordinary way privileges reach a role — would survive and block the drop.
 ///
@@ -391,19 +311,16 @@ pub async fn rename_datatable_role(
 /// MUST restrict this to superadmin paths, and MUST hold [`lock_role_catalog`] on `tx`.
 ///
 /// The per-database passes open their own connections and cannot join `tx`; the lock is what keeps
-/// a concurrent mutation out while they run. On Windmill's own cluster only the final `DROP ROLE`
-/// is on `tx`, so it commits or rolls back with the catalog write that forgets the role; on the
-/// external cluster it runs there, and tolerates a role already gone so a retry after a failed
-/// commit can finish. The passes commit as they go, so callers MUST have disabled the role in an
-/// earlier committed transaction: a failure part-way then leaves a disabled role to retry, not an
-/// enabled one already stripped in some databases.
-pub async fn drop_datatable_role(
+/// a concurrent mutation out while they run. Only the final `DROP ROLE` is on `tx`, so it commits
+/// or rolls back with the catalog write that forgets the role. Those passes commit as they go, so
+/// callers MUST have disabled the role in an earlier committed transaction: a failure part-way
+/// then leaves a disabled role to retry, not an enabled one already stripped in some databases.
+pub async fn drop_instance_role(
     db: &DB,
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    cluster: DatatableRoleCluster,
     name: &str,
 ) -> Result<()> {
-    crate::datatable_roles_oss::drop_datatable_role(db, tx, cluster, name).await
+    crate::datatable_roles_oss::drop_instance_role(db, tx, name).await
 }
 
 #[cfg(test)]

backend/windmill-common/src/datatable_roles_oss.rs

@@ -24,12 +24,12 @@ pub fn datatable_roles_unavailable() -> Error {
 #[cfg(all(feature = "private", feature = "enterprise"))]
 pub(crate) use crate::datatable_roles_ee::{
     can_use_datatable_role, can_use_datatable_role_in_governing_workspace, converge_connect_grants,
-    converge_connect_grants_with, create_datatable_role, delete_role_catalog_entry,
-    drop_datatable_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
+    converge_connect_grants_with, create_instance_role, delete_role_catalog_entry,
+    drop_instance_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
     ensure_instance_db_grant_options_unchecked, forget_datatable_role_everywhere,
     insert_role_catalog_entry, read_role_catalog, read_role_catalog_tx,
-    registered_instance_databases, rename_datatable_role, resolve_datatable_role_connection,
-    role_cluster, set_datatable_role_login, update_role_catalog_entry,
+    registered_instance_databases, rename_instance_role, resolve_datatable_role_connection,
+    set_instance_role_login, update_role_catalog_entry,
 };
 
 #[cfg(not(all(feature = "private", feature = "enterprise")))]
@@ -39,7 +39,7 @@ pub(crate) use ce::*;
 mod ce {
     use super::datatable_roles_unavailable as unavailable;
     use crate::{
-        datatable_roles::{DatatableRoleCatalog, DatatableRoleCluster, InstanceDatatableRole},
+        datatable_roles::{DatatableRoleCatalog, InstanceDatatableRole},
         db::AuthedRef,
         error::Result,
         workspaces::{
@@ -50,31 +50,17 @@ mod ce {
 
     type Tx<'a> = sqlx::Transaction<'a, sqlx::Postgres>;
 
-    pub(crate) async fn read_role_catalog(
-        _db: &DB,
-        _cluster: DatatableRoleCluster,
-    ) -> Result<DatatableRoleCatalog> {
+    pub(crate) async fn read_role_catalog(_db: &DB) -> Result<DatatableRoleCatalog> {
         Err(unavailable())
     }
 
-    pub(crate) async fn read_role_catalog_tx(
-        _tx: &mut Tx<'_>,
-        _cluster: DatatableRoleCluster,
-    ) -> Result<DatatableRoleCatalog> {
-        Err(unavailable())
-    }
-
-    pub(crate) async fn role_cluster(
-        _tx: &mut Tx<'_>,
-        _id: &str,
-    ) -> Result<Option<DatatableRoleCluster>> {
+    pub(crate) async fn read_role_catalog_tx(_tx: &mut Tx<'_>) -> Result<DatatableRoleCatalog> {
         Err(unavailable())
     }
 
     pub(crate) async fn insert_role_catalog_entry(
         _tx: &mut Tx<'_>,
         _id: &str,
-        _cluster: DatatableRoleCluster,
         _role: &InstanceDatatableRole,
     ) -> Result<()> {
         Err(unavailable())
@@ -92,57 +78,43 @@ mod ce {
         Err(unavailable())
     }
 
-    pub(crate) async fn registered_instance_databases(
-        _db: &DB,
-        _cluster: DatatableRoleCluster,
-    ) -> Result<Vec<String>> {
+    pub(crate) async fn registered_instance_databases(_db: &DB) -> Result<Vec<String>> {
         Err(unavailable())
     }
 
-    /// Nothing to converge: with no roles to admit, a managed database keeps the `CONNECT` grants
-    /// it was created with, as it did before roles existed.
-    pub(crate) async fn converge_connect_grants(
-        _db: &DB,
-        _cluster: DatatableRoleCluster,
-        _dbname: &str,
-    ) -> Result<()> {
+    /// Nothing to converge: with no roles to admit, an instance database keeps the `CONNECT`
+    /// grants it was created with, `PUBLIC`'s included, as it did before roles existed.
+    pub(crate) async fn converge_connect_grants(_db: &DB, _dbname: &str) -> Result<()> {
         Ok(())
     }
 
     /// As [`converge_connect_grants`].
     pub(crate) async fn converge_connect_grants_with(
         _db: &DB,
-        _cluster: DatatableRoleCluster,
         _dbname: &str,
         _catalog: &DatatableRoleCatalog,
     ) -> Result<()> {
         Ok(())
     }
 
-    pub(crate) async fn create_datatable_role(
-        _db: &DB,
+    pub(crate) async fn create_instance_role(
         _tx: &mut Tx<'_>,
-        _cluster: DatatableRoleCluster,
         _name: &str,
         _password: &str,
     ) -> Result<()> {
         Err(unavailable())
     }
 
-    pub(crate) async fn set_datatable_role_login(
-        _db: &DB,
+    pub(crate) async fn set_instance_role_login(
         _tx: &mut Tx<'_>,
-        _cluster: DatatableRoleCluster,
         _name: &str,
         _enabled: bool,
     ) -> Result<()> {
         Err(unavailable())
     }
 
-    pub(crate) async fn rename_datatable_role(
-        _db: &DB,
+    pub(crate) async fn rename_instance_role(
         _tx: &mut Tx<'_>,
-        _cluster: DatatableRoleCluster,
         _from: &str,
         _to: &str,
         _password: &str,
@@ -150,18 +122,12 @@ mod ce {
         Err(unavailable())
     }
 
-    pub(crate) async fn drop_datatable_role(
-        _db: &DB,
-        _tx: &mut Tx<'_>,
-        _cluster: DatatableRoleCluster,
-        _name: &str,
-    ) -> Result<()> {
+    pub(crate) async fn drop_instance_role(_db: &DB, _tx: &mut Tx<'_>, _name: &str) -> Result<()> {
         Err(unavailable())
     }
 
     pub(crate) async fn ensure_instance_db_grant_options_unchecked(
         _db: &DB,
-        _cluster: DatatableRoleCluster,
         _dbname: &str,
     ) -> Result<()> {
         Err(unavailable())

backend/windmill-common/src/external_instance_pg.rs

@@ -17,7 +17,7 @@
 //! The cluster may hold data Windmill did not create. Two Windmill instances sharing one is not
 //! supported: each would keep resetting the passwords the other depends on.
 
-use std::collections::{BTreeMap, BTreeSet};
+use std::collections::BTreeMap;
 
 use serde::{Deserialize, Serialize};
 
@@ -41,21 +41,6 @@ pub struct ExternalInstancePgState {
     pub databases: BTreeMap<String, CustomInstanceDb>,
     #[serde(default, skip_serializing_if = "Option::is_none")]
     pub last_setup: Option<ExternalInstancePgSetupReport>,
-    /// The cluster ([`external_instance_pg_address`]) the last successful setup converged. Databases
-    /// are only created on a cluster setup succeeded on: the passwords above exist as soon as setup
-    /// first runs, whether or not the cluster accepted them.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub set_up_for: Option<String>,
-}
-
-/// What identifies the cluster a configuration points at. Other fields (admin login, sslmode) can
-/// change without it becoming another cluster.
-pub fn external_instance_pg_address(config: &ExternalInstancePg) -> String {
-    format!(
-        "{}:{}",
-        config.host.trim().to_lowercase(),
-        config.port.unwrap_or(5432)
-    )
 }
 
 #[derive(Serialize, Deserialize, Clone, Debug)]
@@ -129,6 +114,7 @@ pub(crate) async fn read_external_instance_pg_state<'c>(
     }
 }
 
+/// Authorization: reads the hidden cluster state and checks nothing. Callers MUST be superadmin.
 pub async fn external_instance_pg_status(db: &DB) -> Result<ExternalInstancePgStatus> {
     let configured = read_external_instance_pg_config(db).await?.is_some();
     let state = read_external_instance_pg_state(db).await?;
@@ -139,288 +125,73 @@ pub async fn external_instance_pg_status(db: &DB) -> Result<ExternalInstancePgSt
     })
 }
 
-/// The databases Windmill created on the external cluster, without the passwords kept beside them.
-///
-/// Authorization: names every database across all workspaces, and the workspace each fork copy is
-/// reserved for, and checks nothing. Callers MUST be superadmin or an internal authorization or
-/// lifecycle path that does not return the names to a workspace caller.
-pub async fn external_instance_databases(db: &DB) -> Result<BTreeMap<String, CustomInstanceDb>> {
-    Ok(read_external_instance_pg_state(db).await?.databases)
-}
-
-/// The workspaces whose data tables or Ducklake catalogs name each database on the external cluster,
-/// and the forks whose Ducklake metadata schemas there are still waiting to be dropped: those rows
-/// outlive a settings change, and cleanup cannot drop a schema in a database that is gone. A row
-/// whose schema is already dropped only waits on object storage, which needs no database.
-///
-/// Authorization: reads every workspace's settings and checks nothing. Callers MUST be superadmin
-/// or an internal lifecycle path.
-pub async fn external_instance_database_usages<'c>(
-    db: impl sqlx::PgExecutor<'c>,
-) -> Result<BTreeMap<String, BTreeSet<String>>> {
-    let rows = sqlx::query_as::<_, (String, String)>(
-        "SELECT ws.workspace_id, entry->'database'->>'resource_path'
-         FROM workspace_settings ws
-         CROSS JOIN LATERAL jsonb_each(
-             CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
-                 THEN ws.datatable->'datatables'
-                 ELSE '{}'::jsonb END
-         ) AS dt(k, entry)
-         WHERE entry->'database'->>'resource_type' = 'external_instance'
-           AND entry->'database'->>'resource_path' IS NOT NULL
-         UNION ALL
-         SELECT ws.workspace_id, entry->'catalog'->>'resource_path'
-         FROM workspace_settings ws
-         CROSS JOIN LATERAL jsonb_each(
-             CASE WHEN jsonb_typeof(ws.ducklake->'ducklakes') = 'object'
-                 THEN ws.ducklake->'ducklakes'
-                 ELSE '{}'::jsonb END
-         ) AS dl(k, entry)
-         WHERE entry->'catalog'->>'resource_type' = 'external_instance'
-           AND entry->'catalog'->>'resource_path' IS NOT NULL
-         UNION ALL
-         SELECT workspace_id, substring(catalog FROM length('external_instance:') + 1)
-         FROM fork_ducklake_namespace
-         WHERE catalog LIKE 'external\\_instance:%' AND NOT schema_dropped",
-    )
-    .fetch_all(db)
-    .await?;
-    let mut usages: BTreeMap<String, BTreeSet<String>> = BTreeMap::new();
-    for (workspace_id, dbname) in rows {
-        usages.entry(dbname).or_default().insert(workspace_id);
-    }
-    Ok(usages)
-}
-
-/// Refuse to unset the cluster while Windmill still has databases or data table roles on it, or a
-/// workspace still points at one: every data table there would stop resolving, and every role
-/// would be a login nothing can drop any more. Allowed on every edition, so a
-/// downgraded instance can still clear a setting it no longer uses.
-pub async fn ensure_external_instance_pg_removable(db: &DB) -> Result<()> {
-    ensure_external_instance_pg_unused(db, &format!("removing {EXTERNAL_INSTANCE_PG_SETTING}"))
-        .await
-}
-
-/// Refuse while Windmill has databases or data table roles on the cluster, or a workspace points
-/// at one of its databases. `before` finishes the sentence saying what to do first.
-async fn ensure_external_instance_pg_unused(db: &DB, before: &str) -> Result<()> {
+/// Refuse to unset the cluster while Windmill still has databases on it: every data table and
+/// Ducklake catalog there would stop resolving. Allowed on every edition, so a downgraded
+/// instance can still clear a setting it no longer uses.
+async fn ensure_external_instance_pg_removable(db: &DB) -> Result<()> {
     let state = read_external_instance_pg_state(db).await?;
-    let usages = external_instance_database_usages(db).await?;
-    let roles = sqlx::query_scalar::<_, String>(
-        "SELECT name FROM datatable_role WHERE cluster = 'external_instance' ORDER BY name",
-    )
-    .fetch_all(db)
-    .await?;
-    if state.databases.is_empty() && usages.is_empty() && roles.is_empty() {
+    if state.databases.is_empty() {
         return Ok(());
     }
-    let mut held = vec![];
-    if !(state.databases.is_empty() && usages.is_empty()) {
-        let names = state
-            .databases
-            .keys()
-            .chain(usages.keys())
-            .collect::<BTreeSet<_>>()
-            .into_iter()
-            .cloned()
-            .collect::<Vec<_>>()
-            .join(", ");
-        held.push(format!("databases in use ({names})"));
-    }
-    if !roles.is_empty() {
-        held.push(format!("data table roles ({})", roles.join(", ")));
-    }
-    Err(Error::BadRequest(format!(
-        "The external instance cluster still holds {}. Drop them and repoint the data tables and \
-         Ducklake catalogs using them before {before}.",
-        held.join(" and ")
-    )))
-}
-
-/// Refuse a workspace setting that newly names an `external_instance` database on an edition
-/// without them.
-pub fn ensure_external_instance_available() -> Result<()> {
-    crate::external_instance_pg_oss::ensure_external_instance_available()
-}
-
-/// The connection an `external_instance` database resolves to: `custom_instance_user`, or the
-/// replication user, on the external cluster.
-///
-/// Authorization: returns live credentials and checks nothing. Callers MUST have authorized access
-/// to the data table that names `dbname`.
-pub async fn external_instance_connection_unchecked(
-    db: &DB,
-    dbname: &str,
-    replication: bool,
-) -> Result<crate::PgDatabase> {
-    crate::external_instance_pg_oss::external_instance_connection_unchecked(db, dbname, replication)
-        .await
-}
-
-/// Create `dbname` on the external cluster and register it. Refuses a name already taken there,
-/// whoever took it.
-///
-/// Authorization: checks nothing. Callers MUST be superadmin, or be cloning a data table they may
-/// fork into a `wm_fork_` database.
-pub async fn create_external_instance_database_unchecked(
-    db: &DB,
-    dbname: &str,
-    tag: &str,
-    for_workspace: Option<&str>,
-) -> Result<()> {
-    crate::external_instance_pg_oss::create_external_instance_database_unchecked(
-        db,
-        dbname,
-        tag,
-        for_workspace,
-    )
-    .await
-}
-
-/// Drop `dbname` from the external cluster: only a database Windmill registered creating, and still
-/// carries the mark it set there. Refused while anything uses it
-/// ([`crate::workspaces::managed_database_uses`]), except the `exempt` data table entry: the fork
-/// copy being cleaned up.
-///
-/// Authorization: checks nothing. Callers MUST be superadmin, or be deleting the fork that owns
-/// this `wm_fork_` database.
-pub async fn drop_external_instance_database_unchecked(
-    db: &DB,
-    dbname: &str,
-    exempt: Option<(&str, &str)>,
-) -> Result<()> {
-    crate::external_instance_pg_oss::drop_external_instance_database_unchecked(db, dbname, exempt)
-    .await
-}
-
-/// Serializes everything that changes which databases exist on the external cluster, or which data
-/// tables name them: setup, creates, drops, and data table saves. Held until `tx` ends.
-pub async fn lock_external_instance_pg_state(
-    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-) -> Result<()> {
-    sqlx::query("SELECT pg_advisory_xact_lock(hashtext($1))")
-        .bind(EXTERNAL_INSTANCE_PG_STATE_SETTING)
-        .execute(&mut **tx)
-        .await?;
-    Ok(())
-}
-
-/// Refuse a data table naming `dbname` unless Windmill created it on the external cluster. Takes
-/// the lock drops take, so none can remove the database before `tx`, which saves the data table,
-/// commits.
-pub async fn ensure_external_instance_database_registered(
-    tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
-    dbname: &str,
-) -> Result<()> {
-    lock_external_instance_pg_state(tx).await?;
-    if read_external_instance_pg_state(&mut **tx)
-        .await?
+    let names = state
         .databases
-        .contains_key(dbname)
-    {
-        return Ok(());
-    }
+        .keys()
+        .cloned()
+        .collect::<Vec<_>>()
+        .join(", ");
     Err(Error::BadRequest(format!(
-        "Windmill did not create a database named '{dbname}' on the external instance cluster. \
-         Create it from the instance settings first."
+        "The external instance cluster still holds databases Windmill created ({names}). Drop \
+         them before removing {EXTERNAL_INSTANCE_PG_SETTING}."
     )))
 }
 
-/// Write [`EXTERNAL_INSTANCE_PG_SETTING`]: `None`, null or an empty string unsets it. Every writer
-/// of global settings goes through this for that key — the per-key and bulk endpoints as well as
-/// the declarative sync — instead of writing the row itself.
+/// Check a write to [`EXTERNAL_INSTANCE_PG_SETTING`] before it happens: `None`, null or an empty
+/// string unsets it. Every writer of global settings calls this, the per-key and bulk endpoints
+/// as well as the declarative sync.
 ///
-/// The checks and the write share one transaction holding [`lock_external_instance_pg_state`]. A
-/// check taken outside it could pass while a database create still reads the old cluster, which
-/// would then register a database there after the setting names another one.
-///
-/// Authorization: checks nothing. Callers MUST be superadmin, or the declarative instance config
-/// sync, which applies what the operator deployed.
-pub async fn write_external_instance_pg_setting(
+/// Authorization: reads the hidden cluster state and names the databases on the cluster in its
+/// refusal. Callers MUST be superadmin or an internal server path writing global settings.
+pub async fn check_external_instance_pg_write(
     db: &DB,
     value: Option<&serde_json::Value>,
 ) -> Result<()> {
-    let value = match value {
-        None | Some(serde_json::Value::Null) => None,
-        Some(serde_json::Value::String(s)) if s.trim().is_empty() => None,
-        Some(value) => Some(value),
-    };
-    let mut tx = db.begin().await?;
-    lock_external_instance_pg_state(&mut tx).await?;
     match value {
-        None => {
-            ensure_external_instance_pg_removable(db).await?;
-            sqlx::query("DELETE FROM global_settings WHERE name = $1")
-                .bind(EXTERNAL_INSTANCE_PG_SETTING)
-                .execute(&mut *tx)
-                .await?;
+        None | Some(serde_json::Value::Null) => ensure_external_instance_pg_removable(db).await,
+        Some(serde_json::Value::String(s)) if s.trim().is_empty() => {
+            ensure_external_instance_pg_removable(db).await
         }
         Some(value) => {
             crate::external_instance_pg_oss::validate_external_instance_pg_setting(value)?;
-            ensure_external_instance_pg_not_repointed(db, value).await?;
-            sqlx::query(
-                "INSERT INTO global_settings (name, value) VALUES ($1, $2)
-                 ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value, updated_at = now()",
-            )
-            .bind(EXTERNAL_INSTANCE_PG_SETTING)
-            .bind(value)
-            .execute(&mut *tx)
-            .await?;
+            ensure_same_cluster_while_in_use(db, value).await
         }
     }
-    tx.commit().await?;
-    tracing::info!(
-        "{} global setting {EXTERNAL_INSTANCE_PG_SETTING}",
-        if value.is_some() { "Set" } else { "Unset" }
-    );
-    Ok(())
 }
 
-/// [`write_external_instance_pg_setting`] for a settings diff: writes the key if the diff touches
-/// it, and takes it out of the diff so the generic apply does not write it again.
-///
-/// Authorization: checks nothing. Callers MUST be superadmin, or the declarative instance config
-/// sync, which applies what the operator deployed.
-pub async fn write_external_instance_pg_from_diff(
-    db: &DB,
-    diff: &mut crate::instance_config::SettingsDiff,
-) -> Result<()> {
-    if let Some(value) = diff.upserts.remove(EXTERNAL_INSTANCE_PG_SETTING) {
-        write_external_instance_pg_setting(db, Some(&value)).await?;
-    }
-    if let Some(i) = diff
-        .deletes
-        .iter()
-        .position(|k| k == EXTERNAL_INSTANCE_PG_SETTING)
-    {
-        diff.deletes.remove(i);
-        write_external_instance_pg_setting(db, None).await?;
-    }
-    Ok(())
-}
-
-/// Refuse pointing the setting at another host or port while databases or data table roles live on
-/// the current one. Data tables name databases, and the role catalog names logins, not clusters, so
-/// both would silently resolve to whatever the new cluster holds under the same names. Other fields
-/// (admin login, sslmode) may change freely.
-async fn ensure_external_instance_pg_not_repointed(
-    db: &DB,
-    value: &serde_json::Value,
-) -> Result<()> {
-    let Some(current) = read_external_instance_pg_config(db).await? else {
-        return Ok(());
-    };
-    let Ok(desired) = serde_json::from_value::<ExternalInstancePg>(value.clone()) else {
-        return Ok(());
-    };
-    if external_instance_pg_address(&current) == external_instance_pg_address(&desired) {
+/// Pointing the setting at another cluster strands the databases Windmill created on this one as
+/// surely as unsetting it, and would hand their registry and role passwords to a cluster that has
+/// neither. The login, TLS and maintenance database may change; the host and port may not.
+async fn ensure_same_cluster_while_in_use(db: &DB, value: &serde_json::Value) -> Result<()> {
+    let state = read_external_instance_pg_state(db).await?;
+    if state.databases.is_empty() {
         return Ok(());
     }
-    ensure_external_instance_pg_unused(
-        db,
-        &format!("pointing {EXTERNAL_INSTANCE_PG_SETTING} at another cluster"),
-    )
-    .await
+    let identity = |c: &ExternalInstancePg| (c.host.trim().to_lowercase(), c.port.unwrap_or(5432));
+    let new: ExternalInstancePg = serde_json::from_value(value.clone())
+        .map_err(|e| Error::BadRequest(format!("{EXTERNAL_INSTANCE_PG_SETTING}: {e}")))?;
+    let current = read_external_instance_pg_config(db).await?;
+    if current.is_some_and(|current| identity(&current) == identity(&new)) {
+        return Ok(());
+    }
+    Err(Error::BadRequest(format!(
+        "The external instance cluster still holds databases Windmill created ({}). Drop them \
+         before pointing {EXTERNAL_INSTANCE_PG_SETTING} at another host or port.",
+        state
+            .databases
+            .keys()
+            .cloned()
+            .collect::<Vec<_>>()
+            .join(", ")
+    )))
 }
 
 /// Converge the external cluster on the configured login: check what it can do, create or update

backend/windmill-common/src/external_instance_pg_oss.rs

@@ -19,63 +19,25 @@ pub fn external_instance_pg_unavailable() -> Error {
 
 #[cfg(all(feature = "private", feature = "enterprise"))]
 pub(crate) use crate::external_instance_pg_ee::{
-    create_external_instance_database_unchecked, drop_external_instance_database_unchecked,
-    external_instance_connection_unchecked, setup_external_instance_pg_unchecked,
-    validate_external_instance_pg_setting,
+    setup_external_instance_pg_unchecked, validate_external_instance_pg_setting,
 };
 
-#[cfg(all(feature = "private", feature = "enterprise"))]
-pub(crate) fn ensure_external_instance_available() -> crate::error::Result<()> {
-    Ok(())
-}
-
 #[cfg(not(all(feature = "private", feature = "enterprise")))]
 pub(crate) use ce::*;
 
 #[cfg(not(all(feature = "private", feature = "enterprise")))]
 mod ce {
     use super::external_instance_pg_unavailable as unavailable;
-    use crate::{
-        error::Result, external_instance_pg::ExternalInstancePgSetupReport, PgDatabase, DB,
-    };
+    use crate::{error::Result, external_instance_pg::ExternalInstancePgSetupReport, DB};
 
     pub(crate) fn validate_external_instance_pg_setting(_value: &serde_json::Value) -> Result<()> {
         Err(unavailable())
     }
 
-    pub(crate) fn ensure_external_instance_available() -> Result<()> {
-        Err(unavailable())
-    }
-
     pub(crate) async fn setup_external_instance_pg_unchecked(
         _db: &DB,
         _rotate_passwords: bool,
     ) -> Result<ExternalInstancePgSetupReport> {
         Err(unavailable())
     }
-
-    pub(crate) async fn external_instance_connection_unchecked(
-        _db: &DB,
-        _dbname: &str,
-        _replication: bool,
-    ) -> Result<PgDatabase> {
-        Err(unavailable())
-    }
-
-    pub(crate) async fn create_external_instance_database_unchecked(
-        _db: &DB,
-        _dbname: &str,
-        _tag: &str,
-        _for_workspace: Option<&str>,
-    ) -> Result<()> {
-        Err(unavailable())
-    }
-
-    pub(crate) async fn drop_external_instance_database_unchecked(
-        _db: &DB,
-        _dbname: &str,
-        _exempt: Option<(&str, &str)>,
-    ) -> Result<()> {
-        Err(unavailable())
-    }
 }

backend/windmill-common/src/flow_conversations.rs

@@ -1,12 +1,39 @@
+use std::collections::HashMap;
+
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
+use serde_json::value::RawValue;
 use sqlx::{self, FromRow};
 use uuid::Uuid;
+use windmill_types::s3::S3Object;
 
 use crate::db::DB;
 use crate::error::Result;
 use crate::utils::truncate_with_ellipsis;
 
+/// Changing it detaches every memory stored under a string memory id.
+const MEMORY_ID_NAMESPACE: Uuid = Uuid::from_u128(0x6f1c2d4e_8a3b_5c7d_9e0f_1a2b3c4d5e6f);
+
+/// Memory is stored and carried in `flow_status.memory_id` as a uuid, which names the same memory
+/// wherever it is passed, as a chat conversation id must. Any other string names a memory through a
+/// name-based (v5) uuid scoped to its workspace and flow, so the same key in two flows or two
+/// workspaces names two memories, and chat conversation ids stay unique across workspaces.
+pub fn memory_key(workspace_id: &str, flow_path: &str, memory_id: &str) -> Uuid {
+    let memory_id = memory_id.trim();
+    Uuid::parse_str(memory_id).unwrap_or_else(|_| {
+        use sha1::{Digest, Sha1};
+        let mut hasher = Sha1::new();
+        hasher.update(MEMORY_ID_NAMESPACE.as_bytes());
+        for part in [workspace_id, flow_path, memory_id] {
+            hasher.update(part.as_bytes());
+            hasher.update([0u8]);
+        }
+        let mut bytes = [0u8; 16];
+        bytes.copy_from_slice(&hasher.finalize()[..16]);
+        uuid::Builder::from_sha1_bytes(bytes).into_uuid()
+    })
+}
+
 #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, sqlx::Type)]
 #[sqlx(type_name = "MESSAGE_TYPE", rename_all = "lowercase")]
 #[serde(rename_all = "lowercase")]
@@ -26,8 +53,12 @@ pub struct FlowConversation {
     pub created_at: DateTime<Utc>,
     pub updated_at: DateTime<Utc>,
     pub created_by: String,
+    /// Started from the flow editor's test panel rather than a deployed run.
+    pub is_test: bool,
 }
 
+/// `is_test` is written on insert. An existing conversation of the other kind refuses the
+/// turn, so preview and deployed runs never share one.
 pub async fn get_or_create_conversation_with_id(
     tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
     w_id: &str,
@@ -35,9 +66,10 @@ pub async fn get_or_create_conversation_with_id(
     username: &str,
     title: &str,
     conversation_id: Uuid,
+    is_test: bool,
 ) -> Result<FlowConversation> {
     if let Some(existing) = lock_conversation(tx, w_id, conversation_id).await? {
-        return Ok(existing);
+        return same_kind(existing, is_test);
     }
 
     // Truncate title to 25 characters max
@@ -47,15 +79,16 @@ pub async fn get_or_create_conversation_with_id(
     // wins, the others wait on it, do nothing, and read the row it created.
     let created = sqlx::query_as!(
         FlowConversation,
-        "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title)
-         VALUES ($1, $2, $3, $4, $5)
+        "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title, is_test)
+         VALUES ($1, $2, $3, $4, $5, $6)
          ON CONFLICT (id) DO NOTHING
-         RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by",
+         RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test",
         conversation_id,
         w_id,
         flow_path,
         username,
-        title
+        title,
+        is_test
     )
     .fetch_optional(&mut **tx)
     .await?;
@@ -63,13 +96,29 @@ pub async fn get_or_create_conversation_with_id(
         return Ok(conversation);
     }
 
-    lock_conversation(tx, w_id, conversation_id)
+    // The concurrent first turn that won the insert may have been of the other kind.
+    let existing = lock_conversation(tx, w_id, conversation_id)
         .await?
         .ok_or_else(|| {
             crate::error::Error::BadRequest(format!(
                 "conversation {conversation_id} belongs to another workspace"
             ))
-        })
+        })?;
+    same_kind(existing, is_test)
+}
+
+/// `memory_id` is the caller's to choose, so a preview run could name a deployed
+/// conversation and the reverse. A conversation's kind is fixed at creation and nothing
+/// would show the mixing afterwards, so the turn is refused before it starts.
+fn same_kind(existing: FlowConversation, is_test: bool) -> Result<FlowConversation> {
+    if existing.is_test == is_test {
+        return Ok(existing);
+    }
+    Err(crate::error::Error::BadRequest(if existing.is_test {
+        "this conversation was started from the flow editor's test panel; start a new conversation to run the deployed flow".to_string()
+    } else {
+        "this conversation belongs to the deployed flow; start a new conversation to test from the flow editor".to_string()
+    }))
 }
 
 /// Locked, so a turn orders against retention collecting the conversation
@@ -83,7 +132,7 @@ async fn lock_conversation(
 ) -> Result<Option<FlowConversation>> {
     Ok(sqlx::query_as!(
         FlowConversation,
-        "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by
+        "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test
          FROM flow_conversation
          WHERE id = $1 AND workspace_id = $2
          FOR UPDATE",
@@ -94,6 +143,65 @@ async fn lock_conversation(
     .await?)
 }
 
+/// What a row carries beyond its text. A chat is rebuilt from its rows alone, without
+/// reading jobs, so every tool row carries the model's call and what the model got back: a
+/// Windmill tool's job holds the args its input transforms produced rather than the model's,
+/// and an MCP tool's call sits among every call of the turn in the agent's job. That job's
+/// `reasoning` is one string for the whole turn, where the rows keep it per iteration.
+#[derive(Debug, Clone, Default)]
+pub struct MessageExtras {
+    pub tool_arguments: Option<String>,
+    pub tool_result: Option<String>,
+    pub reasoning: Option<String>,
+    /// The files a user message carried; see `message_attachments`.
+    pub attachments: Vec<MessageAttachment>,
+}
+
+/// The most files a user message keeps references to; the rest are dropped.
+pub const MAX_MESSAGE_ATTACHMENTS: usize = 20;
+
+/// A file a user message carried, as the object-storage reference its run received.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+pub struct MessageAttachment {
+    /// The flow input that held it.
+    pub input: String,
+    pub s3: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub storage: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub filename: Option<String>,
+}
+
+/// The files a run's args carry for its user message: every top-level input other than
+/// `user_message` whose value is an object-storage reference or a list of them, in input
+/// name order, capped at `MAX_MESSAGE_ATTACHMENTS`. Only the reference is kept: `presigned`
+/// grants access to the file, and any other value may be the file's bytes.
+pub fn message_attachments(args: &HashMap<String, Box<RawValue>>) -> Vec<MessageAttachment> {
+    let mut inputs: Vec<_> = args
+        .iter()
+        .filter(|(name, _)| name.as_str() != "user_message")
+        .collect();
+    inputs.sort_by(|a, b| a.0.cmp(b.0));
+    inputs
+        .into_iter()
+        .flat_map(|(name, value)| {
+            serde_json::from_str::<S3Object>(value.get())
+                .map(|object| vec![object])
+                .or_else(|_| serde_json::from_str::<Vec<S3Object>>(value.get()))
+                .unwrap_or_default()
+                .into_iter()
+                .filter(|object| !object.s3.is_empty())
+                .map(move |object| MessageAttachment {
+                    input: name.clone(),
+                    s3: object.s3,
+                    storage: object.storage,
+                    filename: object.filename,
+                })
+        })
+        .take(MAX_MESSAGE_ATTACHMENTS)
+        .collect()
+}
+
 /// Add a message to a conversation using an existing transaction
 /// If the conversation doesn't exist, logs a warning and returns Ok (no error thrown)
 /// This allows memory_id to be used for agent memory without requiring a conversation
@@ -105,6 +213,7 @@ pub async fn add_message_to_conversation_tx(
     message_type: MessageType,
     step_name: Option<&str>,
     success: bool,
+    extras: Option<&MessageExtras>,
 ) -> Result<()> {
     // Check if conversation exists first
     let conversation_exists = sqlx::query!(
@@ -125,14 +234,21 @@ pub async fn add_message_to_conversation_tx(
 
     // Insert the message
     sqlx::query!(
-        "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success)
-         VALUES ($1, $2, $3, $4, $5, $6)",
+        "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success, tool_arguments, tool_result, reasoning, attachments)
+         VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)",
         conversation_id,
         message_type as MessageType,
         content,
         job_id,
         step_name,
-        success
+        success,
+        extras.and_then(|e| e.tool_arguments.as_deref()),
+        extras.and_then(|e| e.tool_result.as_deref()),
+        extras.and_then(|e| e.reasoning.as_deref()),
+        extras
+            .map(|e| &e.attachments)
+            .filter(|attachments| !attachments.is_empty())
+            .map(sqlx::types::Json) as Option<sqlx::types::Json<&Vec<MessageAttachment>>>
     )
     .execute(&mut **tx)
     .await?;
@@ -164,3 +280,67 @@ pub async fn delete_conversation_memory(
 
     Ok(())
 }
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use serde_json::{json, value::to_raw_value};
+
+    fn args(values: serde_json::Value) -> HashMap<String, Box<RawValue>> {
+        values
+            .as_object()
+            .unwrap()
+            .iter()
+            .map(|(name, value)| (name.clone(), to_raw_value(value).unwrap()))
+            .collect()
+    }
+
+    #[test]
+    fn keeps_only_object_storage_references() {
+        let attachments = message_attachments(&args(json!({
+            "user_message": { "s3": "not/an/attachment.png" },
+            "avatar": { "s3": "u/a.png", "storage": "secondary", "presigned": "https://signed" },
+            "files": [
+                { "s3": "u/b.pdf", "filename": "b.pdf" },
+                { "s3": "" }
+            ],
+            "photo": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==",
+            "count": 3
+        })));
+        assert_eq!(
+            serde_json::to_value(&attachments).unwrap(),
+            json!([
+                { "input": "avatar", "s3": "u/a.png", "storage": "secondary" },
+                { "input": "files", "s3": "u/b.pdf", "filename": "b.pdf" }
+            ])
+        );
+    }
+
+    #[test]
+    fn caps_the_references_of_one_message() {
+        let files: Vec<_> = (0..25)
+            .map(|i| json!({ "s3": format!("u/{i}.png") }))
+            .collect();
+        let attachments = message_attachments(&args(json!({ "files": files })));
+        assert_eq!(attachments.len(), MAX_MESSAGE_ATTACHMENTS);
+        assert_eq!(attachments.last().unwrap().s3, "u/19.png");
+    }
+
+    /// A string names a memory only within its workspace and flow; a uuid is used as is.
+    #[test]
+    fn memory_key_scopes_strings_but_not_uuids() {
+        let key = memory_key("ws", "f/support/triage", " customer-1 ");
+        assert_eq!(key, memory_key("ws", "f/support/triage", "customer-1"));
+        assert_ne!(
+            key,
+            memory_key("other_ws", "f/support/triage", "customer-1")
+        );
+        assert_ne!(key, memory_key("ws", "f/sales/triage", "customer-1"));
+        let conversation = Uuid::from_u128(7).to_string();
+        assert_eq!(memory_key("ws", "f/a", &conversation), Uuid::from_u128(7));
+        assert_eq!(
+            memory_key("other_ws", "f/b", &conversation),
+            Uuid::from_u128(7)
+        );
+    }
+}

backend/windmill-common/src/instance_config.rs

@@ -1398,8 +1398,14 @@ pub async fn sync_global_settings_declarative(
     crate::global_settings::parse_allowed_origins_setting(desired.get(origins_key))
         .map_err(|e| anyhow::anyhow!("{origins_key}: {e}"))?;
 
-    let mut diff = diff_global_settings(current, desired, ApplyMode::Replace);
-    crate::external_instance_pg::write_external_instance_pg_from_diff(db, &mut diff).await?;
+    let diff = diff_global_settings(current, desired, ApplyMode::Replace);
+    let external_pg_key = crate::global_settings::EXTERNAL_INSTANCE_PG_SETTING;
+    if diff.deletes.iter().any(|k| k == external_pg_key) {
+        crate::external_instance_pg::check_external_instance_pg_write(db, None).await?;
+    }
+    if let Some(value) = diff.upserts.get(external_pg_key) {
+        crate::external_instance_pg::check_external_instance_pg_write(db, Some(value)).await?;
+    }
     apply_settings_diff(db, &diff).await?;
 
     Ok(())

backend/windmill-common/src/lib.rs

@@ -1086,13 +1086,7 @@ impl PgDatabase {
                 if err_str.contains("password authentication failed for user")
                     && err_str.contains("custom_instance_user")
                 {
-                    // The external instance cluster has a `custom_instance_user` of its own, whose
-                    // password setup manages. Rotating the local one would break every instance
-                    // data table and fix nothing.
-                    let local = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
-                    let on_local_cluster = local.host == self.host
-                        && local.port.unwrap_or(5432) == self.port.unwrap_or(5432);
-                    if let Some(db) = main_db.filter(|_| on_local_cluster) {
+                    if let Some(db) = main_db {
                         tracing::warn!(
                             "custom_instance_user password auth failed, refreshing and retrying..."
                         );
@@ -1485,7 +1479,26 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> {
 }
 
 /// Drop a custom instance database: validate, terminate connections, DROP DATABASE, remove from global_settings.
+///
+/// Authorization: drops any instance database but Windmill's own and checks nothing. Callers MUST
+/// be superadmin, or have established the caller may drop this one — a fork's owner cleaning up
+/// its own copy that nothing else uses.
 pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Result<()> {
+    drop_custom_instance_database_keep_entry(db, dbname).await?;
+    sqlx::query!(
+        r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
+        dbname.trim()
+    )
+    .execute(db)
+    .await?;
+    Ok(())
+}
+
+/// [`drop_custom_instance_database`] leaving its registry entry, for a caller holding row locks in
+/// a transaction: the registry write has to go through that transaction, as waiting on another
+/// connection for a lock the transaction's own peers hold is a deadlock Postgres cannot see. Same
+/// authorization contract.
+pub async fn drop_custom_instance_database_keep_entry(db: &DB, dbname: &str) -> error::Result<()> {
     let dbname = dbname.trim();
     validate_dbname(dbname)?;
 
@@ -1531,14 +1544,6 @@ pub async fn drop_custom_instance_database(db: &DB, dbname: &str) -> error::Resu
         tracing::info!("Database '{}' does not exist, skipping drop", dbname);
     }
 
-    // Always remove from global_settings
-    sqlx::query!(
-        r#"UPDATE global_settings SET value = value #- ARRAY['databases', $1] WHERE name = 'custom_instance_pg_databases'"#,
-        dbname
-    )
-    .execute(db)
-    .await?;
-
     Ok(())
 }
 
@@ -1563,20 +1568,18 @@ pub(crate) fn instance_db_grants(dbname: &str) -> String {
     )
 }
 
-/// Re-apply [`instance_db_grants`] to a managed database provisioned before data table roles
-/// existed, whose grants carry no grant option. Connects as the cluster's administrator — the
-/// database and `public` schema owner — since only it can hand out an option it holds.
+/// Re-apply [`instance_db_grants`] to an instance database provisioned before data table roles
+/// existed, whose grants carry no grant option. Connects as the instance's own Postgres user —
+/// the database and `public` schema owner — since only it can hand out an option it holds.
 ///
-/// Authorization: reaches a managed database with the server's own credentials and checks
+/// Authorization: reaches an instance database with the server's own credentials and checks
 /// nothing. Callers MUST have authorized administration of `dbname` — superadmin, or an admin of
 /// the workspace governing a data table on it.
 pub async fn ensure_instance_db_grant_options_unchecked(
     db: &DB,
-    cluster: crate::datatable_roles::DatatableRoleCluster,
     dbname: &str,
 ) -> error::Result<()> {
-    crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, cluster, dbname)
-        .await
+    crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, dbname).await
 }
 
 /// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
@@ -1653,13 +1656,7 @@ pub async fn create_custom_instance_database(
     // A data table role can only reach a database it may CONNECT to, and PUBLIC's default CONNECT
     // would otherwise let every role in regardless of what this instance defines. Best-effort: a
     // failure here leaves the database usable as `admin`, and the next role change repairs it.
-    if let Err(e) = crate::datatable_roles::converge_connect_grants(
-        db,
-        crate::datatable_roles::DatatableRoleCluster::Instance,
-        dbname,
-    )
-    .await
-    {
+    if let Err(e) = crate::datatable_roles::converge_connect_grants(db, dbname).await {
         tracing::warn!("Could not set CONNECT grants on instance database '{dbname}': {e}");
     }
 
@@ -1667,62 +1664,39 @@ pub async fn create_custom_instance_database(
     Ok(())
 }
 
-/// The system's CA bundle file, for libpq clients that cannot take `sslrootcert=system`: that value
-/// needs libpq 16, and verify-full only.
-pub fn system_ca_bundle() -> Option<std::path::PathBuf> {
-    std::env::var_os("SSL_CERT_FILE")
-        .map(std::path::PathBuf::from)
-        .into_iter()
-        .chain(
-            [
-                "/etc/ssl/certs/ca-certificates.crt",
-                "/etc/pki/tls/certs/ca-bundle.crt",
-                "/etc/ssl/cert.pem",
-                "/etc/ssl/ca-bundle.pem",
-            ]
-            .map(std::path::PathBuf::from),
-        )
-        .find(|path| path.is_file())
-}
-
-/// Refuse a workspace member writing a fork copy into, or pointing a fork at, the managed database
-/// `dbname` of `kind`, unless `w_id` created it for that ([`create_custom_instance_database`], or
-/// its external instance counterpart) and nothing uses it yet. The `wm_fork_` prefix is no
-/// authorization: every database of a cluster answers to the same `custom_instance_user`, so a name
-/// is all it takes to reach another workspace's copy.
+/// Refuse a workspace member writing a fork copy into, or pointing a fork at, the instance database
+/// `dbname`, unless `w_id` created it for that ([`create_custom_instance_database`]) and nothing uses
+/// it yet. The `wm_fork_` prefix is no authorization: every instance database answers to the same
+/// `custom_instance_user`, so a name is all it takes to reach another workspace's copy.
 ///
-/// Authorization: reads the global registries and every workspace's settings, and names other
+/// Authorization: reads the global registry and every workspace's settings, and names other
 /// workspaces in its refusal. Callers MUST have authorized `w_id` for the caller first — a member
 /// of it forking or importing there — and MUST NOT call it on a workspace the caller is not in.
 pub async fn ensure_fork_database_available_to(
     db: &DB,
-    kind: workspaces::DataTableCatalogResourceType,
     dbname: &str,
     w_id: &str,
 ) -> error::Result<()> {
-    let created_for = match kind {
-        workspaces::DataTableCatalogResourceType::ExternalInstance => {
-            external_instance_pg::external_instance_databases(db)
-                .await?
-                .remove(dbname)
-                .and_then(|entry| entry.workspace_id)
-        }
-        _ => sqlx::query_scalar::<_, Option<String>>(
-            "SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
-             WHERE name = 'custom_instance_pg_databases'",
-        )
-        .bind(dbname)
-        .fetch_optional(db)
-        .await?
-        .flatten(),
-    };
+    let created_for = sqlx::query_scalar::<_, Option<String>>(
+        "SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
+         WHERE name = 'custom_instance_pg_databases'",
+    )
+    .bind(dbname)
+    .fetch_optional(db)
+    .await?
+    .flatten();
     if created_for.as_deref() != Some(w_id) {
         return Err(Error::BadRequest(format!(
             "Database '{dbname}' was not created for a fork of workspace '{w_id}'"
         )));
     }
-    let uses =
-        workspaces::managed_database_uses(&mut *db.acquire().await?, kind, dbname, None).await?;
+    let uses = workspaces::managed_database_uses(
+        &mut *db.acquire().await?,
+        workspaces::DataTableCatalogResourceType::Instance,
+        dbname,
+        None,
+    )
+    .await?;
     if !uses.is_empty() {
         return Err(Error::BadRequest(format!(
             "Database '{dbname}' is already in use: {}",

backend/windmill-common/src/utils.rs

@@ -976,18 +976,33 @@ fn six_fields_hint(schedule_str: &str, version: Option<&str>, seconds_required:
 }
 
 impl ScheduleType {
+    /// `NotFound` means the expression has no run left (an expired year, an impossible
+    /// date), and schedule pushes disable the schedule on it. Every other error must stay
+    /// transient: croner fails across a DST jump longer than an hour (Antarctica/Troll)
+    /// and succeeds again once the jump has passed.
     pub fn find_next(
         &self,
         starting_from: &chrono::DateTime<chrono_tz::Tz>,
-    ) -> chrono::DateTime<chrono_tz::Tz> {
+    ) -> Result<chrono::DateTime<chrono_tz::Tz>> {
+        let no_run_left = || {
+            Error::NotFound(format!(
+                "cron: the schedule has no run left after {}",
+                starting_from.format("%Y-%m-%d %H:%M:%S %Z")
+            ))
+        };
         match self {
             ScheduleType::Croner(croner_schedule) => croner_schedule
                 .find_next_occurrence(starting_from, false)
-                .expect("cron: a schedule should have a next event"),
-            ScheduleType::Cron(schedule) => schedule
-                .after(starting_from)
-                .next()
-                .expect("cron: a schedule should have a next event"),
+                .map_err(|e| match e {
+                    croner::errors::CronError::TimeSearchLimitExceeded => no_run_left(),
+                    e => Error::internal_err(format!(
+                        "cron: could not compute the run after {}: {e}",
+                        starting_from.format("%Y-%m-%d %H:%M:%S %Z")
+                    )),
+                }),
+            ScheduleType::Cron(schedule) => {
+                schedule.after(starting_from).next().ok_or_else(no_run_left)
+            }
         }
     }
 
@@ -1709,6 +1724,22 @@ mod tests {
         assert!(!err.contains("6 fields"), "{err}");
     }
 
+    #[test]
+    fn find_next_reports_only_a_cron_with_no_run_left_as_not_found() {
+        use chrono::TimeZone;
+        let troll: chrono_tz::Tz = "Antarctica/Troll".parse().unwrap();
+        // Troll's clocks jump from 01:00 to 03:00 on the last Sunday of March.
+        let before_jump = troll.with_ymd_and_hms(2027, 3, 28, 0, 30, 0).unwrap();
+
+        let expired = ScheduleType::from_str("0 0 9 1 1 * 2026", Some("v1"), true).unwrap();
+        let err = expired.find_next(&before_jump).unwrap_err();
+        assert!(matches!(err, Error::NotFound(_)), "{err}");
+
+        let across_jump = ScheduleType::from_str("0 30 1 * * *", Some("v2"), true).unwrap();
+        let err = across_jump.find_next(&before_jump).unwrap_err();
+        assert!(!matches!(err, Error::NotFound(_)), "{err}");
+    }
+
     /// A worker that restarts must land on the exact same name to reclaim its `worker_ping`
     /// row, while still never colliding with the other workers of its own process. The
     /// suffix must also stay a single `-` segment, which is what the interactive shell tag

backend/windmill-common/src/workspaces.rs

@@ -1397,18 +1397,6 @@ pub enum DataTableCatalogResourceType {
     #[strum(serialize = "postgres")]
     Postgresql,
     Instance,
-    /// On the external instance cluster ([`crate::external_instance_pg`]). Enterprise Edition.
-    #[serde(rename = "external_instance")]
-    #[strum(serialize = "external_instance")]
-    ExternalInstance,
-}
-
-impl DataTableCatalogResourceType {
-    /// A database Windmill created and administers, on its own cluster or the external one, as
-    /// opposed to one a user brought as a resource.
-    pub fn is_windmill_managed(self) -> bool {
-        matches!(self, Self::Instance | Self::ExternalInstance)
-    }
 }
 
 /// Build a self-teaching error for an unresolved `datatable://<name>` reference.
@@ -1551,14 +1539,13 @@ pub async fn lock_fork_datatables(conn: &mut sqlx::PgConnection, w_id: &str) ->
 }
 
 impl GoverningDatatable {
-    /// The Windmill-managed cluster whose data table roles this entry can use. `None` for a
-    /// resource-backed one: roles are logins Windmill creates, and it creates none on a host a
-    /// workspace admin chose.
-    pub fn role_cluster(&self) -> Option<crate::datatable_roles::DatatableRoleCluster> {
+    /// Backed by the Windmill instance's own Postgres, which is the only substrate data table
+    /// roles apply to.
+    pub fn is_instance(&self) -> bool {
         self.datatable
             .database
             .as_ref()
-            .and_then(|d| crate::datatable_roles::DatatableRoleCluster::of(d.resource_type))
+            .is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
     }
 }
 
@@ -1693,8 +1680,7 @@ pub async fn resolve_workspace_governing_datatables(
 }
 
 /// Build the `admin` connection for a governing entry: `custom_instance_user` for an instance
-/// database, on Windmill's cluster or the external one; the user's own resource for a BYO-postgres
-/// one.
+/// database, the user's own resource for a BYO-postgres one.
 async fn resolve_datatable_connection_unchecked(
     db: &DB,
     governing: &GoverningDatatable,
@@ -1705,16 +1691,7 @@ async fn resolve_datatable_connection_unchecked(
         .database
         .as_ref()
         .expect("a governing entry owns a database");
-    if database.resource_type == DataTableCatalogResourceType::ExternalInstance {
-        let pg_creds = crate::external_instance_pg::external_instance_connection_unchecked(
-            db,
-            &database.resource_path,
-            replication,
-        )
-        .await?;
-        serde_json::to_value(&pg_creds)
-            .map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))
-    } else if database.resource_type == DataTableCatalogResourceType::Instance {
+    if database.resource_type == DataTableCatalogResourceType::Instance {
         let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
         pg_creds.dbname = database.resource_path.clone();
         if replication {
@@ -2258,10 +2235,6 @@ pub enum DucklakeCatalogResourceType {
     Postgresql,
     Mysql,
     Instance,
-    /// On the external instance cluster ([`crate::external_instance_pg`]). Enterprise Edition.
-    #[serde(rename = "external_instance")]
-    #[strum(serialize = "external_instance")]
-    ExternalInstance,
 }
 
 #[derive(Deserialize, Serialize)]
@@ -2789,16 +2762,7 @@ async fn ducklake_conn_data(
     let ducklake = serde_json::from_value::<Ducklake>(ducklake)?;
 
     let catalog_resource =
-        if ducklake.catalog.resource_type == DucklakeCatalogResourceType::ExternalInstance {
-            let pg_creds = crate::external_instance_pg::external_instance_connection_unchecked(
-                db,
-                &ducklake.catalog.resource_path,
-                false,
-            )
-            .await?;
-            serde_json::to_value(&pg_creds)
-                .map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))?
-        } else if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
+        if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
             let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
             pg_creds.dbname = ducklake.catalog.resource_path.clone();
             pg_creds.user = Some("custom_instance_user".to_string());
@@ -3179,14 +3143,6 @@ async fn register_fork_ducklake_namespace(
     {
         return Ok(());
     }
-    let mut tx = db.begin().await?;
-    // A row naming an external database counts as a use of it. Written under the lock a drop takes,
-    // and only while the database is still registered, so a drop cannot slip in between the
-    // settings this attach resolved and the row that protects the database.
-    if let Some(dbname) = catalog.strip_prefix("external_instance:") {
-        crate::external_instance_pg::ensure_external_instance_database_registered(&mut tx, dbname)
-            .await?;
-    }
     sqlx::query!(
         "INSERT INTO fork_ducklake_namespace
            (workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)
@@ -3201,10 +3157,9 @@ async fn register_fork_ducklake_namespace(
         &storage_ref,
         data_path,
     )
-    .execute(&mut *tx)
+    .execute(db)
     .await
     .map_err(|e| Error::internal_err(format!("registering fork ducklake namespace: {e:#}")))?;
-    tx.commit().await?;
     let mut locations = FORK_DUCKLAKE_REGISTERED
         .get(w_id)
         .filter(|(_, exp)| *exp > now)

backend/windmill-queue/src/schedule.rs

@@ -166,13 +166,10 @@ pub async fn push_scheduled_job<'c>(
         }
     };
 
-    let next = sched.find_next(&starting_from);
-    // println!("next event ({:?}): {}", tz, next);
-    // println!("next event(UTC): {}", next.with_timezone(&chrono::Utc));
+    let next = sched.find_next(&starting_from)?;
 
     // Scheduled events must be stored in the database in UTC
     let next = next.with_timezone(&chrono::Utc);
-    // panic!("next: {}", next);
     let already_exists: bool = sqlx::query_scalar!(
         // Query plan:
         // - use of the `ix_v2_job_root_by_path` index; hence the `parent_job IS NULL` clause.

backend/windmill-queue/tests/schedule_push.rs

@@ -921,6 +921,47 @@ mod schedule_push {
         Ok(())
     }
 
+    // -----------------------------------------------------------------------
+    // try_schedule_next_job: a cron with no run left disables the schedule
+    // -----------------------------------------------------------------------
+
+    #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))]
+    async fn test_cron_with_no_run_left_disables_schedule(
+        db: Pool<Postgres>,
+    ) -> anyhow::Result<()> {
+        sqlx::query(
+            "INSERT INTO schedule (workspace_id, path, edited_by, edited_at, schedule, timezone, enabled, script_path, is_flow, email, extra_perms, ws_error_handler_muted, no_flow_overlap, permissioned_as, cron_version)
+             VALUES ('test-workspace', 'f/system/test_schedule', 'test-user', now(), '0 0 9 1 1 * 2020', 'UTC', true, 'f/system/test_script', false, 'test@windmill.dev', '{}', false, false, 'u/test-user', 'v1')"
+        )
+        .execute(&db)
+        .await?;
+
+        let schedule = make_schedule(|s| {
+            s.schedule = "0 0 9 1 1 * 2020".to_string();
+            s.cron_version = Some("v1".to_string());
+        });
+        let job = make_completed_job(&schedule);
+
+        let tx = db.begin().await?;
+        let (tx, err) =
+            try_schedule_next_job(&db, tx, &job, &schedule, &schedule.script_path).await;
+        assert!(err.is_none(), "completion must go through, got: {err:?}");
+        tx.commit().await?;
+
+        assert_eq!(count_queued_jobs(&db).await, 0);
+        let (enabled, error): (bool, Option<String>) = sqlx::query_as(
+            "SELECT enabled, error FROM schedule WHERE workspace_id = 'test-workspace' AND path = 'f/system/test_schedule'",
+        )
+        .fetch_one(&db)
+        .await?;
+        assert!(!enabled, "schedule with no run left must be disabled");
+        assert!(
+            error.as_deref().is_some_and(|e| e.contains("no run left")),
+            "error should say why, got: {error:?}"
+        );
+        Ok(())
+    }
+
     // -----------------------------------------------------------------------
     // try_schedule_next_job: disabled schedule leaves no side effects
     // -----------------------------------------------------------------------

backend/windmill-types/src/flows.rs

@@ -1100,7 +1100,8 @@ pub enum FlowModuleValue {
         omit_output_from_conversation: bool,
         /// When set, the agent brain config (provider/model/system prompt/etc.) and tools are
         /// resolved at runtime from this `ai_agent` resource path (hybrid linking). The module's
-        /// `input_transforms` then only carry the flow-local inputs (user_message/user_attachments).
+        /// `input_transforms` then only carry the flow-local inputs: user_message,
+        /// user_attachments, enabled_tools and the history inputs memory_id and previous_messages.
         #[serde(default, skip_serializing_if = "Option::is_none")]
         agent: Option<String>,
         /// Binds an agent's tools to *this* flow's context, keyed by tool id then input key, without

backend/windmill-worker/src/ai/tools.rs

@@ -33,7 +33,7 @@ use windmill_common::{
     client::AuthedClient,
     db::DB,
     error::Error,
-    flow_conversations::MessageType,
+    flow_conversations::{MessageExtras, MessageType},
     flow_status::AgentAction,
     flows::FlowModuleValue,
     worker::{to_raw_value, Connection},
@@ -74,6 +74,9 @@ pub struct ToolExecutionContext<'a> {
     pub stream_event_processor: Option<&'a StreamEventProcessor>,
     pub flow_context: &'a mut FlowContext,
     pub omit_output_from_conversation: bool,
+    /// The thinking that led to this round's calls, stored on the first tool row written.
+    /// None when the round wrote text, whose row carries it.
+    pub reasoning: Option<String>,
     pub previous_result: &'a Option<Box<RawValue>>,
     pub id_context: &'a Option<crate::js_eval::IdContext>,
 
@@ -235,9 +238,24 @@ async fn execute_mcp_tool_call(
                 update_flow_status_module_with_actions_success(ctx.db, parent_job, true).await?;
             }
 
-            // Add tool message to conversation if chat_input_enabled
+            // An MCP tool runs inside the agent's job, whose result holds every call of the
+            // turn and nothing tying one of them to this row: same job id for all of them,
+            // no call id on the row. Kept here so the card shows this call — and the row
+            // names that job, so retention sweeps it with every other row of the turn.
             let content = format!("Used {} tool", tool_call.function.name);
-            add_tool_message_to_chat(ctx, None, &content, true).await;
+            let agent_job_id = ctx.job.id;
+            add_tool_message_to_chat(
+                ctx,
+                Some(agent_job_id),
+                &content,
+                true,
+                Some(MessageExtras {
+                    tool_arguments: Some(tool_call.function.arguments.clone()),
+                    tool_result: Some(result_str),
+                    ..Default::default()
+                }),
+            )
+            .await;
         }
         Err(e) => {
             let error_msg = format!("MCP tool error: {}", e);
@@ -271,8 +289,23 @@ async fn execute_mcp_tool_call(
                 update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?;
             }
 
-            // Add tool message to conversation if chat_input_enabled
-            add_tool_message_to_chat(ctx, None, &error_msg, false).await;
+            // Add tool message to conversation if chat_input_enabled. The row is worded from
+            // the tool, like every other tool row, and the error it failed with is its result
+            // — the one field a call that produced nothing else still has something to put in.
+            let agent_job_id = ctx.job.id;
+            let content = format!("Error executing {}", tool_name);
+            add_tool_message_to_chat(
+                ctx,
+                Some(agent_job_id),
+                &content,
+                false,
+                Some(MessageExtras {
+                    tool_arguments: Some(tool_call.function.arguments.clone()),
+                    tool_result: Some(error_msg.clone()),
+                    ..Default::default()
+                }),
+            )
+            .await;
         }
     }
 
@@ -680,8 +713,8 @@ async fn handle_tool_execution_error(
         update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?;
     }
 
-    // Add tool message to conversation if chat_input_enabled (error case)
-    add_tool_message_to_chat(ctx, Some(job_id), &error_message, false).await;
+    let (content, extras) = windmill_tool_row(tool_call, false, &error_message);
+    add_tool_message_to_chat(ctx, Some(job_id), &content, false, Some(extras)).await;
 
     Ok(())
 }
@@ -782,13 +815,17 @@ async fn handle_tool_execution_success(
         ..Default::default()
     });
 
-    // Stream tool result (success case)
+    let (content, extras) = windmill_tool_row(tool_call, success, &tool_result);
+
+    // The job ran; whether it ran successfully is `success`, and the row stored below is
+    // worded from it. The stream has to carry the same value, or the card the reader watches
+    // and the row that replaces it describe the same call differently.
     if let Some(stream_event_processor) = ctx.stream_event_processor {
         let tool_result_event = StreamingEvent::ToolResult {
             call_id: tool_call.id.clone(),
             function_name: tool_call.function.name.clone(),
             result: tool_result,
-            success: true,
+            success,
         };
         stream_event_processor
             .send(tool_result_event, final_events_str)
@@ -799,28 +836,56 @@ async fn handle_tool_execution_success(
         update_flow_status_module_with_actions_success(ctx.db, parent_job, success).await?;
     }
 
-    // Add tool message to conversation if chat_input_enabled
+    add_tool_message_to_chat(ctx, Some(job_id), &content, success, Some(extras)).await;
+
+    Ok(())
+}
+
+/// A Windmill tool's conversation row: worded from the tool, carrying the model's call and
+/// the exact text the model got back, the same text agent memory keeps for that tool
+/// message, so a card needs no job fetch. The call is the model's arguments, not the job's
+/// args: the step's input transforms add inputs the model never wrote.
+fn windmill_tool_row(
+    tool_call: &OpenAIToolCall,
+    success: bool,
+    sent_to_model: &str,
+) -> (String, MessageExtras) {
     let content = if success {
         format!("Used {} tool", tool_call.function.name)
     } else {
         format!("Error executing {}", tool_call.function.name)
     };
-
-    add_tool_message_to_chat(ctx, Some(job_id), &content, success).await;
-
-    Ok(())
+    let extras = MessageExtras {
+        tool_arguments: Some(tool_call.function.arguments.clone()),
+        tool_result: Some(sent_to_model.to_string()),
+        ..Default::default()
+    };
+    (content, extras)
 }
 
 /// Add tool message to conversation if chat is enabled
 async fn add_tool_message_to_chat(
     ctx: &mut ToolExecutionContext<'_>,
+    // The job this row belongs to: the tool's own where it has one, else the agent's, which
+    // is the job it ran inside. Every row names one so that retention collects the whole
+    // turn — `delete_jobs` removes messages by `job_id = ANY(..)` (there is no FK on the
+    // column; `drop_v2_job_side_table_cascades` dropped it), and a row naming no job would
+    // survive every purge and leave a conversation that can never become empty.
     tool_job_id: Option<Uuid>,
     content: &str,
     success: bool,
+    // The model's call and what it got back; every tool row carries both.
+    extras: Option<MessageExtras>,
 ) {
     if ctx.omit_output_from_conversation {
         return;
     }
+    let extras = match ctx.reasoning.take() {
+        Some(reasoning) => {
+            Some(MessageExtras { reasoning: Some(reasoning), ..extras.unwrap_or_default() })
+        }
+        None => extras,
+    };
 
     let chat_enabled = ctx
         .flow_context
@@ -835,41 +900,74 @@ async fn add_tool_message_to_chat(
             .as_ref()
             .and_then(|fs| fs.memory_id)
         {
-            let db_clone = ctx.db.clone();
             let effective_step_id = ctx
                 .flow_step_id_override
                 .or(ctx.job.flow_step_id.as_deref());
             let step_name = get_step_name_from_flow(ctx.summary.as_deref(), effective_step_id);
-            let content = content.to_string();
 
-            // Spawn task because we do not need to wait for the result
-            tokio::spawn(async move {
-                if let Err(e) = add_message_to_conversation(
-                    &db_clone,
-                    &memory_id,
-                    tool_job_id,
-                    &content,
-                    MessageType::Tool,
-                    &step_name,
-                    success,
-                )
-                .await
-                {
-                    tracing::warn!(
-                        "Failed to add tool message to conversation {}: {}",
-                        memory_id,
-                        e
-                    );
-                }
-            });
+            // Awaited, not spawned: `created_seq` is the transcript's order, so a round's rows
+            // must commit in the order of its calls. Calls run one after another; running them
+            // in parallel would need their rows written in call order all the same.
+            if let Err(e) = add_message_to_conversation(
+                ctx.db,
+                &memory_id,
+                tool_job_id,
+                content,
+                MessageType::Tool,
+                &step_name,
+                success,
+                extras.as_ref(),
+            )
+            .await
+            {
+                tracing::warn!(
+                    "Failed to add tool message to conversation {}: {}",
+                    memory_id,
+                    e
+                );
+            }
         }
     }
 }
 
 #[cfg(test)]
 mod tests {
-    use super::extract_ai_agent_output;
+    use super::{extract_ai_agent_output, windmill_tool_row};
     use serde_json::value::RawValue;
+    use windmill_ai::ai_types::{OpenAIFunction, OpenAIToolCall};
+
+    #[test]
+    fn a_windmill_tool_row_carries_the_models_call_and_what_it_got_back() {
+        let tool_call = OpenAIToolCall {
+            id: "call_1".to_string(),
+            function: OpenAIFunction {
+                name: "get_price".to_string(),
+                arguments: r#"{"item":"widget"}"#.to_string(),
+            },
+            r#type: "function".to_string(),
+            extra_content: None,
+        };
+
+        let (content, extras) = windmill_tool_row(&tool_call, true, r#"{"price":42}"#);
+        assert_eq!(content, "Used get_price tool");
+        assert_eq!(
+            extras.tool_arguments.as_deref(),
+            Some(r#"{"item":"widget"}"#)
+        );
+        assert_eq!(extras.tool_result.as_deref(), Some(r#"{"price":42}"#));
+
+        let (content, extras) =
+            windmill_tool_row(&tool_call, false, "Error running tool: ExecutionErr: boom");
+        assert_eq!(content, "Error executing get_price");
+        assert_eq!(
+            extras.tool_arguments.as_deref(),
+            Some(r#"{"item":"widget"}"#)
+        );
+        assert_eq!(
+            extras.tool_result.as_deref(),
+            Some("Error running tool: ExecutionErr: boom")
+        );
+    }
 
     #[test]
     fn extracts_only_the_output_of_an_agent_result() {

backend/windmill-worker/src/ai/utils.rs

@@ -13,7 +13,7 @@ use windmill_common::flows::FlowModuleValue;
 use windmill_common::{
     db::DB,
     error::Error,
-    flow_conversations::{add_message_to_conversation_tx, MessageType},
+    flow_conversations::{add_message_to_conversation_tx, MessageExtras, MessageType},
     flow_status::AgentAction,
     flows::{InputTransform, Step},
     jobs::JobKind,
@@ -154,6 +154,8 @@ pub async fn get_flow_job_runnable_and_raw_flow(
 pub struct FlowContext {
     pub flow_inputs: Option<HashMap<String, Box<RawValue>>>,
     pub flow_status: Option<windmill_common::flow_status::FlowStatus>,
+    /// Path of the flow the run started from, which scopes a string memory id.
+    pub flow_path: Option<String>,
 }
 
 /// Get flow context (chat settings + args + flow_status) from root flow's job data
@@ -171,7 +173,8 @@ pub async fn get_flow_context(db: &DB, job: &MiniPulledJob) -> FlowContext {
         r#"
         SELECT
             j.args as "args: Json<HashMap<String, Box<RawValue>>>",
-            js.flow_status as "flow_status: Json<windmill_common::flow_status::FlowStatus>"
+            js.flow_status as "flow_status: Json<windmill_common::flow_status::FlowStatus>",
+            j.runnable_path
         FROM v2_job_status js
         INNER JOIN v2_job j ON j.id = js.id
         WHERE js.id = $1
@@ -184,6 +187,7 @@ pub async fn get_flow_context(db: &DB, job: &MiniPulledJob) -> FlowContext {
         Ok(Some(row)) => FlowContext {
             flow_inputs: row.args.map(|j| j.0),
             flow_status: row.flow_status.map(|j| j.0),
+            flow_path: row.runnable_path,
         },
         Ok(None) => {
             tracing::warn!(
@@ -209,6 +213,7 @@ pub async fn add_message_to_conversation(
     message_type: MessageType,
     step_name: &Option<String>,
     success: bool,
+    extras: Option<&MessageExtras>,
 ) -> Result<(), Error> {
     let mut tx = db.begin().await?;
     add_message_to_conversation_tx(
@@ -219,6 +224,7 @@ pub async fn add_message_to_conversation(
         message_type,
         step_name.as_deref(),
         success,
+        extras,
     )
     .await?;
     tx.commit().await?;

backend/windmill-worker/src/ai_executor.rs

@@ -44,7 +44,7 @@ use windmill_common::{
     client::AuthedClient,
     db::DB,
     error::{self, Error},
-    flow_conversations::MessageType,
+    flow_conversations::{memory_key, MessageExtras, MessageType},
     flow_status::AgentAction,
     flows::{AgentTool, FlowModule, FlowModuleValue, InputTransform, ToolValue},
     get_latest_hash_for_path,
@@ -111,6 +111,148 @@ fn prepare_auto_memory_messages_for_persistence(
     non_system_messages[start_idx..].to_vec()
 }
 
+/// The inputs a linked step supplies for itself; the resource holds the rest of the brain.
+const FLOW_LOCAL_AGENT_KEYS: [&str; 5] = [
+    "user_message",
+    "user_attachments",
+    "enabled_tools",
+    "memory_id",
+    "previous_messages",
+];
+
+/// The flow-local inputs that name a conversation, which a saved agent never carries.
+const STEP_HISTORY_KEYS: [&str; 2] = ["memory_id", "previous_messages"];
+
+/// Where one agent invocation's history comes from.
+#[derive(Debug)]
+enum HistorySource<'a> {
+    /// Supplied by the flow and replayed as is: memory is neither read nor written.
+    Messages(&'a [OpenAIMessage]),
+    Window {
+        memory_id: Uuid,
+        context_length: usize,
+    },
+    Stateless,
+}
+
+/// A step's memory id counts only as the step authored it. A static empty value is a form
+/// placeholder, so it reads as unset rather than as an expression that evaluated to nothing, which
+/// runs without memory; an AI-filled value would let the model choose which memory the agent reads.
+fn keep_authored_memory_id(
+    args: &mut AIAgentArgs,
+    step_input_transforms: &HashMap<String, InputTransform>,
+) {
+    match step_input_transforms.get("memory_id") {
+        Some(InputTransform::Javascript { .. }) => {}
+        Some(InputTransform::Static { .. }) if args.memory_id.as_deref() != Some("") => {}
+        _ => args.memory_id = None,
+    }
+}
+
+/// Reconciles the step's history inputs, the agent's memory policy and the run's memory id. A step
+/// holds one of two shapes: an older `auto` or `manual` memory, read as the editor that wrote it
+/// meant it, or the current setting plus the step's own history inputs. Also returns lines for the
+/// job log: an input that went unused, or a policy that remembers ending up stateless.
+fn resolve_history_source<'a>(
+    args: &'a AIAgentArgs,
+    run_memory_id: Option<Uuid>,
+    workspace_id: &str,
+    flow_path: &str,
+) -> (HistorySource<'a>, Vec<&'static str>) {
+    let mut notes = Vec::new();
+    let no_memory_id = "No memory id was passed to this run, so the agent runs without memory.";
+    match &args.memory {
+        // The step's own history inputs came after these, so a step that still holds one reads it
+        // alone: what it did before the editor offered them is what it keeps doing.
+        Some(Memory::Manual { messages }) => {
+            note_unread_step_inputs(&mut notes, args);
+            (HistorySource::Messages(messages), notes)
+        }
+        Some(Memory::Auto { context_length, memory_id }) => {
+            note_unread_step_inputs(&mut notes, args);
+            // An id baked in at save time only ever applied when the run carried none.
+            match run_memory_id.or(*memory_id) {
+                Some(memory_id) => (
+                    HistorySource::Window { memory_id, context_length: *context_length },
+                    notes,
+                ),
+                None => {
+                    notes.push(no_memory_id);
+                    (HistorySource::Stateless, notes)
+                }
+            }
+        }
+        Some(Memory::Window { context_length }) => {
+            if args
+                .previous_messages
+                .as_ref()
+                .is_some_and(|messages| !messages.is_empty())
+            {
+                notes.push("Managed memory is on, so this step's previous messages are ignored.");
+            }
+            let memory_id = match args.memory_id.as_deref() {
+                Some("") => {
+                    notes.push(
+                        "This step's memory id evaluated to an empty value, so the agent runs without memory.",
+                    );
+                    return (HistorySource::Stateless, notes);
+                }
+                Some(step_memory_id) => memory_key(workspace_id, flow_path, step_memory_id),
+                None => match run_memory_id {
+                    Some(memory_id) => memory_id,
+                    None => {
+                        notes.push(no_memory_id);
+                        return (HistorySource::Stateless, notes);
+                    }
+                },
+            };
+            (
+                HistorySource::Window { memory_id, context_length: *context_length },
+                notes,
+            )
+        }
+        Some(Memory::Off) | None => {
+            if args.memory_id.as_deref().is_some_and(|id| !id.is_empty()) {
+                notes.push("Managed memory is off, so this step's memory id is ignored.");
+            }
+            match &args.previous_messages {
+                Some(messages) => (HistorySource::Messages(messages), notes),
+                None => (HistorySource::Stateless, notes),
+            }
+        }
+    }
+}
+
+/// An older memory setting reads neither history input, which is only visible in the job log: the
+/// editor offers them on a step that has been moved to the current settings.
+fn note_unread_step_inputs(notes: &mut Vec<&'static str>, args: &AIAgentArgs) {
+    if args.memory_id.as_deref().is_some_and(|id| !id.is_empty()) {
+        notes.push("This step uses an older memory setting, so its memory id is not read.");
+    }
+    if args
+        .previous_messages
+        .as_ref()
+        .is_some_and(|messages| !messages.is_empty())
+    {
+        notes
+            .push("This step uses an older memory setting, so its previous messages are not read.");
+    }
+}
+
+/// Whether a request has something to ask the model. Only text output sends previous messages, so
+/// an image prompt comes from the user message alone. An empty list is no conversation, except
+/// under a legacy `manual` memory, which ran on whatever list it held.
+fn has_prompt(
+    history: &HistorySource,
+    has_user_message: bool,
+    is_text_output: bool,
+    legacy_list: bool,
+) -> bool {
+    has_user_message
+        || (is_text_output
+            && (legacy_list || matches!(history, HistorySource::Messages(m) if !m.is_empty())))
+}
+
 fn find_module_by_id(
     modules: &Vec<FlowModule>,
     target_id: &str,
@@ -136,14 +278,16 @@ async fn find_ai_agent_tool_module_in_parent_agent(
         return Ok(None);
     };
 
-    let FlowModuleValue::AIAgent { tools, agent, .. } = parent_agent_module.get_value()? else {
+    let FlowModuleValue::AIAgent { tools, agent, tool_inputs, .. } =
+        parent_agent_module.get_value()?
+    else {
         return Ok(None);
     };
 
     // A linked parent carries no tools on the module (they live in the resource, resolved only in
     // the main execution branch). Resolve them from the resource here too, so a nested agent tool
     // of a saved+linked agent can still be located when it runs as its own job.
-    let tools = if let Some(agent_ref) = agent.as_deref() {
+    let mut tools = if let Some(agent_ref) = agent.as_deref() {
         let agent_path = agent_ref
             .trim_start_matches("$res:")
             .trim_start_matches("res://");
@@ -170,6 +314,9 @@ async fn find_ai_agent_tool_module_in_parent_agent(
     } else {
         tools
     };
+    // The nested job reads its history inputs from the tool's transforms, which must carry the
+    // host flow's bindings as the parent evaluated them.
+    overlay_tool_inputs(&mut tools, &tool_inputs);
 
     for tool in tools {
         if tool.id == tool_module_id {
@@ -456,6 +603,7 @@ pub async fn handle_ai_agent_job(
         omit_output_from_conversation,
         agent,
         tool_inputs,
+        input_transforms: step_input_transforms,
         ..
     } = module.get_value()?
     else {
@@ -466,9 +614,11 @@ pub async fn handle_ai_agent_job(
 
     // A linked step takes its brain and tools from the resource and keeps only its own flow-local
     // inputs. The brain and the roster stay rigid; what the step binds to this flow is the message
-    // it asks, which of those tools this use may call, the conversation it is part of, and the
-    // tools' own inputs — the last overlaid from `tool_inputs` below.
-    let (args, tools): (AIAgentArgs, Vec<AgentTool>) = if let Some(agent_ref) = agent.as_deref() {
+    // it asks, which of those tools this use may call, the conversation it is part of (its memory
+    // id and previous messages), and the tools' own inputs — the last overlaid from `tool_inputs`
+    // below.
+    let (mut args, tools): (AIAgentArgs, Vec<AgentTool>) = if let Some(agent_ref) = agent.as_deref()
+    {
         let agent_path = agent_ref
             .trim_start_matches("$res:")
             .trim_start_matches("res://");
@@ -500,6 +650,12 @@ pub async fn handle_ai_agent_job(
             None => Vec::new(),
         };
         overlay_tool_inputs(&mut tools, &tool_inputs);
+        // The resource is not validated against a schema, so a history input it happens to carry
+        // is dropped before interpolation, where a bad `$res:` in it would fail the step. The
+        // other flow-local keys stay: a resource's own user message is the step's fallback.
+        for key in STEP_HISTORY_KEYS {
+            config.remove(key);
+        }
         let brain = transform_json_value(
             "ai_agent",
             client,
@@ -521,7 +677,7 @@ pub async fn handle_ai_agent_job(
         // Only after interpolating the resource: these are caller-controlled and already resolved by
         // build_args_map, so passing them through it again would expand contextual values —
         // `$WM_TOKEN` in a user message would reach the model provider.
-        for key in ["user_message", "user_attachments", "enabled_tools"] {
+        for key in FLOW_LOCAL_AGENT_KEYS {
             if let Some(v) = local_args.get(key) {
                 brain.insert(
                     key.to_string(),
@@ -546,6 +702,8 @@ pub async fn handle_ai_agent_job(
         (args, tools)
     };
 
+    keep_authored_memory_id(&mut args, &step_input_transforms);
+
     // Nesting is capped at flow → agent → nested agent. When this job is itself a nested tool,
     // a linked resource's tool set may still contain AIAgent tools (the editor can't constrain a
     // shared resource); don't advertise them — invoking one would only fail the depth check as a
@@ -1010,8 +1168,18 @@ pub async fn run_agent(
     // Fetch flow context for input transforms context, chat and memory
     let mut flow_context = get_flow_context(db, job).await;
 
-    // Determine if we're using manual messages (which bypasses memory)
-    let use_manual_messages = matches!(args.memory, Some(Memory::Manual { .. }));
+    // The run's memory id is also the chat conversation id, which a step's own memory id never
+    // replaces.
+    let conversation_id = flow_context
+        .flow_status
+        .as_ref()
+        .and_then(|fs| fs.memory_id);
+    let (history, history_notes) = resolve_history_source(
+        args,
+        conversation_id,
+        &job.workspace_id,
+        flow_context.flow_path.as_deref().unwrap_or_default(),
+    );
 
     // Check if user_message is provided and non-empty
     let has_user_message = args
@@ -1020,63 +1188,63 @@ pub async fn run_agent(
         .map(|m| !m.is_empty())
         .unwrap_or(false);
 
-    // Validate: at least one of memory with manual messages or user_message must be provided
-    if !use_manual_messages && !has_user_message {
-        return Err(Error::internal_err(
-            "Either 'memory' with manual messages or 'user_message' must be provided".to_string(),
-        ));
-    }
-
     let is_text_output = output_type == &OutputType::Text;
 
-    // Flow-level memory_id (from chat mode) takes precedence over step-level memory_id
-    let memory_id = flow_context
-        .flow_status
-        .as_ref()
-        .and_then(|fs| fs.memory_id)
-        .or_else(|| {
-            // Extract memory_id from Memory::Auto if present
-            match &args.memory {
-                Some(Memory::Auto { memory_id, .. }) => *memory_id,
-                _ => None,
-            }
-        });
+    if is_text_output {
+        for note in &history_notes {
+            append_logs(&job.id, &job.workspace_id, format!("{note}\n"), conn).await;
+        }
+    } else if !matches!(args.memory, None | Some(Memory::Off))
+        || args.memory_id.is_some()
+        || args.previous_messages.is_some()
+    {
+        append_logs(
+            &job.id,
+            &job.workspace_id,
+            "Image output sends no history, so memory and previous messages are not read.\n",
+            conn,
+        )
+        .await;
+    }
+
+    // A `manual` memory sent whatever list it held, an empty one included, so a step that still has
+    // one keeps running without a user message.
+    let legacy_list = matches!(args.memory, Some(Memory::Manual { .. }));
+    if !has_prompt(&history, has_user_message, is_text_output, legacy_list) {
+        let missing = if !is_text_output {
+            "'user_message' must be provided for image output"
+        } else if matches!(
+            args.memory,
+            Some(Memory::Window { .. } | Memory::Auto { .. })
+        ) {
+            "'user_message' must be provided while managed memory is on"
+        } else {
+            "Either 'previous_messages' or 'user_message' must be provided"
+        };
+        return Err(Error::internal_err(missing.to_string()));
+    }
 
-    // Load messages based on history mode
     if matches!(output_type, OutputType::Text) {
-        match &args.memory {
-            Some(Memory::Manual { messages: manual_messages }) => {
-                // Use explicitly provided messages (bypass memory)
-                if !manual_messages.is_empty() {
-                    messages.extend(manual_messages.clone());
-                }
-            }
-            Some(Memory::Auto { context_length, .. }) => {
-                // Auto mode: load from memory
+        match &history {
+            HistorySource::Messages(provided) => messages.extend(provided.iter().cloned()),
+            HistorySource::Window { memory_id, context_length } => {
                 if let Some(step_id) = effective_flow_step_id {
-                    if let Some(memory_id) = memory_id {
-                        // Read messages from memory
-                        match read_from_memory(db, &job.workspace_id, memory_id, step_id).await {
-                            Ok(Some(loaded_messages)) => {
-                                let messages_to_load = prepare_auto_memory_messages_for_request(
-                                    &loaded_messages,
-                                    *context_length,
-                                );
-                                messages.extend(messages_to_load);
-                            }
-                            Ok(None) => {}
-                            Err(e) => {
-                                tracing::error!(
-                                    "Failed to read memory for step {}: {}",
-                                    step_id,
-                                    e
-                                );
-                            }
+                    match read_from_memory(db, &job.workspace_id, *memory_id, step_id).await {
+                        Ok(Some(loaded_messages)) => {
+                            let messages_to_load = prepare_auto_memory_messages_for_request(
+                                &loaded_messages,
+                                *context_length,
+                            );
+                            messages.extend(messages_to_load);
+                        }
+                        Ok(None) => {}
+                        Err(e) => {
+                            tracing::error!("Failed to read memory for step {}: {}", step_id, e);
                         }
                     }
                 }
             }
-            _ => {}
+            HistorySource::Stateless => {}
         }
     }
 
@@ -1521,30 +1689,35 @@ pub async fn run_agent(
                         ..Default::default()
                     });
                     if persist_output_to_conversation {
-                        if let Some(memory_id) = memory_id {
-                            let agent_job_id = job.id;
-                            let db_clone = db.clone();
-                            let message_content = "Used websearch tool successfully".to_string();
-                            let step_name = step_name.clone();
-                            tokio::spawn(async move {
-                                if let Err(e) = add_message_to_conversation(
-                                    &db_clone,
-                                    &memory_id,
-                                    Some(agent_job_id),
-                                    &message_content,
-                                    MessageType::Tool,
-                                    &step_name,
-                                    true,
-                                )
-                                .await
-                                {
-                                    tracing::warn!(
-                                        "Failed to add websearch tool message to conversation {}: {}",
-                                        memory_id,
-                                        e
-                                    );
-                                }
+                        if let Some(conversation_id) = conversation_id {
+                            // The search ran inside the provider's call, so this job's args
+                            // describe the agent, not the search: its sources reach the row
+                            // only if they are written here.
+                            let extras = (!annotations.is_empty()).then(|| MessageExtras {
+                                tool_result: serde_json::to_string(&annotations).ok(),
+                                ..Default::default()
                             });
+                            // Awaited like every row of the loop, so rows commit in turn order.
+                            // Worded like every other tool row, so a reader recovers the tool
+                            // name from the sentence.
+                            if let Err(e) = add_message_to_conversation(
+                                db,
+                                &conversation_id,
+                                Some(job.id),
+                                "Used websearch tool",
+                                MessageType::Tool,
+                                &step_name,
+                                true,
+                                extras.as_ref(),
+                            )
+                            .await
+                            {
+                                tracing::warn!(
+                                    "Failed to add websearch tool message to conversation {}: {}",
+                                    conversation_id,
+                                    e
+                                );
+                            }
                         }
                     }
                 }
@@ -1573,32 +1746,30 @@ pub async fn run_agent(
 
                     // Add assistant message to conversation if chat_input_enabled
                     if persist_output_to_conversation && !response_content.is_empty() {
-                        if let Some(memory_id) = memory_id {
-                            let agent_job_id = job.id;
-                            let db_clone = db.clone();
-                            let message_content = response_content.clone();
-                            let step_name = step_name.clone();
-
-                            // Spawn task because we do not need to wait for the result
-                            tokio::spawn(async move {
-                                if let Err(e) = add_message_to_conversation(
-                                    &db_clone,
-                                    &memory_id,
-                                    Some(agent_job_id),
-                                    &message_content,
-                                    MessageType::Assistant,
-                                    &step_name,
-                                    true,
-                                )
-                                .await
-                                {
-                                    tracing::warn!(
-                                        "Failed to add assistant message to conversation {}: {}",
-                                        memory_id,
-                                        e
-                                    );
-                                }
+                        if let Some(conversation_id) = conversation_id {
+                            // This iteration's thinking goes on the answer's row; the job
+                            // result only keeps the turn's thinking as one string.
+                            let extras = response_reasoning.clone().map(|reasoning| {
+                                MessageExtras { reasoning: Some(reasoning), ..Default::default() }
                             });
+                            if let Err(e) = add_message_to_conversation(
+                                db,
+                                &conversation_id,
+                                Some(job.id),
+                                response_content,
+                                MessageType::Assistant,
+                                &step_name,
+                                true,
+                                extras.as_ref(),
+                            )
+                            .await
+                            {
+                                tracing::warn!(
+                                    "Failed to add assistant message to conversation {}: {}",
+                                    conversation_id,
+                                    e
+                                );
+                            }
                         }
                     }
                 }
@@ -1637,6 +1808,18 @@ pub async fn run_agent(
                     ..Default::default()
                 });
 
+                // A round's thinking is stored on one row, the first the round writes, which is
+                // where the stream shows it: its text row when it wrote text, else the row of
+                // its first call — the answer row below when that call is the structured-output
+                // tool. Two rows carrying it would show it twice after a reload.
+                let call_reasoning = response_reasoning
+                    .clone()
+                    .filter(|_| response_content.as_deref().unwrap_or("").is_empty());
+                let structured_output_first = structured_output_tool_name
+                    .as_ref()
+                    .zip(tool_calls.first())
+                    .map_or(false, |(name, tc)| tc.function.name == *name);
+
                 // Handle tool calls using extracted tools module
                 let tool_execution_ctx = ToolExecutionContext {
                     db,
@@ -1655,6 +1838,11 @@ pub async fn run_agent(
                     stream_event_processor: stream_event_processor.as_ref(),
                     flow_context: &mut flow_context,
                     omit_output_from_conversation,
+                    reasoning: if structured_output_first {
+                        None
+                    } else {
+                        call_reasoning.clone()
+                    },
                     previous_result: &previous_result,
                     id_context: &id_context,
                     tool_abort_handles: tool_abort_handles.clone(),
@@ -1673,6 +1861,41 @@ pub async fn run_agent(
                     .await?;
 
                 messages.extend(tool_messages);
+
+                // A structured answer is the arguments of the structured-output tool call,
+                // on which the loop ends without a text iteration, so its row is written here.
+                if tool_used_structured_output && persist_output_to_conversation {
+                    if let (Some(conversation_id), Some(OpenAIContent::Text(answer))) =
+                        (conversation_id, tool_content.as_ref())
+                    {
+                        let extras = call_reasoning
+                            .clone()
+                            .filter(|_| structured_output_first)
+                            .map(|reasoning| MessageExtras {
+                                reasoning: Some(reasoning),
+                                ..Default::default()
+                            });
+                        if let Err(e) = add_message_to_conversation(
+                            db,
+                            &conversation_id,
+                            Some(job.id),
+                            answer,
+                            MessageType::Assistant,
+                            &step_name,
+                            true,
+                            extras.as_ref(),
+                        )
+                        .await
+                        {
+                            tracing::warn!(
+                                "Failed to add structured answer to conversation {}: {}",
+                                conversation_id,
+                                e
+                            );
+                        }
+                    }
+                }
+
                 if let Some(tc) = tool_content {
                     content = Some(tc);
                 }
@@ -1692,10 +1915,7 @@ pub async fn run_agent(
 
                 // Add assistant message to conversation if chat_input_enabled
                 if persist_output_to_conversation {
-                    if let Some(memory_id) = memory_id {
-                        let agent_job_id = job.id;
-                        let db_clone = db.clone();
-
+                    if let Some(conversation_id) = conversation_id {
                         // Create extended version with type discriminator for conversation storage
                         // This avoids conflicts with outputs that are of the same format as S3 objects
                         let s3_with_type = S3ObjectWithType {
@@ -1706,26 +1926,24 @@ pub async fn run_agent(
                         let message_content = serde_json::to_string(&s3_with_type)
                             .unwrap_or_else(|_| content.get().to_string());
 
-                        // Spawn task because we do not need to wait for the result
-                        tokio::spawn(async move {
-                            if let Err(e) = add_message_to_conversation(
-                                &db_clone,
-                                &memory_id,
-                                Some(agent_job_id),
-                                &message_content,
-                                MessageType::Assistant,
-                                &step_name,
-                                true,
-                            )
-                            .await
-                            {
-                                tracing::warn!(
-                                    "Failed to add assistant message to conversation {}: {}",
-                                    memory_id,
-                                    e
-                                );
-                            }
-                        });
+                        if let Err(e) = add_message_to_conversation(
+                            db,
+                            &conversation_id,
+                            Some(job.id),
+                            &message_content,
+                            MessageType::Assistant,
+                            &step_name,
+                            true,
+                            None,
+                        )
+                        .await
+                        {
+                            tracing::warn!(
+                                "Failed to add assistant message to conversation {}: {}",
+                                conversation_id,
+                                e
+                            );
+                        }
                     }
                 }
 
@@ -1778,13 +1996,10 @@ pub async fn run_agent(
         }
     }
 
-    // Persist complete conversation to memory at the end (only if in auto mode with context length)
-    // Skip memory persistence if using manual messages (bypass memory entirely)
-    // final_messages contains the complete history (old messages + new ones)
-    if matches!(output_type, OutputType::Text) && !use_manual_messages {
-        if let Some(Memory::Auto { context_length, .. }) = &args.memory {
+    // final_messages holds the complete history: what was loaded plus this run's messages
+    if matches!(output_type, OutputType::Text) {
+        if let HistorySource::Window { memory_id, context_length } = &history {
             if let Some(step_id) = effective_flow_step_id {
-                // Extract OpenAIMessages from final_messages
                 let all_messages: Vec<OpenAIMessage> =
                     final_messages.iter().map(|m| m.message.clone()).collect();
 
@@ -1794,23 +2009,21 @@ pub async fn run_agent(
                         *context_length,
                     );
 
-                    if let Some(memory_id) = memory_id {
-                        if let Err(e) = write_to_memory(
-                            db,
-                            &job.workspace_id,
-                            memory_id,
+                    if let Err(e) = write_to_memory(
+                        db,
+                        &job.workspace_id,
+                        *memory_id,
+                        step_id,
+                        &messages_to_persist,
+                    )
+                    .await
+                    {
+                        tracing::error!(
+                            "Failed to persist {} messages to memory for step {}: {}",
+                            messages_to_persist.len(),
                             step_id,
-                            &messages_to_persist,
-                        )
-                        .await
-                        {
-                            tracing::error!(
-                                "Failed to persist {} messages to memory for step {}: {}",
-                                messages_to_persist.len(),
-                                step_id,
-                                e
-                            );
-                        }
+                            e
+                        );
                     }
                 }
             }
@@ -1870,6 +2083,228 @@ mod tests {
         }
     }
 
+    #[derive(Debug, PartialEq)]
+    enum Resolved {
+        Messages(usize),
+        Window(Uuid, usize),
+        Stateless { noted: bool },
+    }
+
+    /// Every memory shape a worker may still read, resolved against a run with or without a
+    /// memory id. The hashed id is pinned: changing it detaches memories stored under string ids.
+    #[test]
+    fn history_source_resolves_every_memory_shape() {
+        use serde_json::json;
+        let run = Uuid::from_u128(1);
+        let baked = Uuid::from_u128(2);
+        let cust_1 = Uuid::parse_str("0168fcea-ffa7-5c15-bdb0-7709bb5f540d").unwrap();
+        let window = json!({ "kind": "window", "context_length": 10 });
+        let message = json!([{ "role": "user", "content": "earlier" }]);
+        let two_messages = json!([
+            { "role": "user", "content": "earlier" },
+            { "role": "assistant", "content": "reply" }
+        ]);
+        let cases = [
+            (
+                "absent memory is off",
+                json!({}),
+                Some(run),
+                Resolved::Stateless { noted: false },
+            ),
+            (
+                "legacy off",
+                json!({ "memory": { "kind": "off" } }),
+                Some(run),
+                Resolved::Stateless { noted: false },
+            ),
+            (
+                "legacy auto prefers the run's id",
+                json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked } }),
+                Some(run),
+                Resolved::Window(run, 4),
+            ),
+            (
+                "legacy auto falls back to its baked id",
+                json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked } }),
+                None,
+                Resolved::Window(baked, 4),
+            ),
+            (
+                "legacy auto with an empty baked id uses the run's",
+                json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": "" } }),
+                Some(run),
+                Resolved::Window(run, 4),
+            ),
+            (
+                "legacy auto with an empty baked id and no run id is stateless",
+                json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": " " } }),
+                None,
+                Resolved::Stateless { noted: true },
+            ),
+            (
+                "legacy auto without a length is off",
+                json!({ "memory": { "kind": "auto", "memory_id": baked } }),
+                Some(run),
+                Resolved::Stateless { noted: false },
+            ),
+            (
+                "a cleared count is off",
+                json!({ "memory": { "kind": "window", "context_length": null } }),
+                Some(run),
+                Resolved::Stateless { noted: false },
+            ),
+            (
+                "legacy manual replays its messages",
+                json!({ "memory": { "kind": "manual", "messages": message } }),
+                Some(run),
+                Resolved::Messages(1),
+            ),
+            (
+                "window keeps the run's memory",
+                json!({ "memory": window }),
+                Some(run),
+                Resolved::Window(run, 10),
+            ),
+            (
+                "window without a memory id is stateless",
+                json!({ "memory": window }),
+                None,
+                Resolved::Stateless { noted: true },
+            ),
+            (
+                "a step memory id overrides the run's",
+                json!({ "memory": window, "memory_id": "cust_1" }),
+                Some(run),
+                Resolved::Window(cust_1, 10),
+            ),
+            (
+                "a uuid step memory id is used as is",
+                json!({ "memory": window, "memory_id": baked.to_string() }),
+                Some(run),
+                Resolved::Window(baked, 10),
+            ),
+            (
+                "a step memory id evaluating to null is stateless",
+                json!({ "memory": window, "memory_id": null }),
+                Some(run),
+                Resolved::Stateless { noted: true },
+            ),
+            (
+                "an off policy ignores the step memory id, and says so",
+                json!({ "memory": { "kind": "off" }, "memory_id": "cust_1" }),
+                Some(run),
+                Resolved::Stateless { noted: true },
+            ),
+            (
+                "managed memory ignores the step's previous messages",
+                json!({ "memory": window, "memory_id": "cust_1", "previous_messages": message }),
+                Some(run),
+                Resolved::Window(cust_1, 10),
+            ),
+            (
+                "memory that is off sends the step's previous messages",
+                json!({ "previous_messages": message }),
+                Some(run),
+                Resolved::Messages(1),
+            ),
+            (
+                "a previous messages expression that evaluated to null is no history",
+                json!({ "previous_messages": null }),
+                Some(run),
+                Resolved::Stateless { noted: false },
+            ),
+            (
+                "a legacy manual list ignores the step's previous messages",
+                json!({ "memory": { "kind": "manual", "messages": message }, "previous_messages": two_messages }),
+                Some(run),
+                Resolved::Messages(1),
+            ),
+            (
+                "legacy auto ignores a step memory id",
+                json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked }, "memory_id": "cust_1" }),
+                None,
+                Resolved::Window(baked, 4),
+            ),
+        ];
+        for (name, history, run_memory_id, expected) in cases {
+            let mut raw = json!({ "provider": { "kind": "openai", "resource": {}, "model": "m" } });
+            raw.as_object_mut()
+                .unwrap()
+                .extend(history.as_object().unwrap().clone());
+            let args: AIAgentArgs = serde_json::from_value(raw).unwrap();
+            let resolved = match resolve_history_source(&args, run_memory_id, "ws", "f/flow") {
+                (HistorySource::Messages(m), _) => Resolved::Messages(m.len()),
+                (HistorySource::Window { memory_id, context_length }, _) => {
+                    Resolved::Window(memory_id, context_length)
+                }
+                (HistorySource::Stateless, notes) => {
+                    Resolved::Stateless { noted: !notes.is_empty() }
+                }
+            };
+            assert_eq!(resolved, expected, "{name}");
+        }
+    }
+
+    /// A placeholder the form seeds must not read as a memory id that evaluated to nothing, which
+    /// would turn memory off for the step.
+    #[test]
+    fn only_an_expression_can_set_an_empty_step_memory_id() {
+        let transforms = |memory_id: &str| -> HashMap<String, InputTransform> {
+            HashMap::from([(
+                "memory_id".to_string(),
+                serde_json::from_str(memory_id).unwrap(),
+            )])
+        };
+        let args = || -> AIAgentArgs {
+            serde_json::from_value(serde_json::json!({
+                "provider": { "kind": "openai", "resource": {}, "model": "m" },
+                "memory_id": null,
+            }))
+            .unwrap()
+        };
+        for (transform, expected) in [
+            (r#"{ "type": "static" }"#, None),
+            (r#"{ "type": "static", "value": "" }"#, None),
+            (r#"{ "type": "ai" }"#, None),
+            (
+                r#"{ "type": "javascript", "expr": "flow_input.customer_id" }"#,
+                Some(""),
+            ),
+        ] {
+            let mut args = args();
+            keep_authored_memory_id(&mut args, &transforms(transform));
+            assert_eq!(args.memory_id.as_deref(), expected, "{transform}");
+        }
+    }
+
+    /// Only text output sends previous messages, so they never stand in for an image prompt.
+    #[test]
+    fn previous_messages_never_stand_in_for_an_image_prompt() {
+        let args: AIAgentArgs = serde_json::from_value(serde_json::json!({
+            "provider": { "kind": "openai", "resource": {}, "model": "m" },
+            "previous_messages": [{ "role": "user", "content": "earlier" }],
+        }))
+        .unwrap();
+        let (history, _) = resolve_history_source(&args, None, "ws", "f/flow");
+        assert!(has_prompt(&history, false, true, false));
+        assert!(!has_prompt(&history, false, false, false));
+        assert!(has_prompt(&history, true, false, false));
+        assert!(!has_prompt(
+            &HistorySource::Messages(&[]),
+            false,
+            true,
+            false
+        ));
+        // A legacy `manual` memory ran on an empty list alone, and still does for text output.
+        assert!(has_prompt(&HistorySource::Messages(&[]), false, true, true));
+        assert!(!has_prompt(
+            &HistorySource::Messages(&[]),
+            false,
+            false,
+            true
+        ));
+    }
+
     #[test]
     fn reasoning_keeps_every_iteration_in_order() {
         let mut acc = String::new();

backend/windmill-worker/src/duckdb_executor.rs

@@ -2240,99 +2240,11 @@ fn parse_attach_db_resource<'a>(query: &'a str) -> Option<ParsedAttachDbResource
     None
 }
 
-/// The verification a DuckDB postgres attach keeps, as its libpq `sslmode` and `sslrootcert`.
-///
-/// Attaches have always turned verify-ca and verify-full into `require`, which resources rely on.
-/// A connection that explicitly refuses invalid certificates — the external instance cluster's —
-/// keeps its mode instead: under `require` its shared password would go to whichever server
-/// answers. DuckDB's libpq takes one root file, so it gets the system bundle plus the configured
-/// certificate.
-fn pg_attach_verification(res: &PgDatabase) -> Result<Option<(&str, std::path::PathBuf)>> {
-    let mode = match res.sslmode.as_deref() {
-        Some(mode @ ("verify-ca" | "verify-full")) if res.accept_invalid_certs == Some(false) => mode,
-        _ => return Ok(None),
-    };
-    let bundle = windmill_common::system_ca_bundle()
-        .map(std::fs::read_to_string)
-        .transpose()
-        .map_err(|e| Error::ExecutionErr(format!("Failed to read the system CA bundle: {e}")))?
-        .unwrap_or_default();
-    let pem = res.root_certificate_pem.as_deref().unwrap_or_default();
-    if bundle.is_empty() && pem.is_empty() {
-        return Err(Error::ExecutionErr(format!(
-            "sslmode {mode} needs a root certificate, and this worker has no system CA bundle"
-        )));
-    }
-    let roots = format!("{bundle}\n{pem}\n");
-    use sha2::Digest;
-    let dir = std::env::temp_dir().join("windmill-pg-roots");
-    let path = dir.join(format!(
-        "{}.pem",
-        hex::encode(&sha2::Sha256::digest(roots.as_bytes())[..8])
-    ));
-    let write_err = |e: std::io::Error| {
-        Error::ExecutionErr(format!("Failed to write root certificates: {e}"))
-    };
-    if path.is_file() {
-        // Marks it recently used, so pruning takes the others first.
-        let _ = std::fs::File::options()
-            .append(true)
-            .open(&path)
-            .and_then(|f| f.set_modified(std::time::SystemTime::now()));
-    } else {
-        std::fs::create_dir_all(&dir).map_err(write_err)?;
-        // Renamed into place: a job attaching concurrently must never read a half-written file.
-        let partial = path.with_extension(format!("{}.partial", Uuid::new_v4()));
-        std::fs::write(&partial, &roots)
-            .and_then(|()| std::fs::rename(&partial, &path))
-            .map_err(write_err)?;
-        prune_pg_roots(&dir, &path);
-    }
-    Ok(Some((mode, path)))
-}
-
-/// Root files outlive the job: a resource's certificate is workspace-controlled, so each distinct
-/// one would otherwise add a file forever. Keeps the most recently used ones.
-const PG_ROOTS_KEPT: usize = 32;
-
-fn prune_pg_roots(dir: &std::path::Path, keep: &std::path::Path) {
-    let Ok(entries) = std::fs::read_dir(dir) else {
-        return;
-    };
-    let mut files: Vec<(std::time::SystemTime, std::path::PathBuf)> = entries
-        .filter_map(|e| e.ok())
-        .map(|e| e.path())
-        .filter(|p| p.extension().is_some_and(|x| x == "pem") && p != keep)
-        .filter_map(|p| Some((std::fs::metadata(&p).ok()?.modified().ok()?, p)))
-        .collect();
-    if files.len() < PG_ROOTS_KEPT {
-        return;
-    }
-    files.sort();
-    for (_, p) in &files[..=files.len() - PG_ROOTS_KEPT] {
-        let _ = std::fs::remove_file(p);
-    }
-}
-
-fn pg_attach_uri(res: &PgDatabase) -> Result<String> {
-    let uri = res.to_uri();
-    let Some((mode, roots)) = pg_attach_verification(res)? else {
-        return Ok(uri);
-    };
-    let base = uri.strip_suffix("?sslmode=require").ok_or_else(|| {
-        Error::internal_err("unexpected sslmode in a postgres connection URI".to_string())
-    })?;
-    Ok(format!(
-        "{base}?sslmode={mode}&sslrootcert={}",
-        urlencoding::encode(&roots.to_string_lossy())
-    ))
-}
-
 fn format_attach_db_conn_str(db_resource: Value, db_type: &str) -> Result<String> {
     let s = match db_type.to_lowercase().as_str() {
         "postgres" | "postgresql" => {
             let res: PgDatabase = serde_json::from_value(db_resource)?;
-            pg_attach_uri(&res)?
+            res.to_uri()
         }
         #[cfg(feature = "mysql")]
         "mysql" => {
@@ -2479,9 +2391,7 @@ async fn transform_attach_ducklake(
         format!(", {}", user_extra_args)
     };
     let db_type = match ducklake.catalog.resource_type {
-        DucklakeCatalogResourceType::Instance | DucklakeCatalogResourceType::ExternalInstance => {
-            "postgres"
-        }
+        DucklakeCatalogResourceType::Instance => "postgres",
         _ => ducklake.catalog.resource_type.as_ref(),
     };
 
@@ -2597,8 +2507,7 @@ fn fork_defer_statements(
             hidden_passwords.lock().unwrap().push(pwd.to_string());
         }
         let db_type = match a.catalog.resource_type {
-            DucklakeCatalogResourceType::Instance
-            | DucklakeCatalogResourceType::ExternalInstance => "postgres",
+            DucklakeCatalogResourceType::Instance => "postgres",
             _ => a.catalog.resource_type.as_ref(),
         };
         stmts.push(get_attach_db_install_str(db_type)?.to_string());
@@ -2791,21 +2700,10 @@ fn pg_secret_attach_statements(db_resource: Value, alias_name: &str) -> Result<V
     let esc = |s: &str| s.replace('\'', "''");
     // The postgres secret type has no sslmode parameter, so it goes in the ATTACH
     // string; only the libpq values PgDatabase::to_uri collapses to are forwarded.
-    let sslmode = match pg_attach_verification(&res)? {
-        // A libpq keyword/value string: the path is quoted for libpq, then for the DuckDB literal.
-        Some((mode, roots)) => format!(
-            "{mode} sslrootcert=''{}''",
-            roots
-                .to_string_lossy()
-                .replace('\\', "\\\\")
-                .replace('\'', "\\''")
-        ),
-        None => match res.sslmode.as_deref() {
-            Some("disable") => "disable",
-            Some("require") | Some("verify-ca") | Some("verify-full") => "require",
-            _ => "prefer",
-        }
-        .to_string(),
+    let sslmode = match res.sslmode.as_deref() {
+        Some("disable") => "disable",
+        Some("require") | Some("verify-ca") | Some("verify-full") => "require",
+        _ => "prefer",
     };
     let secret_name = datatable_secret_name(alias_name);
     Ok(vec![
@@ -2898,46 +2796,6 @@ pub struct Arg {
 mod tests {
     use super::*;
 
-    #[test]
-    fn pg_attach_keeps_verification_only_when_required() {
-        let pg = |sslmode: &str, accept_invalid_certs: Option<bool>| PgDatabase {
-            host: "db.internal".to_string(),
-            user: Some("custom_instance_user".to_string()),
-            password: Some("pw".to_string()),
-            port: None,
-            sslmode: Some(sslmode.to_string()),
-            dbname: "dt".to_string(),
-            root_certificate_pem: Some("-----BEGIN CERTIFICATE-----test".to_string()),
-            accept_invalid_certs,
-            use_iam_auth: None,
-            region: None,
-        };
-        let uri = pg_attach_uri(&pg("verify-full", Some(false))).unwrap();
-        assert!(uri.contains("?sslmode=verify-full&sslrootcert="), "{uri}");
-        let root = urlencoding::decode(uri.split("sslrootcert=").nth(1).unwrap()).unwrap();
-        let roots = std::fs::read_to_string(root.as_ref()).unwrap();
-        for i in 0..(PG_ROOTS_KEPT + 5) {
-            let mut other = pg("verify-full", Some(false));
-            other.root_certificate_pem = Some(format!("-----BEGIN CERTIFICATE-----{i}"));
-            pg_attach_uri(&other).unwrap();
-        }
-        let kept = std::fs::read_dir(std::env::temp_dir().join("windmill-pg-roots"))
-            .unwrap()
-            .filter(|e| e.as_ref().unwrap().path().extension().is_some_and(|x| x == "pem"))
-            .count();
-        assert!(kept <= PG_ROOTS_KEPT, "{kept} root files kept");
-        assert!(roots.contains("-----BEGIN CERTIFICATE-----test"));
-        let external = serde_json::to_value(pg("verify-full", Some(false))).unwrap();
-        let attach = &pg_secret_attach_statements(external, "dt").unwrap()[3];
-        assert!(
-            attach.starts_with(&format!("ATTACH 'sslmode=verify-full sslrootcert=''{}''", root)),
-            "{attach}"
-        );
-        // A resource that never opted in keeps the historical downgrade.
-        assert!(pg_attach_uri(&pg("verify-full", None)).unwrap().ends_with("?sslmode=require"));
-        assert!(pg_attach_uri(&pg("require", Some(false))).unwrap().ends_with("?sslmode=require"));
-    }
-
     #[test]
     fn attach_datatable_parses_name_and_role() {
         let reference_of = |q: &str| parse_attach_datatable(q).unwrap().reference;

backend/windmill-worker/src/worker_flow.rs

@@ -2236,6 +2236,7 @@ async fn add_tool_message_to_conversation(
                     MessageType::Assistant,
                     None,
                     success,
+                    None,
                 )
                 .await?;
                 tx.commit().await?;

benchmarks/lib.ts

@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
 import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
 import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
 
-export const VERSION = "v1.813.0";
+export const VERSION = "v1.814.0";
 
 export async function login(email: string, password: string): Promise<string> {
   return await windmill.UserService.login({

chat-sdk/README.md

@@ -56,8 +56,10 @@ not a replacement of the previous answer.
 
 The transport also carries the history helpers: `transport.loadMessages(id)` returns
 `UIMessage`s for `useChat({ messages })` or `setMessages`, `transport.listConversations()`
-and `transport.deleteConversation(id)`. Attachments are not supported: `sendMessage` with
-`files` is refused with an explanatory error.
+and `transport.deleteConversation(id)`. A loaded user message lists the files it carried
+in `metadata.attachments`; `WindmillChatApi.attachmentUrl` gives each one's download URL.
+Sending attachments is not supported: `sendMessage` with `files` is refused with an
+explanatory error.
 
 ## assistant-ui
 
@@ -181,7 +183,7 @@ await chat.sendMessage('Hello')
 | `workspace` | Detected inside a raw app. |
 | `token` | A token, or a function returning one (called before every request, so it can fetch a short-lived token from your backend). Omit it inside a raw app. |
 | `history` | `'server'`, `'local'` or `'none'`, see [History](#history). Defaults to `'server'` with a viewer session and `'local'` with an explicit `token`. |
-| `inputs` | Extra flow inputs sent with every message. `sendMessage(text, { inputs })` adds per-message ones. |
+| `inputs` | Extra flow inputs sent with every message. `sendMessage(text, { inputs })` adds per-message ones, and `{ attachments, attachmentsInput }` files, see [Attachments](#attachments). |
 | `storageKey` | Namespace for `local` history, e.g. the signed-in user's id. Local history is per browser and per flow; without it, users sharing a browser share it. |
 | `fetch`, `storage` | Replacements for the globals, for tests and unusual runtimes. |
 | `pageSize` | Messages and conversations per page of server history. Default 50. |
@@ -224,12 +226,41 @@ A turn goes `submitted` (the flow is queued) → `streaming` (the answer is arri
 answer, an `assistant` message with `success: false`. `status: 'error'` (with `error`
 set) means the turn could not run or be followed at all, such as a refused request.
 
-Methods: `sendMessage(text, { inputs? })`, `stop()`, `newConversation()`,
-`selectConversation(id)`, `loadConversations({ page?, perPage? })`,
-`deleteConversation(id)`, `loadOlderMessages()`, `destroy()`. Switching conversations
+Methods: `sendMessage(text, { inputs?, attachments?, attachmentsInput? })`, `stop()`,
+`newConversation()`, `selectConversation(id)`, `loadConversations({ page?, perPage?, kind? })`,
+`deleteConversation(id)`, `renameConversation(id, title)`, `loadOlderMessages()`,
+`destroy()`. `kind` lists the flow editor's test chats (`'test'`), the deployed flow's
+own (`'deployed'`, the server's default) or both (`'all'`); each `Conversation` carries
+`isTest`. A rename keeps the conversation's place in the list. Switching conversations
 stops following the current answer; the flow keeps running and, with server history,
 its answer is there when you come back.
 
+## Attachments
+
+A flow whose AI agent step reads `user_attachments` from an `s3object[]` (or a single
+`s3object`) flow input takes files with a message:
+
+```ts
+await chat.sendMessage('What does this contract say?', {
+  attachments: [{ name: file.name, data: file }], // a Blob/File, or a `data:` URL
+  attachmentsInput: { name: 'files', multiple: true }
+})
+```
+
+Each file is uploaded to the workspace's object storage under
+`windmill_uploads/chat/<turn>/<index>/<name>` and handed to the input as `{ s3, filename }`
+objects (the object for a single-file input). Once the uploads return, the pending user
+message lists them in `attachments`, as `{ input, s3, filename }` references. The name's
+extension is corrected to the file's media type for PNG, JPEG and PDF, because the worker
+reads the type off the key.
+Files need message text to go with them. A failed upload rejects `sendMessage` before any
+run starts, and `stop()` during the upload aborts it; both leave the transcript as it was.
+The chat never deletes uploads, so files of a send that did not run stay in storage. The
+workspace needs object storage set up. With Enterprise advanced storage permissions, the
+user needs read and write on `windmill_uploads/*`, which the default rules grant. The upload goes through
+`job_helpers`, so a restricted token needs `job_helpers:write`; a sandboxed raw app cannot
+request that scope today, so attachments are not available there yet.
+
 ## History
 
 Windmill stores every conversation of a chat-mode flow, and each Windmill user sees

chat-sdk/package-lock.json

@@ -1,12 +1,12 @@
 {
   "name": "windmill-chat",
-  "version": "1.813.0",
+  "version": "1.814.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "windmill-chat",
-      "version": "1.813.0",
+      "version": "1.814.0",
       "license": "Apache-2.0",
       "devDependencies": {
         "@ai-sdk/react": "^4.0.102",

chat-sdk/package.json

@@ -1,7 +1,7 @@
 {
   "name": "windmill-chat",
   "description": "Build chat interfaces on Windmill flows deployed in chat mode, from any frontend or raw app",
-  "version": "1.813.0",
+  "version": "1.814.0",
   "author": "Ruben Fiszel",
   "license": "Apache-2.0",
   "homepage": "https://github.com/windmill-labs/windmill/tree/main/chat-sdk#readme",

chat-sdk/src/ai-sdk.ts

@@ -1,5 +1,10 @@
 import type { ChatTransport, UIMessage, UIMessageChunk, UIMessagePart } from 'ai'
-import { WindmillApiError, WindmillChatApi, type WindmillChatApiOptions } from './api'
+import {
+  WindmillApiError,
+  WindmillChatApi,
+  type FlowConversationMessage,
+  type WindmillChatApiOptions
+} from './api'
 import { followJob } from './follow'
 import type { AgentStreamEvent } from './stream'
 import type { ChatMessage, Conversation } from './types'
@@ -101,7 +106,9 @@ export function createWindmillChatTransport<UI_MESSAGE extends UIMessage = UIMes
           stepName: row.step_name ?? undefined,
           pending: false,
           seq: row.created_seq,
-          tool: toolFromRowContent(row.message_type, row.content, row.success ?? true)
+          reasoning: row.reasoning ?? undefined,
+          attachments: row.attachments ?? undefined,
+          tool: toolFromRow(row)
         }))
       ) as UI_MESSAGE[]
     },
@@ -123,10 +130,20 @@ export function createWindmillChatTransport<UI_MESSAGE extends UIMessage = UIMes
   }
 }
 
-function toolFromRowContent(role: string, content: string, success: boolean): ChatMessage['tool'] {
-  if (role !== 'tool') return undefined
-  const name = /^Used (.+) tool$/.exec(content)?.[1] ?? /^Error executing (.+)$/.exec(content)?.[1]
-  return name ? { name, status: success ? 'success' : 'error' } : undefined
+/** The call a stored tool row carries: its tool, named by the sentence the worker words
+ *  every tool row from, and the model's arguments and what it got back — for a failed
+ *  tool, the result is what it failed with. */
+function toolFromRow(row: FlowConversationMessage): ChatMessage['tool'] {
+  if (row.message_type !== 'tool') return undefined
+  const name =
+    /^Used (.+) tool$/.exec(row.content)?.[1] ?? /^Error executing (.+)$/.exec(row.content)?.[1]
+  if (!name) return undefined
+  return {
+    name,
+    status: (row.success ?? true) ? 'success' : 'error',
+    arguments: row.tool_arguments ?? undefined,
+    result: row.tool_result ?? undefined
+  }
 }
 
 /** Streams a job's answer as AI SDK chunks; resumes from `entry.offset` when the job is already running. */
@@ -292,13 +309,21 @@ class PartWriter {
 
 /**
  * `ChatMessage`s (Windmill's role-per-row model) as `UIMessage`s: an assistant
- * turn becomes one message whose parts carry its text, reasoning and tool calls.
+ * turn becomes one message whose parts carry its text, reasoning and tool calls. A
+ * user message's attachments ride in `metadata.attachments`, as references for
+ * `WindmillChatApi.attachmentUrl`: a `file` part would need a URL the browser can
+ * load unauthenticated.
  */
 export function toUIMessages(messages: ChatMessage[]): UIMessage[] {
   const out: UIMessage[] = []
   for (const m of messages) {
     if (m.role === 'user' || m.role === 'system') {
-      out.push({ id: m.id, role: m.role, parts: [{ type: 'text', text: m.content }] })
+      out.push({
+        id: m.id,
+        role: m.role,
+        ...(m.attachments?.length ? { metadata: { attachments: m.attachments } } : {}),
+        parts: [{ type: 'text', text: m.content }]
+      })
       continue
     }
     let target = out[out.length - 1]
@@ -306,18 +331,18 @@ export function toUIMessages(messages: ChatMessage[]): UIMessage[] {
       target = { id: m.id, role: 'assistant', parts: [] }
       out.push(target)
     }
+    if (m.reasoning) target.parts.push({ type: 'reasoning', text: m.reasoning, state: 'done' })
     if (m.role === 'tool') {
       const toolCallId = m.tool?.callId ?? m.id
       const toolName = m.tool?.name ?? 'tool'
       const input = parseJsonOr(m.tool?.arguments)
       target.parts.push(
         m.success
-          ? { type: 'dynamic-tool', toolName, toolCallId, state: 'output-available', input, output: parseJsonOr(m.tool?.result) ?? m.content }
+          ? { type: 'dynamic-tool', toolName, toolCallId, state: 'output-available', input, output: m.tool?.result !== undefined ? parseJsonOr(m.tool.result) : m.content }
           : { type: 'dynamic-tool', toolName, toolCallId, state: 'output-error', input, errorText: m.tool?.result ?? m.content }
       )
       continue
     }
-    if (m.reasoning) target.parts.push({ type: 'reasoning', text: m.reasoning, state: 'done' })
     if (m.content) target.parts.push({ type: 'text', text: m.content, state: 'done' })
   }
   return out

chat-sdk/src/api.ts

@@ -1,4 +1,4 @@
-import type { FetchLike, TokenSource } from './types'
+import type { ChatAttachment, FetchLike, TokenSource } from './types'
 
 export interface WindmillChatApiOptions {
   baseUrl: string
@@ -28,8 +28,16 @@ export interface FlowConversation {
   created_at: string
   updated_at: string
   created_by: string
+  /** Started from the flow editor's test panel rather than a deployed run. */
+  is_test: boolean
 }
 
+/**
+ * Which conversations a listing holds: the flow editor's test chats, the deployed flow's
+ * own (the server's default), or both.
+ */
+export type ConversationKind = 'test' | 'deployed' | 'all'
+
 export interface FlowConversationMessage {
   id: string
   conversation_id: string
@@ -40,6 +48,14 @@ export interface FlowConversationMessage {
   created_seq: number
   step_name?: string | null
   success?: boolean
+  /** On a tool row, the arguments the model wrote, without the inputs a step wires in; null for a web search. */
+  tool_arguments?: string | null
+  /** On a tool row, the text the model got back or what the call failed with; a web search's citations. */
+  tool_result?: string | null
+  /** On an answer, the thinking that produced it; on a tool row, the thinking that led to the call. */
+  reasoning?: string | null
+  /** The files a user message carried, as object-storage references. */
+  attachments?: ChatAttachment[] | null
 }
 
 export type JobUpdateEvent =
@@ -158,6 +174,17 @@ export class WindmillChatApi {
     return (await res.json()) as FlowJobStatus
   }
 
+  /**
+   * Where a message's attachment downloads from. The endpoint authenticates like every other
+   * request: a consumer holding a token must fetch it with that token, not put the URL in an
+   * `img src`, which would send only the Windmill session cookie.
+   */
+  attachmentUrl(attachment: ChatAttachment): string {
+    const query = new URLSearchParams({ file_key: attachment.s3 })
+    if (attachment.storage) query.set('storage', attachment.storage)
+    return `${this.#baseUrl}/api/w/${encodeURIComponent(this.#workspace)}/job_helpers/download_s3_file?${query}`
+  }
+
   async cancelJob(jobId: string, reason = 'Stopped from the chat'): Promise<void> {
     await this.#request(`jobs_u/queue/cancel/${encodeURIComponent(jobId)}`, {
       method: 'POST',
@@ -167,15 +194,25 @@ export class WindmillChatApi {
 
   async listConversations(
     flowPath: string,
-    options: { page?: number; perPage?: number; signal?: AbortSignal } = {}
+    options: { page?: number; perPage?: number; kind?: ConversationKind; signal?: AbortSignal } = {}
   ): Promise<FlowConversation[]> {
+    const extra: Record<string, string> = { flow_path: flowPath }
+    if (options.kind !== undefined) extra.kind = options.kind
     const res = await this.#request('flow_conversations/list', {
-      query: pagination(options, { flow_path: flowPath }),
+      query: pagination(options, extra),
       signal: options.signal
     })
     return (await res.json()) as FlowConversation[]
   }
 
+  /** Sets a conversation's title. Its place in the list is kept: only a turn moves one. */
+  async renameConversation(conversationId: string, title: string): Promise<void> {
+    await this.#request(`flow_conversations/update/${encodeURIComponent(conversationId)}`, {
+      method: 'POST',
+      body: { title }
+    })
+  }
+
   /**
    * Without `afterSeq`: one page counted from the newest message, returned oldest first.
    * With `afterSeq`: the messages created after that cursor, oldest first.
@@ -193,6 +230,27 @@ export class WindmillChatApi {
     return (await res.json()) as FlowConversationMessage[]
   }
 
+  /**
+   * Puts bytes in the workspace's object storage under `fileKey` and returns the key they were
+   * stored under (the server may rewrite it). Needs the workspace to have object storage set.
+   */
+  async uploadFile(
+    fileKey: string,
+    body: Blob,
+    options: { contentType?: string; signal?: AbortSignal } = {}
+  ): Promise<{ file_key: string }> {
+    const query: Record<string, string> = { file_key: fileKey }
+    if (options.contentType) query.content_type = options.contentType
+    const res = await this.#request('job_helpers/upload_s3_file', {
+      method: 'POST',
+      query,
+      raw: body,
+      contentType: options.contentType || 'application/octet-stream',
+      signal: options.signal
+    })
+    return (await res.json()) as { file_key: string }
+  }
+
   async deleteConversation(conversationId: string): Promise<void> {
     await this.#request(`flow_conversations/delete/${encodeURIComponent(conversationId)}`, {
       method: 'DELETE'
@@ -204,7 +262,11 @@ export class WindmillChatApi {
     init: {
       method?: string
       query?: Record<string, string>
+      /** JSON-encoded. */
       body?: unknown
+      /** Sent as is, under `contentType`. */
+      raw?: Blob
+      contentType?: string
       accept?: string
       signal?: AbortSignal
     } = {}
@@ -215,13 +277,14 @@ export class WindmillChatApi {
     const headers: Record<string, string> = {}
     if (init.accept) headers['Accept'] = init.accept
     if (init.body !== undefined) headers['Content-Type'] = 'application/json'
+    else if (init.raw !== undefined) headers['Content-Type'] = init.contentType ?? 'application/octet-stream'
     const token = typeof this.#token === 'function' ? await this.#token() : this.#token
     if (token) headers['Authorization'] = `Bearer ${token}`
 
     const res = await this.#fetch(url.toString(), {
       method: init.method ?? 'GET',
       headers,
-      body: init.body === undefined ? undefined : JSON.stringify(init.body),
+      body: init.body === undefined ? init.raw : JSON.stringify(init.body),
       // A token must not be paired with ambient cookies; without one, the cookie is
       // the credential and only rides same-origin requests.
       credentials: token ? 'omit' : 'same-origin',

chat-sdk/src/assistant-ui.ts

@@ -53,7 +53,8 @@ export function useWindmillRuntime(options: WindmillRuntimeOptions): AssistantRu
             threads: chat.conversations.map((c) => ({ status: 'regular' as const, id: c.id, title: c.title })),
             onSwitchToNewThread: () => chat.newConversation(),
             onSwitchToThread: (id) => chat.selectConversation(id),
-            onDelete: (id) => chat.deleteConversation(id)
+            onDelete: (id) => chat.deleteConversation(id),
+            onRename: (id, title) => chat.renameConversation(id, title)
           }
         }
       : undefined
@@ -91,6 +92,7 @@ export function toThreadMessage(turn: WindmillTurn): ThreadMessageLike {
   }
   const content: ThreadContentPart[] = []
   for (const m of turn.messages) {
+    if (m.reasoning) content.push({ type: 'reasoning', text: m.reasoning })
     if (m.role === 'tool') {
       const args = parseJsonOr(m.tool?.arguments)
       content.push({
@@ -99,12 +101,12 @@ export function toThreadMessage(turn: WindmillTurn): ThreadMessageLike {
         toolName: m.tool?.name ?? 'tool',
         args: (isJsonObject(args) ? args : args === undefined ? {} : { input: args }) as ToolCallArgs,
         argsText: m.tool?.arguments ?? '',
-        result: m.tool?.status === 'running' ? undefined : (parseJsonOr(m.tool?.result) ?? m.content),
+        result:
+          m.tool?.status === 'running' ? undefined : m.tool?.result !== undefined ? parseJsonOr(m.tool.result) : m.content,
         isError: m.tool?.status === 'error'
       })
       continue
     }
-    if (m.reasoning) content.push({ type: 'reasoning', text: m.reasoning })
     if (m.content) content.push({ type: 'text', text: m.content })
   }
   const last = turn.messages[turn.messages.length - 1]

chat-sdk/src/attachments.ts

@@ -0,0 +1,116 @@
+import type { WindmillChatApi } from './api'
+import type { AttachmentUpload } from './types'
+import { abortError, isAbortError } from './utils'
+
+/**
+ * Where a chat's uploads live in the workspace's object storage. Under `windmill_uploads/`
+ * because the default Enterprise storage permissions grant every user write and read there
+ * and deny any other top-level prefix: a key outside it is refused for non-admins, both on
+ * upload and when the agent's job reads the file back.
+ */
+export const CHAT_UPLOADS_PREFIX = 'windmill_uploads/chat'
+
+/** What an AI agent step reads out of `user_attachments`. */
+export interface UploadedAttachment {
+  s3: string
+  filename: string
+}
+
+/**
+ * The extension each type must be stored under. The worker reads an attachment's media type
+ * from the key's extension only (`mime_guess` in `windmill-ai/src/image_handler.rs`, falling
+ * back to `image/png`), never from the stored content type, so the extension must be true.
+ */
+const EXTENSION_BY_MEDIA_TYPE: Record<string, string> = {
+  'image/png': 'png',
+  'image/jpeg': 'jpg',
+  'application/pdf': 'pdf'
+}
+
+/**
+ * The name an attachment is stored under: the picked name with the extension its media type
+ * needs, e.g. a `photo.webp` re-encoded to PNG becomes `photo.png`. Other types keep their name.
+ */
+export function storedAttachmentName(filename: string, mediaType: string): string {
+  const extension = EXTENSION_BY_MEDIA_TYPE[mediaType]
+  if (!extension) return filename
+  const stem = filename.replace(/\.[^./]+$/, '')
+  return `${stem || filename}.${extension}`
+}
+
+/** The bytes of an attachment as a Blob carrying its media type. */
+export function attachmentBlob(attachment: AttachmentUpload): Blob {
+  const data =
+    typeof attachment.data === 'string'
+      ? dataUrlToBlob(attachment.data, 'application/octet-stream')
+      : attachment.data
+  return attachment.mediaType && attachment.mediaType !== data.type
+    ? new Blob([data], { type: attachment.mediaType })
+    : data
+}
+
+function dataUrlToBlob(dataUrl: string, fallbackType: string): Blob {
+  const comma = dataUrl.indexOf(',')
+  if (!dataUrl.startsWith('data:') || comma === -1) {
+    throw new Error('windmill-chat: an attachment given as a string must be a data: URL')
+  }
+  const header = dataUrl.slice(5, comma)
+  const isBase64 = header.endsWith(';base64')
+  const mediaType = (isBase64 ? header.slice(0, -';base64'.length) : header) || fallbackType
+  const payload = dataUrl.slice(comma + 1)
+  if (!isBase64) return new Blob([decodeURIComponent(payload)], { type: mediaType })
+  const binary = atob(payload)
+  const bytes = new Uint8Array(binary.length)
+  for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i)
+  return new Blob([bytes], { type: mediaType })
+}
+
+/**
+ * Put each attachment in the workspace's object storage and hand back what the agent reads.
+ * The key's turn prefix and per-file index keep two files with the same name, in this turn or
+ * an earlier one, from overwriting each other; the name stays the last segment.
+ */
+export async function uploadAttachments(
+  api: WindmillChatApi,
+  attachments: AttachmentUpload[],
+  turnId: string,
+  signal?: AbortSignal
+): Promise<UploadedAttachment[]> {
+  const prefix = `${CHAT_UPLOADS_PREFIX}/${turnId}`
+  // One failed upload aborts the rest. Nothing already stored is deleted: the chat never
+  // removes objects from the workspace's storage, so a send that does not run leaves them.
+  if (signal?.aborted) throw abortError()
+  const batch = new AbortController()
+  const abortBatch = () => batch.abort()
+  signal?.addEventListener('abort', abortBatch, { once: true })
+  try {
+    const results = await Promise.allSettled(
+      attachments.map(async (attachment, index) => {
+        try {
+          const blob = attachmentBlob(attachment)
+          const filename = storedAttachmentName(
+            attachment.name || `attachment-${index + 1}`,
+            blob.type
+          )
+          const { file_key } = await api.uploadFile(`${prefix}/${index}/${filename}`, blob, {
+            contentType: blob.type,
+            signal: batch.signal
+          })
+          return { s3: file_key, filename }
+        } catch (e) {
+          batch.abort()
+          throw e
+        }
+      })
+    )
+    const reasons = results.flatMap((r) => (r.status === 'rejected' ? [r.reason] : []))
+    // A stop that lands once every upload has answered still withdraws the batch.
+    if (reasons.length === 0 && !signal?.aborted) {
+      return results.flatMap((r) => (r.status === 'fulfilled' ? [r.value] : []))
+    }
+    // The failure that started it, not the aborts it caused in the other uploads.
+    throw reasons.find((reason) => !isAbortError(reason)) ?? reasons[0] ?? abortError()
+  } finally {
+    signal?.removeEventListener('abort', abortBatch)
+  }
+}

chat-sdk/src/chat.ts

@@ -1,12 +1,14 @@
 import {
   WindmillApiError,
   WindmillChatApi,
+  type ConversationKind,
   type FlowConversation,
   type FlowConversationMessage
 } from './api'
 import { resolveConfig, type ResolvedConfig } from './config'
 import { followJob } from './follow'
 import { createLocalHistory, type LocalHistory } from './history'
+import { uploadAttachments } from './attachments'
 import type { AgentStreamEvent } from './stream'
 import type {
   Chat,
@@ -14,12 +16,15 @@ import type {
   ChatOptions,
   ChatState,
   Conversation,
+  SendMessageOptions,
   ToolInvocation
 } from './types'
 import {
   conversationTitle,
+  truncateTitle,
   errorResultMessage,
   extractChatAnswer,
+  abortError,
   isAbortError,
   isErrorResult,
   now,
@@ -39,6 +44,12 @@ interface Turn {
   conversationId: string
   /** Id of the turn's user message; the answer is whatever follows it. */
   userMessageId: string
+  /** The turn opened the conversation; withdrawing it closes the conversation again. */
+  isNew: boolean
+  /** The run was asked for. Before that, a failure or a stop withdraws the turn instead of failing it. */
+  started: boolean
+  /** Both `stop()` and the send's own rejection withdraw; only the first may. */
+  withdrawn: boolean
   jobId?: string
   /** The flow job and its step jobs; a persisted answer carries one of them as `job_id`. */
   jobIds?: Set<string>
@@ -59,6 +70,8 @@ class ChatImpl implements Chat {
   #state: ChatState
   #turn: Turn | undefined
   #page = 1
+  /** The kind the caller last listed, so the refresh after a new turn lists the same rows. */
+  #conversationKind: ConversationKind | undefined
   #persistTimer: ReturnType<typeof setTimeout> | undefined
 
   constructor(options: ChatOptions) {
@@ -97,21 +110,34 @@ class ChatImpl implements Chat {
     }
   }
 
-  sendMessage = async (
-    text: string,
-    options: { inputs?: Record<string, unknown> } = {}
-  ): Promise<void> => {
+  sendMessage = async (text: string, options: SendMessageOptions = {}): Promise<void> => {
     const content = text.trim()
-    if (!content) return
+    if (!content) {
+      // A run needs a message; files alone would otherwise be dropped without a word.
+      if (options.attachments?.length) throw new Error('windmill-chat: attachments need a message to go with them')
+      return
+    }
     if (this.#turn) {
       throw new Error('windmill-chat: a message is already being answered; call stop() first')
     }
+    const attachments = options.attachments ?? []
+    const attachmentsInput = options.attachmentsInput
+    if (attachments.length > 0 && !attachmentsInput) {
+      throw new Error('windmill-chat: attachments need `attachmentsInput`, the flow input that takes them')
+    }
+    if (attachmentsInput && !attachmentsInput.multiple && attachments.length > 1) {
+      // Uploading all of them would run with the first and leave the rest stranded in storage.
+      throw new Error(`windmill-chat: \`${attachmentsInput.name}\` holds one file; got ${attachments.length}`)
+    }
     const isNew = this.#state.conversationId === undefined
     const conversationId = this.#state.conversationId ?? randomId()
     const turn: Turn = {
       controller: new AbortController(),
       conversationId,
       userMessageId: `pending-${randomId()}`,
+      isNew,
+      started: false,
+      withdrawn: false,
       streamedText: false
     }
     this.#turn = turn
@@ -123,7 +149,6 @@ class ChatImpl implements Chat {
     const touched = { ...conversation, updatedAt: timestamp }
     this.#set({
       conversationId,
-      conversations: [touched, ...this.#state.conversations.filter((c) => c.id !== conversationId)],
       messages: [
         ...this.#state.messages,
         { id: turn.userMessageId, role: 'user', content, success: true, createdAt: timestamp, pending: true }
@@ -131,10 +156,32 @@ class ChatImpl implements Chat {
       status: 'submitted',
       error: undefined
     })
-    this.#rememberConversation()
 
     try {
-      const args = { ...this.#config.inputs, ...options.inputs, user_message: content }
+      const args: Record<string, unknown> = { ...this.#config.inputs, ...options.inputs, user_message: content }
+      if (attachmentsInput && attachments.length > 0) {
+        // Uploaded with the turn already shown as submitted: the message is in the transcript
+        // and `stop()` can abort the upload, while a second send is refused as usual.
+        const uploaded = await uploadAttachments(this.#api, attachments, randomId(), turn.controller.signal)
+        args[attachmentsInput.name] = attachmentsInput.multiple ? uploaded : uploaded[0]
+        // Shown on the pending message until its server row replaces it, carrying its own.
+        const carried = uploaded.map((u) => ({ input: attachmentsInput.name, s3: u.s3, filename: u.filename }))
+        if (this.#turnActive(turn)) {
+          this.#set({
+            messages: this.#state.messages.map((m) => (m.id === turn.userMessageId ? { ...m, attachments: carried } : m))
+          })
+        }
+      }
+      // Nothing may start once stop() or a conversation switch has withdrawn the turn, including
+      // a stop from a subscriber told of the attachments just above.
+      if (turn.controller.signal.aborted) throw abortError()
+      turn.started = true
+      // Listed only once the run is asked for: a send that never runs (an upload that failed
+      // or was stopped) then has no conversation entry to take back.
+      this.#set({
+        conversations: [touched, ...this.#state.conversations.filter((c) => c.id !== conversationId)]
+      })
+      this.#rememberConversation()
       const context = { memoryId: conversationId, conversationId, signal: turn.controller.signal }
       turn.jobId = this.#config.run
         ? await this.#config.run(args, context)
@@ -148,6 +195,13 @@ class ChatImpl implements Chat {
       }
       await this.#finishTurn(turn, result, isNew)
     } catch (e) {
+      if (!turn.started) {
+        // Nothing ran: the message is withdrawn rather than shown as a failed turn, and the
+        // caller gets the reason (an upload that failed, or the AbortError of a stop()).
+        if (this.#turn === turn) this.#turn = undefined
+        this.#withdrawTurn(turn)
+        throw e
+      }
       // stop() and a conversation switch abort the turn and settle the state themselves.
       if (turn.controller.signal.aborted || isAbortError(e)) return
       this.#failTurn(turn, e)
@@ -160,6 +214,12 @@ class ChatImpl implements Chat {
     const turn = this.#turn
     if (!turn) return
     this.#detachTurn()
+    if (!turn.started) {
+      // Still uploading its attachments: there is no run to cancel, and the message the
+      // reader took back must not stay in the transcript as sent.
+      this.#withdrawTurn(turn)
+      return
+    }
     if (this.#state.conversationId === turn.conversationId) {
       this.#set({ messages: finalized(this.#state.messages), status: 'idle' })
       this.#persistLocal()
@@ -229,15 +289,21 @@ class ChatImpl implements Chat {
   }
 
   loadConversations = async (
-    options: { page?: number; perPage?: number } = {}
+    options: { page?: number; perPage?: number; kind?: ConversationKind } = {}
   ): Promise<Conversation[]> => {
     const page = options.page ?? 1
+    // A different kind is a different listing: its first rows replace the held ones, on
+    // whichever page they were asked for.
+    const kindChanged = 'kind' in options && options.kind !== this.#conversationKind
+    if ('kind' in options) this.#conversationKind = options.kind
+    const kind = this.#conversationKind
     let conversations: Conversation[]
     if (this.#state.history === 'server') {
       try {
         const rows = await this.#api.listConversations(this.#config.flowPath, {
           page,
-          perPage: options.perPage ?? this.#config.pageSize
+          perPage: options.perPage ?? this.#config.pageSize,
+          kind
         })
         conversations = rows.map(fromConversation)
       } catch (e) {
@@ -247,10 +313,13 @@ class ChatImpl implements Chat {
     } else {
       conversations = this.#state.history === 'local' ? this.#local.listConversations() : []
     }
+    // Another kind was asked for while this list was on its way: its rows are not the
+    // listing any more, whichever response lands last.
+    if (kind !== this.#conversationKind) return conversations
     const known = new Set(this.#state.conversations.map((c) => c.id))
     this.#set({
       conversations:
-        page === 1
+        page === 1 || kindChanged
           ? conversations
           : [...this.#state.conversations, ...conversations.filter((c) => !known.has(c.id))]
     })
@@ -273,6 +342,24 @@ class ChatImpl implements Chat {
     this.#set({ conversations: this.#state.conversations.filter((c) => c.id !== conversationId) })
   }
 
+  renameConversation = async (conversationId: string, title: string): Promise<void> => {
+    // Cut here as the server cuts, so the title shown is the one stored.
+    const trimmed = truncateTitle(title.trim())
+    if (!trimmed) return
+    if (this.#state.history === 'server') {
+      await this.#api.renameConversation(conversationId, trimmed)
+    } else if (this.#state.history === 'local') {
+      this.#local.renameConversation(conversationId, trimmed)
+    }
+    // Patched in place: the server keeps `updated_at` on a rename, so the list order the
+    // next load returns is the one shown now.
+    this.#set({
+      conversations: this.#state.conversations.map((c) =>
+        c.id === conversationId ? { ...c, title: trimmed } : c
+      )
+    })
+  }
+
   loadOlderMessages = async (): Promise<void> => {
     const conversationId = this.#state.conversationId
     if (
@@ -344,16 +431,21 @@ class ChatImpl implements Chat {
           tool: { ...existing.tool!, ...toolPatch }
         }
       } else {
+        // Thinking that produced no text led to this call, and is stored on its row.
+        const a = turn.assistantId ? messages.findIndex((m) => m.id === turn.assistantId) : -1
+        const reasoning = a >= 0 && messages[a].content === '' ? messages.splice(a, 1)[0].reasoning : undefined
         messages.push({
           id: `pending-${randomId()}`,
           role: 'tool',
           content: content ?? '',
+          reasoning,
           success: success ?? true,
           createdAt: now(),
           pending: true,
           tool: { callId, name, status: 'running', ...toolPatch }
         })
       }
+      turn.assistantId = undefined
     }
     const appendAssistant = (text: string, reasoning: string) => {
       const i = turn.assistantId
@@ -388,17 +480,14 @@ class ChatImpl implements Chat {
         case 'reasoning_token_delta':
           appendAssistant('', event.content)
           break
+        // A call completes the round's text: text after it is a new message.
         case 'tool_call':
-          // The round's text is complete; text after the tool result is a new message.
-          turn.assistantId = undefined
           upsertTool(event.call_id, event.function_name, { status: 'running' })
           break
         case 'tool_call_arguments':
-          turn.assistantId = undefined
           upsertTool(event.call_id, event.function_name, { arguments: event.arguments })
           break
         case 'tool_execution':
-          turn.assistantId = undefined
           upsertTool(event.call_id, event.function_name, { status: 'running' })
           break
         case 'tool_result':
@@ -545,6 +634,36 @@ class ChatImpl implements Chat {
     }
   }
 
+  /**
+   * Take back the user message of a turn that never ran. A conversation it would have opened
+   * was never listed (see `sendMessage`), so only the message goes, and, while it is the turn
+   * on screen, the busy status. A switch away mid-upload has already written the message to
+   * local history, so it is removed there too.
+   */
+  #withdrawTurn(turn: Turn): void {
+    if (turn.withdrawn) return
+    turn.withdrawn = true
+    const id = turn.conversationId
+    const withoutTurn = (messages: ChatMessage[]) => messages.filter((m) => m.id !== turn.userMessageId)
+    if (this.#state.conversationId === id) {
+      const messages = withoutTurn(this.#state.messages)
+      // A turn started since, such as a resend right after Stop, owns the status.
+      const newerTurn = this.#turn !== undefined && this.#turn !== turn
+      if (newerTurn) {
+        this.#set({ messages })
+      } else {
+        const unopened = turn.isNew && messages.length === 0
+        this.#set({ messages, status: 'idle', error: undefined, ...(unopened ? { conversationId: undefined } : {}) })
+      }
+      this.#persistLocal()
+    }
+    if (this.#state.history === 'local' && this.#state.conversationId !== id) {
+      const stored = withoutTurn(this.#local.getMessages(id))
+      if (stored.length > 0) this.#local.saveMessages(id, stored)
+      else if (!this.#state.conversations.some((c) => c.id === id)) this.#local.deleteConversation(id)
+    }
+  }
+
   #failTurn(turn: Turn, e: unknown): void {
     if (!this.#turnActive(turn)) return
     const error = toError(e)
@@ -612,22 +731,54 @@ class ChatImpl implements Chat {
     for (const row of rows.map(fromRow)) {
       if (known.has(row.id)) continue
       known.add(row.id)
-      const i = messages.findIndex(
+      let i = messages.findIndex(
         (m) =>
           m.seq === undefined &&
           m.role === row.role &&
           (m.content === row.content || (row.tool !== undefined && m.tool?.name === row.tool.name))
       )
+      // A structured answer streams as the call of the structured-output tool, whose
+      // arguments are the answer's text: its row replaces that call. Only past the newest
+      // user message, where a stopped turn's identical call cannot be.
+      if (i < 0 && row.role === 'assistant') {
+        let j = messages.length - 1
+        while (j >= 0 && messages[j].role !== 'user') {
+          const m = messages[j]
+          if (m.seq === undefined && m.role === 'tool' && m.tool?.arguments === row.content) i = j
+          j--
+        }
+      }
       if (i >= 0) {
         const m = messages[i]
         messages[i] = {
           ...row,
           id: m.id,
           reasoning: m.reasoning ?? row.reasoning,
-          tool: m.tool ? { ...m.tool, status: row.tool?.status ?? m.tool.status } : row.tool
+          // The stream's call wins where it has a value; a stream cut short leaves gaps the row fills.
+          tool:
+            row.role === 'tool' && m.tool
+              ? {
+                  ...m.tool,
+                  arguments: m.tool.arguments ?? row.tool?.arguments,
+                  result: m.tool.result ?? row.tool?.result,
+                  status: row.tool?.status ?? m.tool.status
+                }
+              : row.tool
         }
       } else {
-        messages.push(row)
+        // A tool row nothing streamed, such as a provider-native web search, goes where a
+        // reload puts it: after the last message of a lower `seq`, before the streamed answer.
+        // Any other row closes the turn, a failure included, and stays last: above a streamed
+        // message that never got a row, it would hide the turn's failure.
+        let at = messages.length
+        for (let j = messages.length - 1; row.role === 'tool' && j >= 0; j--) {
+          const seq = messages[j].seq
+          if (seq !== undefined && seq < row.seq!) {
+            at = j + 1
+            break
+          }
+        }
+        messages.splice(at, 0, row)
       }
     }
     this.#set({ messages })
@@ -725,7 +876,19 @@ function fromRow(row: FlowConversationMessage): ChatMessage {
     stepName: row.step_name ?? undefined,
     pending: false,
     seq: row.created_seq,
-    tool: toolName ? { name: toolName, status: success ? 'success' : 'error' } : undefined
+    reasoning: row.reasoning ?? undefined,
+    attachments: row.attachments ?? undefined,
+    // The call the row carries: the model's arguments and what the model got back. For a
+    // failed tool the result is what it failed with, and the row's text names the tool
+    // rather than the reason.
+    tool: toolName
+      ? {
+          name: toolName,
+          status: success ? 'success' : 'error',
+          arguments: row.tool_arguments ?? undefined,
+          result: row.tool_result ?? undefined
+        }
+      : undefined
   }
 }
 
@@ -734,7 +897,8 @@ function fromConversation(row: FlowConversation): Conversation {
     id: row.id,
     title: row.title ?? undefined,
     createdAt: row.created_at,
-    updatedAt: row.updated_at
+    updatedAt: row.updated_at,
+    isTest: row.is_test
   }
 }
 

chat-sdk/src/history.ts

@@ -4,6 +4,8 @@ export interface LocalHistory {
   listConversations(): Conversation[]
   getMessages(conversationId: string): ChatMessage[]
   upsertConversation(conversation: Conversation): void
+  /** Changes a stored conversation's title in place; unlike `upsertConversation`, its position is kept. */
+  renameConversation(conversationId: string, title: string): void
   saveMessages(conversationId: string, messages: ChatMessage[]): void
   deleteConversation(conversationId: string): void
 }
@@ -55,6 +57,11 @@ export function createLocalHistory(storage: StorageLike | undefined, key: string
       }
       write(s)
     },
+    renameConversation(id, title) {
+      const s = read()
+      s.conversations = s.conversations.map((c) => (c.id === id ? { ...c, title } : c))
+      write(s)
+    },
     saveMessages(id, messages) {
       const s = read()
       s.messages[id] = messages.map((m) => ({ ...m, pending: false }))

chat-sdk/src/index.ts

@@ -5,6 +5,7 @@ export {
   WindmillApiError,
   readServerSentEvents,
   type WindmillChatApiOptions,
+  type ConversationKind,
   type FlowConversation,
   type FlowConversationMessage,
   type JobUpdateEvent,
@@ -13,8 +14,12 @@ export {
 export { parseStreamEvents, createStreamEventParser, type AgentStreamEvent } from './stream'
 export { followJob, type FollowEvent } from './follow'
 export { extractChatAnswer, conversationIdFor } from './utils'
+export { storedAttachmentName, uploadAttachments, CHAT_UPLOADS_PREFIX, type UploadedAttachment } from './attachments'
 export type {
+  AttachmentsInput,
+  AttachmentUpload,
   Chat,
+  ChatAttachment,
   ChatMessage,
   ChatOptions,
   ChatRole,
@@ -23,6 +28,7 @@ export type {
   Conversation,
   FetchLike,
   HistoryMode,
+  SendMessageOptions,
   StorageLike,
   TokenSource,
   ToolInvocation

chat-sdk/src/react.ts

@@ -11,6 +11,7 @@ export type UseWindmillChat = ChatState &
     | 'selectConversation'
     | 'loadConversations'
     | 'deleteConversation'
+    | 'renameConversation'
     | 'loadOlderMessages'
   > & { chat: Chat }
 
@@ -65,6 +66,7 @@ export function useWindmillChat(options: ChatOptions): UseWindmillChat {
       selectConversation: chat.selectConversation,
       loadConversations: chat.loadConversations,
       deleteConversation: chat.deleteConversation,
+      renameConversation: chat.renameConversation,
       loadOlderMessages: chat.loadOlderMessages
     }),
     [state, chat]

chat-sdk/src/types.ts

@@ -26,19 +26,23 @@ export interface ToolInvocation {
   status: 'running' | 'success' | 'error'
 }
 
+export interface ChatAttachment { input: string; s3: string; storage?: string; filename?: string }
+
 export interface ChatMessage {
   id: string
   role: ChatRole
   content: string
   /** The model's reasoning summary, when the provider streams one. */
   reasoning?: string
-  /** Set on `tool` messages that came from the live stream. */
+  /** The call on a `tool` message, from the live stream or its stored row; `callId` is only known from the stream. */
   tool?: ToolInvocation
   success: boolean
   createdAt: string
   jobId?: string
   /** The flow step that produced the message. */
   stepName?: string
+  /** The files a user message carried, as object-storage references. */
+  attachments?: ChatAttachment[]
   /** True while the message is optimistic or still streaming. */
   pending: boolean
   /** Id of the persisted row once the server has it; `id` itself never changes, so list keys stay stable. */
@@ -52,6 +56,11 @@ export interface Conversation {
   title: string | undefined
   createdAt: string
   updatedAt: string
+  /**
+   * Started from the flow editor's test panel rather than a deployed run. Known once the
+   * server has listed the conversation; unset for one only this client has seen.
+   */
+  isTest?: boolean
 }
 
 export interface ChatState {
@@ -118,18 +127,61 @@ export interface ChatOptions {
   onError?: (error: Error, turn: { conversationId: string; jobId?: string }) => void
 }
 
+/** A file sent with a message. It is uploaded to the workspace's object storage before the run starts. */
+export interface AttachmentUpload {
+  /** Kept as the last segment of the stored key, its extension corrected to the media type for PNG, JPEG and PDF. */
+  name: string
+  /** The bytes: a Blob, or a `data:` URL of them. */
+  data: Blob | string
+  /** The file's media type. Defaults to the Blob's own type, or the data URL's. */
+  mediaType?: string
+}
+
+/** The flow input the uploaded attachments are handed to: an `s3object` (`multiple: false`) or an `s3object[]`. */
+export interface AttachmentsInput {
+  name: string
+  multiple: boolean
+}
+
+export interface SendMessageOptions {
+  /** Extra flow inputs for this message, on top of `ChatOptions.inputs`. */
+  inputs?: Record<string, unknown>
+  /**
+   * Files to upload and hand to the flow as `{ s3, filename }` objects in `attachmentsInput`,
+   * the way an AI agent step reads `user_attachments`. A failed upload rejects `sendMessage`
+   * and the run never starts; `stop()` during the upload does the same with an `AbortError`.
+   */
+  attachments?: AttachmentUpload[]
+  /** Required with `attachments`, which also need message text. With `multiple: false`, more than one attachment is refused before anything uploads. */
+  attachmentsInput?: AttachmentsInput
+}
+
 export interface Chat {
   getState(): ChatState
   /** Calls `listener` now and on every change; returns the unsubscribe function (Svelte store contract). */
   subscribe(listener: (state: ChatState) => void): () => void
-  /** Sends a message in the current conversation, starting one when there is none. Resolves when the answer is complete. */
-  sendMessage(text: string, options?: { inputs?: Record<string, unknown> }): Promise<void>
+  /**
+   * Sends a message in the current conversation, starting one when there is none. Resolves
+   * when the answer is complete. Rejects when the message could not be sent at all — a turn
+   * already running, an attachment that failed to upload — without touching the transcript.
+   */
+  sendMessage(text: string, options?: SendMessageOptions): Promise<void>
   /** Stops following the answer and asks Windmill to cancel the run. */
   stop(): Promise<void>
   newConversation(): void
   selectConversation(conversationId: string): Promise<void>
-  loadConversations(options?: { page?: number; perPage?: number }): Promise<Conversation[]>
+  /**
+   * `kind` narrows server history to the flow editor's test chats, the deployed flow's
+   * own (the server's default), or both. Local history has no test chats and ignores it.
+   */
+  loadConversations(options?: {
+    page?: number
+    perPage?: number
+    kind?: 'test' | 'deployed' | 'all'
+  }): Promise<Conversation[]>
   deleteConversation(conversationId: string): Promise<void>
+  /** Sets a conversation's title. The list keeps its order: only a turn moves a conversation. */
+  renameConversation(conversationId: string, title: string): Promise<void>
   loadOlderMessages(): Promise<void>
   /** Stops background work (stream, polling) and writes local history out. The chat stays usable. */
   destroy(): void

chat-sdk/src/utils.ts

@@ -69,6 +69,12 @@ export function conversationTitle(firstMessage: string): string {
   return chars.length > 25 ? `${chars.slice(0, 25).join('')}...` : firstMessage
 }
 
+/** The server's bound on a typed title: 252 characters plus an ellipsis fits its 255-char column. */
+export function truncateTitle(title: string): string {
+  const chars = Array.from(title)
+  return chars.length > 252 ? `${chars.slice(0, 252).join('')}...` : title
+}
+
 export function sleep(ms: number, signal?: AbortSignal): Promise<void> {
   return new Promise((resolve, reject) => {
     if (signal?.aborted) return reject(abortError())

chat-sdk/test/ai-sdk.test.ts

@@ -2,7 +2,7 @@ import { describe, expect, test } from 'bun:test'
 import type { UIMessage, UIMessageChunk } from 'ai'
 import { createWindmillChatTransport, toUIMessages } from '../src/ai-sdk'
 import type { ChatMessage } from '../src/types'
-import { fetchMock, json, ndjson, sse, text, type Route } from './support'
+import { fetchMock, json, messageRow, ndjson, sse, text, type Route } from './support'
 
 const FLOW = 'f/chat/agent'
 const run: Route = (c) =>
@@ -141,6 +141,26 @@ describe('createWindmillChatTransport', () => {
     expect(second[1]).toMatchObject({ errorText: 'ExecutionErr: boom' })
   })
 
+  test('loads the attachments of a user row, the call an MCP tool row carries and the reasoning behind an answer', async () => {
+    const { fetch } = fetchMock((c) =>
+      c.method === 'GET' && c.url.pathname.endsWith('/messages')
+        ? json([
+            messageRow(1, 'user', 'hi', { attachments: [{ input: 'files', s3: 'chat/a.png', filename: 'a.png' }] }),
+            messageRow(2, 'tool', 'Used lookup tool', { job_id: 'agent-job', reasoning: 'why', tool_arguments: '{"q":1}', tool_result: '42' }),
+            messageRow(3, 'assistant', 'The answer is 42', { reasoning: 'hmm' })
+          ])
+        : undefined
+    )
+    const transport = createWindmillChatTransport({ baseUrl: 'http://wm.test', workspace: 'ws', flowPath: FLOW, fetch })
+    const ui = await transport.loadMessages('c')
+    expect(ui.map((m) => m.parts.map((p) => p.type))).toEqual([['text'], ['reasoning', 'dynamic-tool', 'reasoning', 'text']])
+    expect(ui[0].metadata).toEqual({ attachments: [{ input: 'files', s3: 'chat/a.png', filename: 'a.png' }] })
+    expect(ui[1].metadata).toBeUndefined()
+    expect(ui[1].parts[0]).toMatchObject({ type: 'reasoning', text: 'why' })
+    expect(ui[1].parts[1]).toMatchObject({ toolName: 'lookup', state: 'output-available', input: { q: 1 }, output: 42 })
+    expect(ui[1].parts[2]).toMatchObject({ type: 'reasoning', text: 'hmm' })
+  })
+
   test('refuses attachments with a clear error', async () => {
     const transport = createWindmillChatTransport({ baseUrl: 'http://wm.test', workspace: 'ws', flowPath: FLOW, fetch: fetchMock().fetch })
     await expect(
@@ -175,4 +195,11 @@ describe('toUIMessages', () => {
     expect(ui[1].parts[0]).toMatchObject({ toolCallId: 'c1', toolName: 'lookup', state: 'output-available', input: { q: 1 }, output: 42 })
     expect(ui[3].parts[0]).toMatchObject({ state: 'output-error', errorText: 'Error executing lookup' })
   })
+
+  test('keeps a stored JSON null result rather than the row text', () => {
+    const ui = toUIMessages([
+      { success: true, createdAt: '2026-01-01T00:00:00Z', pending: false, id: 't1', role: 'tool', content: 'Used notify tool', tool: { callId: 'c1', name: 'notify', status: 'success', arguments: '{}', result: 'null' } }
+    ])
+    expect(ui[0].parts[0]).toMatchObject({ type: 'dynamic-tool', state: 'output-available', output: null })
+  })
 })

chat-sdk/test/api.test.ts

@@ -0,0 +1,14 @@
+import { describe, expect, test } from 'bun:test'
+import { WindmillChatApi } from '../src/api'
+
+describe('WindmillChatApi.attachmentUrl', () => {
+  test('points at the workspace download endpoint, with the storage only when there is one', () => {
+    const api = new WindmillChatApi({ baseUrl: 'https://wm.test/api/', workspace: 'my ws' })
+    expect(api.attachmentUrl({ input: 'files', s3: 'chat/a b&c.png', storage: 'secondary' })).toBe(
+      'https://wm.test/api/w/my%20ws/job_helpers/download_s3_file?file_key=chat%2Fa+b%26c.png&storage=secondary'
+    )
+    expect(api.attachmentUrl({ input: 'files', s3: 'chat/a.png' })).toBe(
+      'https://wm.test/api/w/my%20ws/job_helpers/download_s3_file?file_key=chat%2Fa.png'
+    )
+  })
+})

chat-sdk/test/assistant-ui.test.ts

@@ -31,6 +31,13 @@ describe('assistant-ui conversion', () => {
     expect(toThreadMessage(turns[0])).toMatchObject({ role: 'user', content: [{ type: 'text', text: 'hi' }] })
   })
 
+  test('keeps a stored JSON null result rather than the row text', () => {
+    const turn = groupTurns([
+      { ...base, id: 't1', role: 'tool', content: 'Used notify tool', tool: { callId: 'c1', name: 'notify', status: 'success', arguments: '{}', result: 'null' } }
+    ])[0]
+    expect(toThreadMessage(turn).content).toMatchObject([{ type: 'tool-call', toolName: 'notify', result: null }])
+  })
+
   test('marks a failed answer as incomplete', () => {
     const [turn] = groupTurns([{ ...base, id: 'a', role: 'assistant', content: 'boom', success: false }])
     expect(toThreadMessage(turn).status).toEqual({ type: 'incomplete', reason: 'error', error: 'boom' })

chat-sdk/test/attachments.test.ts

@@ -0,0 +1,444 @@
+import { describe, expect, test } from 'bun:test'
+import { WindmillChatApi } from '../src/api'
+import { storedAttachmentName, uploadAttachments } from '../src/attachments'
+import { createChat } from '../src/chat'
+import type { ChatOptions } from '../src/types'
+import { abortError } from '../src/utils'
+import { fetchMock, json, memoryStorage, sse, text, type RecordedCall, type Route } from './support'
+
+const BASE = 'http://wm.test'
+const FLOW = 'f/chat/agent'
+const UPLOAD_PATH = '/api/w/ws/job_helpers/upload_s3_file'
+
+const run: Route = (c) =>
+  c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
+    ? text('job-1')
+    : undefined
+
+/** Stores under the key it was asked to, like the server with a `file_key`. */
+const upload: Route = (c) =>
+  c.url.pathname === UPLOAD_PATH
+    ? json({ file_key: c.url.searchParams.get('file_key') })
+    : undefined
+
+const answer: Route = (c) =>
+  c.url.pathname === '/api/w/ws/jobs_u/getupdate_sse/job-1'
+    ? sse([
+        {
+          type: 'update',
+          completed: true,
+          only_result: { output: 'ok', messages: [] }
+        }
+      ])
+    : undefined
+
+function options(fetch: ChatOptions['fetch']): ChatOptions {
+  return {
+    flowPath: FLOW,
+    baseUrl: BASE,
+    workspace: 'ws',
+    token: 'tok',
+    fetch,
+    storage: memoryStorage()
+  }
+}
+
+const uploads = (calls: RecordedCall[]) => calls.filter((c) => c.url.pathname === UPLOAD_PATH)
+const runs = (calls: RecordedCall[]) =>
+  calls.filter((c) => c.url.pathname.startsWith('/api/w/ws/jobs/run/'))
+
+const png = new Blob([new Uint8Array([0x89, 0x50, 0x4e, 0x47])], {
+  type: 'image/png'
+})
+const pdf = new Blob(['%PDF-1.7'], { type: 'application/pdf' })
+
+describe('storedAttachmentName', () => {
+  // The worker reads the media type from the key's extension, so it has to match the bytes.
+  test('renames a re-encoded image and gives a bare name its extension', () => {
+    expect(storedAttachmentName('photo.webp', 'image/png')).toBe('photo.png')
+    expect(storedAttachmentName('holiday.png', 'image/jpeg')).toBe('holiday.jpg')
+    expect(storedAttachmentName('contract', 'application/pdf')).toBe('contract.pdf')
+    expect(storedAttachmentName('report.2026.final.webp', 'image/png')).toBe(
+      'report.2026.final.png'
+    )
+  })
+
+  test('leaves a type it does not know alone', () => {
+    expect(storedAttachmentName('notes.csv', 'text/csv')).toBe('notes.csv')
+  })
+})
+
+describe('sendMessage with attachments', () => {
+  test('uploads each file under the turn prefix and hands the list to the input', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+
+    await chat.sendMessage('read these', {
+      inputs: { locale: 'fr' },
+      attachments: [
+        { name: 'photo.webp', data: png },
+        { name: 'contract', data: pdf },
+        // A data URL is decoded to its bytes; the mediaType names what they are.
+        {
+          name: 'photo.webp',
+          data: `data:image/png;base64,${btoa('\x89PNG')}`
+        }
+      ],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+
+    const keys = uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
+    expect(keys).toHaveLength(3)
+    const prefix = keys[0].split('/').slice(0, 3).join('/')
+    expect(prefix).toMatch(/^windmill_uploads\/chat\/[0-9a-f-]{36}$/)
+    expect(keys).toEqual([
+      `${prefix}/0/photo.png`,
+      `${prefix}/1/contract.pdf`,
+      `${prefix}/2/photo.png`
+    ])
+    expect(uploads(calls).map((c) => c.url.searchParams.get('content_type'))).toEqual([
+      'image/png',
+      'application/pdf',
+      'image/png'
+    ])
+    expect(uploads(calls).map((c) => c.headers['content-type'])).toEqual([
+      'image/png',
+      'application/pdf',
+      'image/png'
+    ])
+    expect(new Uint8Array(await uploads(calls)[2].raw!.arrayBuffer())).toEqual(
+      new Uint8Array([0x89, 0x50, 0x4e, 0x47])
+    )
+
+    expect(runs(calls)[0].body).toEqual({
+      locale: 'fr',
+      user_message: 'read these',
+      files: [
+        { s3: `${prefix}/0/photo.png`, filename: 'photo.png' },
+        { s3: `${prefix}/1/contract.pdf`, filename: 'contract.pdf' },
+        { s3: `${prefix}/2/photo.png`, filename: 'photo.png' }
+      ]
+    })
+    expect(chat.getState().status).toBe('idle')
+  })
+
+  test('hands a single object to an input that holds one file', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    await chat.sendMessage('read this', {
+      attachments: [{ name: 'contract.pdf', data: pdf }],
+      attachmentsInput: { name: 'file', multiple: false }
+    })
+    const body = runs(calls)[0].body as Record<string, unknown>
+    expect(body.file).toEqual({
+      s3: expect.stringMatching(/\/0\/contract\.pdf$/),
+      filename: 'contract.pdf'
+    })
+  })
+
+  test('refuses several files for an input that holds one, before uploading any', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    await expect(
+      chat.sendMessage('read these', {
+        attachments: [
+          { name: 'a.pdf', data: pdf },
+          { name: 'b.png', data: png }
+        ],
+        attachmentsInput: { name: 'file', multiple: false }
+      })
+    ).rejects.toThrow('holds one file')
+    expect(calls).toHaveLength(0)
+    expect(chat.getState().messages).toEqual([])
+  })
+
+  test('refuses attachments without message text, before uploading', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    await expect(
+      chat.sendMessage('  ', {
+        attachments: [{ name: 'a.pdf', data: pdf }],
+        attachmentsInput: { name: 'files', multiple: true }
+      })
+    ).rejects.toThrow('need a message')
+    expect(calls).toHaveLength(0)
+  })
+
+  test('refuses attachments without an input to put them in', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    await expect(
+      chat.sendMessage('hi', { attachments: [{ name: 'a.pdf', data: pdf }] })
+    ).rejects.toThrow('attachmentsInput')
+    expect(calls).toHaveLength(0)
+  })
+
+  test('a failed upload rejects without a run, and withdraws the message', async () => {
+    const { fetch, calls } = fetchMock(
+      (c) => (c.url.pathname === UPLOAD_PATH ? text('no object storage', 500) : undefined),
+      run,
+      answer
+    )
+    const chat = createChat(options(fetch))
+    const statuses: string[] = []
+    chat.subscribe((s) => statuses.push(s.status))
+
+    await expect(
+      chat.sendMessage('read this', {
+        attachments: [{ name: 'contract.pdf', data: pdf }],
+        attachmentsInput: { name: 'files', multiple: true }
+      })
+    ).rejects.toThrow('no object storage')
+
+    expect(runs(calls)).toHaveLength(0)
+    // Shown as submitted while uploading, then withdrawn whole: no message, no conversation.
+    expect(statuses).toContain('submitted')
+    const state = chat.getState()
+    expect(state.status).toBe('idle')
+    expect(state.messages).toEqual([])
+    expect(state.conversationId).toBeUndefined()
+    expect(state.conversations).toEqual([])
+    // The chat is free for the next message.
+    await chat.sendMessage('plain')
+    expect(runs(calls)).toHaveLength(1)
+  })
+
+  test('stop() during the upload aborts it and withdraws the message', async () => {
+    const { fetch, calls } = fetchMock(
+      (c) =>
+        c.url.pathname === UPLOAD_PATH
+          ? new Promise((_, reject) =>
+              c.signal!.addEventListener('abort', () => reject(abortError()))
+            )
+          : undefined,
+      run,
+      answer
+    )
+    const chat = createChat(options(fetch))
+    const sending = chat.sendMessage('read this', {
+      attachments: [{ name: 'contract.pdf', data: pdf }],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    await new Promise((r) => setTimeout(r, 0))
+    expect(chat.getState().status).toBe('submitted')
+    await chat.stop()
+    await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
+    expect(runs(calls)).toHaveLength(0)
+    expect(chat.getState()).toMatchObject({
+      status: 'idle',
+      messages: [],
+      conversationId: undefined
+    })
+  })
+
+  test('a switch away mid-upload leaves no conversation behind', async () => {
+    const storage = memoryStorage()
+    const { fetch, calls } = fetchMock(
+      (c) =>
+        c.url.pathname === UPLOAD_PATH
+          ? new Promise((_, reject) =>
+              c.signal!.addEventListener('abort', () => reject(abortError()))
+            )
+          : undefined,
+      run,
+      answer
+    )
+    const chat = createChat({ ...options(fetch), storage })
+    const sending = chat.sendMessage('never runs', {
+      attachments: [{ name: 'contract.pdf', data: pdf }],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    await new Promise((r) => setTimeout(r, 0))
+    const opened = chat.getState().conversationId!
+    chat.newConversation()
+    await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
+
+    expect(runs(calls)).toHaveLength(0)
+    expect(chat.getState().conversations.map((c) => c.id)).not.toContain(opened)
+    const reloaded = createChat({ ...options(fetch), storage })
+    expect((await reloaded.loadConversations()).map((c) => c.id)).not.toContain(opened)
+  })
+
+  test('a failed upload aborts the rest of its batch and deletes nothing', async () => {
+    let first: (r: Response) => void = () => {}
+    const { fetch, calls } = fetchMock(
+      (c) => {
+        if (c.url.pathname !== UPLOAD_PATH) return undefined
+        const key = c.url.searchParams.get('file_key')!
+        // The first file lands after the second has already failed.
+        if (key.includes('/0/')) return new Promise<Response>((resolve) => (first = resolve))
+        setTimeout(() => first(json({ file_key: keys()[0] })), 5)
+        return text('quota exceeded', 507)
+      },
+      run,
+      answer
+    )
+    const keys = () => uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
+    const chat = createChat(options(fetch))
+    await expect(
+      chat.sendMessage('read these', {
+        attachments: [
+          { name: 'a.pdf', data: pdf },
+          { name: 'b.png', data: png }
+        ],
+        attachmentsInput: { name: 'files', multiple: true }
+      })
+    ).rejects.toThrow('quota exceeded')
+    // The upload still in flight when the other failed was told to stop.
+    expect(uploads(calls)[0].signal?.aborted).toBe(true)
+    expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
+    expect(runs(calls)).toHaveLength(0)
+  })
+
+  test('a send made right after stop() is not reset by the stopped upload', async () => {
+    let releaseRun: (r: Response) => void = () => {}
+    const { fetch, calls } = fetchMock(
+      (c) =>
+        c.url.pathname === UPLOAD_PATH
+          ? new Promise((_, reject) =>
+              c.signal!.addEventListener('abort', () => reject(abortError()))
+            )
+          : undefined,
+      (c) =>
+        c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
+          ? new Promise<Response>((resolve) => (releaseRun = resolve))
+          : undefined,
+      answer
+    )
+    const chat = createChat(options(fetch))
+    // An existing conversation, so the stopped turn and the next one share it.
+    const first = chat.sendMessage('first')
+    await new Promise((r) => setTimeout(r, 0))
+    releaseRun(text('job-1'))
+    await first
+    const stopped = chat.sendMessage('with a file', {
+      attachments: [{ name: 'a.pdf', data: pdf }],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    await new Promise((r) => setTimeout(r, 0))
+    void chat.stop()
+    const next = chat.sendMessage('right after')
+    await expect(stopped).rejects.toMatchObject({ name: 'AbortError' })
+    await new Promise((r) => setTimeout(r, 0))
+    expect(chat.getState().status).toBe('submitted')
+    expect(chat.getState().messages.map((m) => m.content)).toContain('right after')
+    expect(chat.getState().messages.map((m) => m.content)).not.toContain('with a file')
+    releaseRun(text('job-1'))
+    await next
+    expect(runs(calls)).toHaveLength(2)
+  })
+
+  test('a conversation is listed only once its run starts', async () => {
+    let failUpload: (r: Response) => void = () => {}
+    const { fetch } = fetchMock(
+      (c) =>
+        c.url.pathname === UPLOAD_PATH
+          ? new Promise<Response>((resolve) => (failUpload = resolve))
+          : undefined,
+      run,
+      answer
+    )
+    const chat = createChat(options(fetch))
+    const sending = chat.sendMessage('read this', {
+      attachments: [{ name: 'a.pdf', data: pdf }],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    await new Promise((r) => setTimeout(r, 0))
+    expect(chat.getState()).toMatchObject({ status: 'submitted', conversations: [] })
+    failUpload(text('boom', 500))
+    await expect(sending).rejects.toThrow('boom')
+    expect(chat.getState().conversations).toEqual([])
+  })
+
+  test('an already aborted signal uploads nothing', async () => {
+    const { fetch, calls } = fetchMock(upload)
+    const api = new WindmillChatApi({ baseUrl: BASE, workspace: 'ws', token: 'tok', fetch })
+    const controller = new AbortController()
+    controller.abort()
+    await expect(
+      uploadAttachments(api, [{ name: 'a.pdf', data: pdf }], 'turn', controller.signal)
+    ).rejects.toMatchObject({ name: 'AbortError' })
+    expect(calls).toHaveLength(0)
+  })
+
+  test('the pending user message carries its uploaded files before the run returns', async () => {
+    let releaseRun: (r: Response) => void = () => {}
+    const { fetch, calls } = fetchMock(
+      upload,
+      (c) =>
+        c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
+          ? new Promise<Response>((resolve) => (releaseRun = resolve))
+          : undefined,
+      answer
+    )
+    const chat = createChat(options(fetch))
+    const sending = chat.sendMessage('read these', {
+      attachments: [
+        { name: 'photo.webp', data: png },
+        { name: 'contract', data: pdf }
+      ],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    while (runs(calls).length === 0) await new Promise((r) => setTimeout(r, 1))
+    const keys = uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
+    const pending = chat.getState().messages.find((m) => m.role === 'user')!
+    expect(pending.pending).toBe(true)
+    expect(pending.attachments).toEqual([
+      { input: 'files', s3: keys[0], filename: 'photo.png' },
+      { input: 'files', s3: keys[1], filename: 'contract.pdf' }
+    ])
+    releaseRun(text('job-1'))
+    await sending
+  })
+
+  test('an explicit mediaType wins over the type a data URL declares', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    await chat.sendMessage('read this', {
+      attachments: [
+        {
+          name: 'contract',
+          data: `data:application/octet-stream;base64,${btoa('%PDF')}`,
+          mediaType: 'application/pdf'
+        }
+      ],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    const call = uploads(calls)[0]
+    expect(call.url.searchParams.get('file_key')).toMatch(/\/0\/contract\.pdf$/)
+    expect(call.url.searchParams.get('content_type')).toBe('application/pdf')
+  })
+
+  test('stop() after the uploads land but before the run starts runs nothing', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    const sending = chat.sendMessage('read this', {
+      attachments: [{ name: 'contract.pdf', data: pdf }],
+      attachmentsInput: { name: 'files', multiple: true }
+    })
+    // The upload responds at once; Stop lands before the send resumes after it.
+    while (uploads(calls).length === 0) await Promise.resolve()
+    await chat.stop()
+    await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
+    expect(runs(calls)).toHaveLength(0)
+    expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
+    expect(chat.getState()).toMatchObject({ status: 'idle', messages: [], conversations: [] })
+  })
+
+  test('a subscriber stopping when the attachments appear runs nothing', async () => {
+    const { fetch, calls } = fetchMock(upload, run, answer)
+    const chat = createChat(options(fetch))
+    chat.subscribe((s) => {
+      if (s.messages.some((m) => m.attachments)) void chat.stop()
+    })
+    await expect(
+      chat.sendMessage('read this', {
+        attachments: [{ name: 'contract.pdf', data: pdf }],
+        attachmentsInput: { name: 'files', multiple: true }
+      })
+    ).rejects.toMatchObject({ name: 'AbortError' })
+    expect(runs(calls)).toHaveLength(0)
+    expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
+    expect(chat.getState()).toMatchObject({ status: 'idle', messages: [], conversations: [] })
+  })
+})

chat-sdk/test/chat.test.ts

@@ -327,6 +327,113 @@ describe('createChat with server history', () => {
     expect(messagesCall.headers.authorization).toBeUndefined()
   })
 
+  test('a persisted row brings back its attachments, reasoning and the call an MCP tool row carries', async () => {
+    const { fetch } = fetchMock(
+      (c) =>
+        c.method === 'GET' && c.url.pathname === '/api/w/ws/flow_conversations/conv-1/messages'
+          ? json([
+              messageRow(1, 'user', 'hi', {
+                attachments: [{ input: 'files', s3: 'chat/a.png', storage: 'secondary', filename: 'a.png' }]
+              }),
+              messageRow(2, 'tool', 'Used lookup tool', {
+                job_id: 'agent-job',
+                tool_arguments: '{"q":1}',
+                tool_result: '42'
+              }),
+              messageRow(3, 'tool', 'Error executing lookup', {
+                job_id: 'agent-job',
+                success: false,
+                tool_arguments: '{"q":2}',
+                tool_result: 'MCP tool error: boom'
+              }),
+              messageRow(4, 'assistant', 'The answer is 42', { reasoning: 'hmm' }),
+              messageRow(5, 'assistant', 'Hello'),
+              messageRow(6, 'tool', 'Used get_price tool', {
+                job_id: 'script-tool-job',
+                tool_arguments: '{"item":"widget"}',
+                tool_result: '{"price":42}'
+              })
+            ])
+          : undefined
+    )
+    const chat = createChat(options({ history: 'server' }, fetch))
+    await chat.selectConversation('conv-1')
+
+    const [user, used, failed, answer, plain, scriptTool] = chat.getState().messages
+    expect(scriptTool).toMatchObject({ jobId: 'script-tool-job', tool: { name: 'get_price', status: 'success', arguments: '{"item":"widget"}', result: '{"price":42}' } })
+    expect(user.attachments).toEqual([{ input: 'files', s3: 'chat/a.png', storage: 'secondary', filename: 'a.png' }])
+    expect(answer.attachments).toBeUndefined()
+    expect(used.tool).toEqual({ name: 'lookup', status: 'success', arguments: '{"q":1}', result: '42' })
+    expect(failed.tool).toEqual({ name: 'lookup', status: 'error', arguments: '{"q":2}', result: 'MCP tool error: boom' })
+    expect(answer.reasoning).toBe('hmm')
+    expect(plain.reasoning).toBeUndefined()
+  })
+
+  test('a row nothing streamed, like a web search, lands before the answer as on reload', async () => {
+    const { fetch } = fetchMock(
+      run,
+      (c) =>
+        c.url.pathname === streamPath
+          ? sse([
+              {
+                type: 'update',
+                new_result_stream: ndjson({ type: 'token_delta', content: 'Rust.' }),
+                stream_offset: 1,
+                completed: true,
+                only_result: { output: 'Rust.', messages: [] }
+              }
+            ])
+          : undefined,
+      (c) =>
+        c.url.pathname.endsWith('/messages')
+          ? json([
+              messageRow(91, 'user', 'hi'),
+              messageRow(92, 'tool', 'Used websearch tool', { job_id: 'step-1', tool_result: '[{"url":"https://example.com"}]' }),
+              messageRow(93, 'assistant', 'Rust.', { job_id: 'step-1' })
+            ])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.sendMessage('hi')
+    expect(chat.getState().messages.map((m) => [m.role, m.content, m.seq])).toEqual([
+      ['user', 'hi', 91],
+      ['tool', 'Used websearch tool', 92],
+      ['assistant', 'Rust.', 93]
+    ])
+  })
+
+  test('a failure row stays after streamed text that never got a row', async () => {
+    const { fetch } = fetchMock(
+      run,
+      (c) =>
+        c.url.pathname === streamPath
+          ? sse([
+              {
+                type: 'update',
+                new_result_stream: ndjson({ type: 'token_delta', content: 'Let me look' }),
+                stream_offset: 1,
+                completed: true,
+                only_result: { error: { name: 'ExecutionErr', message: 'boom' } }
+              }
+            ])
+          : undefined,
+      (c) =>
+        c.url.pathname.endsWith('/messages')
+          ? json([messageRow(91, 'user', 'hi'), messageRow(92, 'assistant', 'boom', { job_id: 'step-1', success: false })])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.sendMessage('hi')
+    const messages = chat.getState().messages
+    expect(messages.map((m) => [m.content, m.success])).toEqual([
+      ['hi', true],
+      ['Let me look', true],
+      ['boom', false]
+    ])
+  })
+
   test('keeps the streamed answer until its row lands, even when a tool row lands first', async () => {
     let messageFetches = 0
     const { fetch } = fetchMock(
@@ -414,6 +521,112 @@ describe('createChat with server history', () => {
     expect(chat.getState().conversations.map((c) => c.id)).toEqual(['c1', 'c2'])
   })
 
+  test('lists one kind of conversation and carries which kind each one is', async () => {
+    const row = (id: string, is_test: boolean) => ({
+      id,
+      workspace_id: 'ws',
+      flow_path: FLOW,
+      title: id,
+      created_at: '2026-01-01T00:00:00Z',
+      updated_at: '2026-01-01T00:00:00Z',
+      created_by: 'admin',
+      is_test
+    })
+    const { fetch, calls } = fetchMock((c) =>
+      c.url.pathname === '/api/w/ws/flow_conversations/list'
+        ? json(c.url.searchParams.get('kind') === 'test' ? [row('t1', true)] : [row('d1', false)])
+        : undefined
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.loadConversations()
+    expect(calls[0].url.searchParams.has('kind')).toBe(false)
+    expect(chat.getState().conversations.map((c) => [c.id, c.isTest])).toEqual([['d1', false]])
+    await chat.loadConversations({ kind: 'test' })
+    expect(calls[1].url.searchParams.get('kind')).toBe('test')
+    expect(chat.getState().conversations.map((c) => [c.id, c.isTest])).toEqual([['t1', true]])
+  })
+
+  test('a list for a kind no longer asked for does not replace the newer one', async () => {
+    const row = (id: string, is_test: boolean) => ({
+      id,
+      workspace_id: 'ws',
+      flow_path: FLOW,
+      title: id,
+      created_at: '2026-01-01T00:00:00Z',
+      updated_at: '2026-01-01T00:00:00Z',
+      created_by: 'admin',
+      is_test
+    })
+    const { fetch } = fetchMock((c) => {
+      if (c.url.pathname !== '/api/w/ws/flow_conversations/list') return undefined
+      if (c.url.searchParams.get('kind') === 'test') {
+        return new Promise<Response>((r) => setTimeout(() => r(json([row('t1', true)])), 50))
+      }
+      return json([row('d1', false)])
+    })
+    const chat = createChat(options({}, fetch))
+    const slow = chat.loadConversations({ kind: 'test' })
+    await chat.loadConversations({ kind: 'deployed' })
+    await slow
+    expect(chat.getState().conversations.map((c) => c.id)).toEqual(['d1'])
+    // Another kind asked for on a later page starts its own listing rather than appending.
+    await chat.loadConversations({ page: 2, kind: 'test' })
+    expect(chat.getState().conversations.map((c) => c.id)).toEqual(['t1'])
+  })
+
+  test('the refresh after a new turn lists the kind last asked for', async () => {
+    const { fetch, calls } = fetchMock(
+      run,
+      (c) =>
+        c.url.pathname === streamPath
+          ? sse([{ type: 'update', completed: true, only_result: { output: 'Hello', messages: [] } }])
+          : undefined,
+      (c) =>
+        c.method === 'GET' && c.url.pathname.endsWith('/messages')
+          ? json([messageRow(11, 'user', 'hi'), messageRow(12, 'assistant', 'Hello', { job_id: 'agent-job' })])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.loadConversations({ kind: 'test' })
+    await chat.sendMessage('hi')
+    const lists = calls.filter((c) => c.url.pathname === '/api/w/ws/flow_conversations/list')
+    expect(lists.length).toBeGreaterThan(1)
+    expect(lists.every((c) => c.url.searchParams.get('kind') === 'test')).toBe(true)
+  })
+
+  test('renaming a conversation keeps its place in the list', async () => {
+    const row = (id: string) => ({
+      id,
+      workspace_id: 'ws',
+      flow_path: FLOW,
+      title: id,
+      created_at: '2026-01-01T00:00:00Z',
+      updated_at: '2026-01-01T00:00:00Z',
+      created_by: 'admin',
+      is_test: false
+    })
+    const { fetch, calls } = fetchMock(
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([row('c1'), row('c2')]) : undefined),
+      (c) =>
+        c.method === 'POST' && c.url.pathname === '/api/w/ws/flow_conversations/update/c2'
+          ? text('Conversation c2 updated')
+          : undefined
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.loadConversations()
+    await chat.renameConversation('c2', '  Budget review  ')
+    expect(calls[1].body).toEqual({ title: 'Budget review' })
+    expect(chat.getState().conversations.map((c) => [c.id, c.title])).toEqual([
+      ['c1', 'c1'],
+      ['c2', 'Budget review']
+    ])
+    // Cut as the server cuts, so what is shown is what is stored.
+    await chat.renameConversation('c2', 'x'.repeat(300))
+    expect(chat.getState().conversations[1].title).toBe('x'.repeat(252) + '...')
+    expect(calls[2].body).toEqual({ title: 'x'.repeat(252) + '...' })
+  })
+
   test('a turn started right after stop() is not touched by the stop sync', async () => {
     let jobs = 0
     const { fetch } = fetchMock(
@@ -583,6 +796,120 @@ describe('createChat with server history', () => {
     expect(chat.getState().messages.map((m) => m.content)).toEqual(['hi', 'Let me check', 'Used search tool', 'Final answer'])
   })
 
+  test('thinking that led to a tool call rides on the call, live and once its row lands', async () => {
+    const { fetch } = fetchMock(
+      run,
+      (c) =>
+        c.url.pathname === streamPath
+          ? sse([
+              {
+                type: 'update',
+                // No `tool_call_arguments`: a stream cut short leaves the call without them.
+                new_result_stream: ndjson(
+                  { type: 'reasoning_token_delta', content: 'r1' },
+                  { type: 'tool_call', call_id: 'c1', function_name: 'lookup' },
+                  { type: 'tool_result', call_id: 'c1', function_name: 'lookup', result: '1', success: true },
+                  { type: 'token_delta', content: 'Final' }
+                ),
+                stream_offset: 4,
+                completed: true,
+                only_result: { output: 'Final', messages: [] }
+              }
+            ])
+          : undefined,
+      (c) =>
+        c.url.pathname.endsWith('/messages')
+          ? json([
+              messageRow(71, 'user', 'hi'),
+              messageRow(72, 'tool', 'Used lookup tool', { job_id: 'step-1', reasoning: 'r1', tool_arguments: '{"q":1}', tool_result: '1' }),
+              messageRow(73, 'assistant', 'Final', { job_id: 'step-1' })
+            ])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.sendMessage('hi')
+    const messages = chat.getState().messages
+    expect(messages.map((m) => [m.role, m.content, m.reasoning, m.seq])).toEqual([
+      ['user', 'hi', undefined, 71],
+      ['tool', 'Used lookup tool', 'r1', 72],
+      ['assistant', 'Final', undefined, 73]
+    ])
+    expect(messages[1].tool).toMatchObject({ callId: 'c1', arguments: '{"q":1}', result: '1', status: 'success' })
+  })
+
+  test('a structured answer row replaces the call it streamed as', async () => {
+    const { fetch } = fetchMock(
+      run,
+      (c) =>
+        c.url.pathname === streamPath
+          ? sse([
+              {
+                type: 'update',
+                new_result_stream: ndjson(
+                  { type: 'reasoning_token_delta', content: 'hmm' },
+                  { type: 'tool_call', call_id: 'c9', function_name: 'structured_output' },
+                  { type: 'tool_call_arguments', call_id: 'c9', function_name: 'structured_output', arguments: '{"n": 1}' },
+                  { type: 'tool_execution', call_id: 'c9', function_name: 'structured_output' }
+                ),
+                stream_offset: 4,
+                completed: true,
+                only_result: { output: { n: 1 }, messages: [] }
+              }
+            ])
+          : undefined,
+      (c) =>
+        c.url.pathname.endsWith('/messages')
+          ? json([messageRow(75, 'user', 'hi'), messageRow(76, 'assistant', '{"n": 1}', { job_id: 'step-1', reasoning: 'hmm' })])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    await chat.sendMessage('hi')
+    expect(chat.getState().messages.map((m) => [m.role, m.content, m.reasoning, m.tool])).toEqual([
+      ['user', 'hi', undefined, undefined],
+      ['assistant', '{"n": 1}', 'hmm', undefined]
+    ])
+  })
+
+  test('a structured answer row leaves a stopped turn its identical call', async () => {
+    const call = (id: string) =>
+      ndjson(
+        { type: 'tool_call', call_id: id, function_name: 'structured_output' },
+        { type: 'tool_call_arguments', call_id: id, function_name: 'structured_output', arguments: '{"ok": true}' }
+      )
+    let jobs = 0
+    const { fetch } = fetchMock(
+      (c) => (c.method === 'POST' && c.url.pathname.includes('/jobs/run/f/') ? text(`job-${++jobs}`) : undefined),
+      (c) =>
+        c.url.pathname.endsWith('/getupdate_sse/job-1') ? sse([{ type: 'update', new_result_stream: call('c1'), stream_offset: 2 }]) : undefined,
+      (c) =>
+        c.url.pathname.endsWith('/getupdate_sse/job-2')
+          ? sse([{ type: 'update', new_result_stream: call('c2'), stream_offset: 2, completed: true, only_result: { output: { ok: true }, messages: [] } }])
+          : undefined,
+      (c) => (c.url.pathname.includes('/queue/cancel/') ? text('ok') : undefined),
+      (c) =>
+        c.url.pathname.endsWith('/messages')
+          ? json([messageRow(41, 'user', 'first'), messageRow(42, 'user', 'again'), messageRow(43, 'assistant', '{"ok": true}', { job_id: 'step-2' })])
+          : undefined,
+      (c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
+    )
+    const chat = createChat(options({}, fetch))
+    const first = chat.sendMessage('first')
+    await new Promise((r) => setTimeout(r, 50))
+    const stopped = chat.stop()
+    await first
+    const second = chat.sendMessage('again')
+    await stopped
+    await second
+    expect(chat.getState().messages.map((m) => [m.role, m.content, m.tool?.callId])).toEqual([
+      ['user', 'first', undefined],
+      ['tool', '', 'c1'],
+      ['user', 'again', undefined],
+      ['assistant', '{"ok": true}', undefined]
+    ])
+  })
+
   test('the stream asks for a server poll interval only when one is set', async () => {
     const answer: Route = (c) =>
       c.url.pathname === streamPath ? sse([{ type: 'update', completed: true, only_result: 'ok' }]) : undefined
@@ -726,6 +1053,27 @@ describe('createChat with server history', () => {
     expect((await again.loadConversations()).map((c) => c.id)).toEqual([newer, older])
   })
 
+  test('renaming a local conversation persists the title without reordering history', async () => {
+    const storage = memoryStorage()
+    const { fetch, calls } = fetchMock(run, (c) =>
+      c.url.pathname === streamPath ? sse([{ type: 'update', completed: true, only_result: 'ok' }]) : undefined
+    )
+    const chat = createChat(options({ token: 'tok', storage }, fetch))
+    await chat.sendMessage('older')
+    const older = chat.getState().conversationId!
+    chat.newConversation()
+    await chat.sendMessage('newer')
+    const newer = chat.getState().conversationId!
+    const before = calls.length
+    await chat.renameConversation(older, 'Renamed')
+    expect(calls.length).toBe(before)
+    const again = createChat(options({ token: 'tok', storage }, fetch))
+    expect((await again.loadConversations()).map((c) => [c.id, c.title])).toEqual([
+      [newer, 'newer'],
+      [older, 'Renamed']
+    ])
+  })
+
   test('destroying the chat mid-turn leaves it idle', async () => {
     const { fetch } = fetchMock(run, (c) =>
       c.url.pathname === streamPath

chat-sdk/test/support.ts

@@ -5,6 +5,9 @@ export interface RecordedCall {
   url: URL
   headers: Record<string, string>
   body: unknown
+  /** A body sent as is rather than as JSON (an upload). */
+  raw?: Blob
+  signal?: AbortSignal
 }
 
 export type Route = (call: RecordedCall) => Response | Promise<Response> | undefined
@@ -20,7 +23,9 @@ export function fetchMock(...routes: Route[]): { fetch: FetchLike; calls: Record
       headers: Object.fromEntries(
         Object.entries((init?.headers as Record<string, string>) ?? {}).map(([k, v]) => [k.toLowerCase(), v])
       ),
-      body: typeof init?.body === 'string' ? JSON.parse(init.body) : undefined
+      body: typeof init?.body === 'string' ? JSON.parse(init.body) : undefined,
+      raw: init?.body instanceof Blob ? init.body : undefined,
+      signal: init?.signal ?? undefined
     }
     calls.push(call)
     for (const route of routes) {