Compare commits

..
Author SHA1 Message Date
Diego Imbert 7577c55b13 Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui 2026-09-18 01:03:57 +02:00
Diego Imbert 51224582c1 Merge commit '2661dadba63efcc0c5db88a7942d66caf34f20b9' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-18 01:00:23 +02:00
Diego Imbert 2661dadba6 Merge commit '8c554ef1337aa6cd99ab869443dfd30becc65a36' into HEAD 2026-09-18 01:00:09 +02:00
Diego ImbertandClaude Opus 5 57211d6c95 fix(datatables): keep only the most recently used DuckDB root certificate files
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-18 00:55:35 +02:00
Diego Imbert 0dbf5eef74 Merge commit 'addd77eaf4' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-18 00:55:35 +02:00
Diego Imbert 8e780a7923 Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui 2026-09-17 18:25:57 +02:00
Diego Imbert 3fdb3047a7 Merge commit '5dfb2f30f74cedaae9f113383e42d3a35cde5189' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-common/src/datatable_roles.rs
2026-09-17 18:23:38 +02:00
Diego Imbert 5dfb2f30f7 Merge commit '76a10f8ee8' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-common/src/lib.rs
2026-09-17 18:22:16 +02:00
Diego Imbert 4a0d7c0c29 Merge commit 'd62807e9edc553b1eb08c10ce28521a71007a263' into HEAD 2026-09-17 18:04:22 +02:00
Diego ImbertandClaude Opus 5 d62807e9ed fix(datatables): stop counting storage-only fork cleanup rows as uses of an external database
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:04:19 +02:00
Diego Imbert 696b4ab5f8 Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui 2026-09-17 16:37:27 +02:00
Diego ImbertandClaude Opus 5 48f026eb0f fix(datatables): refuse rolling back while external data tables are under roles, and type external_instance in the CLI
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:35:45 +02:00
Diego Imbert 0e659898f3 Merge commit 'a296ca016a' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:35:45 +02:00
Diego Imbert a296ca016a Merge commit '7c16d482fd' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:34:50 +02:00
Diego ImbertandClaude Opus 5 947e44fe49 fix(settings): show the server's reason when saving instance settings fails
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:26:29 +02:00
Diego Imbert 489493dba2 Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui 2026-09-17 16:18:12 +02:00
Diego Imbert c55ff3141c Merge commit '5eec203099ba1034039f8b52e81400078cac919d' into HEAD 2026-09-17 16:17:18 +02:00
Diego ImbertandClaude Opus 5 5eec203099 fix(datatables): keep certificate verification when DuckDB attaches an external data table
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:17:15 +02:00
Diego ImbertandClaude Opus 5 0b3882f2b2 fix(datatables): keep certificate verification when DuckDB attaches an external data table
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:17:02 +02:00
Diego Imbert 58731b1828 Merge commit '53d62dd399' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:17:02 +02:00
Diego ImbertandClaude Opus 5 5624d350d5 feat(datatables): manage the external instance cluster's data table roles from the UI
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:09:25 +02:00
Diego ImbertandClaude Opus 5 53d62dd399 fix(datatables): register external fork catalogs under the lifecycle lock, and keep certificate verification in DuckDB attaches
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 16:07:40 +02:00
Diego Imbert e17b76c255 Merge commit '5ddaee5b78' into HEAD
# Conflicts:
#	backend/ee-repo-ref.txt
2026-09-17 16:07:40 +02:00
Diego Imbert cfada61f41 Merge remote-tracking branch 'origin/datatable-external-instance-4' into datatable-external-instance-ui 2026-09-17 16:00:06 +02:00
Diego ImbertandClaude Opus 5 0406f133c5 feat(datatables): per-cluster data table role catalogs, with roles on the external instance cluster
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:59:22 +02:00
Diego ImbertandClaude Opus 5 aebae79397 feat(datatables): configure the external instance cluster and pick its databases from the UI
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:45:22 +02:00
Diego ImbertandClaude Opus 5 b62d8b246a fix(datatables): protect external databases pending fork cleanup, and describe Ducklake usage in the API
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 15:30:25 +02:00
Diego Imbert 5bc2f13565 Merge commit '3981b2eb1b3bf77cdc1a5cbf43090e6b9706aa1f' into datatable-external-instance-3 2026-09-17 15:28:31 +02:00
Diego Imbert 0c53990d18 Merge commit '9b0046d8671697e6f6568c89170052367139d157' into datatable-external-instance-3
# Conflicts:
#	backend/ee-repo-ref.txt
#	backend/windmill-common/src/external_instance_pg.rs
2026-09-17 11:50:21 +02:00
Diego ImbertandClaude Opus 5 67f488c676 feat(datatables): Ducklake catalogs on the external instance cluster
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 11:47:25 +02:00
193 changed files with 2589 additions and 8176 deletions
+1 -1
View File
@@ -1,3 +1,3 @@
{
".": "1.814.0"
".": "1.813.0"
}
-26
View File
@@ -1,31 +1,5 @@
# Changelog
## [1.814.0](https://github.com/windmill-labs/windmill/compare/v1.813.0...v1.814.0) (2026-09-17)
### Features
* **ai-chat:** add list_workers and list_data_metrics global tools ([#11143](https://github.com/windmill-labs/windmill/issues/11143)) ([e954d33](https://github.com/windmill-labs/windmill/commit/e954d33613e4ff5027667eb8f646615d9bbd499d))
* **ai-chat:** merge get_job_logs and get_flow_run_details into get_run ([#11172](https://github.com/windmill-labs/windmill/issues/11172)) ([5bb37ca](https://github.com/windmill-labs/windmill/commit/5bb37ca3388666fba72c55534e37f37bb3e9299e))
* allow git sync auto-pull, promotion and PRs on Pro licenses ([#11173](https://github.com/windmill-labs/windmill/issues/11173)) ([02e47de](https://github.com/windmill-labs/windmill/commit/02e47de8b4c4f3f54753aabf8c67bc8e71ffb957))
* badge chat-input flows on the home list ([#11164](https://github.com/windmill-labs/windmill/issues/11164)) ([3d08197](https://github.com/windmill-labs/windmill/commit/3d0819718221f885b61e73d02b43dcc853c7d02a))
* collect flow conversations and agent memory once their last message goes ([#11178](https://github.com/windmill-labs/windmill/issues/11178)) ([23c24a9](https://github.com/windmill-labs/windmill/commit/23c24a9688d4c8c462f53221334d538280f16bca))
* flow chat model picker on a shared model-settings component ([#11187](https://github.com/windmill-labs/windmill/issues/11187)) ([189793c](https://github.com/windmill-labs/windmill/commit/189793c2e4db7f1c853695ebcc895c1ec82ed19f))
* keep flow inputs and seed the agent when chat mode is enabled ([#11177](https://github.com/windmill-labs/windmill/issues/11177)) ([68f2248](https://github.com/windmill-labs/windmill/commit/68f2248018fc218a090bf939e1eb22ff97d5bc22))
* let plan mode search and read connected mcp servers ([#11205](https://github.com/windmill-labs/windmill/issues/11205)) ([5371519](https://github.com/windmill-labs/windmill/commit/5371519f0f5ce7750982dcdb374dca72115902e7))
* let test_run_flow name the conversation of a chat-mode test run ([#11198](https://github.com/windmill-labs/windmill/issues/11198)) ([6e1ef93](https://github.com/windmill-labs/windmill/commit/6e1ef93f329cb396ffc3df3304d592e8fa0e0e71))
* managed memory with an inherited or custom memory id per step ([#11118](https://github.com/windmill-labs/windmill/issues/11118)) ([c297ed0](https://github.com/windmill-labs/windmill/commit/c297ed0052d998fb8f063faa2a36c6eb03e327be))
* render the flow chat through the shared session chat components ([#11175](https://github.com/windmill-labs/windmill/issues/11175)) ([a9ec0ae](https://github.com/windmill-labs/windmill/commit/a9ec0aec3ac0c6b0f7919d0eb2168816923826d7))
* show flow step detail inside the graph tab on narrow detail layouts ([#11168](https://github.com/windmill-labs/windmill/issues/11168)) ([64dffe6](https://github.com/windmill-labs/windmill/commit/64dffe6106ad6a55b61a423c855a4b5b0cef533e))
* store mcp tool call, result and reasoning on flow conversation rows ([#11176](https://github.com/windmill-labs/windmill/issues/11176)) ([a571117](https://github.com/windmill-labs/windmill/commit/a571117f3fd2cef14c920770645c60ee358fdfdd))
* tell test flow conversations from deployed ones and rename a chat ([#11179](https://github.com/windmill-labs/windmill/issues/11179)) ([4eab995](https://github.com/windmill-labs/windmill/commit/4eab995cf7cf091a5e4640da4cb77e0921bb7fdf))
### Bug Fixes
* disable a schedule whose cron has no run left instead of panicking ([#11195](https://github.com/windmill-labs/windmill/issues/11195)) ([381d447](https://github.com/windmill-labs/windmill/commit/381d4470ef699ea82283742132e56556b95d2bd2))
* skip expiry notifications for app embed and SDK tokens ([#11169](https://github.com/windmill-labs/windmill/issues/11169)) ([9d348f8](https://github.com/windmill-labs/windmill/commit/9d348f84c7830f36b6153472556fd70e3d84cd24))
## [1.813.0](https://github.com/windmill-labs/windmill/compare/v1.812.0...v1.813.0) (2026-09-16)
-4
View File
@@ -175,10 +175,6 @@ the decrypted value, exactly as against a real backend. The chat's read path pas
Seed a recognizable secret (the existing fixture uses `sk_live_do_not_leak_me`) and
assert it via `valueExcludes` to catch a leak.
`toolExpect.toolCallArgs` entries support `sharedByAtLeast: <n>`: at least `n` recorded
calls to that tool must carry the same non-blank string in the field. Use it for calls that
have to share an identifier, like two test runs of one chat conversation.
`toolExpect.toolCallArgs` entries additionally support `fieldMustBeAbsent: true`: no
recorded call to that tool may pass the field at all (an explicit `null` counts as
passing it). Use it for partial-update tools, where supplying a field the model could
+12 -1
View File
@@ -1,5 +1,5 @@
import { randomUUID } from 'node:crypto'
import type { BackendValidationSettings } from '../../core/backendValidation'
import { buildWorkspaceId } from './workspaceId'
interface CompletedJobResultMaybe {
completed: boolean
@@ -24,6 +24,7 @@ export interface CompletedPreviewJob {
const tokenCache = new Map<string, Promise<string>>()
const sharedWorkspaceQueue = new Map<string, Promise<void>>()
const managedSharedWorkspacePrefixes = ['f/evals/']
const DEFAULT_WORKSPACE_PREFIX = 'ai-evals'
export class BackendPreviewClient {
constructor(private readonly settings: BackendValidationSettings) {}
@@ -440,6 +441,16 @@ async function withSharedWorkspaceLock<T>(workspaceId: string, body: () => Promi
}
}
function buildWorkspaceId(caseId: string, attempt: number): string {
const caseSlug = caseId
.toLowerCase()
.replace(/[^a-z0-9-]+/g, '-')
.replace(/^-+|-+$/g, '')
.slice(0, 30)
const suffix = randomUUID().slice(0, 8)
return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || 'case'}-a${attempt}-${suffix}`
}
function extractFolderName(path: string): string | null {
if (!path.startsWith('f/')) {
return null
-48
View File
@@ -11,7 +11,6 @@ import type {
Script
} from '../../../frontend/src/lib/gen'
import type {
DataMetric,
DataTableTables,
DataTableTableSchema,
EndpointTool,
@@ -113,9 +112,6 @@ export interface BenchmarkWorkspaceRunnables {
aiProviders?: BenchmarkWorkspaceAiProvider[]
resources?: BenchmarkWorkspaceResource[]
datatables?: BenchmarkDatatableSeed[]
/** DuckLake catalog names, as `list_ducklakes` reports them. */
ducklakes?: string[]
dataMetrics?: DataMetric[]
jobs?: BenchmarkWorkspaceJob[]
}
@@ -677,27 +673,6 @@ export function listBenchmarkDatatables(workspace: string): DataTableTables[] |
}))
}
// ============= DuckLake catalogs and declared metrics =============
/** Seeded DuckLake names, or `null` for a non-benchmark workspace. */
export function listBenchmarkDucklakes(workspace: string): string[] | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
return runnables ? (runnables.ducklakes ?? []) : null
}
/**
* Seeded metric declarations, or `null` for a non-benchmark workspace.
*
* The `table` / `path_prefix` filters are ignored: which rows a filter selects is
* `canonical_table_path`'s business and is pinned by `ducklakeTools.test.ts`.
* Re-deriving it here would give the eval its own copy of that spec to drift from,
* and the case this serves measures whether the model reaches for the tool at all.
*/
export function listBenchmarkDataMetrics(workspace: string): DataMetric[] | null {
const runnables = benchmarkWorkspaceRunnables.get(workspace)
return runnables ? (runnables.dataMetrics ?? []) : null
}
export function getBenchmarkDatatableSchema(input: {
workspace: string
datatableName: string
@@ -865,29 +840,6 @@ export function runBenchmarkFlowByPath(input: {
})
}
/**
* Mirror `JobService.runFlowPreview` for benchmark workspaces, including the server's
* refusal of a chat-enabled flow run that names no conversation (`memory_id`).
*/
export function runBenchmarkFlowPreview(input: {
workspace: string
memoryId?: string
requestBody?: { path?: string; value?: { chat_input_enabled?: boolean }; args?: unknown }
}): string {
if (input.requestBody?.value?.chat_input_enabled && !input.memoryId) {
throw new Error('Bad request: memory_id is required for chat-enabled flows')
}
const args = (input.requestBody?.args ?? {}) as Record<string, unknown>
return createBenchmarkCompletedJob({
workspace: input.workspace,
jobKind: 'flowpreview',
success: true,
args,
result: { path: input.requestBody?.path, args, mocked: true },
logs: 'Mock benchmark flow preview completed successfully.'
})
}
export function previewBenchmarkSchedule(input: {
requestBody?: Record<string, unknown>
}): Record<string, unknown> {
@@ -76,9 +76,7 @@ vi.mock('$lib/gen', async () => {
listBenchmarkPlainResources,
listBenchmarkApps,
listBenchmarkDatatables,
listBenchmarkDataMetrics,
listBenchmarkDrafts,
listBenchmarkDucklakes,
listBenchmarkFlows,
listBenchmarkJobs,
listBenchmarkScripts,
@@ -89,7 +87,6 @@ vi.mock('$lib/gen', async () => {
previewBenchmarkSchedule,
runBenchmarkDatatableSql,
runBenchmarkFlowByPath,
runBenchmarkFlowPreview,
runBenchmarkScriptByPath,
runBenchmarkScriptPreview,
updateBenchmarkDraft,
@@ -296,14 +293,6 @@ vi.mock('$lib/gen', async () => {
args: data.requestBody
})
: actual.JobService.runScriptByPath(data),
runFlowPreview: async (data: {
workspace: string
memoryId?: string
requestBody?: { path?: string; value?: { chat_input_enabled?: boolean }; args?: unknown }
}) =>
hasBenchmarkWorkspace(data.workspace)
? runBenchmarkFlowPreview(data)
: actual.JobService.runFlowPreview(data as any),
runFlowByPath: async (data: {
workspace: string
path: string
@@ -352,10 +341,6 @@ vi.mock('$lib/gen', async () => {
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkDatatables(data.workspace) ?? [])
: actual.WorkspaceService.listDataTableTables(data),
listDucklakes: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? (listBenchmarkDucklakes(data.workspace) ?? [])
: actual.WorkspaceService.listDucklakes(data),
getDataTableTableSchema: async (data: {
workspace: string
datatableName: string
@@ -371,12 +356,6 @@ vi.mock('$lib/gen', async () => {
})
: actual.WorkspaceService.getDataTableTableSchema(data)
}),
DataMetricService: wrapService(actual.DataMetricService, {
listDataMetrics: async (data: { workspace: string }) =>
hasBenchmarkWorkspace(data.workspace)
? { metrics: listBenchmarkDataMetrics(data.workspace) ?? [] }
: actual.DataMetricService.listDataMetrics(data)
}),
ScheduleService: wrapService(actual.ScheduleService, {
existsSchedule: async (data: { workspace: string; path: string }) =>
hasBenchmarkWorkspace(data.workspace) ? false : actual.ScheduleService.existsSchedule(data),
+12 -1
View File
@@ -1,8 +1,9 @@
import { randomUUID } from "node:crypto";
import type { WindmillBackendSettings } from "../../core/windmillBackendSettings";
import { buildWorkspaceId } from "./workspaceId";
const tokenCache = new Map<string, Promise<string>>();
const sharedWorkspaceQueue = new Map<string, Promise<void>>();
const DEFAULT_WORKSPACE_PREFIX = "ai-evals";
export class WindmillBackendClient {
constructor(private readonly settings: WindmillBackendSettings) {}
@@ -178,6 +179,16 @@ async function withSharedWorkspaceLock<T>(
}
}
function buildWorkspaceId(caseId: string, attempt: number): string {
const caseSlug = caseId
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/^-+|-+$/g, "")
.slice(0, 30);
const suffix = randomUUID().slice(0, 8);
return `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || "case"}-a${attempt}-${suffix}`;
}
async function expectOk(response: Response, context: string): Promise<void> {
if (response.ok) {
return;
@@ -1,21 +0,0 @@
import { describe, expect, it } from "bun:test";
import { buildWorkspaceId } from "./workspaceId";
describe("buildWorkspaceId", () => {
// `workspace.proper_id` rejects `--`, which a case id can carry itself and
// which truncating a slug on a hyphen produces once the suffix adds its own.
// One id per shape: cut landing on a hyphen, cut landing mid-word, no cut, and
// a doubled hyphen no cut ever reaches.
it("stays within the id length cap and the proper_id format", () => {
for (const caseId of [
"global-test6-secret-variable-draft",
"global-test23-datatable-query-select",
"short",
"global--test-foo",
]) {
const id = buildWorkspaceId(caseId, 1);
expect(id.length).toBeLessThanOrEqual(50);
expect(id).toMatch(/^\w+(-\w+)*$/);
}
});
});
-22
View File
@@ -1,22 +0,0 @@
import { randomUUID } from "node:crypto";
const DEFAULT_WORKSPACE_PREFIX = "ai-evals";
// A workspace id must be at most 50 characters AND match `^\w+(-\w+)*$`
// (`workspace.proper_id`), so the case slug yields to the random suffix that
// makes the id unique, and no hyphen may end up doubled — neither one already in
// the case id nor one a truncation leaves for the suffix to follow.
const MAX_WORKSPACE_ID_LENGTH = 50;
export function buildWorkspaceId(caseId: string, attempt: number): string {
const caseSlug = caseId
.toLowerCase()
.replace(/[^a-z0-9-]+/g, "-")
.replace(/-{2,}/g, "-")
.replace(/^-+|-+$/g, "");
const suffix = `-a${attempt}-${randomUUID().slice(0, 8)}`;
const head = `${DEFAULT_WORKSPACE_PREFIX}-${caseSlug || "case"}`;
return `${head
.slice(0, MAX_WORKSPACE_ID_LENGTH - suffix.length)
.replace(/-+$/, "")}${suffix}`;
}
+12 -58
View File
@@ -1919,11 +1919,10 @@
- when the lookup fails, tells the user instead of inventing table names
- does not write scripts or resources to answer a read-only question
# --- Dedicated tools preferred over the API catalog ---
# The harness serves worker/queue reads itself (benchmark fetch handlers in
# adapters/frontend), so these cases do not require an mcp-enabled eval backend.
# The stale `api-catalog` in the id below is kept so results stay comparable
# across benchmark runs.
# --- API catalog (search_api_endpoints / call_api_get / call_api_endpoint) ---
# The harness serves the catalog and the executed calls itself (mock
# listMcpTools + benchmark fetch handlers in adapters/frontend), so these cases
# do not require an mcp-enabled eval backend.
- id: global-test30-api-catalog-workers
prompt: |-
@@ -1935,42 +1934,23 @@
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- list_workers
forbiddenToolsUsed:
- search_api_endpoints
- call_api_get
forbiddenToolsUsed:
- call_api_endpoint
- write_script
- deploy_workspace_item
toolCallArgs:
- tool: call_api_get
field: name
stringIncludesAnyOf:
- listWorkers
# Read-only workspace inspection produces no draft; validate via tool use.
skipJudge: true
judgeChecklist:
- reads worker state through list_workers instead of guessing or fabricating
- discovers the workers endpoint through the API catalog instead of guessing or fabricating
- reports worker status from the returned data
- id: global-test37-ducklake-declared-measure
prompt: |-
We track orders in the main ducklake. Write me a duckdb script that reports total
revenue by month. Keep it as a draft, don't deploy it.
initial: ai_evals/fixtures/frontend/global/initial/ducklake_orders_metrics.json
runtime:
maxTurns: 8
validate:
draftCountExactly: 1
toolExpect:
requiredToolsUsed:
- list_data_metrics
forbiddenToolsUsed:
- deploy_workspace_item
- delete_workspace_item
# The judge runs: the point is not that the tool was called but that the number it
# describes is the declared one. `revenue` excludes test rows, so an aggregate that
# reproduces it without the filter is plausible, runnable and wrong.
judgeChecklist:
- totals revenue with the declared sum over the amount column rather than an invented aggregate over a guessed column
- excludes test orders from the total, as the declared revenue measure does
- groups by month using the declared order_month expression over order_date
- does not introduce column names absent from the declarations
- id: global-test31-draft-test-run-not-deployed
prompt: |-
Update `f/evals/global/format_greeting` so the provided name is uppercased in the greeting, then run it with name "ada" to check it works.
@@ -2160,32 +2140,6 @@
- creates an AI draft of f/evals/global/process_invoice applying 8% tax
- does not deploy or save the draft
- id: global-test38-chat-flow-follow-up-same-conversation
prompt: |-
I want to check that my support chat flow `f/evals/global/support_chat` remembers what was said.
Test it: first send "My name is Ada", then send "What is my name?" as a follow-up in the same chat.
initial: ai_evals/fixtures/frontend/global/initial/support_chat_flow.json
runtime:
maxTurns: 8
validate:
draftCountExactly: 0
toolExpect:
requiredToolsUsed:
- test_run_flow
# A chat flow's memory lives in its conversation, so a follow-up only reaches the first
# turn's history when both test runs name the same conversation.
toolCallArgs:
- tool: test_run_flow
field: memory_id
sharedByAtLeast: 2
forbiddenToolsUsed:
- run_flow
- deploy_workspace_item
# The judge cannot observe runs; what this case guards is the conversation the runs share.
skipJudge: true
judgeChecklist:
- test-runs the chat flow twice, the second message as a follow-up in the first run's conversation
- id: global-undo-created-draft
prompt: |-
Create a draft Postgres resource at `u/admin/scratch_db` for host db.example.com port 5432, database `orders`, user `app`, and tell me what fields it ended up with.
-7
View File
@@ -182,13 +182,6 @@ export interface ToolCallArgumentRule {
* the point is that the model filled it in at all rather than what it said.
*/
nonEmpty?: boolean;
/**
* Existential over calls: at least this many recorded calls to `tool` carry the
* same non-blank string in `field`. Use when calls have to share an identifier —
* e.g. test runs that continue one conversation — while a retry with a rejected
* value in between is still acceptable.
*/
sharedByAtLeast?: number;
/**
* Universal over calls: no recorded call to `tool` may pass `field` at all.
* For partial-update tools, where supplying a field the model could not have
-26
View File
@@ -396,32 +396,6 @@ describe("validateToolExpectations", () => {
expect(nonEmptyCheck?.details).toContain("blank on 1 of 2");
});
it("requires sharedByAtLeast calls to carry one value, not merely a value each", () => {
const run = (ids: (string | undefined)[]) =>
validateToolExpectations({
run: {
success: true,
actual: {},
assistantMessageCount: 1,
toolCallCount: ids.length,
toolsUsed: ["test_run_flow"],
toolCallDetails: ids.map((memory_id) => ({
name: "test_run_flow",
arguments: { path: "f/chat", memory_id },
})),
skillsInvoked: [],
},
toolExpect: {
toolCallArgs: [{ tool: "test_run_flow", field: "memory_id", sharedByAtLeast: 2 }],
},
}).find((c) => c.name.includes("is shared by at least 2 calls"))?.passed;
expect(run(["a", "b"])).toBe(false);
expect(run(["a"])).toBe(false);
expect(run([undefined, undefined])).toBe(false);
expect(run(["rejected", "a", "a"])).toBe(true);
});
it("passes nonEmpty when every call filled the field", () => {
const checks = validateToolExpectations({
run: {
-17
View File
@@ -320,23 +320,6 @@ export function validateToolExpectations(input: {
);
}
if (rule.sharedByAtLeast !== undefined) {
const counts = new Map<string, number>();
for (const value of values) {
if (typeof value === "string" && value.trim().length > 0) {
counts.set(value, (counts.get(value) ?? 0) + 1);
}
}
const mostShared = Math.max(0, ...counts.values());
checks.push(
check(
`${rule.tool}.${rule.field} is shared by at least ${rule.sharedByAtLeast} calls`,
mostShared >= rule.sharedByAtLeast,
`most calls sharing one value: ${mostShared}; values: ${summarizeToolValues(values)}`
)
);
}
if (rule.fieldMustBeAbsent) {
// Anything other than `undefined` was supplied — an explicit `null` is the
// model passing the field, not omitting it.
@@ -1,36 +0,0 @@
{
"workspace": {
"ducklakes": ["main"],
"dataMetrics": [
{
"script_path": "f/analytics/orders_pipeline",
"table_path": "main/main.orders",
"kind": "measure",
"name": "revenue",
"expr": "sum(amount)",
"filter": "not is_test"
},
{
"script_path": "f/analytics/orders_pipeline",
"table_path": "main/main.orders",
"kind": "measure",
"name": "order_count",
"expr": "count(*)"
},
{
"script_path": "f/analytics/orders_pipeline",
"table_path": "main/main.orders",
"kind": "dimension",
"name": "order_month",
"expr": "date_trunc('month', order_date)"
},
{
"script_path": "f/analytics/orders_pipeline",
"table_path": "main/main.orders",
"kind": "dimension",
"name": "region",
"expr": "region"
}
]
}
}
@@ -1,59 +0,0 @@
{
"workspace": {
"flows": [
{
"path": "f/evals/global/support_chat",
"summary": "Support chat",
"description": "Answers customer questions in a chat, remembering earlier messages.",
"schema": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"user_message": {
"type": "string",
"description": "Message from user"
}
},
"required": ["user_message"]
},
"value": {
"chat_input_enabled": true,
"modules": [
{
"id": "assistant",
"summary": "Support assistant",
"value": {
"type": "aiagent",
"tools": [],
"input_transforms": {
"provider": {
"type": "static",
"value": {
"kind": "anthropic",
"model": "claude-haiku-4-5-20251001",
"resource": "$res:f/evals/ai/anthropic"
}
},
"user_message": {
"type": "javascript",
"expr": "flow_input.user_message"
},
"system_prompt": {
"type": "static",
"value": "You are a friendly support assistant. Keep answers short."
},
"memory": {
"type": "static",
"value": { "kind": "auto", "context_length": 10 }
},
"streaming": { "type": "static", "value": true },
"output_type": { "type": "static", "value": "text" }
}
}
}
]
}
}
]
}
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n FROM flow_conversation_message\n WHERE conversation_id = $1\n AND created_seq > $2\n ORDER BY created_seq ASC\n LIMIT $3\n ",
"query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success\n FROM (\n SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success\n FROM flow_conversation_message\n WHERE conversation_id = $1\n ORDER BY created_seq DESC\n LIMIT $2 OFFSET $3\n ) AS messages\n ORDER BY created_seq ASC\n ",
"describe": {
"columns": [
{
@@ -58,26 +58,6 @@
"ordinal": 8,
"name": "success",
"type_info": "Bool"
},
{
"ordinal": 9,
"name": "tool_arguments",
"type_info": "Text"
},
{
"ordinal": 10,
"name": "tool_result",
"type_info": "Text"
},
{
"ordinal": 11,
"name": "reasoning",
"type_info": "Text"
},
{
"ordinal": 12,
"name": "attachments",
"type_info": "Jsonb"
}
],
"parameters": {
@@ -96,12 +76,8 @@
false,
false,
true,
false,
true,
true,
true,
true
false
]
},
"hash": "d6fa78c43b6c5f8040d7bccb29ad8627be1dac6fbe0097735a52f47c173f51c9"
"hash": "1c3473a0f9f6b6148b2c975f9f05bdefedf8a51c4e6ddf0eca367b9cc778d051"
}
@@ -1,24 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "UPDATE flow_conversation SET title = $1, updated_at = updated_at\n WHERE id = $2 AND workspace_id = $3\n RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Varchar",
"Uuid",
"Text"
]
},
"nullable": [
false
]
},
"hash": "5b9c9eb64051f291fed4be9bc0b0cc0aef2e7bde732899976eddac36a2da7658"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test\n FROM flow_conversation\n WHERE id = $1 AND workspace_id = $2",
"query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by\n FROM flow_conversation\n WHERE id = $1 AND workspace_id = $2\n FOR UPDATE",
"describe": {
"columns": [
{
@@ -37,11 +37,6 @@
"ordinal": 6,
"name": "created_by",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "is_test",
"type_info": "Bool"
}
],
"parameters": {
@@ -57,9 +52,8 @@
true,
false,
false,
false,
false
]
},
"hash": "48c8522a4fed219c5011f4ba63c81cfe028a8b2a32bd790840cef65c452a8c31"
"hash": "6f32c1feed096ff706ae359ad6a3ca33b3f82ca38289dfa4a69aa95041027d57"
}
@@ -1,38 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, name, enabled, pwd FROM datatable_role",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "name",
"type_info": "Varchar"
},
{
"ordinal": 2,
"name": "enabled",
"type_info": "Bool"
},
{
"ordinal": 3,
"name": "pwd",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
false,
false,
false,
true
]
},
"hash": "71ee2cb6661cca1fa4d8874a7f6d368347c59f36fd87df6dc7996152ccb84af0"
}
@@ -1,29 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT ws.workspace_id AS \"workspace_id!\", dt.key AS \"datatable!\"\n FROM workspace_settings ws\n JOIN workspace w ON w.id = ws.workspace_id AND w.deleted = false\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE ws.workspace_id <> $1\n AND dt.value->'database'->>'resource_type' = 'instance'\n AND dt.value->'database'->>'resource_path' = $2\n ORDER BY ws.workspace_id, dt.key\n ",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "workspace_id!",
"type_info": "Varchar"
},
{
"ordinal": 1,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
false,
null
]
},
"hash": "79799b5a2e499df6c28e286c42b9ad2db940c2455ab19cc95e5198baf96d5629"
}
@@ -1,17 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO datatable_role (id, name, enabled, pwd) VALUES ($1, $2, $3, $4)",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Varchar",
"Bool",
"Text"
]
},
"nullable": []
},
"hash": "86af9d51a158ea5cb6161461ecddf2a63695f8cbf8af648da5a0a77a5b9d02ba"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success, tool_arguments, tool_result, reasoning, attachments)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)",
"query": "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success)\n VALUES ($1, $2, $3, $4, $5, $6)",
"describe": {
"columns": [],
"parameters": {
@@ -21,14 +21,10 @@
"Text",
"Uuid",
"Varchar",
"Bool",
"Text",
"Text",
"Text",
"Jsonb"
"Bool"
]
},
"nullable": []
},
"hash": "12329c3359a7944ab5fa3aa27ddca1b26f340ccf574b9fa07641fe88b2d2987c"
"hash": "b1a9a433e577133869c067b2ce383fc6ce4e9df307feb5fd3edc0d1276d61ff1"
}
@@ -1,20 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT jsonb_object_keys(value->'databases') FROM global_settings\n WHERE name = 'custom_instance_pg_databases'",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "jsonb_object_keys",
"type_info": "Text"
}
],
"parameters": {
"Left": []
},
"nullable": [
null
]
},
"hash": "b9842d2d8abf382bd82d8fa1de012373638be391f884f81dc387ffc465badac6"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title, is_test)\n VALUES ($1, $2, $3, $4, $5, $6)\n ON CONFLICT (id) DO NOTHING\n RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test",
"query": "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title)\n VALUES ($1, $2, $3, $4, $5)\n ON CONFLICT (id) DO NOTHING\n RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by",
"describe": {
"columns": [
{
@@ -37,11 +37,6 @@
"ordinal": 6,
"name": "created_by",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "is_test",
"type_info": "Bool"
}
],
"parameters": {
@@ -50,8 +45,7 @@
"Varchar",
"Varchar",
"Varchar",
"Varchar",
"Bool"
"Varchar"
]
},
"nullable": [
@@ -61,9 +55,8 @@
true,
false,
false,
false,
false
]
},
"hash": "6d259b8cce5da5fecefe4ce322789b6b2cc43b51f2056c677d58f39c31fb26cb"
"hash": "c1e3ed3ecc3bcb98f60ba8196d33fee4a74f61b061e5025ecb75882208b3ba8f"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test\n FROM flow_conversation\n WHERE id = $1 AND workspace_id = $2\n FOR UPDATE",
"query": "SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by\n FROM flow_conversation\n WHERE id = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
@@ -37,11 +37,6 @@
"ordinal": 6,
"name": "created_by",
"type_info": "Varchar"
},
{
"ordinal": 7,
"name": "is_test",
"type_info": "Bool"
}
],
"parameters": {
@@ -57,9 +52,8 @@
true,
false,
false,
false,
false
]
},
"hash": "dd84f9dfb238d18cb74f9e43228345427131bf8008eba6920021d6a449791534"
"hash": "c383cc023714b361d10c10e8fef1fc148ab1da942951ee9ffdddaecee76a6be9"
}
@@ -1,24 +0,0 @@
{
"db_name": "PostgreSQL",
"query": "SELECT dt.key AS \"datatable!\"\n FROM workspace_settings ws\n CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt\n WHERE ws.workspace_id = $1\n AND dt.key <> $2\n AND NOT dt.value ? 'permissions'\n AND dt.value->'database'->>'resource_type' = 'instance'\n AND dt.value->'database'->>'resource_path' = $3\n ORDER BY dt.key",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "datatable!",
"type_info": "Text"
}
],
"parameters": {
"Left": [
"Text",
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "d48ca62c86b1af7a9dd2450c1c28dc45020a2a553d8874c49f9eafedea5a9d40"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "\n SELECT\n j.args as \"args: Json<HashMap<String, Box<RawValue>>>\",\n js.flow_status as \"flow_status: Json<windmill_common::flow_status::FlowStatus>\",\n j.runnable_path\n FROM v2_job_status js\n INNER JOIN v2_job j ON j.id = js.id\n WHERE js.id = $1\n ",
"query": "\n SELECT\n j.args as \"args: Json<HashMap<String, Box<RawValue>>>\",\n js.flow_status as \"flow_status: Json<windmill_common::flow_status::FlowStatus>\"\n FROM v2_job_status js\n INNER JOIN v2_job j ON j.id = js.id\n WHERE js.id = $1\n ",
"describe": {
"columns": [
{
@@ -12,11 +12,6 @@
"ordinal": 1,
"name": "flow_status: Json<windmill_common::flow_status::FlowStatus>",
"type_info": "Jsonb"
},
{
"ordinal": 2,
"name": "runnable_path",
"type_info": "Varchar"
}
],
"parameters": {
@@ -25,10 +20,9 @@
]
},
"nullable": [
true,
true,
true
]
},
"hash": "9008f9abb70a9a07e38acb20bea6a710d0efd77dac4aedeb88d72240e816530b"
"hash": "dd89d652154748d6d7e625e31778f6885d0ee62d29a4b8894a4b459dd215a103"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n FROM (\n SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments\n FROM flow_conversation_message\n WHERE conversation_id = $1\n ORDER BY created_seq DESC\n LIMIT $2 OFFSET $3\n ) AS messages\n ORDER BY created_seq ASC\n ",
"query": "SELECT id, conversation_id, message_type as \"message_type: MessageType\", content, job_id, created_at, created_seq, step_name, success\n FROM flow_conversation_message\n WHERE conversation_id = $1\n AND created_seq > $2\n ORDER BY created_seq ASC\n LIMIT $3\n ",
"describe": {
"columns": [
{
@@ -58,26 +58,6 @@
"ordinal": 8,
"name": "success",
"type_info": "Bool"
},
{
"ordinal": 9,
"name": "tool_arguments",
"type_info": "Text"
},
{
"ordinal": 10,
"name": "tool_result",
"type_info": "Text"
},
{
"ordinal": 11,
"name": "reasoning",
"type_info": "Text"
},
{
"ordinal": 12,
"name": "attachments",
"type_info": "Jsonb"
}
],
"parameters": {
@@ -96,12 +76,8 @@
false,
false,
true,
false,
true,
true,
true,
true
false
]
},
"hash": "a4a823f70b3dbe6aaf4a61c98345e94c5042fd5e6351fea139a66ecb1fb812ab"
"hash": "e8802be9203c1e88a06e337260ccca029380139f89a01a89033e36a6ed9ac082"
}
+125 -126
View File
@@ -728,7 +728,7 @@ dependencies = [
"futures-lite 2.6.1",
"parking",
"polling 3.11.0",
"rustix 1.1.5",
"rustix 1.1.4",
"slab",
"windows-sys 0.61.2",
]
@@ -873,7 +873,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -1976,7 +1976,7 @@ dependencies = [
"prettyplease 0.3.0",
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -2000,7 +2000,7 @@ dependencies = [
"proc-macro-crate",
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -2144,7 +2144,7 @@ checksum = "6a1f896587b6f2c069c73d2f0913e2d590c3990285cd2f0b6aa02b786b4c679c"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -2164,9 +2164,9 @@ dependencies = [
[[package]]
name = "bytes-str"
version = "0.2.9"
version = "0.2.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4dde6d05e75a31ec9610eb6446a6f0a10dd30ff5100d720fee4c7c7a9008b5ba"
checksum = "577d2bf5650f8554d5a372af5ac93535110a0fc75b3e702bb853369febf227c2"
dependencies = [
"bytes",
"serde",
@@ -2338,9 +2338,9 @@ dependencies = [
[[package]]
name = "cfg-if"
version = "1.0.5"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "cfg_aliases"
@@ -2454,7 +2454,7 @@ dependencies = [
"heck",
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -3047,7 +3047,7 @@ dependencies = [
"proc-macro2",
"quote",
"strsim 0.11.1",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -3102,7 +3102,7 @@ checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785"
dependencies = [
"darling_core 0.24.1",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -4580,7 +4580,7 @@ checksum = "e01a3366d27ee9890022452ee61b2b63a67e6f13f58900b651ff5665f0bb1fab"
dependencies = [
"libc",
"option-ext",
"redox_users 0.5.3",
"redox_users 0.5.2",
"windows-sys 0.61.2",
]
@@ -4603,7 +4603,7 @@ checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -4850,7 +4850,7 @@ checksum = "a65863d15a4ce2888bd2f0f543cc963d3879c3a022c8ee43f6141d479a3ac815"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -5198,7 +5198,7 @@ version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8640e34b88f7652208ce9e88b1a37a2ae95227d84abec377ccd3c5cfeb141ed4"
dependencies = [
"rustix 1.1.5",
"rustix 1.1.4",
"windows-sys 0.59.0",
]
@@ -5319,7 +5319,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -9438,7 +9438,7 @@ dependencies = [
"concurrent-queue",
"hermit-abi 0.5.3",
"pin-project-lite",
"rustix 1.1.5",
"rustix 1.1.4",
"windows-sys 0.61.2",
]
@@ -9574,7 +9574,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2bfe0f4c752e450fc2faf62654f1c134747922825d5b04ca717b8874f41a40c0"
dependencies = [
"proc-macro2",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -10235,10 +10235,11 @@ dependencies = [
[[package]]
name = "redox_users"
version = "0.5.3"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "60dc65c0ff1a7ae1294b0c67b9f14baf70b644404010370171787bfac1038fc0"
checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac"
dependencies = [
"getrandom 0.2.17",
"libredox",
"thiserror 2.0.20",
]
@@ -10260,7 +10261,7 @@ checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -10579,7 +10580,7 @@ dependencies = [
"proc-macro2",
"quote",
"serde_json",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -10797,9 +10798,9 @@ dependencies = [
[[package]]
name = "rustix"
version = "1.1.5"
version = "1.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
dependencies = [
"bitflags 2.13.2",
"errno",
@@ -11216,7 +11217,7 @@ dependencies = [
"proc-macro2",
"quote",
"serde_derive_internals 0.30.0",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -11415,7 +11416,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -11437,7 +11438,7 @@ checksum = "f852137cce035d6a4df67ccce505ff6b3e9fd3a10e3e52b24dc71e650bb1a9bd"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -11491,7 +11492,7 @@ checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -11559,7 +11560,7 @@ dependencies = [
"darling 0.24.1",
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -12707,9 +12708,9 @@ dependencies = [
[[package]]
name = "syn"
version = "3.0.6"
version = "3.0.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
dependencies = [
"proc-macro2",
"quote",
@@ -12744,7 +12745,7 @@ checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -13035,7 +13036,7 @@ dependencies = [
"fastrand 2.5.0",
"getrandom 0.4.3",
"once_cell",
"rustix 1.1.5",
"rustix 1.1.4",
"windows-sys 0.61.2",
]
@@ -13054,7 +13055,7 @@ version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874"
dependencies = [
"rustix 1.1.5",
"rustix 1.1.4",
"windows-sys 0.61.2",
]
@@ -13114,7 +13115,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -14074,7 +14075,7 @@ checksum = "f153acc4e99a5f2a5aefa09fb078be54e26271b2813f6041200b224c098d8328"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -14178,9 +14179,9 @@ checksum = "81b79ad29b5e19de4260020f8919b443b2ef0277d242ce532ec7b7a2cc8b6007"
[[package]]
name = "unicode-ident"
version = "1.0.26"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unicode-normalization"
@@ -14545,7 +14546,7 @@ dependencies = [
"bumpalo",
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
"wasm-bindgen-shared",
]
@@ -14588,7 +14589,7 @@ checksum = "8c89dcab8b516b6b603baca9d550b7282d68fcc7f367e3956cff7ebf406a3f12"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
@@ -14793,7 +14794,7 @@ dependencies = [
[[package]]
name = "windmill"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-nats",
@@ -14881,7 +14882,7 @@ dependencies = [
[[package]]
name = "windmill-ai"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"async-stream",
"async-trait",
@@ -14895,7 +14896,6 @@ dependencies = [
"eventsource-stream",
"futures",
"http 1.5.0",
"indexmap 2.14.2",
"lazy_static",
"mime_guess",
"reqwest 0.13.5",
@@ -14915,7 +14915,7 @@ dependencies = [
[[package]]
name = "windmill-alerting"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -14928,7 +14928,7 @@ dependencies = [
[[package]]
name = "windmill-api"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"argon2",
@@ -15068,7 +15068,7 @@ dependencies = [
[[package]]
name = "windmill-api-agent-workers"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15091,7 +15091,7 @@ dependencies = [
[[package]]
name = "windmill-api-assets"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15108,7 +15108,7 @@ dependencies = [
[[package]]
name = "windmill-api-auth"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15134,7 +15134,7 @@ dependencies = [
[[package]]
name = "windmill-api-client"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"reqwest 0.12.28",
"serde",
@@ -15144,7 +15144,7 @@ dependencies = [
[[package]]
name = "windmill-api-configs"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15161,7 +15161,7 @@ dependencies = [
[[package]]
name = "windmill-api-debug"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"base64 0.22.1",
@@ -15183,7 +15183,7 @@ dependencies = [
[[package]]
name = "windmill-api-embeddings"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15206,7 +15206,7 @@ dependencies = [
[[package]]
name = "windmill-api-flow-conversations"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15222,7 +15222,7 @@ dependencies = [
[[package]]
name = "windmill-api-flows"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15244,7 +15244,7 @@ dependencies = [
[[package]]
name = "windmill-api-groups"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15266,7 +15266,7 @@ dependencies = [
[[package]]
name = "windmill-api-inputs"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15280,7 +15280,7 @@ dependencies = [
[[package]]
name = "windmill-api-integration-tests"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-nats",
@@ -15315,7 +15315,7 @@ dependencies = [
[[package]]
name = "windmill-api-jobs"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15340,7 +15340,7 @@ dependencies = [
[[package]]
name = "windmill-api-npm-proxy"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15368,7 +15368,7 @@ dependencies = [
[[package]]
name = "windmill-api-openapi"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15390,7 +15390,7 @@ dependencies = [
[[package]]
name = "windmill-api-schedule"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15410,7 +15410,7 @@ dependencies = [
[[package]]
name = "windmill-api-scripts"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15448,7 +15448,7 @@ dependencies = [
[[package]]
name = "windmill-api-settings"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15477,7 +15477,7 @@ dependencies = [
[[package]]
name = "windmill-api-sse"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"lazy_static",
"serde",
@@ -15489,7 +15489,7 @@ dependencies = [
[[package]]
name = "windmill-api-users"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"argon2",
"axum 0.8.9",
@@ -15513,7 +15513,7 @@ dependencies = [
[[package]]
name = "windmill-api-workers"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15527,7 +15527,7 @@ dependencies = [
[[package]]
name = "windmill-api-workspaces"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"axum 0.8.9",
"chrono",
@@ -15562,7 +15562,7 @@ dependencies = [
[[package]]
name = "windmill-audit"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"chrono",
"lazy_static",
@@ -15576,7 +15576,7 @@ dependencies = [
[[package]]
name = "windmill-autoscaling"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"axum 0.8.9",
@@ -15595,7 +15595,7 @@ dependencies = [
[[package]]
name = "windmill-common"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"aes-gcm",
"aho-corasick",
@@ -15667,7 +15667,6 @@ dependencies = [
"serde",
"serde_json",
"serde_yml",
"sha1",
"sha2 0.10.9",
"size",
"spki",
@@ -15703,7 +15702,7 @@ dependencies = [
[[package]]
name = "windmill-dep-map"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"chrono",
"futures",
@@ -15723,7 +15722,7 @@ dependencies = [
[[package]]
name = "windmill-git-sync"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"regex",
"serde",
@@ -15740,7 +15739,7 @@ dependencies = [
[[package]]
name = "windmill-indexer"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"astral-tokio-tar",
@@ -15767,7 +15766,7 @@ dependencies = [
[[package]]
name = "windmill-jseval"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"futures",
@@ -15784,7 +15783,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"itertools 0.14.0",
"lazy_static",
@@ -15800,7 +15799,7 @@ dependencies = [
[[package]]
name = "windmill-mcp"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15821,7 +15820,7 @@ dependencies = [
[[package]]
name = "windmill-native-triggers"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -15852,7 +15851,7 @@ dependencies = [
[[package]]
name = "windmill-oauth"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"arc-swap",
@@ -15877,7 +15876,7 @@ dependencies = [
[[package]]
name = "windmill-object-store"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-stream",
@@ -15912,7 +15911,7 @@ dependencies = [
[[package]]
name = "windmill-operator"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"futures",
@@ -15930,7 +15929,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"convert_case 0.6.0",
"serde",
@@ -15939,7 +15938,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15951,7 +15950,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -15963,7 +15962,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"gosyn",
@@ -15975,7 +15974,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -15987,7 +15986,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -15999,7 +15998,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -16010,7 +16009,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -16021,7 +16020,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -16033,7 +16032,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -16044,7 +16043,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16066,7 +16065,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -16078,7 +16077,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16092,7 +16091,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"convert_case 0.6.0",
@@ -16109,7 +16108,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16122,7 +16121,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde",
@@ -16134,7 +16133,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16152,7 +16151,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -16168,7 +16167,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -16184,7 +16183,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -16198,7 +16197,7 @@ dependencies = [
[[package]]
name = "windmill-queue"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16237,7 +16236,7 @@ dependencies = [
[[package]]
name = "windmill-runtime-nativets"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"const_format",
@@ -16277,7 +16276,7 @@ dependencies = [
[[package]]
name = "windmill-sql-datatype-parser-wasm"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"getrandom 0.3.4",
"wasm-bindgen",
@@ -16288,7 +16287,7 @@ dependencies = [
[[package]]
name = "windmill-store"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -16323,7 +16322,7 @@ dependencies = [
[[package]]
name = "windmill-test-utils"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16347,7 +16346,7 @@ dependencies = [
[[package]]
name = "windmill-trigger"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16380,7 +16379,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-amqp"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16407,7 +16406,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-azure"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16440,7 +16439,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-email"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16460,7 +16459,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-gcp"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16494,7 +16493,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-http"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16530,7 +16529,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-kafka"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16553,7 +16552,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-mqtt"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16577,7 +16576,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-nats"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-nats",
@@ -16601,7 +16600,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-postgres"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16636,7 +16635,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-sqs"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16664,7 +16663,7 @@ dependencies = [
[[package]]
name = "windmill-trigger-websocket"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-trait",
@@ -16689,7 +16688,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"bitflags 2.13.2",
@@ -16708,7 +16707,7 @@ dependencies = [
[[package]]
name = "windmill-worker"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-once-cell",
@@ -16826,7 +16825,7 @@ dependencies = [
[[package]]
name = "windmill-worker-volumes"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"bytes",
"futures",
@@ -17459,7 +17458,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156"
dependencies = [
"libc",
"rustix 1.1.5",
"rustix 1.1.4",
]
[[package]]
@@ -17520,7 +17519,7 @@ checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
"synstructure 0.14.0",
]
@@ -17561,7 +17560,7 @@ checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
"synstructure 0.14.0",
]
@@ -17617,7 +17616,7 @@ checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da"
dependencies = [
"proc-macro2",
"quote",
"syn 3.0.6",
"syn 3.0.5",
]
[[package]]
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.814.0"
version = "1.813.0"
authors.workspace = true
edition.workspace = true
@@ -88,7 +88,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.814.0"
version = "1.813.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
+1 -1
View File
@@ -1 +1 @@
220301c27272b6c9e38401884c45cd959bd02a09
151be033ea2bf2f768f381a8cd3b888bdd366cf9
@@ -1,3 +0,0 @@
ALTER TABLE flow_conversation_message DROP COLUMN tool_arguments;
ALTER TABLE flow_conversation_message DROP COLUMN tool_result;
ALTER TABLE flow_conversation_message DROP COLUMN reasoning;
@@ -1,12 +0,0 @@
-- A chat is rebuilt from its rows without reading jobs, so every tool row carries its call:
-- the arguments the model wrote and the text the model got back, or what the call failed
-- with. A script or flow tool's job holds the args its input transforms produced, not the
-- model's; an MCP tool runs inside the agent's job, whose result lists every call of the
-- turn with nothing tying one to a row. A provider-native web search carries only its
-- citations, the provider never returning the query.
ALTER TABLE flow_conversation_message ADD COLUMN tool_arguments TEXT;
ALTER TABLE flow_conversation_message ADD COLUMN tool_result TEXT;
-- The thinking behind this row. The agent job keeps the turn's thinking as one string;
-- the rows keep it per iteration, next to the answer or tool call it led to.
ALTER TABLE flow_conversation_message ADD COLUMN reasoning TEXT;
@@ -1 +0,0 @@
ALTER TABLE flow_conversation DROP COLUMN is_test;
@@ -1,26 +0,0 @@
-- A chat run from the flow editor's test panel is stored exactly like one from the
-- deployed flow, so the two were indistinguishable once written. Marking them lets the
-- lists tell a trial apart from a real conversation.
ALTER TABLE flow_conversation ADD COLUMN is_test BOOLEAN NOT NULL DEFAULT false;
-- Existing rows: a conversation whose messages came from a flowpreview run was a test.
-- Derived once here because the job is purged on retention, after which the origin of an
-- old conversation is unknowable.
--
-- Walked to the root job rather than matched directly: an existing message row never holds
-- the flow job itself. The rows point at the step that produced them — the AI agent's job
-- for an answer, the tool's own job for a tool call — whose kind is never 'flowpreview'.
--
-- `root_job` first, matching `get_root_job_id` (windmill-worker/src/common.rs): only it
-- reaches the top of the run. `flow_innermost_root_job` stops at the closest flow scope by
-- design, so an agent inside a subflow would land on that subflow's 'flow' row and the
-- conversation would read as deployed.
UPDATE flow_conversation c
SET is_test = true
WHERE EXISTS (
SELECT 1 FROM flow_conversation_message m
JOIN v2_job j ON j.id = m.job_id
JOIN v2_job root
ON root.id = coalesce(j.root_job, j.flow_innermost_root_job, j.parent_job, j.id)
WHERE m.conversation_id = c.id AND root.kind = 'flowpreview'
);
@@ -1 +0,0 @@
ALTER TABLE flow_conversation_message DROP COLUMN attachments;
@@ -1,4 +0,0 @@
-- The files a user message carried, as object-storage references: `[{input, s3, storage?,
-- filename?}]`. Only references, never file bytes and never a presigned URL, so a
-- transcript can show a message's files without reading its run's args.
ALTER TABLE flow_conversation_message ADD COLUMN attachments JSONB;
@@ -0,0 +1,22 @@
-- Roles on the external cluster are live logins there; dropping the column would forget them.
LOCK TABLE datatable_role;
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM datatable_role WHERE cluster <> 'instance') THEN
RAISE EXCEPTION 'datatable_role holds roles on the external instance cluster. Delete them in instance settings first.';
END IF;
-- Before this, only data tables on Windmill's own cluster could be under roles, and a role
-- block left with just `admin` survives deleting every external role.
IF EXISTS (
SELECT 1 FROM workspace_settings ws,
jsonb_each(CASE WHEN jsonb_typeof(ws.datatable->'datatables') = 'object'
THEN ws.datatable->'datatables' ELSE '{}'::jsonb END) dt
WHERE dt.value->'database'->>'resource_type' = 'external_instance'
AND dt.value ? 'permissions'
) THEN
RAISE EXCEPTION 'external instance data tables are still under roles. Turn their roles off first.';
END IF;
END $$;
ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_cluster_name_key;
ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_name_key UNIQUE (name);
ALTER TABLE datatable_role DROP COLUMN cluster;
@@ -0,0 +1,8 @@
-- A data table role is a Postgres login on one cluster: Windmill's own ('instance'), or the external
-- instance cluster ('external_instance'). Role names are the cluster's own key, so they are unique
-- per cluster rather than across the instance.
ALTER TABLE datatable_role
ADD COLUMN cluster VARCHAR(20) NOT NULL DEFAULT 'instance'
CHECK (cluster IN ('instance', 'external_instance'));
ALTER TABLE datatable_role DROP CONSTRAINT datatable_role_name_key;
ALTER TABLE datatable_role ADD CONSTRAINT datatable_role_cluster_name_key UNIQUE (cluster, name);
+24 -24
View File
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
[[package]]
name = "windmill-common"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"aho-corasick",
"anyhow",
@@ -6274,7 +6274,7 @@ dependencies = [
[[package]]
name = "windmill-macros"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"proc-macro2",
"quote",
@@ -6286,7 +6286,7 @@ dependencies = [
[[package]]
name = "windmill-parser"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"convert_case",
"serde",
@@ -6295,7 +6295,7 @@ dependencies = [
[[package]]
name = "windmill-parser-bash"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6307,7 +6307,7 @@ dependencies = [
[[package]]
name = "windmill-parser-csharp"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6319,7 +6319,7 @@ dependencies = [
[[package]]
name = "windmill-parser-go"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"gosyn",
@@ -6331,7 +6331,7 @@ dependencies = [
[[package]]
name = "windmill-parser-graphql"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6343,7 +6343,7 @@ dependencies = [
[[package]]
name = "windmill-parser-java"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6355,7 +6355,7 @@ dependencies = [
[[package]]
name = "windmill-parser-nu"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"nu-parser",
@@ -6366,7 +6366,7 @@ dependencies = [
[[package]]
name = "windmill-parser-php"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6377,7 +6377,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"itertools 0.14.0",
@@ -6389,7 +6389,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6400,7 +6400,7 @@ dependencies = [
[[package]]
name = "windmill-parser-py-imports"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"async-recursion",
@@ -6422,7 +6422,7 @@ dependencies = [
[[package]]
name = "windmill-parser-r"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde_json",
@@ -6434,7 +6434,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ruby"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6448,7 +6448,7 @@ dependencies = [
[[package]]
name = "windmill-parser-rust"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"convert_case",
@@ -6465,7 +6465,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6478,7 +6478,7 @@ dependencies = [
[[package]]
name = "windmill-parser-sql-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde",
@@ -6490,7 +6490,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6508,7 +6508,7 @@ dependencies = [
[[package]]
name = "windmill-parser-ts-asset"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"serde-wasm-bindgen",
@@ -6524,7 +6524,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wac"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"rustpython-ast",
@@ -6540,7 +6540,7 @@ dependencies = [
[[package]]
name = "windmill-parser-wasm"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"getrandom 0.2.17",
@@ -6572,7 +6572,7 @@ dependencies = [
[[package]]
name = "windmill-parser-yaml"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"lazy_static",
@@ -6586,7 +6586,7 @@ dependencies = [
[[package]]
name = "windmill-types"
version = "1.814.0"
version = "1.813.0"
dependencies = [
"anyhow",
"bitflags",
@@ -12,7 +12,7 @@ resolver = "2"
members = ["."]
[workspace.package]
version = "1.814.0"
version = "1.813.0"
edition = "2021"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
+2 -2
View File
@@ -99,9 +99,9 @@ email_trigger: path(char), local_part(char), workspaced_local_part(bool), script
favorite: usr(char), workspace_id(char), path(char), favorite_kind(favorite_kind)
flow: workspace_id(char), path(char), summary(text), description(text), value(jsonb), edited_by(char), edited_at(ts), archived(bool), schema(json), extra_perms(jsonb), dependency_job(uuid), draft_only(bool), tag(char), ws_error_handler_muted(bool), dedicated_worker(bool), timeout(int), visible_to_runner_only(bool), concurrency_key(char), versions(bigint[]), on_behalf_of(varchar), on_behalf_of_email(text), lock_error_logs(text), labels(text[])
FK: (workspace_id) -> workspace(id)
flow_conversation: id(uuid), workspace_id(char), flow_path(char), title(char), created_at(ts), updated_at(ts), created_by(char), is_test(bool)
flow_conversation: id(uuid), workspace_id(char), flow_path(char), title(char), created_at(ts), updated_at(ts), created_by(char)
FK: (workspace_id) -> workspace(id)
flow_conversation_message: id(uuid), conversation_id(uuid), message_type(message_type), content(text), job_id(uuid), created_at(ts), created_seq(int8), step_name(char), success(bool), tool_arguments(text), tool_result(text), reasoning(text), attachments(jsonb)
flow_conversation_message: id(uuid), conversation_id(uuid), message_type(message_type), content(text), job_id(uuid), created_at(ts), created_seq(int8), step_name(char), success(bool)
FK: (conversation_id) -> flow_conversation(id) | (job_id) -> v2_job(id)
flow_iterator_data: job_id(uuid), itered(jsonb)
flow_node: id(bigint), workspace_id(char), hash(bigint), path(char), lock(text), code(text), flow(jsonb), hash_v2(char(64))
-60
View File
@@ -1199,63 +1199,3 @@ async fn test_wm_labels_from_result_merged_with_static_labels(
Ok(())
}
/// `tag` lives only on `v2_job`, which count_jobs joins only when `tags` is set.
#[sqlx::test(fixtures("base"))]
async fn test_count_completed_jobs_tags_filter(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let client = windmill_api_client::create_client(
&format!("http://localhost:{port}"),
"SECRET_TOKEN".to_string(),
);
for (ws, tag, status) in [
("test-workspace", "deno", "success"),
("test-workspace", "deno", "failure"),
("test-workspace", "python3", "success"),
("other-workspace", "deno", "success"),
] {
let id = uuid::Uuid::new_v4();
sqlx::query("INSERT INTO v2_job (id, workspace_id, tag) VALUES ($1, $2, $3)")
.bind(id)
.bind(ws)
.bind(tag)
.execute(&db)
.await?;
sqlx::query(
"INSERT INTO v2_job_completed (id, workspace_id, status, duration_ms) VALUES ($1, $2, $3::job_status, 0)",
)
.bind(id)
.bind(ws)
.bind(status)
.execute(&db)
.await?;
}
for (query, expected) in [
("", 3),
("tags=deno", 2),
("tags=deno&success=true", 1),
("tags=deno,python3&completed_after_s_ago=3600", 3),
] {
let response = client
.client()
.get(format!(
"{}/w/test-workspace/jobs/completed/count_jobs?{query}",
client.baseurl()
))
.send()
.await?;
assert!(
response.status().is_success(),
"{query}: {}",
response.text().await?
);
assert_eq!(response.json::<i64>().await?, expected, "{query}");
}
Ok(())
}
-2
View File
@@ -258,7 +258,6 @@ async fn test_new_turns_wait_for_conversation_cleanup_and_recreate(
"test-user",
"hi again",
conv_id,
false,
)
.await?;
windmill_common::flow_conversations::add_message_to_conversation_tx(
@@ -269,7 +268,6 @@ async fn test_new_turns_wait_for_conversation_cleanup_and_recreate(
windmill_common::flow_conversations::MessageType::User,
None,
true,
None,
)
.await?;
tx.commit().await?;
-1
View File
@@ -23,7 +23,6 @@ async-trait.workspace = true
async-stream.workspace = true
base64.workspace = true
bytes.workspace = true
indexmap.workspace = true
eventsource-stream.workspace = true
futures.workspace = true
http.workspace = true
+2 -6
View File
@@ -1074,8 +1074,7 @@ impl BedrockQueryBuilder {
let mut accumulated_text = String::new();
let mut events_str = String::new();
let mut accumulated_tool_calls: indexmap::IndexMap<String, StreamingToolCall> =
indexmap::IndexMap::new();
let mut accumulated_tool_calls: HashMap<String, StreamingToolCall> = HashMap::new();
let mut current_tool_use_id: Option<String> = None;
let mut usage: Option<TokenUsage> = None;
// Claude reasoning block for the turn (only populated when thinking is on),
@@ -1264,10 +1263,7 @@ mod tests {
// recovers the uncached share by subtracting the details back out.
assert_eq!(usage["usage"]["prompt_tokens"], 1010);
assert_eq!(usage["usage"]["completion_tokens"], 7);
assert_eq!(
usage["usage"]["prompt_tokens_details"]["cached_tokens"],
900
);
assert_eq!(usage["usage"]["prompt_tokens_details"]["cached_tokens"], 900);
assert_eq!(
usage["usage"]["prompt_tokens_details"]["cache_write_tokens"],
100
+8 -11
View File
@@ -1,7 +1,6 @@
use std::collections::HashMap;
use eventsource_stream::Eventsource;
use indexmap::IndexMap;
use reqwest::Response;
use serde::Deserialize;
use tokio_stream::StreamExt;
@@ -138,9 +137,7 @@ pub struct OpenAISSEParser {
pub accumulated_content: String,
/// The thinking streamed before the answer, kept so it can be stored with it.
pub accumulated_reasoning: String,
// Insertion-ordered in every parser: tool calls run and are persisted in the order the
// stream showed them, and a chat attaches a round's thinking to its first call.
pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
pub events_str: String,
pub stream_event_processor: Box<dyn StreamEventSink>,
/// Token usage from final chunk (when stream_options.include_usage is true)
@@ -152,7 +149,7 @@ impl OpenAISSEParser {
Self {
accumulated_content: String::new(),
accumulated_reasoning: String::new(),
accumulated_tool_calls: IndexMap::new(),
accumulated_tool_calls: HashMap::new(),
events_str: String::new(),
stream_event_processor,
usage: None,
@@ -362,7 +359,7 @@ pub struct AnthropicSSEParser {
pub accumulated_content: String,
/// The thinking streamed before the answer, kept so it can be stored with it.
pub accumulated_reasoning: String,
pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
pub events_str: String,
pub stream_event_processor: Box<dyn StreamEventSink>,
/// Track content block types by index
@@ -385,7 +382,7 @@ impl AnthropicSSEParser {
Self {
accumulated_content: String::new(),
accumulated_reasoning: String::new(),
accumulated_tool_calls: IndexMap::new(),
accumulated_tool_calls: HashMap::new(),
events_str: String::new(),
stream_event_processor,
content_blocks: HashMap::new(),
@@ -604,7 +601,7 @@ pub struct GeminiSSEParser {
pub accumulated_content: String,
/// The thinking streamed before the answer, kept so it can be stored with it.
pub accumulated_reasoning: String,
pub accumulated_tool_calls: IndexMap<i64, OpenAIToolCall>,
pub accumulated_tool_calls: HashMap<i64, OpenAIToolCall>,
pub events_str: String,
pub stream_event_processor: Box<dyn StreamEventSink>,
tool_call_index: i64,
@@ -618,7 +615,7 @@ impl GeminiSSEParser {
Self {
accumulated_content: String::new(),
accumulated_reasoning: String::new(),
accumulated_tool_calls: IndexMap::new(),
accumulated_tool_calls: HashMap::new(),
events_str: String::new(),
stream_event_processor,
tool_call_index: 0,
@@ -836,7 +833,7 @@ pub struct OpenAIResponsesSSEParser {
pub accumulated_content: String,
/// The reasoning summary streamed before the answer, kept so it can be stored with it.
pub accumulated_reasoning: String,
pub accumulated_tool_calls: IndexMap<String, OpenAIToolCall>,
pub accumulated_tool_calls: HashMap<String, OpenAIToolCall>,
/// Maps item_id -> (name, call_id) for function calls
tool_call_metadata: HashMap<String, (String, String)>,
/// Maps item_id -> accumulated arguments
@@ -858,7 +855,7 @@ impl OpenAIResponsesSSEParser {
Self {
accumulated_content: String::new(),
accumulated_reasoning: String::new(),
accumulated_tool_calls: IndexMap::new(),
accumulated_tool_calls: HashMap::new(),
tool_call_metadata: HashMap::new(),
tool_call_arguments: HashMap::new(),
events_str: String::new(),
+6 -56
View File
@@ -78,50 +78,17 @@ impl Default for OutputType {
#[serde(tag = "kind", rename_all = "lowercase")]
pub enum Memory {
Off,
Window {
#[serde(default, deserialize_with = "deserialize_null_as_zero")]
context_length: usize,
},
/// Written before `window`. Its `memory_id` stays a fallback behind the run's memory id.
Auto {
#[serde(default, deserialize_with = "deserialize_null_as_zero")]
#[serde(default)]
context_length: usize,
#[serde(default, deserialize_with = "deserialize_blank_as_none")]
#[serde(default)]
memory_id: Option<Uuid>,
},
/// Written before a step had history inputs of its own, and read on its own where it remains.
Manual {
messages: Vec<OpenAIMessage>,
},
}
// An editor form can leave `""` in a legacy baked id it never filled; it means no id rather than
// failing every run of the step.
fn deserialize_blank_as_none<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Option<Uuid>, D::Error> {
match <Option<String> as serde::Deserialize>::deserialize(deserializer)? {
Some(id) if !id.trim().is_empty() => Uuid::parse_str(id.trim())
.map(Some)
.map_err(serde::de::Error::custom),
_ => Ok(None),
}
}
// A count the editor's number field was cleared of is stored as `null`, which `default` does not
// cover; it reads as 0, memory off, rather than failing every run of the step.
fn deserialize_null_as_zero<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<usize, D::Error> {
<Option<usize> as serde::Deserialize>::deserialize(deserializer).map(Option::unwrap_or_default)
}
fn deserialize_present<'de, D: serde::Deserializer<'de>>(
deserializer: D,
) -> Result<Option<serde_json::Value>, D::Error> {
<serde_json::Value as serde::Deserialize>::deserialize(deserializer).map(Some)
}
#[derive(Deserialize)]
struct AIAgentArgsRaw {
provider: ProviderWithResource,
@@ -136,12 +103,6 @@ struct AIAgentArgsRaw {
streaming: Option<bool>,
max_iterations: Option<usize>,
memory: Option<Memory>,
// A null must stay distinguishable from an absent key: a step whose own memory id evaluates to
// nothing runs stateless instead of falling back to the run's memory id.
#[serde(default, deserialize_with = "deserialize_present")]
memory_id: Option<serde_json::Value>,
#[serde(default)]
previous_messages: Option<Vec<OpenAIMessage>>,
enabled_tools: Option<Vec<String>>,
// Legacy field for backward compatibility
messages_context_length: Option<usize>,
@@ -163,10 +124,6 @@ pub struct AIAgentArgs {
pub streaming: Option<bool>,
pub max_iterations: Option<usize>,
pub memory: Option<Memory>,
/// Memory id set on the step, overriding the run's. Empty when its expression produced none.
pub memory_id: Option<String>,
/// History supplied by the flow, replayed without reading or writing memory.
pub previous_messages: Option<Vec<OpenAIMessage>>,
/// Which of the agent's tools this run may call; `narrow_roster` holds what the names are and
/// what `None` means.
pub enabled_tools: Option<Vec<String>>,
@@ -182,17 +139,12 @@ impl From<AIAgentArgsRaw> for AIAgentArgs {
});
// Backward compatibility: if context_length is 0, use off mode
let memory = memory.map(|memory| match memory {
Memory::Auto { context_length: 0, .. } | Memory::Window { context_length: 0 } => {
let memory = memory.map(|memory| {
if let Memory::Auto { context_length: 0, .. } = memory {
Memory::Off
} else {
memory
}
memory => memory,
});
let memory_id = raw.memory_id.map(|value| match value {
serde_json::Value::Null => String::new(),
serde_json::Value::String(s) => s.trim().to_string(),
value => value.to_string(),
});
AIAgentArgs {
@@ -207,8 +159,6 @@ impl From<AIAgentArgsRaw> for AIAgentArgs {
streaming: raw.streaming,
max_iterations: raw.max_iterations,
memory,
memory_id,
previous_messages: raw.previous_messages,
enabled_tools: raw.enabled_tools,
credentials_check: raw.credentials_check.unwrap_or(false),
}
@@ -1,6 +1,6 @@
use axum::{
extract::{Path, Query},
routing::{delete, get, post},
routing::{delete, get},
Extension, Json, Router,
};
use chrono::{DateTime, Utc};
@@ -15,14 +15,13 @@ use windmill_common::{
db::{UserDB, DB},
error::{JsonResult, Result},
flow_conversations::MessageType,
utils::{not_found_if_none, paginate, truncate_with_ellipsis, Pagination},
utils::{not_found_if_none, paginate, Pagination},
};
pub fn workspaced_service() -> Router {
Router::new()
.route("/list", get(list_conversations))
.route("/delete/{conversation_id}", delete(delete_conversation))
.route("/update/{conversation_id}", post(update_conversation))
.route("/{conversation_id}/messages", get(list_messages))
}
@@ -37,37 +36,11 @@ pub struct FlowConversationMessage {
pub created_seq: i64,
pub step_name: Option<String>,
pub success: bool,
/// On a tool row, the arguments the model wrote. For a Windmill tool these exclude the
/// inputs its step wires in. Null for a web search, whose query the provider does not
/// return.
pub tool_arguments: Option<String>,
/// On a tool row, the text the model got back, or what the call failed with; a web
/// search's citations.
pub tool_result: Option<String>,
/// On an answer, the thinking that produced it; on a tool row, the thinking that led to
/// the call. The agent job keeps the turn's thinking as one string.
pub reasoning: Option<String>,
/// The files a user message carried, as object-storage references
/// (`[{input, s3, storage?, filename?}]`).
pub attachments: Option<sqlx::types::JsonValue>,
}
/// Which conversations a listing holds. A test chat was started from the editor's test
/// panel; a deployed one from the flow itself.
#[derive(Deserialize, Default, Clone, Copy)]
#[serde(rename_all = "lowercase")]
pub enum ConversationKind {
Test,
/// The default: a deployed flow's chat should not surface someone's trial runs.
#[default]
Deployed,
All,
}
#[derive(Deserialize)]
pub struct ListConversationsQuery {
pub flow_path: Option<String>,
pub kind: Option<ConversationKind>,
}
#[derive(Deserialize)]
@@ -94,7 +67,6 @@ async fn list_conversations(
"created_at",
"updated_at",
"created_by",
"is_test",
])
.and_where_eq("workspace_id", "?".bind(&w_id));
@@ -102,16 +74,6 @@ async fn list_conversations(
sqlb.and_where_eq("flow_path", "?".bind(flow_path));
}
match query.kind.unwrap_or_default() {
ConversationKind::Test => {
sqlb.and_where_eq("is_test", "true");
}
ConversationKind::Deployed => {
sqlb.and_where_eq("is_test", "false");
}
ConversationKind::All => {}
}
sqlb.order_by("updated_at", true)
.limit(per_page as i64)
.offset(offset as i64);
@@ -139,7 +101,7 @@ async fn delete_conversation(
// Verify the conversation exists and belongs to the user
let conversation = sqlx::query_as!(
FlowConversation,
"SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test
"SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by
FROM flow_conversation
WHERE id = $1 AND workspace_id = $2",
conversation_id,
@@ -186,50 +148,6 @@ async fn delete_conversation(
Ok(format!("Conversation {} deleted", conversation_id))
}
#[derive(Deserialize)]
pub struct UpdateConversation {
pub title: String,
}
async fn update_conversation(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Path((w_id, conversation_id)): Path<(String, Uuid)>,
Json(update): Json<UpdateConversation>,
) -> Result<String> {
// Postgres refuses a NUL in a text column, so it must not reach the query as a 500.
if update.title.contains('\0') {
return Err(windmill_common::error::Error::BadRequest(
"title cannot contain a NUL character".to_string(),
));
}
// The column is VARCHAR(255) and the helper appends an ellipsis to what it cuts, so the
// bound it takes is three short of the column's. A longer title would otherwise reach
// Postgres as a 22001 and come back a 500.
let title = truncate_with_ellipsis(update.title.trim(), 252);
let mut tx = user_db.clone().begin(&authed).await?;
// `updated_at` is kept: the list is ordered by it, and a rename must not move the
// chat to the top the way a new turn does.
let updated = sqlx::query_scalar!(
"UPDATE flow_conversation SET title = $1, updated_at = updated_at
WHERE id = $2 AND workspace_id = $3
RETURNING id",
title,
conversation_id,
&w_id
)
.fetch_optional(&mut *tx)
.await?;
not_found_if_none(updated, "Conversation", conversation_id.to_string())?;
tx.commit().await?;
Ok(format!("Conversation {} updated", conversation_id))
}
async fn list_messages(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
@@ -260,7 +178,7 @@ async fn list_messages(
let messages = if let Some(after_seq) = query.after_seq {
sqlx::query_as!(
FlowConversationMessage,
r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success
FROM flow_conversation_message
WHERE conversation_id = $1
AND created_seq > $2
@@ -277,9 +195,9 @@ async fn list_messages(
// Fetch messages for this conversation, oldest first, but reverse the order of the messages for easy rendering on the frontend
sqlx::query_as!(
FlowConversationMessage,
r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
r#"SELECT id, conversation_id, message_type as "message_type: MessageType", content, job_id, created_at, created_seq, step_name, success
FROM (
SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success, tool_arguments, tool_result, reasoning, attachments
SELECT id, conversation_id, message_type, content, job_id, created_at, created_seq, step_name, success
FROM flow_conversation_message
WHERE conversation_id = $1
ORDER BY created_seq DESC
@@ -391,7 +391,11 @@ async fn concurrent_role_creations_both_survive(db: Pool<Postgres>) -> anyhow::R
assert_eq!(a.0, 200, "{}", a.1);
assert_eq!(b.0, 200, "{}", b.1);
let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
let catalog = windmill_common::datatable_roles::read_role_catalog(
&db,
windmill_common::datatable_roles::DatatableRoleCluster::Instance,
)
.await?;
let recorded: Vec<&str> = catalog.values().map(|r| r.name.as_str()).collect();
for name in &names {
assert!(
@@ -460,7 +464,11 @@ async fn a_role_delete_that_fails_part_way_leaves_the_role_disabled(
let body = resp.text().await?;
assert_eq!(status, 400, "{body}");
let catalog = windmill_common::datatable_roles::read_role_catalog(&db).await?;
let catalog = windmill_common::datatable_roles::read_role_catalog(
&db,
windmill_common::datatable_roles::DatatableRoleCluster::Instance,
)
.await?;
let role = catalog
.get(&id)
.expect("a failed delete keeps the entry to retry");
+6 -17
View File
@@ -26,9 +26,7 @@ use windmill_api_auth::{check_scopes, get_scope_tags, ApiAuthed};
use windmill_common::{
db::{UserDB, UserDbWithAuthed},
error::{self, Error},
flow_conversations::{
add_message_to_conversation_tx, message_attachments, MessageExtras, MessageType,
},
flow_conversations::{add_message_to_conversation_tx, MessageType},
get_latest_flow_version_info_for_path,
jobs::{
check_tag_available_for_workspace_internal, format_result, script_path_to_payload,
@@ -655,11 +653,9 @@ pub async fn set_flow_memory_id(
pub async fn process_flow_run_query_params(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
job_id: Uuid,
w_id: &str,
flow_path: &str,
run_query: &RunJobQuery,
) -> error::Result<()> {
if let Some(memory_id) = run_query.memory_key(w_id, flow_path) {
if let Some(memory_id) = run_query.memory_id {
set_flow_memory_id(tx, job_id, memory_id).await?;
}
Ok(())
@@ -673,13 +669,10 @@ pub async fn handle_chat_conversation_messages(
run_query: &RunJobQuery,
user_message_raw: Option<&Box<serde_json::value::RawValue>>,
job_id: Uuid,
is_test: bool,
// The run's args, for the files the message carried.
args: &HashMap<String, Box<serde_json::value::RawValue>>,
) -> error::Result<()> {
// Names the query parameter rather than the field: it is not a flow argument, and
// supplying it as one is the first thing tried on reading `memory_id is required`.
let memory_id = run_query.memory_key(w_id, flow_path).ok_or_else(|| {
let memory_id = run_query.memory_id.ok_or_else(|| {
windmill_common::error::Error::BadRequest(
"memory_id is required for chat-enabled flows. Pass it as the `memory_id` query \
parameter, not as a flow argument: it names the conversation the turn belongs to, \
@@ -708,12 +701,11 @@ pub async fn handle_chat_conversation_messages(
&authed.username,
&user_message,
memory_id,
is_test,
)
.await?;
// The run this message started. The row keeps the files the message carried as
// references; its args are the only record of every other flow input, and nothing
// The run this message started. Its args are the only record of what the message
// carried besides its text — attachments and every other flow input and nothing
// written later points at them: an assistant row holds the AI agent step's job.
add_message_to_conversation_tx(
tx,
@@ -723,7 +715,6 @@ pub async fn handle_chat_conversation_messages(
MessageType::User,
None,
true,
Some(&MessageExtras { attachments: message_attachments(args), ..Default::default() }),
)
.await?;
@@ -831,7 +822,7 @@ pub async fn run_flow<'c>(
.await?;
// Set memory_id if provided (for agent memory)
if let Some(memory_id) = run_query.memory_key(w_id, flow_path) {
if let Some(memory_id) = run_query.memory_id {
set_flow_memory_id(&mut tx, uuid, memory_id).await?;
}
@@ -845,8 +836,6 @@ pub async fn run_flow<'c>(
&run_query,
args.args.get("user_message"),
uuid,
false,
&args.args,
)
.await?;
}
+1 -13
View File
@@ -47,25 +47,13 @@ pub struct RunJobQuery {
pub cache_ignore_s3_path: Option<bool>,
pub skip_preprocessor: Option<bool>,
pub poll_delay_ms: Option<u64>,
/// Any string; see [`RunJobQuery::memory_key`].
pub memory_id: Option<String>,
pub memory_id: Option<Uuid>,
pub trigger_external_id: Option<String>,
pub service_name: Option<String>,
pub suspended_mode: Option<bool>,
}
impl RunJobQuery {
/// The memory id as stored in `flow_status.memory_id`: a uuid is kept, any other string hashed
/// within the workspace and the flow being run.
pub fn memory_key(&self, workspace_id: &str, flow_path: &str) -> Option<Uuid> {
self.memory_id
.as_deref()
.filter(|memory_id| !memory_id.trim().is_empty())
.map(|memory_id| {
windmill_common::flow_conversations::memory_key(workspace_id, flow_path, memory_id)
})
}
pub async fn get_scheduled_for(
&self,
db: &DB,
+18 -26
View File
@@ -42,6 +42,7 @@ use axum::{
routing::{get, post},
Json, Router,
};
use serde_json::json;
use serde::{Deserialize, Serialize};
use windmill_ai::ai_cache::bump_instance_ai_config_revision;
@@ -1719,15 +1720,7 @@ async fn list_custom_instance_pg_databases(
))
})?;
if !windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
// Which workspace reserved a fork copy is nobody else's business: it would enumerate every
// pending fork on the instance.
for entry in result.values_mut() {
entry.workspace_id = None;
}
return Ok(Json(result));
}
{
if windmill_api_auth::is_super_admin_authed(&db, &authed).await? {
// Enrich each database with the list of workspaces referencing it through
// either a ducklake catalog or a datatable database whose resource_type is
// 'instance'. Not stored in DB to avoid drift.
@@ -1926,6 +1919,16 @@ async fn setup_custom_instance_pg_database(
// Before anything is recorded: the status written below replaces the registry entry, and with it
// the workspace a fork copy is reserved for.
require_super_admin(&db, &authed).await?;
// A re-run keeps the fork reservation: without it, the workspace the copy was made for could no
// longer import into it or finish its fork.
let workspace_id = sqlx::query_scalar::<_, Option<String>>(
"SELECT value->'databases'->$1->>'workspace_id' FROM global_settings
WHERE name = 'custom_instance_pg_databases'",
)
.bind(&dbname)
.fetch_optional(&db)
.await?
.flatten();
let mut logs = CustomInstanceDbLogs::default();
let result = setup_custom_instance_pg_database_inner(authed, &db, &dbname, &mut logs).await;
let success = result.is_ok();
@@ -1936,25 +1939,14 @@ async fn setup_custom_instance_pg_database(
error,
tag: body.tag,
used_by_workspaces: vec![],
workspace_id: None,
workspace_id,
};
let status_json = serde_json::to_value(&status).map_err(to_anyhow)?;
// The fork reservation is carried over inside the write, from whatever the row holds then: a
// rename migrating it while the setup above ran would otherwise be overwritten with the value
// this request started from, stranding the copy under the archived workspace.
let saved = sqlx::query_scalar::<_, serde_json::Value>(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}',
COALESCE(value->'databases', '{}'::jsonb)
|| jsonb_build_object($1::text, $2::jsonb || jsonb_build_object(
'workspace_id', value->'databases'->$1::text->'workspace_id')))
WHERE name = 'custom_instance_pg_databases'
RETURNING value->'databases'->$1::text"#,
)
.bind(&dbname)
.bind(&status_json)
.fetch_one(&db)
.await?;
let status: CustomInstanceDb = serde_json::from_value(saved).map_err(to_anyhow)?;
// Save that the database was setup successfully
sqlx::query!(
r#"UPDATE global_settings SET value = jsonb_set(value, '{databases}', (COALESCE(value->'databases', '{}'::jsonb) || to_jsonb($1::json))) WHERE name = 'custom_instance_pg_databases'"#,
json!({ dbname: status_json })
).execute(&db).await?;
Ok(Json(status))
}
@@ -6,7 +6,7 @@
* LICENSE-AGPL for a copy of the license.
*/
//! Ownership and grants on the objects of an instance data table.
//! Ownership and grants on the objects of a data table on a cluster Windmill manages.
//!
//! [`datatable_permissions`](crate::datatable_permissions) decides who may connect as which role;
//! this decides what each role may then touch. Every change is a real `GRANT`, `REVOKE`,
@@ -33,7 +33,7 @@ use windmill_audit::audit_oss::audit_log;
use windmill_audit::ActionKind;
use windmill_common::datatable_roles::{
lock_role_catalog, quote_ident, read_role_catalog, read_role_catalog_tx, DatatableRoleCatalog,
ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
DatatableRoleCluster, ADMIN_DATATABLE_ROLE, CUSTOM_INSTANCE_USER,
};
use windmill_common::error::{pg_error_message, Error, JsonResult, Result};
use windmill_common::workspaces::{resolve_governing_datatable, DataTable, GoverningDatatable};
@@ -295,22 +295,22 @@ fn role_names(catalog: &DatatableRoleCatalog) -> Vec<String> {
names
}
fn ensure_instance(governing: &GoverningDatatable) -> Result<()> {
if governing.is_instance() {
return Ok(());
}
Err(Error::BadRequest(format!(
"Data table '{}' is backed by a Postgres resource, so its access is managed on that \
server directly. Only a data table on the Windmill instance's own database has data \
table roles to grant to.",
governing.name
)))
/// The cluster whose roles the data table's grants name.
fn ensure_managed(governing: &GoverningDatatable) -> Result<DatatableRoleCluster> {
governing.role_cluster().ok_or_else(|| {
Error::BadRequest(format!(
"Data table '{}' is backed by a Postgres resource, so its access is managed on that \
server directly. Only a data table on a database Windmill manages has data table \
roles to grant to.",
governing.name
))
})
}
/// The data table's `admin` connection, and the notices Postgres sends on it.
///
/// Authorization: connects as `custom_instance_user` with the instance's own credentials and checks
/// nothing. Callers MUST have authorized the request first — a request about to be refused must
/// Authorization: connects as `custom_instance_user` with the cluster's stored credentials and
/// checks nothing. Callers MUST have authorized the request first — a request about to be refused must
/// not get as far as this connection.
async fn connect_as_admin_unchecked(
db: &DB,
@@ -320,21 +320,33 @@ async fn connect_as_admin_unchecked(
mpsc::UnboundedReceiver<DbError>,
String,
)> {
ensure_instance(governing)?;
let cluster = ensure_managed(governing)?;
// Built from the authorized entry, never by resolving the settings again: a save in between
// could point the entry at a resource on another server and back, and this connection would
// then alter a database the later checks of the entry never see.
let mut pg = PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
pg.dbname = governing
let dbname = governing
.datatable
.database
.as_ref()
.expect("a governing entry owns a database")
.resource_path
.clone();
pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
let dbname = pg.dbname.clone();
let pg = match cluster {
DatatableRoleCluster::Instance => {
let mut pg =
PgDatabase::parse_uri(&windmill_common::get_database_url().await?.as_str().await)?;
pg.dbname = dbname.clone();
pg.user = Some(CUSTOM_INSTANCE_USER.to_string());
pg.password = Some(windmill_common::utils::get_custom_pg_instance_password(db).await?);
pg
}
DatatableRoleCluster::ExternalInstance => {
windmill_common::external_instance_pg::external_instance_connection_unchecked(
db, &dbname, false,
)
.await?
}
};
let (client, mut connection) = pg.connect(Some(db)).await?;
// Unbounded: the driver must never wait on the receiver, which only drains once the statement
// the driver is carrying has completed.
@@ -1020,12 +1032,12 @@ async fn get_datatable_acl(
let target: AclTarget = query.try_into()?;
let governing = resolve_governing_datatable(&db, &w_id, &datatable_name).await?;
ensure_reaches_governing_datatable(&db, &w_id, &datatable_name, &governing, &authed).await?;
ensure_instance(&governing)?;
let cluster = ensure_managed(&governing)?;
let editable = ensure_governs_datatable(&db, &authed, &w_id, &governing)
.await
.is_ok();
let roles = if editable {
role_names(&read_role_catalog(&db).await?)
role_names(&read_role_catalog(&db, cluster).await?)
} else {
vec![]
};
@@ -1310,7 +1322,7 @@ async fn authorize_acl_change(
) -> Result<GoverningDatatable> {
let governing = resolve_governing_datatable(db, w_id, datatable_name).await?;
ensure_governs_datatable(db, authed, w_id, &governing).await?;
ensure_instance(&governing)?;
ensure_managed(&governing)?;
Ok(governing)
}
@@ -1320,8 +1332,14 @@ static APPLY_SLOT: tokio::sync::Semaphore = tokio::sync::Semaphore::const_new(1)
/// provisioned before data table roles gave `custom_instance_user` none. Adds that option to its
/// database and `public` privileges, and nothing else: default privileges are left alone, since a
/// schema's change of owner is planned against them. Best-effort, as a grant it fails to enable is
/// refused when it runs.
async fn ensure_grant_options(client: &tokio_postgres::Client, db: &DB, dbname: &str) {
/// refused when it runs. An external instance database was created with the options, so one
/// missing there is someone's deliberate revoke and is left alone.
async fn ensure_grant_options(
client: &tokio_postgres::Client,
db: &DB,
cluster: DatatableRoleCluster,
dbname: &str,
) {
let held = client
.query_one(
"SELECT has_database_privilege(current_database(), 'CONNECT WITH GRANT OPTION')
@@ -1333,7 +1351,7 @@ async fn ensure_grant_options(client: &tokio_postgres::Client, db: &DB, dbname:
)
.await
.is_ok_and(|row| row.get::<_, bool>(0));
if held {
if held || cluster != DatatableRoleCluster::Instance {
return;
}
if let Err(e) = grant_options_as_server(db, dbname).await {
@@ -1559,7 +1577,8 @@ async fn plan_datatable_acl(
) -> JsonResult<AclPlan> {
crate::datatable_acl_oss::ensure_datatable_acl_available()?;
let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
let catalog = read_role_catalog(&db).await?;
let cluster = ensure_managed(&governing)?;
let catalog = read_role_catalog(&db, cluster).await?;
let (client, _notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
Ok(Json(
build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?,
@@ -1583,6 +1602,7 @@ async fn apply_datatable_acl(
// connection could wait forever on a pool that concurrent applies, queued on the same locks,
// have exhausted.
let governing = authorize_acl_change(&db, &authed, &w_id, &datatable_name).await?;
let cluster = ensure_managed(&governing)?;
// Applies queue on an instance-wide lock while each holds a direct connection to the instance's
// Postgres; unbounded, the queue alone could exhaust its connection limit. One at a time per
// server, and the ones waiting hold no connection at all.
@@ -1591,7 +1611,7 @@ async fn apply_datatable_acl(
.await
.map_err(|e| Error::internal_err(format!("ACL apply slot closed: {e}")))?;
let (mut client, mut notices, dbname) = connect_as_admin_unchecked(&db, &governing).await?;
ensure_grant_options(&client, &db, &dbname).await;
ensure_grant_options(&client, &db, cluster, &dbname).await;
// Held until the change is committed: a role renamed or dropped meanwhile would change what
// the plan names, and a settings save could move the entry onto another database. Taken in the
@@ -1606,7 +1626,7 @@ async fn apply_datatable_acl(
.fetch_optional(&mut *tx)
.await?
.flatten();
let catalog = read_role_catalog_tx(&mut tx).await?;
let catalog = read_role_catalog_tx(&mut tx, cluster).await?;
let plan = build_plan(&client, &dbname, &catalog, &req.target, &req.change).await?;
if !entry_unchanged(&governing, entry_now) || &plan.statements != confirmed {
+70 -102
View File
@@ -2263,7 +2263,8 @@ struct DataTableTables {
schemas: TableListMap,
#[serde(skip_serializing_if = "Option::is_none")]
error: Option<String>,
/// On the instance database: the only kind that can be under roles or have its access edited.
/// On a database Windmill manages, on its own cluster or the external one: the only kinds that
/// can be under roles or have their access edited.
instance: bool,
permissioned: bool,
/// The roles this caller may connect as, by name; empty when not under roles.
@@ -2520,7 +2521,7 @@ async fn list_one_datatable_tables(
};
let result: Result<()> = async {
let governing = resolve_governing_datatable(db, w_id, &entry.datatable_name).await?;
entry.instance = governing.is_instance();
entry.instance = governing.role_cluster().is_some();
let usable =
crate::datatable_permissions_oss::usable_datatable_roles(db, authed, w_id, &governing)
.await?;
@@ -3370,7 +3371,7 @@ fn apply_pg_tls_env(
cmd.env("PGSSLROOTCERT", "system");
}
Some("verify-ca") => {
if let Some(bundle) = system_ca_bundle() {
if let Some(bundle) = windmill_common::system_ca_bundle() {
cmd.env("PGSSLROOTCERT", bundle);
}
}
@@ -3380,21 +3381,6 @@ fn apply_pg_tls_env(
Ok(None)
}
fn system_ca_bundle() -> Option<std::path::PathBuf> {
std::env::var_os("SSL_CERT_FILE")
.map(std::path::PathBuf::from)
.into_iter()
.chain(
[
"/etc/ssl/certs/ca-certificates.crt",
"/etc/pki/tls/certs/ca-bundle.crt",
"/etc/ssl/cert.pem",
"/etc/ssl/ca-bundle.pem",
]
.map(std::path::PathBuf::from),
)
.find(|path| path.is_file())
}
#[cfg(test)]
mod pg_tls_env_tests {
@@ -3695,7 +3681,6 @@ async fn import_pg_database(
}
let schema_only = req.fork_behavior == DataTableForkBehavior::SchemaOnly;
let mut fork_lock: Option<Transaction<'_, Postgres>> = None;
let source_pg = resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.source).await?;
let mut target_pg =
resolve_pg_source_checked(&db, &user_db, &authed, &w_id, &req.target).await?;
@@ -3709,10 +3694,6 @@ async fn import_pg_database(
));
}
if let Some(kind) = managed_datatable_source_kind(&db, &w_id, &req.target).await? {
// Held until the restore is done, as fork finalization takes it: a fork must not
// commit this database while `psql` is still filling it.
let mut tx = db.begin().await?;
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
windmill_common::ensure_fork_database_available_to(
&db,
kind,
@@ -3720,7 +3701,6 @@ async fn import_pg_database(
&w_id,
)
.await?;
fork_lock = Some(tx);
}
}
target_pg.dbname = override_dbname.clone();
@@ -3744,9 +3724,6 @@ async fn import_pg_database(
)
.await?;
pg_import_dump(&target_pg, &dump_file).await?;
if let Some(tx) = fork_lock {
tx.commit().await?;
}
Ok(format!(
"Imported from '{}' into '{}'",
@@ -3852,22 +3829,37 @@ async fn edit_ducklake_config(
let old_ducklakes: HashMap<String, Ducklake> =
serde_json::from_value(old_ducklakes).unwrap_or_default();
// Check that non-superadmins are not abusing Instance databases
if !is_superadmin {
for (name, dl) in new_config.settings.ducklakes.iter() {
if dl.catalog.resource_type == DucklakeCatalogResourceType::Instance {
let old_dl = old_ducklakes.get(name);
if old_dl.is_none()
|| old_dl.unwrap().catalog.resource_type
!= DucklakeCatalogResourceType::Instance
|| old_dl.unwrap().catalog.resource_path != dl.catalog.resource_path
{
return Err(Error::BadRequest(
"Only superadmins can create or modify ducklakes with Instance databases"
.to_string(),
));
}
}
// Check that non-superadmins are not abusing Instance databases. An unchanged catalog is left
// alone either way, so a downgraded instance can still save lakes that already name an
// external instance database.
for (name, dl) in new_config.settings.ducklakes.iter() {
let kind = &dl.catalog.resource_type;
if !matches!(
kind,
DucklakeCatalogResourceType::Instance | DucklakeCatalogResourceType::ExternalInstance
) {
continue;
}
let unchanged = old_ducklakes.get(name).is_some_and(|old| {
&old.catalog.resource_type == kind
&& old.catalog.resource_path == dl.catalog.resource_path
});
if unchanged {
continue;
}
if *kind == DucklakeCatalogResourceType::ExternalInstance {
windmill_common::external_instance_pg::ensure_external_instance_available()?;
windmill_common::external_instance_pg::ensure_external_instance_database_registered(
&mut tx,
&dl.catalog.resource_path,
)
.await?;
}
if !is_superadmin {
return Err(Error::BadRequest(
"Only superadmins can create or modify ducklakes with Instance databases"
.to_string(),
));
}
}
@@ -3926,8 +3918,6 @@ async fn edit_datatable_config(
let is_superadmin = require_super_admin(&db, &authed).await.is_ok();
let mut tx = db.begin().await?;
// Ahead of the settings row, as fork cleanup of this workspace takes the two.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
// Read under the row lock this transaction will write with. `permissions`, `reference` and
// `forked_from` are carried across from what this read returns, so a permissions save
@@ -4061,18 +4051,20 @@ async fn edit_datatable_config(
None => None,
};
// Carrying the block onto a resource-backed entry would produce a data table the chokepoint
// refuses on every job — a save that succeeds and breaks everything afterwards. Refuse it
// instead: turning roles off first is one step, and it keeps discarding an access decision
// something somebody chose rather than a side effect of moving a database.
// refuses on every job — a save that succeeds and breaks everything afterwards — and onto
// the other managed cluster, one whose role ids name nothing in that cluster's catalog.
// Refuse it instead: turning roles off first is one step, and it keeps discarding an access
// decision something somebody chose rather than a side effect of moving a database.
let old_kind = old.and_then(|old| old.database.as_ref()).map(|d| d.resource_type);
if dt.permissions.is_some()
&& dt
.database
.as_ref()
.is_some_and(|d| d.resource_type != DataTableCatalogResourceType::Instance)
.is_some_and(|d| Some(d.resource_type) != old_kind)
{
return Err(Error::BadRequest(format!(
"Data table '{name}' is under roles, which only a data table on the instance \
database can be. Turn its roles off before moving it to a PostgreSQL resource."
"Data table '{name}' is under roles, which belong to the cluster its database is \
on. Turn its roles off before moving it to another kind of database."
)));
}
// A pointer names no database of its own, so the form's empty `database` is correct there.
@@ -4117,14 +4109,6 @@ async fn edit_datatable_config(
if unchanged {
continue;
}
// Before the registration check, whose refusal would otherwise tell a workspace admin
// which databases exist on the cluster.
if !is_superadmin {
return Err(Error::BadRequest(
"Only superadmins can create or modify data tables with Instance databases"
.to_string(),
));
}
if database.resource_type == DataTableCatalogResourceType::ExternalInstance {
windmill_common::external_instance_pg::ensure_external_instance_available()?;
windmill_common::external_instance_pg::ensure_external_instance_database_registered(
@@ -4133,6 +4117,12 @@ async fn edit_datatable_config(
)
.await?;
}
if !is_superadmin {
return Err(Error::BadRequest(
"Only superadmins can create or modify data tables with Instance databases"
.to_string(),
));
}
}
// Worked out from the locked entries rather than taken from `deleted_datatables`: a settings
@@ -4151,7 +4141,7 @@ async fn edit_datatable_config(
// entry through a declared rename alone, and a settings sync never declares one, so an entry
// without roles that newly points at such a database — a name added, or an existing one
// repointed — would answer everyone there as `admin`. That holds whichever workspace governs it.
let newly_pointed: Vec<(&String, &str)> = new_config
let newly_pointed: Vec<(&String, DataTableCatalogResourceType, &str)> = new_config
.settings
.datatables
.iter()
@@ -4160,7 +4150,7 @@ async fn edit_datatable_config(
let db = dt
.database
.as_ref()
.filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
.filter(|d| d.resource_type.is_windmill_managed())?;
let lookup = rename_src
.get(name.as_str())
.copied()
@@ -4172,66 +4162,44 @@ async fn edit_datatable_config(
old_db.resource_type != db.resource_type
|| old_db.resource_path != db.resource_path
});
repointed.then_some((name, db.resource_path.as_str()))
repointed.then_some((name, db.resource_type, db.resource_path.as_str()))
})
.collect();
// Another workspace turning roles on for the same database holds only its own settings row, so
// without this the scan below could read past its uncommitted write. Every managed database
// this save newly names is locked, not just the ones the scan is about: fork cleanup takes the
// same lock to decide nothing uses the database it is dropping.
let newly_named: std::collections::BTreeSet<&str> = new_config
.settings
.datatables
.iter()
.filter_map(|(name, dt)| {
let db = dt
.database
.as_ref()
.filter(|d| d.resource_type == DataTableCatalogResourceType::Instance)?;
let lookup = rename_src
.get(name.as_str())
.copied()
.unwrap_or(name.as_str());
old_datatables
.get(lookup)
.and_then(|old| old.database.as_ref())
.is_none_or(|old_db| {
old_db.resource_type != db.resource_type
|| old_db.resource_path != db.resource_path
})
.then_some(db.resource_path.as_str())
})
.collect();
// without this the scan below could read past its uncommitted write.
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut *tx,
newly_pointed
.iter()
.map(|(_, dbname)| *dbname)
.chain(newly_named.iter().copied()),
newly_pointed.iter().map(|(_, _, dbname)| *dbname),
)
.await?;
let governed_elsewhere: Vec<String> = if newly_pointed.is_empty() {
let governed_elsewhere: Vec<(String, String)> = if newly_pointed.is_empty() {
vec![]
} else {
sqlx::query_scalar(
"SELECT DISTINCT dt.value->'database'->>'resource_path' FROM workspace_settings ws
sqlx::query_as(
"SELECT DISTINCT dt.value->'database'->>'resource_type',
dt.value->'database'->>'resource_path'
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(COALESCE(ws.datatable->'datatables', '{}'::jsonb)) dt
WHERE ws.workspace_id <> $1 AND dt.value ? 'permissions'
AND dt.value->'database'->>'resource_type' = 'instance'",
AND dt.value->'database'->>'resource_type' IN ('instance', 'external_instance')",
)
.bind(&w_id)
.fetch_all(&mut *tx)
.await?
};
for (name, dbname) in newly_pointed {
for (name, kind, dbname) in newly_pointed {
let governed_here = old_datatables.values().any(|old| {
old.permissions.is_some()
&& old.database.as_ref().is_some_and(|d| {
d.resource_type == DataTableCatalogResourceType::Instance
&& d.resource_path == dbname
})
&& old
.database
.as_ref()
.is_some_and(|d| d.resource_type == kind && d.resource_path == dbname)
});
if governed_here || governed_elsewhere.iter().any(|g| g == dbname) {
if governed_here
|| governed_elsewhere
.iter()
.any(|(k, p)| k == kind.as_ref() && p == dbname)
{
return Err(Error::BadRequest(format!(
"Data table '{name}' would point at database '{dbname}', which a data table under \
roles uses, without carrying those roles: everyone reaching '{name}' would connect \
@@ -1438,19 +1438,7 @@ pub async fn drop_forked_datatable_databases(
// from gaining such a pointer before the drop.
let dropped = async {
let mut tx = db.begin().await?;
// The three locks a settings save takes, in its order: this workspace's data
// tables, its settings row, and the database itself. Without them a save could
// rename this entry, or point another one here, either side of the check below.
windmill_common::workspaces::lock_fork_datatables(&mut tx, &w_id).await?;
sqlx::query("SELECT 1 FROM workspace_settings WHERE workspace_id = $1 FOR UPDATE")
.bind(&w_id)
.fetch_optional(&mut *tx)
.await?;
windmill_common::datatable_roles::lock_instance_databases_governance(
&mut tx,
[db_to_drop.as_str()],
)
.await?;
if database.resource_type
== windmill_common::workspaces::DataTableCatalogResourceType::ExternalInstance
{
@@ -1476,16 +1464,6 @@ pub async fn drop_forked_datatable_databases(
}
windmill_common::drop_custom_instance_database(&db, db_to_drop).await?;
}
// The entry goes with the database: a fork this one is cloned into afterwards must
// not inherit a pointer at a data table whose database is gone.
sqlx::query(
"UPDATE workspace_settings SET datatable = datatable #- ARRAY['datatables', $2]
WHERE workspace_id = $1",
)
.bind(&w_id)
.bind(dt_name)
.execute(&mut *tx)
.await?;
tx.commit().await?;
Ok::<_, Error>(())
}
@@ -1857,7 +1835,17 @@ async fn resolve_fork_catalog_pg(
"ducklake://{ducklake_name}: malformed registry catalog identity `{catalog}`"
))
})?;
let catalog_resource = if resource_type == "instance" {
let catalog_resource = if resource_type == "external_instance" {
serde_json::to_value(
windmill_common::external_instance_pg::external_instance_connection_unchecked(
db,
resource_path,
false,
)
.await?,
)
.map_err(|e| Error::internal_err(format!("serializing pg creds: {e}")))?
} else if resource_type == "instance" {
let mut pg_creds = windmill_common::PgDatabase::parse_uri(
&windmill_common::get_database_url().await?.as_str().await,
)?;
+54 -112
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.814.0
version: 1.813.0
title: Windmill API
contact:
@@ -1611,7 +1611,7 @@ paths:
/settings/external_instance_pg/databases:
get:
summary: Lists the databases Windmill created on the external instance cluster, with the workspaces whose data tables use each
summary: Lists the databases Windmill created on the external instance cluster, with the workspaces whose data tables, Ducklake catalogs or pending fork cleanups use each
operationId: listExternalInstancePgDatabases
tags:
- setting
@@ -1653,7 +1653,7 @@ paths:
application/json:
schema: {}
delete:
summary: Drops a database Windmill created on the external instance cluster, refused while a data table uses it
summary: Drops a database Windmill created on the external instance cluster, refused while a data table, Ducklake catalog or pending fork cleanup uses it
operationId: dropExternalInstancePgDatabase
tags:
- setting
@@ -1688,13 +1688,19 @@ paths:
/settings/datatable_roles:
get:
summary: list the instance's data table roles
summary: list the data table roles of one Windmill-managed Postgres cluster
operationId: listInstanceDatatableRoles
tags:
- setting
parameters:
- in: query
name: cluster
required: false
schema:
$ref: "#/components/schemas/DatatableRoleCluster"
responses:
"200":
description: the instance role catalog
description: the cluster's role catalog
content:
application/json:
schema:
@@ -1702,7 +1708,7 @@ paths:
items:
$ref: "#/components/schemas/InstanceDatatableRole"
post:
summary: create a data table role on the instance's Postgres cluster
summary: create a data table role on a Windmill-managed Postgres cluster
operationId: createInstanceDatatableRole
tags:
- setting
@@ -1716,6 +1722,8 @@ paths:
properties:
name:
type: string
cluster:
$ref: "#/components/schemas/DatatableRoleCluster"
responses:
"200":
description: the created role
@@ -11665,10 +11673,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: script args
@@ -11705,10 +11714,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: script args
@@ -11745,10 +11755,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
responses:
"200":
@@ -11771,10 +11782,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
- name: poll_delay_ms
description: delay between polling for job updates in milliseconds
in: query
@@ -11812,10 +11824,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
- name: poll_delay_ms
description: delay between polling for job updates in milliseconds
in: query
@@ -11851,10 +11864,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
- name: poll_delay_ms
description: delay between polling for job updates in milliseconds
in: query
@@ -11898,10 +11912,11 @@ paths:
- $ref: "#/components/parameters/NewJobId"
- $ref: "#/components/parameters/SkipPreprocessor"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
- name: poll_delay_ms
description: delay between polling for job updates in milliseconds
in: query
@@ -12855,15 +12870,6 @@ paths:
in: query
schema:
type: string
- name: kind
description: which conversations to list - the flow editor's test chats, the deployed flow's own (the default), or both
in: query
schema:
type: string
enum:
- test
- deployed
- all
responses:
"200":
description: flow conversations list
@@ -12874,40 +12880,6 @@ paths:
items:
$ref: "#/components/schemas/FlowConversation"
/w/{workspace}/flow_conversations/update/{conversation_id}:
post:
summary: rename flow conversation
operationId: updateFlowConversation
tags:
- flow_conversations
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: conversation_id
description: conversation id
in: path
required: true
schema:
type: string
format: uuid
requestBody:
required: true
content:
application/json:
schema:
type: object
required: [title]
properties:
title:
type: string
description: the chat's name
responses:
"200":
description: flow conversation updated
content:
text/plain:
schema:
type: string
/w/{workspace}/flow_conversations/delete/{conversation_id}:
delete:
summary: delete flow conversation
@@ -15302,10 +15274,11 @@ paths:
schema:
type: boolean
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: flow args
required: true
@@ -15359,10 +15332,11 @@ paths:
schema:
type: boolean
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: flow args
required: true
@@ -15850,10 +15824,11 @@ paths:
type: boolean
- $ref: "#/components/parameters/NewJobId"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: preview
@@ -15881,10 +15856,11 @@ paths:
parameters:
- $ref: "#/components/parameters/WorkspaceId"
- name: memory_id
description: Memory id for the flow's AI agent steps. A uuid is used as is; any other string is hashed within the workspace and flow, so the same string always names the same memory of that flow.
description: memory ID for chat-enabled flows
in: query
schema:
type: string
format: uuid
requestBody:
description: preview
@@ -28731,7 +28707,7 @@ components:
FlowConversation:
type: object
required:
[id, workspace_id, flow_path, created_at, updated_at, created_by, is_test]
[id, workspace_id, flow_path, created_at, updated_at, created_by]
properties:
id:
type: string
@@ -28758,9 +28734,6 @@ components:
created_by:
type: string
description: Username who created the conversation
is_test:
type: boolean
description: Started from the flow editor's test panel rather than a deployed run
FlowConversationMessage:
type: object
@@ -28800,50 +28773,6 @@ components:
success:
type: boolean
description: Whether the message is a success
tool_arguments:
type: string
nullable: true
description: >-
On a tool row, the arguments the model wrote for the call. For a script, flow or
AI agent tool these exclude the inputs its step wires in, which only the tool's
job holds. Null for a provider-native web search, whose query the provider does
not return.
tool_result:
type: string
nullable: true
description: >-
On a tool row, the text the model got back from the call, or what the call
failed with — the row's own text names the tool rather than the reason. For a
provider-native web search, its citations.
reasoning:
type: string
nullable: true
description: >-
On an answer, the thinking that produced it; on a tool row, the thinking that
led to the call. Each round's thinking is on one row. The agent job's result
keeps the turn's thinking as a single string.
attachments:
type: array
nullable: true
description: >-
The files a user message carried, as object-storage references: every flow
input other than user_message that held one or a list of them, at most 20. Never
file bytes or a presigned URL.
items:
type: object
required: [input, s3]
properties:
input:
type: string
description: The flow input that held the file
s3:
type: string
description: The file's key in object storage
storage:
type: string
description: The secondary storage holding the file, absent for the primary one
filename:
type: string
EndpointTool:
type: object
@@ -33836,9 +33765,17 @@ components:
last_setup:
$ref: "#/components/schemas/ExternalInstancePgSetupReport"
DatatableRoleCluster:
type: string
description: >-
The Windmill-managed Postgres cluster a data table role is a login on: Windmill's own
(behind `instance` data tables) or the external instance cluster (behind
`external_instance` ones). Defaults to `instance`.
enum: [instance, external_instance]
InstanceDatatableRole:
type: object
required: [id, name, enabled]
required: [id, name, enabled, cluster]
properties:
id:
type: string
@@ -33846,6 +33783,8 @@ components:
type: string
enabled:
type: boolean
cluster:
$ref: "#/components/schemas/DatatableRoleCluster"
DatatableRoleTenants:
type: object
@@ -33867,8 +33806,10 @@ components:
supported:
type: boolean
description: >-
Whether this data table can be put under roles at all. Only one backed by the
instance database can: a role is a login on that cluster.
Whether this data table can be put under roles at all. Only one on a database Windmill
manages can: a role is a login on that database's cluster.
cluster:
$ref: "#/components/schemas/DatatableRoleCluster"
permissioned:
type: boolean
default_role:
@@ -34168,7 +34109,7 @@ components:
type: array
items:
type: string
description: Workspaces that reference this database through a ducklake catalog or a datatable database of the kind being listed — 'instance' for the instance databases endpoint, 'external_instance' for the external cluster one. Computed at request time, not persisted, and only returned to superadmins.
description: Workspaces that reference this database via a ducklake catalog or datatable database with resource_type 'instance'. Computed at request time, not persisted.
workspace_id:
type: string
description: The workspace a member created this database for as a fork copy. Only that workspace can import into it or point a fork at it.
@@ -36145,6 +36086,7 @@ components:
- postgresql
- mysql
- instance
- external_instance
resource_path:
type: string
required:
+6 -7
View File
@@ -4310,11 +4310,11 @@ async fn execute_component(
}
}
let flow_path = payload
let is_flow = payload
.path
.as_deref()
.and_then(|path| path.strip_prefix("flow/"))
.map(str::to_string);
.as_ref()
.map(|p| p.starts_with("flow/"))
.unwrap_or(false);
// Tag for inline-script jobs is read from the deployed policy in run mode;
// only preview mode (editor) honors the client-supplied tag. This applies to
@@ -4444,9 +4444,8 @@ async fn execute_component(
// Apply runnable query parameters if provided
if let Some(ref run_query) = payload.run_query_params {
if let Some(flow_path) = flow_path.as_deref() {
crate::jobs::process_flow_run_query_params(&mut tx, uuid, &w_id, flow_path, run_query)
.await?;
if is_flow {
crate::jobs::process_flow_run_query_params(&mut tx, uuid, run_query).await?;
}
}
+26 -51
View File
@@ -4369,12 +4369,12 @@ async fn count_completed_jobs_detail(
Query(query): Query<CountCompletedJobsQuery>,
) -> error::JsonResult<i64> {
let mut sqlb = SqlBuilder::select_from("v2_job_completed");
//FOR RLS
sqlb.join("v2_job USING (id)");
sqlb.field("COUNT(*) as count");
// Filtering on v2_job.workspace_id instead would keep the planner off
// ix_job_workspace_id_completed_at_all and scan the whole retention window.
if !(w_id == "admins" && query.all_workspaces.unwrap_or(false)) {
sqlb.and_where_eq("v2_job_completed.workspace_id", "?".bind(&w_id));
sqlb.and_where_eq("v2_job.workspace_id", "?".bind(&w_id));
}
if let Some(after_s_ago) = query.completed_after_s_ago {
@@ -4393,7 +4393,6 @@ async fn count_completed_jobs_detail(
}
if let Some(tags) = query.tags {
sqlb.join("v2_job USING (id)");
sqlb.and_where_in(
"v2_job.tag",
&tags.split(",").map(|t| quote(t)).collect::<Vec<_>>(),
@@ -4401,19 +4400,7 @@ async fn count_completed_jobs_detail(
}
let sql = sqlb.sql()?;
let mut tx = db.begin().await?;
set_list_jobs_statement_timeout(&mut tx).await?;
let stats = sqlx::query_scalar::<_, i64>(&sql)
.fetch_one(&mut *tx)
.await
.map_err(|e| {
list_jobs_timeout_error(
e,
"Counting completed jobs",
"Lower completed_after_s_ago or narrow the filters.",
)
})?;
tx.commit().await?;
let stats = sqlx::query_scalar::<_, i64>(&sql).fetch_one(&db).await?;
Ok(Json(stats))
}
@@ -4442,33 +4429,6 @@ lazy_static::lazy_static! {
.unwrap_or(30);
}
/// A client that gives up does not cancel its query, so without this bound every retry of a
/// slow filter stacks another scan running until the connection-wide 5min timeout.
async fn set_list_jobs_statement_timeout(tx: &mut Transaction<'_, Postgres>) -> error::Result<()> {
let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
if timeout_secs > 0 {
sqlx::query(&format!("SET LOCAL statement_timeout = '{timeout_secs}s'"))
.execute(&mut **tx)
.await?;
}
Ok(())
}
fn list_jobs_timeout_error(e: sqlx::Error, action: &str, hint: &str) -> Error {
let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
match e {
sqlx::Error::Database(ref db_err)
if timeout_secs > 0 && db_err.code().as_deref() == Some("57014") =>
{
Error::Generic(
StatusCode::BAD_REQUEST,
format!("{action} took more than {timeout_secs}s and was stopped. {hint}"),
)
}
e => e.into(),
}
}
async fn list_jobs(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
@@ -4585,14 +4545,32 @@ async fn list_jobs(
};
// tracing::info!("sql: {}", &sql);
let mut tx: Transaction<'_, Postgres> = user_db.begin(&authed).await?;
set_list_jobs_statement_timeout(&mut tx).await?;
// A client that gives up does not cancel its query, so without this bound every retry of a
// slow filter stacks another scan running until the connection-wide 5min timeout.
let timeout_secs = *LIST_JOBS_STATEMENT_TIMEOUT_SECS;
if timeout_secs > 0 {
sqlx::query(&format!("SET LOCAL statement_timeout = '{timeout_secs}s'"))
.execute(&mut *tx)
.await?;
}
let jobs: Vec<UnifiedJob> = sqlx::query_as(&sql)
.fetch_all(&mut *tx)
.warn_after_seconds_with_sql(5, format!("list_jobs: {}", sql))
.await
.map_err(|e| {
list_jobs_timeout_error(e, "Listing jobs", "Set a start date or narrow the filters.")
.map_err(|e| match e {
sqlx::Error::Database(ref db_err)
if timeout_secs > 0 && db_err.code().as_deref() == Some("57014") =>
{
Error::Generic(
StatusCode::BAD_REQUEST,
format!(
"Listing jobs took more than {timeout_secs}s and was stopped. Set a start date or narrow the filters."
),
)
}
e => e.into(),
})?;
tx.commit().await?;
@@ -9568,7 +9546,7 @@ async fn run_preview_flow_job(
.await?;
// Set memory_id if provided (for agent memory)
if let Some(memory_id) = run_query.memory_key(&w_id, &flow_path) {
if let Some(memory_id) = run_query.memory_id {
set_flow_memory_id(&mut tx, uuid, memory_id).await?;
}
@@ -9582,9 +9560,6 @@ async fn run_preview_flow_job(
&run_query,
user_message.as_ref(),
uuid,
// Run from the editor's test panel: a trial, not a real conversation.
true,
&flow_args,
)
.await?;
}
-1
View File
@@ -33,7 +33,6 @@ path = "src/lib.rs"
tar.workspace = true
hmac.workspace = true
sha2.workspace = true
sha1.workspace = true
thiserror.workspace = true
anyhow.workspace = true
serde.workspace = true
+127 -44
View File
@@ -6,22 +6,67 @@
* LICENSE-AGPL for a copy of the license.
*/
//! The instance's data table role catalog.
//! The instance's data table role catalogs.
//!
//! A data table role is a real Postgres login role on the Windmill cluster, named exactly as the
//! user named it, shared by every instance database. Windmill decides who may ask for a role (the
//! per-data-table tenant lists in [`crate::workspaces`]); Postgres decides what the role may then
//! touch. The catalog here is only the first half's vocabulary plus the cluster provisioning.
//! A data table role is a real Postgres login role on one cluster — Windmill's own, or the external
//! instance cluster — named exactly as the user named it, shared by every database Windmill manages
//! on that cluster. Each cluster has its own catalog: a role exists where it was created and nowhere
//! else. Windmill decides who may ask for a role (the per-data-table tenant lists in
//! [`crate::workspaces`]); Postgres decides what the role may then touch. The catalog here is only
//! the first half's vocabulary plus the cluster provisioning.
//!
//! Entries are keyed by a generated id so a rename moves nothing else: tenants name the id.
use std::collections::BTreeMap;
use serde::{Deserialize, Serialize};
use crate::{
error::{Error, Result},
workspaces::DataTableCatalogResourceType,
DB,
};
/// The cluster a role catalog belongs to.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub enum DatatableRoleCluster {
/// Windmill's own Postgres, behind `instance` data tables.
#[default]
Instance,
/// The external instance cluster, behind `external_instance` data tables.
ExternalInstance,
}
impl DatatableRoleCluster {
pub fn as_str(self) -> &'static str {
match self {
Self::Instance => "instance",
Self::ExternalInstance => "external_instance",
}
}
pub fn parse(value: &str) -> Result<Self> {
match value {
"instance" => Ok(Self::Instance),
"external_instance" => Ok(Self::ExternalInstance),
other => Err(Error::BadRequest(format!(
"Unknown data table role cluster '{other}': expected instance or external_instance"
))),
}
}
/// The cluster whose roles a data table on `kind` can use. `None` for a resource-backed one,
/// which is never under roles.
pub fn of(kind: DataTableCatalogResourceType) -> Option<Self> {
match kind {
DataTableCatalogResourceType::Instance => Some(Self::Instance),
DataTableCatalogResourceType::ExternalInstance => Some(Self::ExternalInstance),
DataTableCatalogResourceType::Postgresql => None,
}
}
}
/// The connection every data table resolved to before roles existed (`custom_instance_user`). It
/// owns every pre-existing object, so it is a reserved name rather than a catalog entry: never
/// created, renamed or dropped.
@@ -164,28 +209,42 @@ pub async fn lock_instance_databases_governance<'a>(
/// need the names — but callers MUST NOT let `pwd` reach a response, a log line, an audit record
/// or an export. Nothing about who may call it: the credential is the whole risk, and `Debug` is
/// hand-written to redact it for the same reason.
pub async fn read_role_catalog(db: &DB) -> Result<DatatableRoleCatalog> {
crate::datatable_roles_oss::read_role_catalog(db).await
pub async fn read_role_catalog(
db: &DB,
cluster: DatatableRoleCluster,
) -> Result<DatatableRoleCatalog> {
crate::datatable_roles_oss::read_role_catalog(db, cluster).await
}
/// As [`read_role_catalog`], reading inside the caller's transaction so the value is the one
/// [`lock_role_catalog`] is protecting. Same disclosure contract.
pub async fn read_role_catalog_tx(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
cluster: DatatableRoleCluster,
) -> Result<DatatableRoleCatalog> {
crate::datatable_roles_oss::read_role_catalog_tx(tx).await
crate::datatable_roles_oss::read_role_catalog_tx(tx, cluster).await
}
/// Record a role, in the caller's transaction so it commits with the `CREATE ROLE` it describes.
/// The cluster a role belongs to, or `None` if no role has this id.
pub async fn role_cluster(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
id: &str,
) -> Result<Option<DatatableRoleCluster>> {
crate::datatable_roles_oss::role_cluster(tx, id).await
}
/// Record a role, in the caller's transaction. On Windmill's own cluster that commits it with the
/// `CREATE ROLE` it describes; on the external cluster the role already exists by then.
///
/// Authorization: writes a generated Postgres credential. Callers MUST restrict this to superadmin
/// paths and MUST hold [`lock_role_catalog`] on `tx`.
pub async fn insert_role_catalog_entry(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
id: &str,
cluster: DatatableRoleCluster,
role: &InstanceDatatableRole,
) -> Result<()> {
crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, role).await
crate::datatable_roles_oss::insert_role_catalog_entry(tx, id, cluster, role).await
}
/// Update a role's recorded name, login flag and password. Same contract as
@@ -215,7 +274,7 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
.find(|(_, role)| role.name == name)
.ok_or_else(|| {
Error::NotFound(format!(
"'{name}' is not a data table role of this instance. Defined roles: {}.",
"'{name}' is not a data table role of this database's cluster. Defined roles: {}.",
catalog
.values()
.map(|r| r.name.as_str())
@@ -231,70 +290,90 @@ pub fn role_id_by_name<'a>(catalog: &'a DatatableRoleCatalog, name: &str) -> Res
Ok(entry.0.as_str())
}
/// Every instance database the registry knows about. Role provisioning has to reach all of them:
/// a role that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
/// Every database Windmill manages on `cluster`. Role provisioning has to reach all of them: a role
/// that cannot `CONNECT` to a database is refused by Postgres before any grant matters.
///
/// Authorization: checks nothing, and names every instance database across all workspaces. Callers
/// Authorization: checks nothing, and names every managed database across all workspaces. Callers
/// MUST be superadmin-gated or keep the names server-side; never return them to a workspace caller.
pub async fn registered_instance_databases(db: &DB) -> Result<Vec<String>> {
crate::datatable_roles_oss::registered_instance_databases(db).await
pub async fn registered_instance_databases(
db: &DB,
cluster: DatatableRoleCluster,
) -> Result<Vec<String>> {
crate::datatable_roles_oss::registered_instance_databases(db, cluster).await
}
/// `CONNECT` on `dbname` for every enabled role, and none for `PUBLIC`. Run at role creation, at
/// database creation, and lazily whenever an instance data table is administered, so a database
/// provisioned before a role existed is repaired rather than left silently unreachable.
/// `CONNECT` on `dbname` for every enabled role of `cluster`, and none for `PUBLIC`. Run at role
/// creation, at database creation, and lazily whenever a managed data table is administered, so a
/// database provisioned before a role existed is repaired rather than left silently unreachable.
///
/// Authorization: rewrites a database's ACL with the server's own credentials and checks nothing.
/// Callers MUST have authorized administration of `dbname` — superadmin, or an admin of the
/// workspace governing a data table on it.
pub async fn converge_connect_grants(db: &DB, dbname: &str) -> Result<()> {
crate::datatable_roles_oss::converge_connect_grants(db, dbname).await
pub async fn converge_connect_grants(
db: &DB,
cluster: DatatableRoleCluster,
dbname: &str,
) -> Result<()> {
crate::datatable_roles_oss::converge_connect_grants(db, cluster, dbname).await
}
/// As [`converge_connect_grants`], with a catalog the caller already read. Same contract.
/// As [`converge_connect_grants`], with the catalog of `cluster` the caller already read. Same
/// contract.
pub async fn converge_connect_grants_with(
db: &DB,
cluster: DatatableRoleCluster,
dbname: &str,
catalog: &DatatableRoleCatalog,
) -> Result<()> {
crate::datatable_roles_oss::converge_connect_grants_with(db, dbname, catalog).await
crate::datatable_roles_oss::converge_connect_grants_with(db, cluster, dbname, catalog).await
}
/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user`, and `CONNECT` on
/// every registered database. No privileges beyond that — an admin grants them through SQL or the
/// ACL editor.
/// `CREATE ROLE <name> LOGIN PASSWORD ...; GRANT <name> TO custom_instance_user` on `cluster`. No
/// privileges beyond that — an admin grants them through SQL or the ACL editor.
///
/// On Windmill's own cluster the DDL runs on `tx`, so it commits with the catalog row. The external
/// cluster is another server: the role is created there before `tx` commits, and callers MUST drop
/// it again ([`drop_datatable_role`]) if `tx` then fails to commit.
///
/// Authorization: creates a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn create_instance_role(
/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
pub async fn create_datatable_role(
db: &DB,
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
cluster: DatatableRoleCluster,
name: &str,
password: &str,
) -> Result<()> {
crate::datatable_roles_oss::create_instance_role(tx, name, password).await
crate::datatable_roles_oss::create_datatable_role(db, tx, cluster, name, password).await
}
/// Authorization: alters a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn set_instance_role_login(
/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
pub async fn set_datatable_role_login(
db: &DB,
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
cluster: DatatableRoleCluster,
name: &str,
enabled: bool,
) -> Result<()> {
crate::datatable_roles_oss::set_instance_role_login(tx, name, enabled).await
crate::datatable_roles_oss::set_datatable_role_login(db, tx, cluster, name, enabled).await
}
/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one.
/// A rename discards an md5-hashed password, so the caller has to hand over a fresh one. On the
/// external cluster the rename lands before `tx` commits, and callers MUST rename it back if `tx`
/// then fails to commit.
///
/// Authorization: renames a cluster-wide Postgres login. Callers MUST restrict this to superadmin
/// paths, and MUST hold [`lock_role_catalog`] on the same transaction.
pub async fn rename_instance_role(
/// paths, and MUST hold [`lock_role_catalog`] on `tx`.
pub async fn rename_datatable_role(
db: &DB,
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
cluster: DatatableRoleCluster,
from: &str,
to: &str,
password: &str,
) -> Result<()> {
crate::datatable_roles_oss::rename_instance_role(tx, from, to, password).await
crate::datatable_roles_oss::rename_datatable_role(db, tx, cluster, from, to, password).await
}
/// A role owning anything in any database blocks its own `DROP ROLE`, and both its objects and the
@@ -302,8 +381,9 @@ pub async fn rename_instance_role(
/// registry. An unreachable database aborts the whole delete: dropping the role while one database
/// still holds objects owned by it leaves those objects owned by a numeric OID nobody can name.
///
/// Each pass runs as the instance's own Postgres user rather than `custom_instance_user`, which
/// owns the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
/// Each pass runs as the cluster's administrator rather than `custom_instance_user`: on Windmill's
/// own cluster the instance's Postgres user, on the external one its configured admin login. Both
/// own the databases and can therefore revoke a grant whoever made it. `custom_instance_user`
/// could only undo what it granted itself, so a privilege planted by an operator in psql — the
/// ordinary way privileges reach a role — would survive and block the drop.
///
@@ -311,16 +391,19 @@ pub async fn rename_instance_role(
/// MUST restrict this to superadmin paths, and MUST hold [`lock_role_catalog`] on `tx`.
///
/// The per-database passes open their own connections and cannot join `tx`; the lock is what keeps
/// a concurrent mutation out while they run. Only the final `DROP ROLE` is on `tx`, so it commits
/// or rolls back with the catalog write that forgets the role. Those passes commit as they go, so
/// callers MUST have disabled the role in an earlier committed transaction: a failure part-way
/// then leaves a disabled role to retry, not an enabled one already stripped in some databases.
pub async fn drop_instance_role(
/// a concurrent mutation out while they run. On Windmill's own cluster only the final `DROP ROLE`
/// is on `tx`, so it commits or rolls back with the catalog write that forgets the role; on the
/// external cluster it runs there, and tolerates a role already gone so a retry after a failed
/// commit can finish. The passes commit as they go, so callers MUST have disabled the role in an
/// earlier committed transaction: a failure part-way then leaves a disabled role to retry, not an
/// enabled one already stripped in some databases.
pub async fn drop_datatable_role(
db: &DB,
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
cluster: DatatableRoleCluster,
name: &str,
) -> Result<()> {
crate::datatable_roles_oss::drop_instance_role(db, tx, name).await
crate::datatable_roles_oss::drop_datatable_role(db, tx, cluster, name).await
}
#[cfg(test)]
@@ -24,12 +24,12 @@ pub fn datatable_roles_unavailable() -> Error {
#[cfg(all(feature = "private", feature = "enterprise"))]
pub(crate) use crate::datatable_roles_ee::{
can_use_datatable_role, can_use_datatable_role_in_governing_workspace, converge_connect_grants,
converge_connect_grants_with, create_instance_role, delete_role_catalog_entry,
drop_instance_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
converge_connect_grants_with, create_datatable_role, delete_role_catalog_entry,
drop_datatable_role, ensure_can_use_datatable_role, ensure_datatable_admin_access,
ensure_instance_db_grant_options_unchecked, forget_datatable_role_everywhere,
insert_role_catalog_entry, read_role_catalog, read_role_catalog_tx,
registered_instance_databases, rename_instance_role, resolve_datatable_role_connection,
set_instance_role_login, update_role_catalog_entry,
registered_instance_databases, rename_datatable_role, resolve_datatable_role_connection,
role_cluster, set_datatable_role_login, update_role_catalog_entry,
};
#[cfg(not(all(feature = "private", feature = "enterprise")))]
@@ -39,7 +39,7 @@ pub(crate) use ce::*;
mod ce {
use super::datatable_roles_unavailable as unavailable;
use crate::{
datatable_roles::{DatatableRoleCatalog, InstanceDatatableRole},
datatable_roles::{DatatableRoleCatalog, DatatableRoleCluster, InstanceDatatableRole},
db::AuthedRef,
error::Result,
workspaces::{
@@ -50,17 +50,31 @@ mod ce {
type Tx<'a> = sqlx::Transaction<'a, sqlx::Postgres>;
pub(crate) async fn read_role_catalog(_db: &DB) -> Result<DatatableRoleCatalog> {
pub(crate) async fn read_role_catalog(
_db: &DB,
_cluster: DatatableRoleCluster,
) -> Result<DatatableRoleCatalog> {
Err(unavailable())
}
pub(crate) async fn read_role_catalog_tx(_tx: &mut Tx<'_>) -> Result<DatatableRoleCatalog> {
pub(crate) async fn read_role_catalog_tx(
_tx: &mut Tx<'_>,
_cluster: DatatableRoleCluster,
) -> Result<DatatableRoleCatalog> {
Err(unavailable())
}
pub(crate) async fn role_cluster(
_tx: &mut Tx<'_>,
_id: &str,
) -> Result<Option<DatatableRoleCluster>> {
Err(unavailable())
}
pub(crate) async fn insert_role_catalog_entry(
_tx: &mut Tx<'_>,
_id: &str,
_cluster: DatatableRoleCluster,
_role: &InstanceDatatableRole,
) -> Result<()> {
Err(unavailable())
@@ -78,43 +92,57 @@ mod ce {
Err(unavailable())
}
pub(crate) async fn registered_instance_databases(_db: &DB) -> Result<Vec<String>> {
pub(crate) async fn registered_instance_databases(
_db: &DB,
_cluster: DatatableRoleCluster,
) -> Result<Vec<String>> {
Err(unavailable())
}
/// Nothing to converge: with no roles to admit, an instance database keeps the `CONNECT`
/// grants it was created with, `PUBLIC`'s included, as it did before roles existed.
pub(crate) async fn converge_connect_grants(_db: &DB, _dbname: &str) -> Result<()> {
/// Nothing to converge: with no roles to admit, a managed database keeps the `CONNECT` grants
/// it was created with, as it did before roles existed.
pub(crate) async fn converge_connect_grants(
_db: &DB,
_cluster: DatatableRoleCluster,
_dbname: &str,
) -> Result<()> {
Ok(())
}
/// As [`converge_connect_grants`].
pub(crate) async fn converge_connect_grants_with(
_db: &DB,
_cluster: DatatableRoleCluster,
_dbname: &str,
_catalog: &DatatableRoleCatalog,
) -> Result<()> {
Ok(())
}
pub(crate) async fn create_instance_role(
pub(crate) async fn create_datatable_role(
_db: &DB,
_tx: &mut Tx<'_>,
_cluster: DatatableRoleCluster,
_name: &str,
_password: &str,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn set_instance_role_login(
pub(crate) async fn set_datatable_role_login(
_db: &DB,
_tx: &mut Tx<'_>,
_cluster: DatatableRoleCluster,
_name: &str,
_enabled: bool,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn rename_instance_role(
pub(crate) async fn rename_datatable_role(
_db: &DB,
_tx: &mut Tx<'_>,
_cluster: DatatableRoleCluster,
_from: &str,
_to: &str,
_password: &str,
@@ -122,12 +150,18 @@ mod ce {
Err(unavailable())
}
pub(crate) async fn drop_instance_role(_db: &DB, _tx: &mut Tx<'_>, _name: &str) -> Result<()> {
pub(crate) async fn drop_datatable_role(
_db: &DB,
_tx: &mut Tx<'_>,
_cluster: DatatableRoleCluster,
_name: &str,
) -> Result<()> {
Err(unavailable())
}
pub(crate) async fn ensure_instance_db_grant_options_unchecked(
_db: &DB,
_cluster: DatatableRoleCluster,
_dbname: &str,
) -> Result<()> {
Err(unavailable())
@@ -148,7 +148,10 @@ pub async fn external_instance_databases(db: &DB) -> Result<BTreeMap<String, Cus
Ok(read_external_instance_pg_state(db).await?.databases)
}
/// The workspaces whose data tables name each database on the external cluster.
/// The workspaces whose data tables or Ducklake catalogs name each database on the external cluster,
/// and the forks whose Ducklake metadata schemas there are still waiting to be dropped: those rows
/// outlive a settings change, and cleanup cannot drop a schema in a database that is gone. A row
/// whose schema is already dropped only waits on object storage, which needs no database.
///
/// Authorization: reads every workspace's settings and checks nothing. Callers MUST be superadmin
/// or an internal lifecycle path.
@@ -164,7 +167,21 @@ pub async fn external_instance_database_usages<'c>(
ELSE '{}'::jsonb END
) AS dt(k, entry)
WHERE entry->'database'->>'resource_type' = 'external_instance'
AND entry->'database'->>'resource_path' IS NOT NULL",
AND entry->'database'->>'resource_path' IS NOT NULL
UNION ALL
SELECT ws.workspace_id, entry->'catalog'->>'resource_path'
FROM workspace_settings ws
CROSS JOIN LATERAL jsonb_each(
CASE WHEN jsonb_typeof(ws.ducklake->'ducklakes') = 'object'
THEN ws.ducklake->'ducklakes'
ELSE '{}'::jsonb END
) AS dl(k, entry)
WHERE entry->'catalog'->>'resource_type' = 'external_instance'
AND entry->'catalog'->>'resource_path' IS NOT NULL
UNION ALL
SELECT workspace_id, substring(catalog FROM length('external_instance:') + 1)
FROM fork_ducklake_namespace
WHERE catalog LIKE 'external\\_instance:%' AND NOT schema_dropped",
)
.fetch_all(db)
.await?;
@@ -175,27 +192,48 @@ pub async fn external_instance_database_usages<'c>(
Ok(usages)
}
/// Refuse to unset the cluster while Windmill still has databases on it, or a workspace still
/// points at one: every data table there would stop resolving. Allowed on every edition, so a
/// Refuse to unset the cluster while Windmill still has databases or data table roles on it, or a
/// workspace still points at one: every data table there would stop resolving, and every role
/// would be a login nothing can drop any more. Allowed on every edition, so a
/// downgraded instance can still clear a setting it no longer uses.
pub async fn ensure_external_instance_pg_removable(conn: &mut sqlx::PgConnection) -> Result<()> {
let state = read_external_instance_pg_state(&mut *conn).await?;
let usages = external_instance_database_usages(&mut *conn).await?;
if state.databases.is_empty() && usages.is_empty() {
pub async fn ensure_external_instance_pg_removable(db: &DB) -> Result<()> {
ensure_external_instance_pg_unused(db, &format!("removing {EXTERNAL_INSTANCE_PG_SETTING}"))
.await
}
/// Refuse while Windmill has databases or data table roles on the cluster, or a workspace points
/// at one of its databases. `before` finishes the sentence saying what to do first.
async fn ensure_external_instance_pg_unused(db: &DB, before: &str) -> Result<()> {
let state = read_external_instance_pg_state(db).await?;
let usages = external_instance_database_usages(db).await?;
let roles = sqlx::query_scalar::<_, String>(
"SELECT name FROM datatable_role WHERE cluster = 'external_instance' ORDER BY name",
)
.fetch_all(db)
.await?;
if state.databases.is_empty() && usages.is_empty() && roles.is_empty() {
return Ok(());
}
let names = state
.databases
.keys()
.chain(usages.keys())
.collect::<BTreeSet<_>>()
.into_iter()
.cloned()
.collect::<Vec<_>>()
.join(", ");
let mut held = vec![];
if !(state.databases.is_empty() && usages.is_empty()) {
let names = state
.databases
.keys()
.chain(usages.keys())
.collect::<BTreeSet<_>>()
.into_iter()
.cloned()
.collect::<Vec<_>>()
.join(", ");
held.push(format!("databases in use ({names})"));
}
if !roles.is_empty() {
held.push(format!("data table roles ({})", roles.join(", ")));
}
Err(Error::BadRequest(format!(
"The external instance cluster still holds databases in use ({names}). Drop them and \
repoint the data tables using them before removing {EXTERNAL_INSTANCE_PG_SETTING}."
"The external instance cluster still holds {}. Drop them and repoint the data tables and \
Ducklake catalogs using them before {before}.",
held.join(" and ")
)))
}
@@ -270,9 +308,6 @@ pub async fn lock_external_instance_pg_state(
/// Refuse a data table naming `dbname` unless Windmill created it on the external cluster. Takes
/// the lock drops take, so none can remove the database before `tx`, which saves the data table,
/// commits.
///
/// Authorization: its refusal says whether Windmill created a database of that name, which is
/// instance-wide knowledge. Callers MUST have authorized the caller as superadmin first.
pub async fn ensure_external_instance_database_registered(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
dbname: &str,
@@ -312,12 +347,9 @@ pub async fn write_external_instance_pg_setting(
};
let mut tx = db.begin().await?;
lock_external_instance_pg_state(&mut tx).await?;
// Every check runs on this transaction's own connection: it holds the advisory lock, and
// taking a second connection from the pool while other writers queue on that lock is how a
// small pool deadlocks.
match value {
None => {
ensure_external_instance_pg_removable(&mut tx).await?;
ensure_external_instance_pg_removable(db).await?;
sqlx::query("DELETE FROM global_settings WHERE name = $1")
.bind(EXTERNAL_INSTANCE_PG_SETTING)
.execute(&mut *tx)
@@ -325,7 +357,7 @@ pub async fn write_external_instance_pg_setting(
}
Some(value) => {
crate::external_instance_pg_oss::validate_external_instance_pg_setting(value)?;
ensure_external_instance_pg_not_repointed(&mut tx, value).await?;
ensure_external_instance_pg_not_repointed(db, value).await?;
sqlx::query(
"INSERT INTO global_settings (name, value) VALUES ($1, $2)
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value, updated_at = now()",
@@ -367,14 +399,15 @@ pub async fn write_external_instance_pg_from_diff(
Ok(())
}
/// Refuse pointing the setting at another host or port while databases live on the current one.
/// Data tables name databases, not clusters, so they would silently resolve to whatever the new
/// cluster holds under the same names. Other fields (admin login, sslmode) may change freely.
/// Refuse pointing the setting at another host or port while databases or data table roles live on
/// the current one. Data tables name databases, and the role catalog names logins, not clusters, so
/// both would silently resolve to whatever the new cluster holds under the same names. Other fields
/// (admin login, sslmode) may change freely.
async fn ensure_external_instance_pg_not_repointed(
conn: &mut sqlx::PgConnection,
db: &DB,
value: &serde_json::Value,
) -> Result<()> {
let Some(current) = read_external_instance_pg_config(&mut *conn).await? else {
let Some(current) = read_external_instance_pg_config(db).await? else {
return Ok(());
};
let Ok(desired) = serde_json::from_value::<ExternalInstancePg>(value.clone()) else {
@@ -383,17 +416,11 @@ async fn ensure_external_instance_pg_not_repointed(
if external_instance_pg_address(&current) == external_instance_pg_address(&desired) {
return Ok(());
}
let state = read_external_instance_pg_state(&mut *conn).await?;
let usages = external_instance_database_usages(&mut *conn).await?;
if state.databases.is_empty() && usages.is_empty() {
return Ok(());
}
Err(Error::BadRequest(format!(
"The external instance cluster at {}:{} still holds databases in use. Drop them and repoint \
what uses them before pointing {EXTERNAL_INSTANCE_PG_SETTING} at another cluster.",
current.host.trim(),
current.port.unwrap_or(5432)
)))
ensure_external_instance_pg_unused(
db,
&format!("pointing {EXTERNAL_INSTANCE_PG_SETTING} at another cluster"),
)
.await
}
/// Converge the external cluster on the configured login: check what it can do, create or update
+11 -191
View File
@@ -1,39 +1,12 @@
use std::collections::HashMap;
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use serde_json::value::RawValue;
use sqlx::{self, FromRow};
use uuid::Uuid;
use windmill_types::s3::S3Object;
use crate::db::DB;
use crate::error::Result;
use crate::utils::truncate_with_ellipsis;
/// Changing it detaches every memory stored under a string memory id.
const MEMORY_ID_NAMESPACE: Uuid = Uuid::from_u128(0x6f1c2d4e_8a3b_5c7d_9e0f_1a2b3c4d5e6f);
/// Memory is stored and carried in `flow_status.memory_id` as a uuid, which names the same memory
/// wherever it is passed, as a chat conversation id must. Any other string names a memory through a
/// name-based (v5) uuid scoped to its workspace and flow, so the same key in two flows or two
/// workspaces names two memories, and chat conversation ids stay unique across workspaces.
pub fn memory_key(workspace_id: &str, flow_path: &str, memory_id: &str) -> Uuid {
let memory_id = memory_id.trim();
Uuid::parse_str(memory_id).unwrap_or_else(|_| {
use sha1::{Digest, Sha1};
let mut hasher = Sha1::new();
hasher.update(MEMORY_ID_NAMESPACE.as_bytes());
for part in [workspace_id, flow_path, memory_id] {
hasher.update(part.as_bytes());
hasher.update([0u8]);
}
let mut bytes = [0u8; 16];
bytes.copy_from_slice(&hasher.finalize()[..16]);
uuid::Builder::from_sha1_bytes(bytes).into_uuid()
})
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, sqlx::Type)]
#[sqlx(type_name = "MESSAGE_TYPE", rename_all = "lowercase")]
#[serde(rename_all = "lowercase")]
@@ -53,12 +26,8 @@ pub struct FlowConversation {
pub created_at: DateTime<Utc>,
pub updated_at: DateTime<Utc>,
pub created_by: String,
/// Started from the flow editor's test panel rather than a deployed run.
pub is_test: bool,
}
/// `is_test` is written on insert. An existing conversation of the other kind refuses the
/// turn, so preview and deployed runs never share one.
pub async fn get_or_create_conversation_with_id(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
w_id: &str,
@@ -66,10 +35,9 @@ pub async fn get_or_create_conversation_with_id(
username: &str,
title: &str,
conversation_id: Uuid,
is_test: bool,
) -> Result<FlowConversation> {
if let Some(existing) = lock_conversation(tx, w_id, conversation_id).await? {
return same_kind(existing, is_test);
return Ok(existing);
}
// Truncate title to 25 characters max
@@ -79,16 +47,15 @@ pub async fn get_or_create_conversation_with_id(
// wins, the others wait on it, do nothing, and read the row it created.
let created = sqlx::query_as!(
FlowConversation,
"INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title, is_test)
VALUES ($1, $2, $3, $4, $5, $6)
"INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by, title)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (id) DO NOTHING
RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test",
RETURNING id, workspace_id, flow_path, title, created_at, updated_at, created_by",
conversation_id,
w_id,
flow_path,
username,
title,
is_test
title
)
.fetch_optional(&mut **tx)
.await?;
@@ -96,29 +63,13 @@ pub async fn get_or_create_conversation_with_id(
return Ok(conversation);
}
// The concurrent first turn that won the insert may have been of the other kind.
let existing = lock_conversation(tx, w_id, conversation_id)
lock_conversation(tx, w_id, conversation_id)
.await?
.ok_or_else(|| {
crate::error::Error::BadRequest(format!(
"conversation {conversation_id} belongs to another workspace"
))
})?;
same_kind(existing, is_test)
}
/// `memory_id` is the caller's to choose, so a preview run could name a deployed
/// conversation and the reverse. A conversation's kind is fixed at creation and nothing
/// would show the mixing afterwards, so the turn is refused before it starts.
fn same_kind(existing: FlowConversation, is_test: bool) -> Result<FlowConversation> {
if existing.is_test == is_test {
return Ok(existing);
}
Err(crate::error::Error::BadRequest(if existing.is_test {
"this conversation was started from the flow editor's test panel; start a new conversation to run the deployed flow".to_string()
} else {
"this conversation belongs to the deployed flow; start a new conversation to test from the flow editor".to_string()
}))
})
}
/// Locked, so a turn orders against retention collecting the conversation
@@ -132,7 +83,7 @@ async fn lock_conversation(
) -> Result<Option<FlowConversation>> {
Ok(sqlx::query_as!(
FlowConversation,
"SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by, is_test
"SELECT id, workspace_id, flow_path, title, created_at, updated_at, created_by
FROM flow_conversation
WHERE id = $1 AND workspace_id = $2
FOR UPDATE",
@@ -143,65 +94,6 @@ async fn lock_conversation(
.await?)
}
/// What a row carries beyond its text. A chat is rebuilt from its rows alone, without
/// reading jobs, so every tool row carries the model's call and what the model got back: a
/// Windmill tool's job holds the args its input transforms produced rather than the model's,
/// and an MCP tool's call sits among every call of the turn in the agent's job. That job's
/// `reasoning` is one string for the whole turn, where the rows keep it per iteration.
#[derive(Debug, Clone, Default)]
pub struct MessageExtras {
pub tool_arguments: Option<String>,
pub tool_result: Option<String>,
pub reasoning: Option<String>,
/// The files a user message carried; see `message_attachments`.
pub attachments: Vec<MessageAttachment>,
}
/// The most files a user message keeps references to; the rest are dropped.
pub const MAX_MESSAGE_ATTACHMENTS: usize = 20;
/// A file a user message carried, as the object-storage reference its run received.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct MessageAttachment {
/// The flow input that held it.
pub input: String,
pub s3: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub storage: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub filename: Option<String>,
}
/// The files a run's args carry for its user message: every top-level input other than
/// `user_message` whose value is an object-storage reference or a list of them, in input
/// name order, capped at `MAX_MESSAGE_ATTACHMENTS`. Only the reference is kept: `presigned`
/// grants access to the file, and any other value may be the file's bytes.
pub fn message_attachments(args: &HashMap<String, Box<RawValue>>) -> Vec<MessageAttachment> {
let mut inputs: Vec<_> = args
.iter()
.filter(|(name, _)| name.as_str() != "user_message")
.collect();
inputs.sort_by(|a, b| a.0.cmp(b.0));
inputs
.into_iter()
.flat_map(|(name, value)| {
serde_json::from_str::<S3Object>(value.get())
.map(|object| vec![object])
.or_else(|_| serde_json::from_str::<Vec<S3Object>>(value.get()))
.unwrap_or_default()
.into_iter()
.filter(|object| !object.s3.is_empty())
.map(move |object| MessageAttachment {
input: name.clone(),
s3: object.s3,
storage: object.storage,
filename: object.filename,
})
})
.take(MAX_MESSAGE_ATTACHMENTS)
.collect()
}
/// Add a message to a conversation using an existing transaction
/// If the conversation doesn't exist, logs a warning and returns Ok (no error thrown)
/// This allows memory_id to be used for agent memory without requiring a conversation
@@ -213,7 +105,6 @@ pub async fn add_message_to_conversation_tx(
message_type: MessageType,
step_name: Option<&str>,
success: bool,
extras: Option<&MessageExtras>,
) -> Result<()> {
// Check if conversation exists first
let conversation_exists = sqlx::query!(
@@ -234,21 +125,14 @@ pub async fn add_message_to_conversation_tx(
// Insert the message
sqlx::query!(
"INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success, tool_arguments, tool_result, reasoning, attachments)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)",
"INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id, step_name, success)
VALUES ($1, $2, $3, $4, $5, $6)",
conversation_id,
message_type as MessageType,
content,
job_id,
step_name,
success,
extras.and_then(|e| e.tool_arguments.as_deref()),
extras.and_then(|e| e.tool_result.as_deref()),
extras.and_then(|e| e.reasoning.as_deref()),
extras
.map(|e| &e.attachments)
.filter(|attachments| !attachments.is_empty())
.map(sqlx::types::Json) as Option<sqlx::types::Json<&Vec<MessageAttachment>>>
success
)
.execute(&mut **tx)
.await?;
@@ -280,67 +164,3 @@ pub async fn delete_conversation_memory(
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::{json, value::to_raw_value};
fn args(values: serde_json::Value) -> HashMap<String, Box<RawValue>> {
values
.as_object()
.unwrap()
.iter()
.map(|(name, value)| (name.clone(), to_raw_value(value).unwrap()))
.collect()
}
#[test]
fn keeps_only_object_storage_references() {
let attachments = message_attachments(&args(json!({
"user_message": { "s3": "not/an/attachment.png" },
"avatar": { "s3": "u/a.png", "storage": "secondary", "presigned": "https://signed" },
"files": [
{ "s3": "u/b.pdf", "filename": "b.pdf" },
{ "s3": "" }
],
"photo": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==",
"count": 3
})));
assert_eq!(
serde_json::to_value(&attachments).unwrap(),
json!([
{ "input": "avatar", "s3": "u/a.png", "storage": "secondary" },
{ "input": "files", "s3": "u/b.pdf", "filename": "b.pdf" }
])
);
}
#[test]
fn caps_the_references_of_one_message() {
let files: Vec<_> = (0..25)
.map(|i| json!({ "s3": format!("u/{i}.png") }))
.collect();
let attachments = message_attachments(&args(json!({ "files": files })));
assert_eq!(attachments.len(), MAX_MESSAGE_ATTACHMENTS);
assert_eq!(attachments.last().unwrap().s3, "u/19.png");
}
/// A string names a memory only within its workspace and flow; a uuid is used as is.
#[test]
fn memory_key_scopes_strings_but_not_uuids() {
let key = memory_key("ws", "f/support/triage", " customer-1 ");
assert_eq!(key, memory_key("ws", "f/support/triage", "customer-1"));
assert_ne!(
key,
memory_key("other_ws", "f/support/triage", "customer-1")
);
assert_ne!(key, memory_key("ws", "f/sales/triage", "customer-1"));
let conversation = Uuid::from_u128(7).to_string();
assert_eq!(memory_key("ws", "f/a", &conversation), Uuid::from_u128(7));
assert_eq!(
memory_key("other_ws", "f/b", &conversation),
Uuid::from_u128(7)
);
}
}
+32 -6
View File
@@ -1563,18 +1563,20 @@ pub(crate) fn instance_db_grants(dbname: &str) -> String {
)
}
/// Re-apply [`instance_db_grants`] to an instance database provisioned before data table roles
/// existed, whose grants carry no grant option. Connects as the instance's own Postgres user —
/// the database and `public` schema owner — since only it can hand out an option it holds.
/// Re-apply [`instance_db_grants`] to a managed database provisioned before data table roles
/// existed, whose grants carry no grant option. Connects as the cluster's administrator — the
/// database and `public` schema owner — since only it can hand out an option it holds.
///
/// Authorization: reaches an instance database with the server's own credentials and checks
/// Authorization: reaches a managed database with the server's own credentials and checks
/// nothing. Callers MUST have authorized administration of `dbname` — superadmin, or an admin of
/// the workspace governing a data table on it.
pub async fn ensure_instance_db_grant_options_unchecked(
db: &DB,
cluster: crate::datatable_roles::DatatableRoleCluster,
dbname: &str,
) -> error::Result<()> {
crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, dbname).await
crate::datatable_roles_oss::ensure_instance_db_grant_options_unchecked(db, cluster, dbname)
.await
}
/// Create a custom instance database: CREATE DATABASE, grant permissions, register in global_settings.
@@ -1651,7 +1653,13 @@ pub async fn create_custom_instance_database(
// A data table role can only reach a database it may CONNECT to, and PUBLIC's default CONNECT
// would otherwise let every role in regardless of what this instance defines. Best-effort: a
// failure here leaves the database usable as `admin`, and the next role change repairs it.
if let Err(e) = crate::datatable_roles::converge_connect_grants(db, dbname).await {
if let Err(e) = crate::datatable_roles::converge_connect_grants(
db,
crate::datatable_roles::DatatableRoleCluster::Instance,
dbname,
)
.await
{
tracing::warn!("Could not set CONNECT grants on instance database '{dbname}': {e}");
}
@@ -1659,6 +1667,24 @@ pub async fn create_custom_instance_database(
Ok(())
}
/// The system's CA bundle file, for libpq clients that cannot take `sslrootcert=system`: that value
/// needs libpq 16, and verify-full only.
pub fn system_ca_bundle() -> Option<std::path::PathBuf> {
std::env::var_os("SSL_CERT_FILE")
.map(std::path::PathBuf::from)
.into_iter()
.chain(
[
"/etc/ssl/certs/ca-certificates.crt",
"/etc/pki/tls/certs/ca-bundle.crt",
"/etc/ssl/cert.pem",
"/etc/ssl/ca-bundle.pem",
]
.map(std::path::PathBuf::from),
)
.find(|path| path.is_file())
}
/// Refuse a workspace member writing a fork copy into, or pointing a fork at, the managed database
/// `dbname` of `kind`, unless `w_id` created it for that ([`create_custom_instance_database`], or
/// its external instance counterpart) and nothing uses it yet. The `wm_fork_` prefix is no
+6 -37
View File
@@ -976,33 +976,18 @@ fn six_fields_hint(schedule_str: &str, version: Option<&str>, seconds_required:
}
impl ScheduleType {
/// `NotFound` means the expression has no run left (an expired year, an impossible
/// date), and schedule pushes disable the schedule on it. Every other error must stay
/// transient: croner fails across a DST jump longer than an hour (Antarctica/Troll)
/// and succeeds again once the jump has passed.
pub fn find_next(
&self,
starting_from: &chrono::DateTime<chrono_tz::Tz>,
) -> Result<chrono::DateTime<chrono_tz::Tz>> {
let no_run_left = || {
Error::NotFound(format!(
"cron: the schedule has no run left after {}",
starting_from.format("%Y-%m-%d %H:%M:%S %Z")
))
};
) -> chrono::DateTime<chrono_tz::Tz> {
match self {
ScheduleType::Croner(croner_schedule) => croner_schedule
.find_next_occurrence(starting_from, false)
.map_err(|e| match e {
croner::errors::CronError::TimeSearchLimitExceeded => no_run_left(),
e => Error::internal_err(format!(
"cron: could not compute the run after {}: {e}",
starting_from.format("%Y-%m-%d %H:%M:%S %Z")
)),
}),
ScheduleType::Cron(schedule) => {
schedule.after(starting_from).next().ok_or_else(no_run_left)
}
.expect("cron: a schedule should have a next event"),
ScheduleType::Cron(schedule) => schedule
.after(starting_from)
.next()
.expect("cron: a schedule should have a next event"),
}
}
@@ -1724,22 +1709,6 @@ mod tests {
assert!(!err.contains("6 fields"), "{err}");
}
#[test]
fn find_next_reports_only_a_cron_with_no_run_left_as_not_found() {
use chrono::TimeZone;
let troll: chrono_tz::Tz = "Antarctica/Troll".parse().unwrap();
// Troll's clocks jump from 01:00 to 03:00 on the last Sunday of March.
let before_jump = troll.with_ymd_and_hms(2027, 3, 28, 0, 30, 0).unwrap();
let expired = ScheduleType::from_str("0 0 9 1 1 * 2026", Some("v1"), true).unwrap();
let err = expired.find_next(&before_jump).unwrap_err();
assert!(matches!(err, Error::NotFound(_)), "{err}");
let across_jump = ScheduleType::from_str("0 30 1 * * *", Some("v2"), true).unwrap();
let err = across_jump.find_next(&before_jump).unwrap_err();
assert!(!matches!(err, Error::NotFound(_)), "{err}");
}
/// A worker that restarts must land on the exact same name to reclaim its `worker_ping`
/// row, while still never colliding with the other workers of its own process. The
/// suffix must also stay a single `-` segment, which is what the interactive shell tag
+29 -6
View File
@@ -1551,13 +1551,14 @@ pub async fn lock_fork_datatables(conn: &mut sqlx::PgConnection, w_id: &str) ->
}
impl GoverningDatatable {
/// Backed by the Windmill instance's own Postgres, which is the only substrate data table
/// roles apply to.
pub fn is_instance(&self) -> bool {
/// The Windmill-managed cluster whose data table roles this entry can use. `None` for a
/// resource-backed one: roles are logins Windmill creates, and it creates none on a host a
/// workspace admin chose.
pub fn role_cluster(&self) -> Option<crate::datatable_roles::DatatableRoleCluster> {
self.datatable
.database
.as_ref()
.is_some_and(|d| d.resource_type == DataTableCatalogResourceType::Instance)
.and_then(|d| crate::datatable_roles::DatatableRoleCluster::of(d.resource_type))
}
}
@@ -2257,6 +2258,10 @@ pub enum DucklakeCatalogResourceType {
Postgresql,
Mysql,
Instance,
/// On the external instance cluster ([`crate::external_instance_pg`]). Enterprise Edition.
#[serde(rename = "external_instance")]
#[strum(serialize = "external_instance")]
ExternalInstance,
}
#[derive(Deserialize, Serialize)]
@@ -2784,7 +2789,16 @@ async fn ducklake_conn_data(
let ducklake = serde_json::from_value::<Ducklake>(ducklake)?;
let catalog_resource =
if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
if ducklake.catalog.resource_type == DucklakeCatalogResourceType::ExternalInstance {
let pg_creds = crate::external_instance_pg::external_instance_connection_unchecked(
db,
&ducklake.catalog.resource_path,
false,
)
.await?;
serde_json::to_value(&pg_creds)
.map_err(|e| Error::internal_err(format!("Error serializing pg creds: {}", e)))?
} else if ducklake.catalog.resource_type == DucklakeCatalogResourceType::Instance {
let mut pg_creds = PgDatabase::parse_uri(&get_database_url().await?.as_str().await)?;
pg_creds.dbname = ducklake.catalog.resource_path.clone();
pg_creds.user = Some("custom_instance_user".to_string());
@@ -3165,6 +3179,14 @@ async fn register_fork_ducklake_namespace(
{
return Ok(());
}
let mut tx = db.begin().await?;
// A row naming an external database counts as a use of it. Written under the lock a drop takes,
// and only while the database is still registered, so a drop cannot slip in between the
// settings this attach resolved and the row that protects the database.
if let Some(dbname) = catalog.strip_prefix("external_instance:") {
crate::external_instance_pg::ensure_external_instance_database_registered(&mut tx, dbname)
.await?;
}
sqlx::query!(
"INSERT INTO fork_ducklake_namespace
(workspace_id, ducklake_name, metadata_schema, catalog, storage, storage_ref, data_path)
@@ -3179,9 +3201,10 @@ async fn register_fork_ducklake_namespace(
&storage_ref,
data_path,
)
.execute(db)
.execute(&mut *tx)
.await
.map_err(|e| Error::internal_err(format!("registering fork ducklake namespace: {e:#}")))?;
tx.commit().await?;
let mut locations = FORK_DUCKLAKE_REGISTERED
.get(w_id)
.filter(|(_, exp)| *exp > now)
+4 -1
View File
@@ -166,10 +166,13 @@ pub async fn push_scheduled_job<'c>(
}
};
let next = sched.find_next(&starting_from)?;
let next = sched.find_next(&starting_from);
// println!("next event ({:?}): {}", tz, next);
// println!("next event(UTC): {}", next.with_timezone(&chrono::Utc));
// Scheduled events must be stored in the database in UTC
let next = next.with_timezone(&chrono::Utc);
// panic!("next: {}", next);
let already_exists: bool = sqlx::query_scalar!(
// Query plan:
// - use of the `ix_v2_job_root_by_path` index; hence the `parent_job IS NULL` clause.
@@ -921,47 +921,6 @@ mod schedule_push {
Ok(())
}
// -----------------------------------------------------------------------
// try_schedule_next_job: a cron with no run left disables the schedule
// -----------------------------------------------------------------------
#[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))]
async fn test_cron_with_no_run_left_disables_schedule(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
sqlx::query(
"INSERT INTO schedule (workspace_id, path, edited_by, edited_at, schedule, timezone, enabled, script_path, is_flow, email, extra_perms, ws_error_handler_muted, no_flow_overlap, permissioned_as, cron_version)
VALUES ('test-workspace', 'f/system/test_schedule', 'test-user', now(), '0 0 9 1 1 * 2020', 'UTC', true, 'f/system/test_script', false, 'test@windmill.dev', '{}', false, false, 'u/test-user', 'v1')"
)
.execute(&db)
.await?;
let schedule = make_schedule(|s| {
s.schedule = "0 0 9 1 1 * 2020".to_string();
s.cron_version = Some("v1".to_string());
});
let job = make_completed_job(&schedule);
let tx = db.begin().await?;
let (tx, err) =
try_schedule_next_job(&db, tx, &job, &schedule, &schedule.script_path).await;
assert!(err.is_none(), "completion must go through, got: {err:?}");
tx.commit().await?;
assert_eq!(count_queued_jobs(&db).await, 0);
let (enabled, error): (bool, Option<String>) = sqlx::query_as(
"SELECT enabled, error FROM schedule WHERE workspace_id = 'test-workspace' AND path = 'f/system/test_schedule'",
)
.fetch_one(&db)
.await?;
assert!(!enabled, "schedule with no run left must be disabled");
assert!(
error.as_deref().is_some_and(|e| e.contains("no run left")),
"error should say why, got: {error:?}"
);
Ok(())
}
// -----------------------------------------------------------------------
// try_schedule_next_job: disabled schedule leaves no side effects
// -----------------------------------------------------------------------
+1 -2
View File
@@ -1100,8 +1100,7 @@ pub enum FlowModuleValue {
omit_output_from_conversation: bool,
/// When set, the agent brain config (provider/model/system prompt/etc.) and tools are
/// resolved at runtime from this `ai_agent` resource path (hybrid linking). The module's
/// `input_transforms` then only carry the flow-local inputs: user_message,
/// user_attachments, enabled_tools and the history inputs memory_id and previous_messages.
/// `input_transforms` then only carry the flow-local inputs (user_message/user_attachments).
#[serde(default, skip_serializing_if = "Option::is_none")]
agent: Option<String>,
/// Binds an agent's tools to *this* flow's context, keyed by tool id then input key, without
+37 -135
View File
@@ -33,7 +33,7 @@ use windmill_common::{
client::AuthedClient,
db::DB,
error::Error,
flow_conversations::{MessageExtras, MessageType},
flow_conversations::MessageType,
flow_status::AgentAction,
flows::FlowModuleValue,
worker::{to_raw_value, Connection},
@@ -74,9 +74,6 @@ pub struct ToolExecutionContext<'a> {
pub stream_event_processor: Option<&'a StreamEventProcessor>,
pub flow_context: &'a mut FlowContext,
pub omit_output_from_conversation: bool,
/// The thinking that led to this round's calls, stored on the first tool row written.
/// None when the round wrote text, whose row carries it.
pub reasoning: Option<String>,
pub previous_result: &'a Option<Box<RawValue>>,
pub id_context: &'a Option<crate::js_eval::IdContext>,
@@ -238,24 +235,9 @@ async fn execute_mcp_tool_call(
update_flow_status_module_with_actions_success(ctx.db, parent_job, true).await?;
}
// An MCP tool runs inside the agent's job, whose result holds every call of the
// turn and nothing tying one of them to this row: same job id for all of them,
// no call id on the row. Kept here so the card shows this call — and the row
// names that job, so retention sweeps it with every other row of the turn.
// Add tool message to conversation if chat_input_enabled
let content = format!("Used {} tool", tool_call.function.name);
let agent_job_id = ctx.job.id;
add_tool_message_to_chat(
ctx,
Some(agent_job_id),
&content,
true,
Some(MessageExtras {
tool_arguments: Some(tool_call.function.arguments.clone()),
tool_result: Some(result_str),
..Default::default()
}),
)
.await;
add_tool_message_to_chat(ctx, None, &content, true).await;
}
Err(e) => {
let error_msg = format!("MCP tool error: {}", e);
@@ -289,23 +271,8 @@ async fn execute_mcp_tool_call(
update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?;
}
// Add tool message to conversation if chat_input_enabled. The row is worded from
// the tool, like every other tool row, and the error it failed with is its result
// — the one field a call that produced nothing else still has something to put in.
let agent_job_id = ctx.job.id;
let content = format!("Error executing {}", tool_name);
add_tool_message_to_chat(
ctx,
Some(agent_job_id),
&content,
false,
Some(MessageExtras {
tool_arguments: Some(tool_call.function.arguments.clone()),
tool_result: Some(error_msg.clone()),
..Default::default()
}),
)
.await;
// Add tool message to conversation if chat_input_enabled
add_tool_message_to_chat(ctx, None, &error_msg, false).await;
}
}
@@ -713,8 +680,8 @@ async fn handle_tool_execution_error(
update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?;
}
let (content, extras) = windmill_tool_row(tool_call, false, &error_message);
add_tool_message_to_chat(ctx, Some(job_id), &content, false, Some(extras)).await;
// Add tool message to conversation if chat_input_enabled (error case)
add_tool_message_to_chat(ctx, Some(job_id), &error_message, false).await;
Ok(())
}
@@ -815,17 +782,13 @@ async fn handle_tool_execution_success(
..Default::default()
});
let (content, extras) = windmill_tool_row(tool_call, success, &tool_result);
// The job ran; whether it ran successfully is `success`, and the row stored below is
// worded from it. The stream has to carry the same value, or the card the reader watches
// and the row that replaces it describe the same call differently.
// Stream tool result (success case)
if let Some(stream_event_processor) = ctx.stream_event_processor {
let tool_result_event = StreamingEvent::ToolResult {
call_id: tool_call.id.clone(),
function_name: tool_call.function.name.clone(),
result: tool_result,
success,
success: true,
};
stream_event_processor
.send(tool_result_event, final_events_str)
@@ -836,56 +799,28 @@ async fn handle_tool_execution_success(
update_flow_status_module_with_actions_success(ctx.db, parent_job, success).await?;
}
add_tool_message_to_chat(ctx, Some(job_id), &content, success, Some(extras)).await;
Ok(())
}
/// A Windmill tool's conversation row: worded from the tool, carrying the model's call and
/// the exact text the model got back, the same text agent memory keeps for that tool
/// message, so a card needs no job fetch. The call is the model's arguments, not the job's
/// args: the step's input transforms add inputs the model never wrote.
fn windmill_tool_row(
tool_call: &OpenAIToolCall,
success: bool,
sent_to_model: &str,
) -> (String, MessageExtras) {
// Add tool message to conversation if chat_input_enabled
let content = if success {
format!("Used {} tool", tool_call.function.name)
} else {
format!("Error executing {}", tool_call.function.name)
};
let extras = MessageExtras {
tool_arguments: Some(tool_call.function.arguments.clone()),
tool_result: Some(sent_to_model.to_string()),
..Default::default()
};
(content, extras)
add_tool_message_to_chat(ctx, Some(job_id), &content, success).await;
Ok(())
}
/// Add tool message to conversation if chat is enabled
async fn add_tool_message_to_chat(
ctx: &mut ToolExecutionContext<'_>,
// The job this row belongs to: the tool's own where it has one, else the agent's, which
// is the job it ran inside. Every row names one so that retention collects the whole
// turn — `delete_jobs` removes messages by `job_id = ANY(..)` (there is no FK on the
// column; `drop_v2_job_side_table_cascades` dropped it), and a row naming no job would
// survive every purge and leave a conversation that can never become empty.
tool_job_id: Option<Uuid>,
content: &str,
success: bool,
// The model's call and what it got back; every tool row carries both.
extras: Option<MessageExtras>,
) {
if ctx.omit_output_from_conversation {
return;
}
let extras = match ctx.reasoning.take() {
Some(reasoning) => {
Some(MessageExtras { reasoning: Some(reasoning), ..extras.unwrap_or_default() })
}
None => extras,
};
let chat_enabled = ctx
.flow_context
@@ -900,74 +835,41 @@ async fn add_tool_message_to_chat(
.as_ref()
.and_then(|fs| fs.memory_id)
{
let db_clone = ctx.db.clone();
let effective_step_id = ctx
.flow_step_id_override
.or(ctx.job.flow_step_id.as_deref());
let step_name = get_step_name_from_flow(ctx.summary.as_deref(), effective_step_id);
let content = content.to_string();
// Awaited, not spawned: `created_seq` is the transcript's order, so a round's rows
// must commit in the order of its calls. Calls run one after another; running them
// in parallel would need their rows written in call order all the same.
if let Err(e) = add_message_to_conversation(
ctx.db,
&memory_id,
tool_job_id,
content,
MessageType::Tool,
&step_name,
success,
extras.as_ref(),
)
.await
{
tracing::warn!(
"Failed to add tool message to conversation {}: {}",
memory_id,
e
);
}
// Spawn task because we do not need to wait for the result
tokio::spawn(async move {
if let Err(e) = add_message_to_conversation(
&db_clone,
&memory_id,
tool_job_id,
&content,
MessageType::Tool,
&step_name,
success,
)
.await
{
tracing::warn!(
"Failed to add tool message to conversation {}: {}",
memory_id,
e
);
}
});
}
}
}
#[cfg(test)]
mod tests {
use super::{extract_ai_agent_output, windmill_tool_row};
use super::extract_ai_agent_output;
use serde_json::value::RawValue;
use windmill_ai::ai_types::{OpenAIFunction, OpenAIToolCall};
#[test]
fn a_windmill_tool_row_carries_the_models_call_and_what_it_got_back() {
let tool_call = OpenAIToolCall {
id: "call_1".to_string(),
function: OpenAIFunction {
name: "get_price".to_string(),
arguments: r#"{"item":"widget"}"#.to_string(),
},
r#type: "function".to_string(),
extra_content: None,
};
let (content, extras) = windmill_tool_row(&tool_call, true, r#"{"price":42}"#);
assert_eq!(content, "Used get_price tool");
assert_eq!(
extras.tool_arguments.as_deref(),
Some(r#"{"item":"widget"}"#)
);
assert_eq!(extras.tool_result.as_deref(), Some(r#"{"price":42}"#));
let (content, extras) =
windmill_tool_row(&tool_call, false, "Error running tool: ExecutionErr: boom");
assert_eq!(content, "Error executing get_price");
assert_eq!(
extras.tool_arguments.as_deref(),
Some(r#"{"item":"widget"}"#)
);
assert_eq!(
extras.tool_result.as_deref(),
Some("Error running tool: ExecutionErr: boom")
);
}
#[test]
fn extracts_only_the_output_of_an_agent_result() {
+2 -8
View File
@@ -13,7 +13,7 @@ use windmill_common::flows::FlowModuleValue;
use windmill_common::{
db::DB,
error::Error,
flow_conversations::{add_message_to_conversation_tx, MessageExtras, MessageType},
flow_conversations::{add_message_to_conversation_tx, MessageType},
flow_status::AgentAction,
flows::{InputTransform, Step},
jobs::JobKind,
@@ -154,8 +154,6 @@ pub async fn get_flow_job_runnable_and_raw_flow(
pub struct FlowContext {
pub flow_inputs: Option<HashMap<String, Box<RawValue>>>,
pub flow_status: Option<windmill_common::flow_status::FlowStatus>,
/// Path of the flow the run started from, which scopes a string memory id.
pub flow_path: Option<String>,
}
/// Get flow context (chat settings + args + flow_status) from root flow's job data
@@ -173,8 +171,7 @@ pub async fn get_flow_context(db: &DB, job: &MiniPulledJob) -> FlowContext {
r#"
SELECT
j.args as "args: Json<HashMap<String, Box<RawValue>>>",
js.flow_status as "flow_status: Json<windmill_common::flow_status::FlowStatus>",
j.runnable_path
js.flow_status as "flow_status: Json<windmill_common::flow_status::FlowStatus>"
FROM v2_job_status js
INNER JOIN v2_job j ON j.id = js.id
WHERE js.id = $1
@@ -187,7 +184,6 @@ pub async fn get_flow_context(db: &DB, job: &MiniPulledJob) -> FlowContext {
Ok(Some(row)) => FlowContext {
flow_inputs: row.args.map(|j| j.0),
flow_status: row.flow_status.map(|j| j.0),
flow_path: row.runnable_path,
},
Ok(None) => {
tracing::warn!(
@@ -213,7 +209,6 @@ pub async fn add_message_to_conversation(
message_type: MessageType,
step_name: &Option<String>,
success: bool,
extras: Option<&MessageExtras>,
) -> Result<(), Error> {
let mut tx = db.begin().await?;
add_message_to_conversation_tx(
@@ -224,7 +219,6 @@ pub async fn add_message_to_conversation(
message_type,
step_name.as_deref(),
success,
extras,
)
.await?;
tx.commit().await?;
+151 -586
View File
@@ -44,7 +44,7 @@ use windmill_common::{
client::AuthedClient,
db::DB,
error::{self, Error},
flow_conversations::{memory_key, MessageExtras, MessageType},
flow_conversations::MessageType,
flow_status::AgentAction,
flows::{AgentTool, FlowModule, FlowModuleValue, InputTransform, ToolValue},
get_latest_hash_for_path,
@@ -111,148 +111,6 @@ fn prepare_auto_memory_messages_for_persistence(
non_system_messages[start_idx..].to_vec()
}
/// The inputs a linked step supplies for itself; the resource holds the rest of the brain.
const FLOW_LOCAL_AGENT_KEYS: [&str; 5] = [
"user_message",
"user_attachments",
"enabled_tools",
"memory_id",
"previous_messages",
];
/// The flow-local inputs that name a conversation, which a saved agent never carries.
const STEP_HISTORY_KEYS: [&str; 2] = ["memory_id", "previous_messages"];
/// Where one agent invocation's history comes from.
#[derive(Debug)]
enum HistorySource<'a> {
/// Supplied by the flow and replayed as is: memory is neither read nor written.
Messages(&'a [OpenAIMessage]),
Window {
memory_id: Uuid,
context_length: usize,
},
Stateless,
}
/// A step's memory id counts only as the step authored it. A static empty value is a form
/// placeholder, so it reads as unset rather than as an expression that evaluated to nothing, which
/// runs without memory; an AI-filled value would let the model choose which memory the agent reads.
fn keep_authored_memory_id(
args: &mut AIAgentArgs,
step_input_transforms: &HashMap<String, InputTransform>,
) {
match step_input_transforms.get("memory_id") {
Some(InputTransform::Javascript { .. }) => {}
Some(InputTransform::Static { .. }) if args.memory_id.as_deref() != Some("") => {}
_ => args.memory_id = None,
}
}
/// Reconciles the step's history inputs, the agent's memory policy and the run's memory id. A step
/// holds one of two shapes: an older `auto` or `manual` memory, read as the editor that wrote it
/// meant it, or the current setting plus the step's own history inputs. Also returns lines for the
/// job log: an input that went unused, or a policy that remembers ending up stateless.
fn resolve_history_source<'a>(
args: &'a AIAgentArgs,
run_memory_id: Option<Uuid>,
workspace_id: &str,
flow_path: &str,
) -> (HistorySource<'a>, Vec<&'static str>) {
let mut notes = Vec::new();
let no_memory_id = "No memory id was passed to this run, so the agent runs without memory.";
match &args.memory {
// The step's own history inputs came after these, so a step that still holds one reads it
// alone: what it did before the editor offered them is what it keeps doing.
Some(Memory::Manual { messages }) => {
note_unread_step_inputs(&mut notes, args);
(HistorySource::Messages(messages), notes)
}
Some(Memory::Auto { context_length, memory_id }) => {
note_unread_step_inputs(&mut notes, args);
// An id baked in at save time only ever applied when the run carried none.
match run_memory_id.or(*memory_id) {
Some(memory_id) => (
HistorySource::Window { memory_id, context_length: *context_length },
notes,
),
None => {
notes.push(no_memory_id);
(HistorySource::Stateless, notes)
}
}
}
Some(Memory::Window { context_length }) => {
if args
.previous_messages
.as_ref()
.is_some_and(|messages| !messages.is_empty())
{
notes.push("Managed memory is on, so this step's previous messages are ignored.");
}
let memory_id = match args.memory_id.as_deref() {
Some("") => {
notes.push(
"This step's memory id evaluated to an empty value, so the agent runs without memory.",
);
return (HistorySource::Stateless, notes);
}
Some(step_memory_id) => memory_key(workspace_id, flow_path, step_memory_id),
None => match run_memory_id {
Some(memory_id) => memory_id,
None => {
notes.push(no_memory_id);
return (HistorySource::Stateless, notes);
}
},
};
(
HistorySource::Window { memory_id, context_length: *context_length },
notes,
)
}
Some(Memory::Off) | None => {
if args.memory_id.as_deref().is_some_and(|id| !id.is_empty()) {
notes.push("Managed memory is off, so this step's memory id is ignored.");
}
match &args.previous_messages {
Some(messages) => (HistorySource::Messages(messages), notes),
None => (HistorySource::Stateless, notes),
}
}
}
}
/// An older memory setting reads neither history input, which is only visible in the job log: the
/// editor offers them on a step that has been moved to the current settings.
fn note_unread_step_inputs(notes: &mut Vec<&'static str>, args: &AIAgentArgs) {
if args.memory_id.as_deref().is_some_and(|id| !id.is_empty()) {
notes.push("This step uses an older memory setting, so its memory id is not read.");
}
if args
.previous_messages
.as_ref()
.is_some_and(|messages| !messages.is_empty())
{
notes
.push("This step uses an older memory setting, so its previous messages are not read.");
}
}
/// Whether a request has something to ask the model. Only text output sends previous messages, so
/// an image prompt comes from the user message alone. An empty list is no conversation, except
/// under a legacy `manual` memory, which ran on whatever list it held.
fn has_prompt(
history: &HistorySource,
has_user_message: bool,
is_text_output: bool,
legacy_list: bool,
) -> bool {
has_user_message
|| (is_text_output
&& (legacy_list || matches!(history, HistorySource::Messages(m) if !m.is_empty())))
}
fn find_module_by_id(
modules: &Vec<FlowModule>,
target_id: &str,
@@ -278,16 +136,14 @@ async fn find_ai_agent_tool_module_in_parent_agent(
return Ok(None);
};
let FlowModuleValue::AIAgent { tools, agent, tool_inputs, .. } =
parent_agent_module.get_value()?
else {
let FlowModuleValue::AIAgent { tools, agent, .. } = parent_agent_module.get_value()? else {
return Ok(None);
};
// A linked parent carries no tools on the module (they live in the resource, resolved only in
// the main execution branch). Resolve them from the resource here too, so a nested agent tool
// of a saved+linked agent can still be located when it runs as its own job.
let mut tools = if let Some(agent_ref) = agent.as_deref() {
let tools = if let Some(agent_ref) = agent.as_deref() {
let agent_path = agent_ref
.trim_start_matches("$res:")
.trim_start_matches("res://");
@@ -314,9 +170,6 @@ async fn find_ai_agent_tool_module_in_parent_agent(
} else {
tools
};
// The nested job reads its history inputs from the tool's transforms, which must carry the
// host flow's bindings as the parent evaluated them.
overlay_tool_inputs(&mut tools, &tool_inputs);
for tool in tools {
if tool.id == tool_module_id {
@@ -603,7 +456,6 @@ pub async fn handle_ai_agent_job(
omit_output_from_conversation,
agent,
tool_inputs,
input_transforms: step_input_transforms,
..
} = module.get_value()?
else {
@@ -614,11 +466,9 @@ pub async fn handle_ai_agent_job(
// A linked step takes its brain and tools from the resource and keeps only its own flow-local
// inputs. The brain and the roster stay rigid; what the step binds to this flow is the message
// it asks, which of those tools this use may call, the conversation it is part of (its memory
// id and previous messages), and the tools' own inputs — the last overlaid from `tool_inputs`
// below.
let (mut args, tools): (AIAgentArgs, Vec<AgentTool>) = if let Some(agent_ref) = agent.as_deref()
{
// it asks, which of those tools this use may call, the conversation it is part of, and the
// tools' own inputs — the last overlaid from `tool_inputs` below.
let (args, tools): (AIAgentArgs, Vec<AgentTool>) = if let Some(agent_ref) = agent.as_deref() {
let agent_path = agent_ref
.trim_start_matches("$res:")
.trim_start_matches("res://");
@@ -650,12 +500,6 @@ pub async fn handle_ai_agent_job(
None => Vec::new(),
};
overlay_tool_inputs(&mut tools, &tool_inputs);
// The resource is not validated against a schema, so a history input it happens to carry
// is dropped before interpolation, where a bad `$res:` in it would fail the step. The
// other flow-local keys stay: a resource's own user message is the step's fallback.
for key in STEP_HISTORY_KEYS {
config.remove(key);
}
let brain = transform_json_value(
"ai_agent",
client,
@@ -677,7 +521,7 @@ pub async fn handle_ai_agent_job(
// Only after interpolating the resource: these are caller-controlled and already resolved by
// build_args_map, so passing them through it again would expand contextual values —
// `$WM_TOKEN` in a user message would reach the model provider.
for key in FLOW_LOCAL_AGENT_KEYS {
for key in ["user_message", "user_attachments", "enabled_tools"] {
if let Some(v) = local_args.get(key) {
brain.insert(
key.to_string(),
@@ -702,8 +546,6 @@ pub async fn handle_ai_agent_job(
(args, tools)
};
keep_authored_memory_id(&mut args, &step_input_transforms);
// Nesting is capped at flow → agent → nested agent. When this job is itself a nested tool,
// a linked resource's tool set may still contain AIAgent tools (the editor can't constrain a
// shared resource); don't advertise them — invoking one would only fail the depth check as a
@@ -1168,18 +1010,8 @@ pub async fn run_agent(
// Fetch flow context for input transforms context, chat and memory
let mut flow_context = get_flow_context(db, job).await;
// The run's memory id is also the chat conversation id, which a step's own memory id never
// replaces.
let conversation_id = flow_context
.flow_status
.as_ref()
.and_then(|fs| fs.memory_id);
let (history, history_notes) = resolve_history_source(
args,
conversation_id,
&job.workspace_id,
flow_context.flow_path.as_deref().unwrap_or_default(),
);
// Determine if we're using manual messages (which bypasses memory)
let use_manual_messages = matches!(args.memory, Some(Memory::Manual { .. }));
// Check if user_message is provided and non-empty
let has_user_message = args
@@ -1188,63 +1020,63 @@ pub async fn run_agent(
.map(|m| !m.is_empty())
.unwrap_or(false);
// Validate: at least one of memory with manual messages or user_message must be provided
if !use_manual_messages && !has_user_message {
return Err(Error::internal_err(
"Either 'memory' with manual messages or 'user_message' must be provided".to_string(),
));
}
let is_text_output = output_type == &OutputType::Text;
if is_text_output {
for note in &history_notes {
append_logs(&job.id, &job.workspace_id, format!("{note}\n"), conn).await;
}
} else if !matches!(args.memory, None | Some(Memory::Off))
|| args.memory_id.is_some()
|| args.previous_messages.is_some()
{
append_logs(
&job.id,
&job.workspace_id,
"Image output sends no history, so memory and previous messages are not read.\n",
conn,
)
.await;
}
// A `manual` memory sent whatever list it held, an empty one included, so a step that still has
// one keeps running without a user message.
let legacy_list = matches!(args.memory, Some(Memory::Manual { .. }));
if !has_prompt(&history, has_user_message, is_text_output, legacy_list) {
let missing = if !is_text_output {
"'user_message' must be provided for image output"
} else if matches!(
args.memory,
Some(Memory::Window { .. } | Memory::Auto { .. })
) {
"'user_message' must be provided while managed memory is on"
} else {
"Either 'previous_messages' or 'user_message' must be provided"
};
return Err(Error::internal_err(missing.to_string()));
}
// Flow-level memory_id (from chat mode) takes precedence over step-level memory_id
let memory_id = flow_context
.flow_status
.as_ref()
.and_then(|fs| fs.memory_id)
.or_else(|| {
// Extract memory_id from Memory::Auto if present
match &args.memory {
Some(Memory::Auto { memory_id, .. }) => *memory_id,
_ => None,
}
});
// Load messages based on history mode
if matches!(output_type, OutputType::Text) {
match &history {
HistorySource::Messages(provided) => messages.extend(provided.iter().cloned()),
HistorySource::Window { memory_id, context_length } => {
match &args.memory {
Some(Memory::Manual { messages: manual_messages }) => {
// Use explicitly provided messages (bypass memory)
if !manual_messages.is_empty() {
messages.extend(manual_messages.clone());
}
}
Some(Memory::Auto { context_length, .. }) => {
// Auto mode: load from memory
if let Some(step_id) = effective_flow_step_id {
match read_from_memory(db, &job.workspace_id, *memory_id, step_id).await {
Ok(Some(loaded_messages)) => {
let messages_to_load = prepare_auto_memory_messages_for_request(
&loaded_messages,
*context_length,
);
messages.extend(messages_to_load);
}
Ok(None) => {}
Err(e) => {
tracing::error!("Failed to read memory for step {}: {}", step_id, e);
if let Some(memory_id) = memory_id {
// Read messages from memory
match read_from_memory(db, &job.workspace_id, memory_id, step_id).await {
Ok(Some(loaded_messages)) => {
let messages_to_load = prepare_auto_memory_messages_for_request(
&loaded_messages,
*context_length,
);
messages.extend(messages_to_load);
}
Ok(None) => {}
Err(e) => {
tracing::error!(
"Failed to read memory for step {}: {}",
step_id,
e
);
}
}
}
}
}
HistorySource::Stateless => {}
_ => {}
}
}
@@ -1689,35 +1521,30 @@ pub async fn run_agent(
..Default::default()
});
if persist_output_to_conversation {
if let Some(conversation_id) = conversation_id {
// The search ran inside the provider's call, so this job's args
// describe the agent, not the search: its sources reach the row
// only if they are written here.
let extras = (!annotations.is_empty()).then(|| MessageExtras {
tool_result: serde_json::to_string(&annotations).ok(),
..Default::default()
if let Some(memory_id) = memory_id {
let agent_job_id = job.id;
let db_clone = db.clone();
let message_content = "Used websearch tool successfully".to_string();
let step_name = step_name.clone();
tokio::spawn(async move {
if let Err(e) = add_message_to_conversation(
&db_clone,
&memory_id,
Some(agent_job_id),
&message_content,
MessageType::Tool,
&step_name,
true,
)
.await
{
tracing::warn!(
"Failed to add websearch tool message to conversation {}: {}",
memory_id,
e
);
}
});
// Awaited like every row of the loop, so rows commit in turn order.
// Worded like every other tool row, so a reader recovers the tool
// name from the sentence.
if let Err(e) = add_message_to_conversation(
db,
&conversation_id,
Some(job.id),
"Used websearch tool",
MessageType::Tool,
&step_name,
true,
extras.as_ref(),
)
.await
{
tracing::warn!(
"Failed to add websearch tool message to conversation {}: {}",
conversation_id,
e
);
}
}
}
}
@@ -1746,30 +1573,32 @@ pub async fn run_agent(
// Add assistant message to conversation if chat_input_enabled
if persist_output_to_conversation && !response_content.is_empty() {
if let Some(conversation_id) = conversation_id {
// This iteration's thinking goes on the answer's row; the job
// result only keeps the turn's thinking as one string.
let extras = response_reasoning.clone().map(|reasoning| {
MessageExtras { reasoning: Some(reasoning), ..Default::default() }
if let Some(memory_id) = memory_id {
let agent_job_id = job.id;
let db_clone = db.clone();
let message_content = response_content.clone();
let step_name = step_name.clone();
// Spawn task because we do not need to wait for the result
tokio::spawn(async move {
if let Err(e) = add_message_to_conversation(
&db_clone,
&memory_id,
Some(agent_job_id),
&message_content,
MessageType::Assistant,
&step_name,
true,
)
.await
{
tracing::warn!(
"Failed to add assistant message to conversation {}: {}",
memory_id,
e
);
}
});
if let Err(e) = add_message_to_conversation(
db,
&conversation_id,
Some(job.id),
response_content,
MessageType::Assistant,
&step_name,
true,
extras.as_ref(),
)
.await
{
tracing::warn!(
"Failed to add assistant message to conversation {}: {}",
conversation_id,
e
);
}
}
}
}
@@ -1808,18 +1637,6 @@ pub async fn run_agent(
..Default::default()
});
// A round's thinking is stored on one row, the first the round writes, which is
// where the stream shows it: its text row when it wrote text, else the row of
// its first call — the answer row below when that call is the structured-output
// tool. Two rows carrying it would show it twice after a reload.
let call_reasoning = response_reasoning
.clone()
.filter(|_| response_content.as_deref().unwrap_or("").is_empty());
let structured_output_first = structured_output_tool_name
.as_ref()
.zip(tool_calls.first())
.map_or(false, |(name, tc)| tc.function.name == *name);
// Handle tool calls using extracted tools module
let tool_execution_ctx = ToolExecutionContext {
db,
@@ -1838,11 +1655,6 @@ pub async fn run_agent(
stream_event_processor: stream_event_processor.as_ref(),
flow_context: &mut flow_context,
omit_output_from_conversation,
reasoning: if structured_output_first {
None
} else {
call_reasoning.clone()
},
previous_result: &previous_result,
id_context: &id_context,
tool_abort_handles: tool_abort_handles.clone(),
@@ -1861,41 +1673,6 @@ pub async fn run_agent(
.await?;
messages.extend(tool_messages);
// A structured answer is the arguments of the structured-output tool call,
// on which the loop ends without a text iteration, so its row is written here.
if tool_used_structured_output && persist_output_to_conversation {
if let (Some(conversation_id), Some(OpenAIContent::Text(answer))) =
(conversation_id, tool_content.as_ref())
{
let extras = call_reasoning
.clone()
.filter(|_| structured_output_first)
.map(|reasoning| MessageExtras {
reasoning: Some(reasoning),
..Default::default()
});
if let Err(e) = add_message_to_conversation(
db,
&conversation_id,
Some(job.id),
answer,
MessageType::Assistant,
&step_name,
true,
extras.as_ref(),
)
.await
{
tracing::warn!(
"Failed to add structured answer to conversation {}: {}",
conversation_id,
e
);
}
}
}
if let Some(tc) = tool_content {
content = Some(tc);
}
@@ -1915,7 +1692,10 @@ pub async fn run_agent(
// Add assistant message to conversation if chat_input_enabled
if persist_output_to_conversation {
if let Some(conversation_id) = conversation_id {
if let Some(memory_id) = memory_id {
let agent_job_id = job.id;
let db_clone = db.clone();
// Create extended version with type discriminator for conversation storage
// This avoids conflicts with outputs that are of the same format as S3 objects
let s3_with_type = S3ObjectWithType {
@@ -1926,24 +1706,26 @@ pub async fn run_agent(
let message_content = serde_json::to_string(&s3_with_type)
.unwrap_or_else(|_| content.get().to_string());
if let Err(e) = add_message_to_conversation(
db,
&conversation_id,
Some(job.id),
&message_content,
MessageType::Assistant,
&step_name,
true,
None,
)
.await
{
tracing::warn!(
"Failed to add assistant message to conversation {}: {}",
conversation_id,
e
);
}
// Spawn task because we do not need to wait for the result
tokio::spawn(async move {
if let Err(e) = add_message_to_conversation(
&db_clone,
&memory_id,
Some(agent_job_id),
&message_content,
MessageType::Assistant,
&step_name,
true,
)
.await
{
tracing::warn!(
"Failed to add assistant message to conversation {}: {}",
memory_id,
e
);
}
});
}
}
@@ -1996,10 +1778,13 @@ pub async fn run_agent(
}
}
// final_messages holds the complete history: what was loaded plus this run's messages
if matches!(output_type, OutputType::Text) {
if let HistorySource::Window { memory_id, context_length } = &history {
// Persist complete conversation to memory at the end (only if in auto mode with context length)
// Skip memory persistence if using manual messages (bypass memory entirely)
// final_messages contains the complete history (old messages + new ones)
if matches!(output_type, OutputType::Text) && !use_manual_messages {
if let Some(Memory::Auto { context_length, .. }) = &args.memory {
if let Some(step_id) = effective_flow_step_id {
// Extract OpenAIMessages from final_messages
let all_messages: Vec<OpenAIMessage> =
final_messages.iter().map(|m| m.message.clone()).collect();
@@ -2009,21 +1794,23 @@ pub async fn run_agent(
*context_length,
);
if let Err(e) = write_to_memory(
db,
&job.workspace_id,
*memory_id,
step_id,
&messages_to_persist,
)
.await
{
tracing::error!(
"Failed to persist {} messages to memory for step {}: {}",
messages_to_persist.len(),
if let Some(memory_id) = memory_id {
if let Err(e) = write_to_memory(
db,
&job.workspace_id,
memory_id,
step_id,
e
);
&messages_to_persist,
)
.await
{
tracing::error!(
"Failed to persist {} messages to memory for step {}: {}",
messages_to_persist.len(),
step_id,
e
);
}
}
}
}
@@ -2083,228 +1870,6 @@ mod tests {
}
}
#[derive(Debug, PartialEq)]
enum Resolved {
Messages(usize),
Window(Uuid, usize),
Stateless { noted: bool },
}
/// Every memory shape a worker may still read, resolved against a run with or without a
/// memory id. The hashed id is pinned: changing it detaches memories stored under string ids.
#[test]
fn history_source_resolves_every_memory_shape() {
use serde_json::json;
let run = Uuid::from_u128(1);
let baked = Uuid::from_u128(2);
let cust_1 = Uuid::parse_str("0168fcea-ffa7-5c15-bdb0-7709bb5f540d").unwrap();
let window = json!({ "kind": "window", "context_length": 10 });
let message = json!([{ "role": "user", "content": "earlier" }]);
let two_messages = json!([
{ "role": "user", "content": "earlier" },
{ "role": "assistant", "content": "reply" }
]);
let cases = [
(
"absent memory is off",
json!({}),
Some(run),
Resolved::Stateless { noted: false },
),
(
"legacy off",
json!({ "memory": { "kind": "off" } }),
Some(run),
Resolved::Stateless { noted: false },
),
(
"legacy auto prefers the run's id",
json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked } }),
Some(run),
Resolved::Window(run, 4),
),
(
"legacy auto falls back to its baked id",
json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked } }),
None,
Resolved::Window(baked, 4),
),
(
"legacy auto with an empty baked id uses the run's",
json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": "" } }),
Some(run),
Resolved::Window(run, 4),
),
(
"legacy auto with an empty baked id and no run id is stateless",
json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": " " } }),
None,
Resolved::Stateless { noted: true },
),
(
"legacy auto without a length is off",
json!({ "memory": { "kind": "auto", "memory_id": baked } }),
Some(run),
Resolved::Stateless { noted: false },
),
(
"a cleared count is off",
json!({ "memory": { "kind": "window", "context_length": null } }),
Some(run),
Resolved::Stateless { noted: false },
),
(
"legacy manual replays its messages",
json!({ "memory": { "kind": "manual", "messages": message } }),
Some(run),
Resolved::Messages(1),
),
(
"window keeps the run's memory",
json!({ "memory": window }),
Some(run),
Resolved::Window(run, 10),
),
(
"window without a memory id is stateless",
json!({ "memory": window }),
None,
Resolved::Stateless { noted: true },
),
(
"a step memory id overrides the run's",
json!({ "memory": window, "memory_id": "cust_1" }),
Some(run),
Resolved::Window(cust_1, 10),
),
(
"a uuid step memory id is used as is",
json!({ "memory": window, "memory_id": baked.to_string() }),
Some(run),
Resolved::Window(baked, 10),
),
(
"a step memory id evaluating to null is stateless",
json!({ "memory": window, "memory_id": null }),
Some(run),
Resolved::Stateless { noted: true },
),
(
"an off policy ignores the step memory id, and says so",
json!({ "memory": { "kind": "off" }, "memory_id": "cust_1" }),
Some(run),
Resolved::Stateless { noted: true },
),
(
"managed memory ignores the step's previous messages",
json!({ "memory": window, "memory_id": "cust_1", "previous_messages": message }),
Some(run),
Resolved::Window(cust_1, 10),
),
(
"memory that is off sends the step's previous messages",
json!({ "previous_messages": message }),
Some(run),
Resolved::Messages(1),
),
(
"a previous messages expression that evaluated to null is no history",
json!({ "previous_messages": null }),
Some(run),
Resolved::Stateless { noted: false },
),
(
"a legacy manual list ignores the step's previous messages",
json!({ "memory": { "kind": "manual", "messages": message }, "previous_messages": two_messages }),
Some(run),
Resolved::Messages(1),
),
(
"legacy auto ignores a step memory id",
json!({ "memory": { "kind": "auto", "context_length": 4, "memory_id": baked }, "memory_id": "cust_1" }),
None,
Resolved::Window(baked, 4),
),
];
for (name, history, run_memory_id, expected) in cases {
let mut raw = json!({ "provider": { "kind": "openai", "resource": {}, "model": "m" } });
raw.as_object_mut()
.unwrap()
.extend(history.as_object().unwrap().clone());
let args: AIAgentArgs = serde_json::from_value(raw).unwrap();
let resolved = match resolve_history_source(&args, run_memory_id, "ws", "f/flow") {
(HistorySource::Messages(m), _) => Resolved::Messages(m.len()),
(HistorySource::Window { memory_id, context_length }, _) => {
Resolved::Window(memory_id, context_length)
}
(HistorySource::Stateless, notes) => {
Resolved::Stateless { noted: !notes.is_empty() }
}
};
assert_eq!(resolved, expected, "{name}");
}
}
/// A placeholder the form seeds must not read as a memory id that evaluated to nothing, which
/// would turn memory off for the step.
#[test]
fn only_an_expression_can_set_an_empty_step_memory_id() {
let transforms = |memory_id: &str| -> HashMap<String, InputTransform> {
HashMap::from([(
"memory_id".to_string(),
serde_json::from_str(memory_id).unwrap(),
)])
};
let args = || -> AIAgentArgs {
serde_json::from_value(serde_json::json!({
"provider": { "kind": "openai", "resource": {}, "model": "m" },
"memory_id": null,
}))
.unwrap()
};
for (transform, expected) in [
(r#"{ "type": "static" }"#, None),
(r#"{ "type": "static", "value": "" }"#, None),
(r#"{ "type": "ai" }"#, None),
(
r#"{ "type": "javascript", "expr": "flow_input.customer_id" }"#,
Some(""),
),
] {
let mut args = args();
keep_authored_memory_id(&mut args, &transforms(transform));
assert_eq!(args.memory_id.as_deref(), expected, "{transform}");
}
}
/// Only text output sends previous messages, so they never stand in for an image prompt.
#[test]
fn previous_messages_never_stand_in_for_an_image_prompt() {
let args: AIAgentArgs = serde_json::from_value(serde_json::json!({
"provider": { "kind": "openai", "resource": {}, "model": "m" },
"previous_messages": [{ "role": "user", "content": "earlier" }],
}))
.unwrap();
let (history, _) = resolve_history_source(&args, None, "ws", "f/flow");
assert!(has_prompt(&history, false, true, false));
assert!(!has_prompt(&history, false, false, false));
assert!(has_prompt(&history, true, false, false));
assert!(!has_prompt(
&HistorySource::Messages(&[]),
false,
true,
false
));
// A legacy `manual` memory ran on an empty list alone, and still does for text output.
assert!(has_prompt(&HistorySource::Messages(&[]), false, true, true));
assert!(!has_prompt(
&HistorySource::Messages(&[]),
false,
false,
true
));
}
#[test]
fn reasoning_keeps_every_iteration_in_order() {
let mut acc = String::new();
+149 -7
View File
@@ -2240,11 +2240,99 @@ fn parse_attach_db_resource<'a>(query: &'a str) -> Option<ParsedAttachDbResource
None
}
/// The verification a DuckDB postgres attach keeps, as its libpq `sslmode` and `sslrootcert`.
///
/// Attaches have always turned verify-ca and verify-full into `require`, which resources rely on.
/// A connection that explicitly refuses invalid certificates — the external instance cluster's —
/// keeps its mode instead: under `require` its shared password would go to whichever server
/// answers. DuckDB's libpq takes one root file, so it gets the system bundle plus the configured
/// certificate.
fn pg_attach_verification(res: &PgDatabase) -> Result<Option<(&str, std::path::PathBuf)>> {
let mode = match res.sslmode.as_deref() {
Some(mode @ ("verify-ca" | "verify-full")) if res.accept_invalid_certs == Some(false) => mode,
_ => return Ok(None),
};
let bundle = windmill_common::system_ca_bundle()
.map(std::fs::read_to_string)
.transpose()
.map_err(|e| Error::ExecutionErr(format!("Failed to read the system CA bundle: {e}")))?
.unwrap_or_default();
let pem = res.root_certificate_pem.as_deref().unwrap_or_default();
if bundle.is_empty() && pem.is_empty() {
return Err(Error::ExecutionErr(format!(
"sslmode {mode} needs a root certificate, and this worker has no system CA bundle"
)));
}
let roots = format!("{bundle}\n{pem}\n");
use sha2::Digest;
let dir = std::env::temp_dir().join("windmill-pg-roots");
let path = dir.join(format!(
"{}.pem",
hex::encode(&sha2::Sha256::digest(roots.as_bytes())[..8])
));
let write_err = |e: std::io::Error| {
Error::ExecutionErr(format!("Failed to write root certificates: {e}"))
};
if path.is_file() {
// Marks it recently used, so pruning takes the others first.
let _ = std::fs::File::options()
.append(true)
.open(&path)
.and_then(|f| f.set_modified(std::time::SystemTime::now()));
} else {
std::fs::create_dir_all(&dir).map_err(write_err)?;
// Renamed into place: a job attaching concurrently must never read a half-written file.
let partial = path.with_extension(format!("{}.partial", Uuid::new_v4()));
std::fs::write(&partial, &roots)
.and_then(|()| std::fs::rename(&partial, &path))
.map_err(write_err)?;
prune_pg_roots(&dir, &path);
}
Ok(Some((mode, path)))
}
/// Root files outlive the job: a resource's certificate is workspace-controlled, so each distinct
/// one would otherwise add a file forever. Keeps the most recently used ones.
const PG_ROOTS_KEPT: usize = 32;
fn prune_pg_roots(dir: &std::path::Path, keep: &std::path::Path) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
let mut files: Vec<(std::time::SystemTime, std::path::PathBuf)> = entries
.filter_map(|e| e.ok())
.map(|e| e.path())
.filter(|p| p.extension().is_some_and(|x| x == "pem") && p != keep)
.filter_map(|p| Some((std::fs::metadata(&p).ok()?.modified().ok()?, p)))
.collect();
if files.len() < PG_ROOTS_KEPT {
return;
}
files.sort();
for (_, p) in &files[..=files.len() - PG_ROOTS_KEPT] {
let _ = std::fs::remove_file(p);
}
}
fn pg_attach_uri(res: &PgDatabase) -> Result<String> {
let uri = res.to_uri();
let Some((mode, roots)) = pg_attach_verification(res)? else {
return Ok(uri);
};
let base = uri.strip_suffix("?sslmode=require").ok_or_else(|| {
Error::internal_err("unexpected sslmode in a postgres connection URI".to_string())
})?;
Ok(format!(
"{base}?sslmode={mode}&sslrootcert={}",
urlencoding::encode(&roots.to_string_lossy())
))
}
fn format_attach_db_conn_str(db_resource: Value, db_type: &str) -> Result<String> {
let s = match db_type.to_lowercase().as_str() {
"postgres" | "postgresql" => {
let res: PgDatabase = serde_json::from_value(db_resource)?;
res.to_uri()
pg_attach_uri(&res)?
}
#[cfg(feature = "mysql")]
"mysql" => {
@@ -2391,7 +2479,9 @@ async fn transform_attach_ducklake(
format!(", {}", user_extra_args)
};
let db_type = match ducklake.catalog.resource_type {
DucklakeCatalogResourceType::Instance => "postgres",
DucklakeCatalogResourceType::Instance | DucklakeCatalogResourceType::ExternalInstance => {
"postgres"
}
_ => ducklake.catalog.resource_type.as_ref(),
};
@@ -2507,7 +2597,8 @@ fn fork_defer_statements(
hidden_passwords.lock().unwrap().push(pwd.to_string());
}
let db_type = match a.catalog.resource_type {
DucklakeCatalogResourceType::Instance => "postgres",
DucklakeCatalogResourceType::Instance
| DucklakeCatalogResourceType::ExternalInstance => "postgres",
_ => a.catalog.resource_type.as_ref(),
};
stmts.push(get_attach_db_install_str(db_type)?.to_string());
@@ -2700,10 +2791,21 @@ fn pg_secret_attach_statements(db_resource: Value, alias_name: &str) -> Result<V
let esc = |s: &str| s.replace('\'', "''");
// The postgres secret type has no sslmode parameter, so it goes in the ATTACH
// string; only the libpq values PgDatabase::to_uri collapses to are forwarded.
let sslmode = match res.sslmode.as_deref() {
Some("disable") => "disable",
Some("require") | Some("verify-ca") | Some("verify-full") => "require",
_ => "prefer",
let sslmode = match pg_attach_verification(&res)? {
// A libpq keyword/value string: the path is quoted for libpq, then for the DuckDB literal.
Some((mode, roots)) => format!(
"{mode} sslrootcert=''{}''",
roots
.to_string_lossy()
.replace('\\', "\\\\")
.replace('\'', "\\''")
),
None => match res.sslmode.as_deref() {
Some("disable") => "disable",
Some("require") | Some("verify-ca") | Some("verify-full") => "require",
_ => "prefer",
}
.to_string(),
};
let secret_name = datatable_secret_name(alias_name);
Ok(vec![
@@ -2796,6 +2898,46 @@ pub struct Arg {
mod tests {
use super::*;
#[test]
fn pg_attach_keeps_verification_only_when_required() {
let pg = |sslmode: &str, accept_invalid_certs: Option<bool>| PgDatabase {
host: "db.internal".to_string(),
user: Some("custom_instance_user".to_string()),
password: Some("pw".to_string()),
port: None,
sslmode: Some(sslmode.to_string()),
dbname: "dt".to_string(),
root_certificate_pem: Some("-----BEGIN CERTIFICATE-----test".to_string()),
accept_invalid_certs,
use_iam_auth: None,
region: None,
};
let uri = pg_attach_uri(&pg("verify-full", Some(false))).unwrap();
assert!(uri.contains("?sslmode=verify-full&sslrootcert="), "{uri}");
let root = urlencoding::decode(uri.split("sslrootcert=").nth(1).unwrap()).unwrap();
let roots = std::fs::read_to_string(root.as_ref()).unwrap();
for i in 0..(PG_ROOTS_KEPT + 5) {
let mut other = pg("verify-full", Some(false));
other.root_certificate_pem = Some(format!("-----BEGIN CERTIFICATE-----{i}"));
pg_attach_uri(&other).unwrap();
}
let kept = std::fs::read_dir(std::env::temp_dir().join("windmill-pg-roots"))
.unwrap()
.filter(|e| e.as_ref().unwrap().path().extension().is_some_and(|x| x == "pem"))
.count();
assert!(kept <= PG_ROOTS_KEPT, "{kept} root files kept");
assert!(roots.contains("-----BEGIN CERTIFICATE-----test"));
let external = serde_json::to_value(pg("verify-full", Some(false))).unwrap();
let attach = &pg_secret_attach_statements(external, "dt").unwrap()[3];
assert!(
attach.starts_with(&format!("ATTACH 'sslmode=verify-full sslrootcert=''{}''", root)),
"{attach}"
);
// A resource that never opted in keeps the historical downgrade.
assert!(pg_attach_uri(&pg("verify-full", None)).unwrap().ends_with("?sslmode=require"));
assert!(pg_attach_uri(&pg("require", Some(false))).unwrap().ends_with("?sslmode=require"));
}
#[test]
fn attach_datatable_parses_name_and_role() {
let reference_of = |q: &str| parse_attach_datatable(q).unwrap().reference;
@@ -2236,7 +2236,6 @@ async fn add_tool_message_to_conversation(
MessageType::Assistant,
None,
success,
None,
)
.await?;
tx.commit().await?;
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.814.0";
export const VERSION = "v1.813.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
+6 -37
View File
@@ -56,10 +56,8 @@ not a replacement of the previous answer.
The transport also carries the history helpers: `transport.loadMessages(id)` returns
`UIMessage`s for `useChat({ messages })` or `setMessages`, `transport.listConversations()`
and `transport.deleteConversation(id)`. A loaded user message lists the files it carried
in `metadata.attachments`; `WindmillChatApi.attachmentUrl` gives each one's download URL.
Sending attachments is not supported: `sendMessage` with `files` is refused with an
explanatory error.
and `transport.deleteConversation(id)`. Attachments are not supported: `sendMessage` with
`files` is refused with an explanatory error.
## assistant-ui
@@ -183,7 +181,7 @@ await chat.sendMessage('Hello')
| `workspace` | Detected inside a raw app. |
| `token` | A token, or a function returning one (called before every request, so it can fetch a short-lived token from your backend). Omit it inside a raw app. |
| `history` | `'server'`, `'local'` or `'none'`, see [History](#history). Defaults to `'server'` with a viewer session and `'local'` with an explicit `token`. |
| `inputs` | Extra flow inputs sent with every message. `sendMessage(text, { inputs })` adds per-message ones, and `{ attachments, attachmentsInput }` files, see [Attachments](#attachments). |
| `inputs` | Extra flow inputs sent with every message. `sendMessage(text, { inputs })` adds per-message ones. |
| `storageKey` | Namespace for `local` history, e.g. the signed-in user's id. Local history is per browser and per flow; without it, users sharing a browser share it. |
| `fetch`, `storage` | Replacements for the globals, for tests and unusual runtimes. |
| `pageSize` | Messages and conversations per page of server history. Default 50. |
@@ -226,41 +224,12 @@ A turn goes `submitted` (the flow is queued) → `streaming` (the answer is arri
answer, an `assistant` message with `success: false`. `status: 'error'` (with `error`
set) means the turn could not run or be followed at all, such as a refused request.
Methods: `sendMessage(text, { inputs?, attachments?, attachmentsInput? })`, `stop()`,
`newConversation()`, `selectConversation(id)`, `loadConversations({ page?, perPage?, kind? })`,
`deleteConversation(id)`, `renameConversation(id, title)`, `loadOlderMessages()`,
`destroy()`. `kind` lists the flow editor's test chats (`'test'`), the deployed flow's
own (`'deployed'`, the server's default) or both (`'all'`); each `Conversation` carries
`isTest`. A rename keeps the conversation's place in the list. Switching conversations
Methods: `sendMessage(text, { inputs? })`, `stop()`, `newConversation()`,
`selectConversation(id)`, `loadConversations({ page?, perPage? })`,
`deleteConversation(id)`, `loadOlderMessages()`, `destroy()`. Switching conversations
stops following the current answer; the flow keeps running and, with server history,
its answer is there when you come back.
## Attachments
A flow whose AI agent step reads `user_attachments` from an `s3object[]` (or a single
`s3object`) flow input takes files with a message:
```ts
await chat.sendMessage('What does this contract say?', {
attachments: [{ name: file.name, data: file }], // a Blob/File, or a `data:` URL
attachmentsInput: { name: 'files', multiple: true }
})
```
Each file is uploaded to the workspace's object storage under
`windmill_uploads/chat/<turn>/<index>/<name>` and handed to the input as `{ s3, filename }`
objects (the object for a single-file input). Once the uploads return, the pending user
message lists them in `attachments`, as `{ input, s3, filename }` references. The name's
extension is corrected to the file's media type for PNG, JPEG and PDF, because the worker
reads the type off the key.
Files need message text to go with them. A failed upload rejects `sendMessage` before any
run starts, and `stop()` during the upload aborts it; both leave the transcript as it was.
The chat never deletes uploads, so files of a send that did not run stay in storage. The
workspace needs object storage set up. With Enterprise advanced storage permissions, the
user needs read and write on `windmill_uploads/*`, which the default rules grant. The upload goes through
`job_helpers`, so a restricted token needs `job_helpers:write`; a sandboxed raw app cannot
request that scope today, so attachments are not available there yet.
## History
Windmill stores every conversation of a chat-mode flow, and each Windmill user sees
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "windmill-chat",
"version": "1.814.0",
"version": "1.813.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "windmill-chat",
"version": "1.814.0",
"version": "1.813.0",
"license": "Apache-2.0",
"devDependencies": {
"@ai-sdk/react": "^4.0.102",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"name": "windmill-chat",
"description": "Build chat interfaces on Windmill flows deployed in chat mode, from any frontend or raw app",
"version": "1.814.0",
"version": "1.813.0",
"author": "Ruben Fiszel",
"license": "Apache-2.0",
"homepage": "https://github.com/windmill-labs/windmill/tree/main/chat-sdk#readme",
+10 -35
View File
@@ -1,10 +1,5 @@
import type { ChatTransport, UIMessage, UIMessageChunk, UIMessagePart } from 'ai'
import {
WindmillApiError,
WindmillChatApi,
type FlowConversationMessage,
type WindmillChatApiOptions
} from './api'
import { WindmillApiError, WindmillChatApi, type WindmillChatApiOptions } from './api'
import { followJob } from './follow'
import type { AgentStreamEvent } from './stream'
import type { ChatMessage, Conversation } from './types'
@@ -106,9 +101,7 @@ export function createWindmillChatTransport<UI_MESSAGE extends UIMessage = UIMes
stepName: row.step_name ?? undefined,
pending: false,
seq: row.created_seq,
reasoning: row.reasoning ?? undefined,
attachments: row.attachments ?? undefined,
tool: toolFromRow(row)
tool: toolFromRowContent(row.message_type, row.content, row.success ?? true)
}))
) as UI_MESSAGE[]
},
@@ -130,20 +123,10 @@ export function createWindmillChatTransport<UI_MESSAGE extends UIMessage = UIMes
}
}
/** The call a stored tool row carries: its tool, named by the sentence the worker words
* every tool row from, and the model's arguments and what it got back for a failed
* tool, the result is what it failed with. */
function toolFromRow(row: FlowConversationMessage): ChatMessage['tool'] {
if (row.message_type !== 'tool') return undefined
const name =
/^Used (.+) tool$/.exec(row.content)?.[1] ?? /^Error executing (.+)$/.exec(row.content)?.[1]
if (!name) return undefined
return {
name,
status: (row.success ?? true) ? 'success' : 'error',
arguments: row.tool_arguments ?? undefined,
result: row.tool_result ?? undefined
}
function toolFromRowContent(role: string, content: string, success: boolean): ChatMessage['tool'] {
if (role !== 'tool') return undefined
const name = /^Used (.+) tool$/.exec(content)?.[1] ?? /^Error executing (.+)$/.exec(content)?.[1]
return name ? { name, status: success ? 'success' : 'error' } : undefined
}
/** Streams a job's answer as AI SDK chunks; resumes from `entry.offset` when the job is already running. */
@@ -309,21 +292,13 @@ class PartWriter {
/**
* `ChatMessage`s (Windmill's role-per-row model) as `UIMessage`s: an assistant
* turn becomes one message whose parts carry its text, reasoning and tool calls. A
* user message's attachments ride in `metadata.attachments`, as references for
* `WindmillChatApi.attachmentUrl`: a `file` part would need a URL the browser can
* load unauthenticated.
* turn becomes one message whose parts carry its text, reasoning and tool calls.
*/
export function toUIMessages(messages: ChatMessage[]): UIMessage[] {
const out: UIMessage[] = []
for (const m of messages) {
if (m.role === 'user' || m.role === 'system') {
out.push({
id: m.id,
role: m.role,
...(m.attachments?.length ? { metadata: { attachments: m.attachments } } : {}),
parts: [{ type: 'text', text: m.content }]
})
out.push({ id: m.id, role: m.role, parts: [{ type: 'text', text: m.content }] })
continue
}
let target = out[out.length - 1]
@@ -331,18 +306,18 @@ export function toUIMessages(messages: ChatMessage[]): UIMessage[] {
target = { id: m.id, role: 'assistant', parts: [] }
out.push(target)
}
if (m.reasoning) target.parts.push({ type: 'reasoning', text: m.reasoning, state: 'done' })
if (m.role === 'tool') {
const toolCallId = m.tool?.callId ?? m.id
const toolName = m.tool?.name ?? 'tool'
const input = parseJsonOr(m.tool?.arguments)
target.parts.push(
m.success
? { type: 'dynamic-tool', toolName, toolCallId, state: 'output-available', input, output: m.tool?.result !== undefined ? parseJsonOr(m.tool.result) : m.content }
? { type: 'dynamic-tool', toolName, toolCallId, state: 'output-available', input, output: parseJsonOr(m.tool?.result) ?? m.content }
: { type: 'dynamic-tool', toolName, toolCallId, state: 'output-error', input, errorText: m.tool?.result ?? m.content }
)
continue
}
if (m.reasoning) target.parts.push({ type: 'reasoning', text: m.reasoning, state: 'done' })
if (m.content) target.parts.push({ type: 'text', text: m.content, state: 'done' })
}
return out
+4 -67
View File
@@ -1,4 +1,4 @@
import type { ChatAttachment, FetchLike, TokenSource } from './types'
import type { FetchLike, TokenSource } from './types'
export interface WindmillChatApiOptions {
baseUrl: string
@@ -28,16 +28,8 @@ export interface FlowConversation {
created_at: string
updated_at: string
created_by: string
/** Started from the flow editor's test panel rather than a deployed run. */
is_test: boolean
}
/**
* Which conversations a listing holds: the flow editor's test chats, the deployed flow's
* own (the server's default), or both.
*/
export type ConversationKind = 'test' | 'deployed' | 'all'
export interface FlowConversationMessage {
id: string
conversation_id: string
@@ -48,14 +40,6 @@ export interface FlowConversationMessage {
created_seq: number
step_name?: string | null
success?: boolean
/** On a tool row, the arguments the model wrote, without the inputs a step wires in; null for a web search. */
tool_arguments?: string | null
/** On a tool row, the text the model got back or what the call failed with; a web search's citations. */
tool_result?: string | null
/** On an answer, the thinking that produced it; on a tool row, the thinking that led to the call. */
reasoning?: string | null
/** The files a user message carried, as object-storage references. */
attachments?: ChatAttachment[] | null
}
export type JobUpdateEvent =
@@ -174,17 +158,6 @@ export class WindmillChatApi {
return (await res.json()) as FlowJobStatus
}
/**
* Where a message's attachment downloads from. The endpoint authenticates like every other
* request: a consumer holding a token must fetch it with that token, not put the URL in an
* `img src`, which would send only the Windmill session cookie.
*/
attachmentUrl(attachment: ChatAttachment): string {
const query = new URLSearchParams({ file_key: attachment.s3 })
if (attachment.storage) query.set('storage', attachment.storage)
return `${this.#baseUrl}/api/w/${encodeURIComponent(this.#workspace)}/job_helpers/download_s3_file?${query}`
}
async cancelJob(jobId: string, reason = 'Stopped from the chat'): Promise<void> {
await this.#request(`jobs_u/queue/cancel/${encodeURIComponent(jobId)}`, {
method: 'POST',
@@ -194,25 +167,15 @@ export class WindmillChatApi {
async listConversations(
flowPath: string,
options: { page?: number; perPage?: number; kind?: ConversationKind; signal?: AbortSignal } = {}
options: { page?: number; perPage?: number; signal?: AbortSignal } = {}
): Promise<FlowConversation[]> {
const extra: Record<string, string> = { flow_path: flowPath }
if (options.kind !== undefined) extra.kind = options.kind
const res = await this.#request('flow_conversations/list', {
query: pagination(options, extra),
query: pagination(options, { flow_path: flowPath }),
signal: options.signal
})
return (await res.json()) as FlowConversation[]
}
/** Sets a conversation's title. Its place in the list is kept: only a turn moves one. */
async renameConversation(conversationId: string, title: string): Promise<void> {
await this.#request(`flow_conversations/update/${encodeURIComponent(conversationId)}`, {
method: 'POST',
body: { title }
})
}
/**
* Without `afterSeq`: one page counted from the newest message, returned oldest first.
* With `afterSeq`: the messages created after that cursor, oldest first.
@@ -230,27 +193,6 @@ export class WindmillChatApi {
return (await res.json()) as FlowConversationMessage[]
}
/**
* Puts bytes in the workspace's object storage under `fileKey` and returns the key they were
* stored under (the server may rewrite it). Needs the workspace to have object storage set.
*/
async uploadFile(
fileKey: string,
body: Blob,
options: { contentType?: string; signal?: AbortSignal } = {}
): Promise<{ file_key: string }> {
const query: Record<string, string> = { file_key: fileKey }
if (options.contentType) query.content_type = options.contentType
const res = await this.#request('job_helpers/upload_s3_file', {
method: 'POST',
query,
raw: body,
contentType: options.contentType || 'application/octet-stream',
signal: options.signal
})
return (await res.json()) as { file_key: string }
}
async deleteConversation(conversationId: string): Promise<void> {
await this.#request(`flow_conversations/delete/${encodeURIComponent(conversationId)}`, {
method: 'DELETE'
@@ -262,11 +204,7 @@ export class WindmillChatApi {
init: {
method?: string
query?: Record<string, string>
/** JSON-encoded. */
body?: unknown
/** Sent as is, under `contentType`. */
raw?: Blob
contentType?: string
accept?: string
signal?: AbortSignal
} = {}
@@ -277,14 +215,13 @@ export class WindmillChatApi {
const headers: Record<string, string> = {}
if (init.accept) headers['Accept'] = init.accept
if (init.body !== undefined) headers['Content-Type'] = 'application/json'
else if (init.raw !== undefined) headers['Content-Type'] = init.contentType ?? 'application/octet-stream'
const token = typeof this.#token === 'function' ? await this.#token() : this.#token
if (token) headers['Authorization'] = `Bearer ${token}`
const res = await this.#fetch(url.toString(), {
method: init.method ?? 'GET',
headers,
body: init.body === undefined ? init.raw : JSON.stringify(init.body),
body: init.body === undefined ? undefined : JSON.stringify(init.body),
// A token must not be paired with ambient cookies; without one, the cookie is
// the credential and only rides same-origin requests.
credentials: token ? 'omit' : 'same-origin',
+3 -5
View File
@@ -53,8 +53,7 @@ export function useWindmillRuntime(options: WindmillRuntimeOptions): AssistantRu
threads: chat.conversations.map((c) => ({ status: 'regular' as const, id: c.id, title: c.title })),
onSwitchToNewThread: () => chat.newConversation(),
onSwitchToThread: (id) => chat.selectConversation(id),
onDelete: (id) => chat.deleteConversation(id),
onRename: (id, title) => chat.renameConversation(id, title)
onDelete: (id) => chat.deleteConversation(id)
}
}
: undefined
@@ -92,7 +91,6 @@ export function toThreadMessage(turn: WindmillTurn): ThreadMessageLike {
}
const content: ThreadContentPart[] = []
for (const m of turn.messages) {
if (m.reasoning) content.push({ type: 'reasoning', text: m.reasoning })
if (m.role === 'tool') {
const args = parseJsonOr(m.tool?.arguments)
content.push({
@@ -101,12 +99,12 @@ export function toThreadMessage(turn: WindmillTurn): ThreadMessageLike {
toolName: m.tool?.name ?? 'tool',
args: (isJsonObject(args) ? args : args === undefined ? {} : { input: args }) as ToolCallArgs,
argsText: m.tool?.arguments ?? '',
result:
m.tool?.status === 'running' ? undefined : m.tool?.result !== undefined ? parseJsonOr(m.tool.result) : m.content,
result: m.tool?.status === 'running' ? undefined : (parseJsonOr(m.tool?.result) ?? m.content),
isError: m.tool?.status === 'error'
})
continue
}
if (m.reasoning) content.push({ type: 'reasoning', text: m.reasoning })
if (m.content) content.push({ type: 'text', text: m.content })
}
const last = turn.messages[turn.messages.length - 1]
-116
View File
@@ -1,116 +0,0 @@
import type { WindmillChatApi } from './api'
import type { AttachmentUpload } from './types'
import { abortError, isAbortError } from './utils'
/**
* Where a chat's uploads live in the workspace's object storage. Under `windmill_uploads/`
* because the default Enterprise storage permissions grant every user write and read there
* and deny any other top-level prefix: a key outside it is refused for non-admins, both on
* upload and when the agent's job reads the file back.
*/
export const CHAT_UPLOADS_PREFIX = 'windmill_uploads/chat'
/** What an AI agent step reads out of `user_attachments`. */
export interface UploadedAttachment {
s3: string
filename: string
}
/**
* The extension each type must be stored under. The worker reads an attachment's media type
* from the key's extension only (`mime_guess` in `windmill-ai/src/image_handler.rs`, falling
* back to `image/png`), never from the stored content type, so the extension must be true.
*/
const EXTENSION_BY_MEDIA_TYPE: Record<string, string> = {
'image/png': 'png',
'image/jpeg': 'jpg',
'application/pdf': 'pdf'
}
/**
* The name an attachment is stored under: the picked name with the extension its media type
* needs, e.g. a `photo.webp` re-encoded to PNG becomes `photo.png`. Other types keep their name.
*/
export function storedAttachmentName(filename: string, mediaType: string): string {
const extension = EXTENSION_BY_MEDIA_TYPE[mediaType]
if (!extension) return filename
const stem = filename.replace(/\.[^./]+$/, '')
return `${stem || filename}.${extension}`
}
/** The bytes of an attachment as a Blob carrying its media type. */
export function attachmentBlob(attachment: AttachmentUpload): Blob {
const data =
typeof attachment.data === 'string'
? dataUrlToBlob(attachment.data, 'application/octet-stream')
: attachment.data
return attachment.mediaType && attachment.mediaType !== data.type
? new Blob([data], { type: attachment.mediaType })
: data
}
function dataUrlToBlob(dataUrl: string, fallbackType: string): Blob {
const comma = dataUrl.indexOf(',')
if (!dataUrl.startsWith('data:') || comma === -1) {
throw new Error('windmill-chat: an attachment given as a string must be a data: URL')
}
const header = dataUrl.slice(5, comma)
const isBase64 = header.endsWith(';base64')
const mediaType = (isBase64 ? header.slice(0, -';base64'.length) : header) || fallbackType
const payload = dataUrl.slice(comma + 1)
if (!isBase64) return new Blob([decodeURIComponent(payload)], { type: mediaType })
const binary = atob(payload)
const bytes = new Uint8Array(binary.length)
for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i)
return new Blob([bytes], { type: mediaType })
}
/**
* Put each attachment in the workspace's object storage and hand back what the agent reads.
* The key's turn prefix and per-file index keep two files with the same name, in this turn or
* an earlier one, from overwriting each other; the name stays the last segment.
*/
export async function uploadAttachments(
api: WindmillChatApi,
attachments: AttachmentUpload[],
turnId: string,
signal?: AbortSignal
): Promise<UploadedAttachment[]> {
const prefix = `${CHAT_UPLOADS_PREFIX}/${turnId}`
// One failed upload aborts the rest. Nothing already stored is deleted: the chat never
// removes objects from the workspace's storage, so a send that does not run leaves them.
if (signal?.aborted) throw abortError()
const batch = new AbortController()
const abortBatch = () => batch.abort()
signal?.addEventListener('abort', abortBatch, { once: true })
try {
const results = await Promise.allSettled(
attachments.map(async (attachment, index) => {
try {
const blob = attachmentBlob(attachment)
const filename = storedAttachmentName(
attachment.name || `attachment-${index + 1}`,
blob.type
)
const { file_key } = await api.uploadFile(`${prefix}/${index}/${filename}`, blob, {
contentType: blob.type,
signal: batch.signal
})
return { s3: file_key, filename }
} catch (e) {
batch.abort()
throw e
}
})
)
const reasons = results.flatMap((r) => (r.status === 'rejected' ? [r.reason] : []))
// A stop that lands once every upload has answered still withdraws the batch.
if (reasons.length === 0 && !signal?.aborted) {
return results.flatMap((r) => (r.status === 'fulfilled' ? [r.value] : []))
}
// The failure that started it, not the aborts it caused in the other uploads.
throw reasons.find((reason) => !isAbortError(reason)) ?? reasons[0] ?? abortError()
} finally {
signal?.removeEventListener('abort', abortBatch)
}
}
+20 -184
View File
@@ -1,14 +1,12 @@
import {
WindmillApiError,
WindmillChatApi,
type ConversationKind,
type FlowConversation,
type FlowConversationMessage
} from './api'
import { resolveConfig, type ResolvedConfig } from './config'
import { followJob } from './follow'
import { createLocalHistory, type LocalHistory } from './history'
import { uploadAttachments } from './attachments'
import type { AgentStreamEvent } from './stream'
import type {
Chat,
@@ -16,15 +14,12 @@ import type {
ChatOptions,
ChatState,
Conversation,
SendMessageOptions,
ToolInvocation
} from './types'
import {
conversationTitle,
truncateTitle,
errorResultMessage,
extractChatAnswer,
abortError,
isAbortError,
isErrorResult,
now,
@@ -44,12 +39,6 @@ interface Turn {
conversationId: string
/** Id of the turn's user message; the answer is whatever follows it. */
userMessageId: string
/** The turn opened the conversation; withdrawing it closes the conversation again. */
isNew: boolean
/** The run was asked for. Before that, a failure or a stop withdraws the turn instead of failing it. */
started: boolean
/** Both `stop()` and the send's own rejection withdraw; only the first may. */
withdrawn: boolean
jobId?: string
/** The flow job and its step jobs; a persisted answer carries one of them as `job_id`. */
jobIds?: Set<string>
@@ -70,8 +59,6 @@ class ChatImpl implements Chat {
#state: ChatState
#turn: Turn | undefined
#page = 1
/** The kind the caller last listed, so the refresh after a new turn lists the same rows. */
#conversationKind: ConversationKind | undefined
#persistTimer: ReturnType<typeof setTimeout> | undefined
constructor(options: ChatOptions) {
@@ -110,34 +97,21 @@ class ChatImpl implements Chat {
}
}
sendMessage = async (text: string, options: SendMessageOptions = {}): Promise<void> => {
sendMessage = async (
text: string,
options: { inputs?: Record<string, unknown> } = {}
): Promise<void> => {
const content = text.trim()
if (!content) {
// A run needs a message; files alone would otherwise be dropped without a word.
if (options.attachments?.length) throw new Error('windmill-chat: attachments need a message to go with them')
return
}
if (!content) return
if (this.#turn) {
throw new Error('windmill-chat: a message is already being answered; call stop() first')
}
const attachments = options.attachments ?? []
const attachmentsInput = options.attachmentsInput
if (attachments.length > 0 && !attachmentsInput) {
throw new Error('windmill-chat: attachments need `attachmentsInput`, the flow input that takes them')
}
if (attachmentsInput && !attachmentsInput.multiple && attachments.length > 1) {
// Uploading all of them would run with the first and leave the rest stranded in storage.
throw new Error(`windmill-chat: \`${attachmentsInput.name}\` holds one file; got ${attachments.length}`)
}
const isNew = this.#state.conversationId === undefined
const conversationId = this.#state.conversationId ?? randomId()
const turn: Turn = {
controller: new AbortController(),
conversationId,
userMessageId: `pending-${randomId()}`,
isNew,
started: false,
withdrawn: false,
streamedText: false
}
this.#turn = turn
@@ -149,6 +123,7 @@ class ChatImpl implements Chat {
const touched = { ...conversation, updatedAt: timestamp }
this.#set({
conversationId,
conversations: [touched, ...this.#state.conversations.filter((c) => c.id !== conversationId)],
messages: [
...this.#state.messages,
{ id: turn.userMessageId, role: 'user', content, success: true, createdAt: timestamp, pending: true }
@@ -156,32 +131,10 @@ class ChatImpl implements Chat {
status: 'submitted',
error: undefined
})
this.#rememberConversation()
try {
const args: Record<string, unknown> = { ...this.#config.inputs, ...options.inputs, user_message: content }
if (attachmentsInput && attachments.length > 0) {
// Uploaded with the turn already shown as submitted: the message is in the transcript
// and `stop()` can abort the upload, while a second send is refused as usual.
const uploaded = await uploadAttachments(this.#api, attachments, randomId(), turn.controller.signal)
args[attachmentsInput.name] = attachmentsInput.multiple ? uploaded : uploaded[0]
// Shown on the pending message until its server row replaces it, carrying its own.
const carried = uploaded.map((u) => ({ input: attachmentsInput.name, s3: u.s3, filename: u.filename }))
if (this.#turnActive(turn)) {
this.#set({
messages: this.#state.messages.map((m) => (m.id === turn.userMessageId ? { ...m, attachments: carried } : m))
})
}
}
// Nothing may start once stop() or a conversation switch has withdrawn the turn, including
// a stop from a subscriber told of the attachments just above.
if (turn.controller.signal.aborted) throw abortError()
turn.started = true
// Listed only once the run is asked for: a send that never runs (an upload that failed
// or was stopped) then has no conversation entry to take back.
this.#set({
conversations: [touched, ...this.#state.conversations.filter((c) => c.id !== conversationId)]
})
this.#rememberConversation()
const args = { ...this.#config.inputs, ...options.inputs, user_message: content }
const context = { memoryId: conversationId, conversationId, signal: turn.controller.signal }
turn.jobId = this.#config.run
? await this.#config.run(args, context)
@@ -195,13 +148,6 @@ class ChatImpl implements Chat {
}
await this.#finishTurn(turn, result, isNew)
} catch (e) {
if (!turn.started) {
// Nothing ran: the message is withdrawn rather than shown as a failed turn, and the
// caller gets the reason (an upload that failed, or the AbortError of a stop()).
if (this.#turn === turn) this.#turn = undefined
this.#withdrawTurn(turn)
throw e
}
// stop() and a conversation switch abort the turn and settle the state themselves.
if (turn.controller.signal.aborted || isAbortError(e)) return
this.#failTurn(turn, e)
@@ -214,12 +160,6 @@ class ChatImpl implements Chat {
const turn = this.#turn
if (!turn) return
this.#detachTurn()
if (!turn.started) {
// Still uploading its attachments: there is no run to cancel, and the message the
// reader took back must not stay in the transcript as sent.
this.#withdrawTurn(turn)
return
}
if (this.#state.conversationId === turn.conversationId) {
this.#set({ messages: finalized(this.#state.messages), status: 'idle' })
this.#persistLocal()
@@ -289,21 +229,15 @@ class ChatImpl implements Chat {
}
loadConversations = async (
options: { page?: number; perPage?: number; kind?: ConversationKind } = {}
options: { page?: number; perPage?: number } = {}
): Promise<Conversation[]> => {
const page = options.page ?? 1
// A different kind is a different listing: its first rows replace the held ones, on
// whichever page they were asked for.
const kindChanged = 'kind' in options && options.kind !== this.#conversationKind
if ('kind' in options) this.#conversationKind = options.kind
const kind = this.#conversationKind
let conversations: Conversation[]
if (this.#state.history === 'server') {
try {
const rows = await this.#api.listConversations(this.#config.flowPath, {
page,
perPage: options.perPage ?? this.#config.pageSize,
kind
perPage: options.perPage ?? this.#config.pageSize
})
conversations = rows.map(fromConversation)
} catch (e) {
@@ -313,13 +247,10 @@ class ChatImpl implements Chat {
} else {
conversations = this.#state.history === 'local' ? this.#local.listConversations() : []
}
// Another kind was asked for while this list was on its way: its rows are not the
// listing any more, whichever response lands last.
if (kind !== this.#conversationKind) return conversations
const known = new Set(this.#state.conversations.map((c) => c.id))
this.#set({
conversations:
page === 1 || kindChanged
page === 1
? conversations
: [...this.#state.conversations, ...conversations.filter((c) => !known.has(c.id))]
})
@@ -342,24 +273,6 @@ class ChatImpl implements Chat {
this.#set({ conversations: this.#state.conversations.filter((c) => c.id !== conversationId) })
}
renameConversation = async (conversationId: string, title: string): Promise<void> => {
// Cut here as the server cuts, so the title shown is the one stored.
const trimmed = truncateTitle(title.trim())
if (!trimmed) return
if (this.#state.history === 'server') {
await this.#api.renameConversation(conversationId, trimmed)
} else if (this.#state.history === 'local') {
this.#local.renameConversation(conversationId, trimmed)
}
// Patched in place: the server keeps `updated_at` on a rename, so the list order the
// next load returns is the one shown now.
this.#set({
conversations: this.#state.conversations.map((c) =>
c.id === conversationId ? { ...c, title: trimmed } : c
)
})
}
loadOlderMessages = async (): Promise<void> => {
const conversationId = this.#state.conversationId
if (
@@ -431,21 +344,16 @@ class ChatImpl implements Chat {
tool: { ...existing.tool!, ...toolPatch }
}
} else {
// Thinking that produced no text led to this call, and is stored on its row.
const a = turn.assistantId ? messages.findIndex((m) => m.id === turn.assistantId) : -1
const reasoning = a >= 0 && messages[a].content === '' ? messages.splice(a, 1)[0].reasoning : undefined
messages.push({
id: `pending-${randomId()}`,
role: 'tool',
content: content ?? '',
reasoning,
success: success ?? true,
createdAt: now(),
pending: true,
tool: { callId, name, status: 'running', ...toolPatch }
})
}
turn.assistantId = undefined
}
const appendAssistant = (text: string, reasoning: string) => {
const i = turn.assistantId
@@ -480,14 +388,17 @@ class ChatImpl implements Chat {
case 'reasoning_token_delta':
appendAssistant('', event.content)
break
// A call completes the round's text: text after it is a new message.
case 'tool_call':
// The round's text is complete; text after the tool result is a new message.
turn.assistantId = undefined
upsertTool(event.call_id, event.function_name, { status: 'running' })
break
case 'tool_call_arguments':
turn.assistantId = undefined
upsertTool(event.call_id, event.function_name, { arguments: event.arguments })
break
case 'tool_execution':
turn.assistantId = undefined
upsertTool(event.call_id, event.function_name, { status: 'running' })
break
case 'tool_result':
@@ -634,36 +545,6 @@ class ChatImpl implements Chat {
}
}
/**
* Take back the user message of a turn that never ran. A conversation it would have opened
* was never listed (see `sendMessage`), so only the message goes, and, while it is the turn
* on screen, the busy status. A switch away mid-upload has already written the message to
* local history, so it is removed there too.
*/
#withdrawTurn(turn: Turn): void {
if (turn.withdrawn) return
turn.withdrawn = true
const id = turn.conversationId
const withoutTurn = (messages: ChatMessage[]) => messages.filter((m) => m.id !== turn.userMessageId)
if (this.#state.conversationId === id) {
const messages = withoutTurn(this.#state.messages)
// A turn started since, such as a resend right after Stop, owns the status.
const newerTurn = this.#turn !== undefined && this.#turn !== turn
if (newerTurn) {
this.#set({ messages })
} else {
const unopened = turn.isNew && messages.length === 0
this.#set({ messages, status: 'idle', error: undefined, ...(unopened ? { conversationId: undefined } : {}) })
}
this.#persistLocal()
}
if (this.#state.history === 'local' && this.#state.conversationId !== id) {
const stored = withoutTurn(this.#local.getMessages(id))
if (stored.length > 0) this.#local.saveMessages(id, stored)
else if (!this.#state.conversations.some((c) => c.id === id)) this.#local.deleteConversation(id)
}
}
#failTurn(turn: Turn, e: unknown): void {
if (!this.#turnActive(turn)) return
const error = toError(e)
@@ -731,54 +612,22 @@ class ChatImpl implements Chat {
for (const row of rows.map(fromRow)) {
if (known.has(row.id)) continue
known.add(row.id)
let i = messages.findIndex(
const i = messages.findIndex(
(m) =>
m.seq === undefined &&
m.role === row.role &&
(m.content === row.content || (row.tool !== undefined && m.tool?.name === row.tool.name))
)
// A structured answer streams as the call of the structured-output tool, whose
// arguments are the answer's text: its row replaces that call. Only past the newest
// user message, where a stopped turn's identical call cannot be.
if (i < 0 && row.role === 'assistant') {
let j = messages.length - 1
while (j >= 0 && messages[j].role !== 'user') {
const m = messages[j]
if (m.seq === undefined && m.role === 'tool' && m.tool?.arguments === row.content) i = j
j--
}
}
if (i >= 0) {
const m = messages[i]
messages[i] = {
...row,
id: m.id,
reasoning: m.reasoning ?? row.reasoning,
// The stream's call wins where it has a value; a stream cut short leaves gaps the row fills.
tool:
row.role === 'tool' && m.tool
? {
...m.tool,
arguments: m.tool.arguments ?? row.tool?.arguments,
result: m.tool.result ?? row.tool?.result,
status: row.tool?.status ?? m.tool.status
}
: row.tool
tool: m.tool ? { ...m.tool, status: row.tool?.status ?? m.tool.status } : row.tool
}
} else {
// A tool row nothing streamed, such as a provider-native web search, goes where a
// reload puts it: after the last message of a lower `seq`, before the streamed answer.
// Any other row closes the turn, a failure included, and stays last: above a streamed
// message that never got a row, it would hide the turn's failure.
let at = messages.length
for (let j = messages.length - 1; row.role === 'tool' && j >= 0; j--) {
const seq = messages[j].seq
if (seq !== undefined && seq < row.seq!) {
at = j + 1
break
}
}
messages.splice(at, 0, row)
messages.push(row)
}
}
this.#set({ messages })
@@ -876,19 +725,7 @@ function fromRow(row: FlowConversationMessage): ChatMessage {
stepName: row.step_name ?? undefined,
pending: false,
seq: row.created_seq,
reasoning: row.reasoning ?? undefined,
attachments: row.attachments ?? undefined,
// The call the row carries: the model's arguments and what the model got back. For a
// failed tool the result is what it failed with, and the row's text names the tool
// rather than the reason.
tool: toolName
? {
name: toolName,
status: success ? 'success' : 'error',
arguments: row.tool_arguments ?? undefined,
result: row.tool_result ?? undefined
}
: undefined
tool: toolName ? { name: toolName, status: success ? 'success' : 'error' } : undefined
}
}
@@ -897,8 +734,7 @@ function fromConversation(row: FlowConversation): Conversation {
id: row.id,
title: row.title ?? undefined,
createdAt: row.created_at,
updatedAt: row.updated_at,
isTest: row.is_test
updatedAt: row.updated_at
}
}
-7
View File
@@ -4,8 +4,6 @@ export interface LocalHistory {
listConversations(): Conversation[]
getMessages(conversationId: string): ChatMessage[]
upsertConversation(conversation: Conversation): void
/** Changes a stored conversation's title in place; unlike `upsertConversation`, its position is kept. */
renameConversation(conversationId: string, title: string): void
saveMessages(conversationId: string, messages: ChatMessage[]): void
deleteConversation(conversationId: string): void
}
@@ -57,11 +55,6 @@ export function createLocalHistory(storage: StorageLike | undefined, key: string
}
write(s)
},
renameConversation(id, title) {
const s = read()
s.conversations = s.conversations.map((c) => (c.id === id ? { ...c, title } : c))
write(s)
},
saveMessages(id, messages) {
const s = read()
s.messages[id] = messages.map((m) => ({ ...m, pending: false }))
-6
View File
@@ -5,7 +5,6 @@ export {
WindmillApiError,
readServerSentEvents,
type WindmillChatApiOptions,
type ConversationKind,
type FlowConversation,
type FlowConversationMessage,
type JobUpdateEvent,
@@ -14,12 +13,8 @@ export {
export { parseStreamEvents, createStreamEventParser, type AgentStreamEvent } from './stream'
export { followJob, type FollowEvent } from './follow'
export { extractChatAnswer, conversationIdFor } from './utils'
export { storedAttachmentName, uploadAttachments, CHAT_UPLOADS_PREFIX, type UploadedAttachment } from './attachments'
export type {
AttachmentsInput,
AttachmentUpload,
Chat,
ChatAttachment,
ChatMessage,
ChatOptions,
ChatRole,
@@ -28,7 +23,6 @@ export type {
Conversation,
FetchLike,
HistoryMode,
SendMessageOptions,
StorageLike,
TokenSource,
ToolInvocation
-2
View File
@@ -11,7 +11,6 @@ export type UseWindmillChat = ChatState &
| 'selectConversation'
| 'loadConversations'
| 'deleteConversation'
| 'renameConversation'
| 'loadOlderMessages'
> & { chat: Chat }
@@ -66,7 +65,6 @@ export function useWindmillChat(options: ChatOptions): UseWindmillChat {
selectConversation: chat.selectConversation,
loadConversations: chat.loadConversations,
deleteConversation: chat.deleteConversation,
renameConversation: chat.renameConversation,
loadOlderMessages: chat.loadOlderMessages
}),
[state, chat]
+4 -56
View File
@@ -26,23 +26,19 @@ export interface ToolInvocation {
status: 'running' | 'success' | 'error'
}
export interface ChatAttachment { input: string; s3: string; storage?: string; filename?: string }
export interface ChatMessage {
id: string
role: ChatRole
content: string
/** The model's reasoning summary, when the provider streams one. */
reasoning?: string
/** The call on a `tool` message, from the live stream or its stored row; `callId` is only known from the stream. */
/** Set on `tool` messages that came from the live stream. */
tool?: ToolInvocation
success: boolean
createdAt: string
jobId?: string
/** The flow step that produced the message. */
stepName?: string
/** The files a user message carried, as object-storage references. */
attachments?: ChatAttachment[]
/** True while the message is optimistic or still streaming. */
pending: boolean
/** Id of the persisted row once the server has it; `id` itself never changes, so list keys stay stable. */
@@ -56,11 +52,6 @@ export interface Conversation {
title: string | undefined
createdAt: string
updatedAt: string
/**
* Started from the flow editor's test panel rather than a deployed run. Known once the
* server has listed the conversation; unset for one only this client has seen.
*/
isTest?: boolean
}
export interface ChatState {
@@ -127,61 +118,18 @@ export interface ChatOptions {
onError?: (error: Error, turn: { conversationId: string; jobId?: string }) => void
}
/** A file sent with a message. It is uploaded to the workspace's object storage before the run starts. */
export interface AttachmentUpload {
/** Kept as the last segment of the stored key, its extension corrected to the media type for PNG, JPEG and PDF. */
name: string
/** The bytes: a Blob, or a `data:` URL of them. */
data: Blob | string
/** The file's media type. Defaults to the Blob's own type, or the data URL's. */
mediaType?: string
}
/** The flow input the uploaded attachments are handed to: an `s3object` (`multiple: false`) or an `s3object[]`. */
export interface AttachmentsInput {
name: string
multiple: boolean
}
export interface SendMessageOptions {
/** Extra flow inputs for this message, on top of `ChatOptions.inputs`. */
inputs?: Record<string, unknown>
/**
* Files to upload and hand to the flow as `{ s3, filename }` objects in `attachmentsInput`,
* the way an AI agent step reads `user_attachments`. A failed upload rejects `sendMessage`
* and the run never starts; `stop()` during the upload does the same with an `AbortError`.
*/
attachments?: AttachmentUpload[]
/** Required with `attachments`, which also need message text. With `multiple: false`, more than one attachment is refused before anything uploads. */
attachmentsInput?: AttachmentsInput
}
export interface Chat {
getState(): ChatState
/** Calls `listener` now and on every change; returns the unsubscribe function (Svelte store contract). */
subscribe(listener: (state: ChatState) => void): () => void
/**
* Sends a message in the current conversation, starting one when there is none. Resolves
* when the answer is complete. Rejects when the message could not be sent at all a turn
* already running, an attachment that failed to upload without touching the transcript.
*/
sendMessage(text: string, options?: SendMessageOptions): Promise<void>
/** Sends a message in the current conversation, starting one when there is none. Resolves when the answer is complete. */
sendMessage(text: string, options?: { inputs?: Record<string, unknown> }): Promise<void>
/** Stops following the answer and asks Windmill to cancel the run. */
stop(): Promise<void>
newConversation(): void
selectConversation(conversationId: string): Promise<void>
/**
* `kind` narrows server history to the flow editor's test chats, the deployed flow's
* own (the server's default), or both. Local history has no test chats and ignores it.
*/
loadConversations(options?: {
page?: number
perPage?: number
kind?: 'test' | 'deployed' | 'all'
}): Promise<Conversation[]>
loadConversations(options?: { page?: number; perPage?: number }): Promise<Conversation[]>
deleteConversation(conversationId: string): Promise<void>
/** Sets a conversation's title. The list keeps its order: only a turn moves a conversation. */
renameConversation(conversationId: string, title: string): Promise<void>
loadOlderMessages(): Promise<void>
/** Stops background work (stream, polling) and writes local history out. The chat stays usable. */
destroy(): void
-6
View File
@@ -69,12 +69,6 @@ export function conversationTitle(firstMessage: string): string {
return chars.length > 25 ? `${chars.slice(0, 25).join('')}...` : firstMessage
}
/** The server's bound on a typed title: 252 characters plus an ellipsis fits its 255-char column. */
export function truncateTitle(title: string): string {
const chars = Array.from(title)
return chars.length > 252 ? `${chars.slice(0, 252).join('')}...` : title
}
export function sleep(ms: number, signal?: AbortSignal): Promise<void> {
return new Promise((resolve, reject) => {
if (signal?.aborted) return reject(abortError())
+1 -28
View File
@@ -2,7 +2,7 @@ import { describe, expect, test } from 'bun:test'
import type { UIMessage, UIMessageChunk } from 'ai'
import { createWindmillChatTransport, toUIMessages } from '../src/ai-sdk'
import type { ChatMessage } from '../src/types'
import { fetchMock, json, messageRow, ndjson, sse, text, type Route } from './support'
import { fetchMock, json, ndjson, sse, text, type Route } from './support'
const FLOW = 'f/chat/agent'
const run: Route = (c) =>
@@ -141,26 +141,6 @@ describe('createWindmillChatTransport', () => {
expect(second[1]).toMatchObject({ errorText: 'ExecutionErr: boom' })
})
test('loads the attachments of a user row, the call an MCP tool row carries and the reasoning behind an answer', async () => {
const { fetch } = fetchMock((c) =>
c.method === 'GET' && c.url.pathname.endsWith('/messages')
? json([
messageRow(1, 'user', 'hi', { attachments: [{ input: 'files', s3: 'chat/a.png', filename: 'a.png' }] }),
messageRow(2, 'tool', 'Used lookup tool', { job_id: 'agent-job', reasoning: 'why', tool_arguments: '{"q":1}', tool_result: '42' }),
messageRow(3, 'assistant', 'The answer is 42', { reasoning: 'hmm' })
])
: undefined
)
const transport = createWindmillChatTransport({ baseUrl: 'http://wm.test', workspace: 'ws', flowPath: FLOW, fetch })
const ui = await transport.loadMessages('c')
expect(ui.map((m) => m.parts.map((p) => p.type))).toEqual([['text'], ['reasoning', 'dynamic-tool', 'reasoning', 'text']])
expect(ui[0].metadata).toEqual({ attachments: [{ input: 'files', s3: 'chat/a.png', filename: 'a.png' }] })
expect(ui[1].metadata).toBeUndefined()
expect(ui[1].parts[0]).toMatchObject({ type: 'reasoning', text: 'why' })
expect(ui[1].parts[1]).toMatchObject({ toolName: 'lookup', state: 'output-available', input: { q: 1 }, output: 42 })
expect(ui[1].parts[2]).toMatchObject({ type: 'reasoning', text: 'hmm' })
})
test('refuses attachments with a clear error', async () => {
const transport = createWindmillChatTransport({ baseUrl: 'http://wm.test', workspace: 'ws', flowPath: FLOW, fetch: fetchMock().fetch })
await expect(
@@ -195,11 +175,4 @@ describe('toUIMessages', () => {
expect(ui[1].parts[0]).toMatchObject({ toolCallId: 'c1', toolName: 'lookup', state: 'output-available', input: { q: 1 }, output: 42 })
expect(ui[3].parts[0]).toMatchObject({ state: 'output-error', errorText: 'Error executing lookup' })
})
test('keeps a stored JSON null result rather than the row text', () => {
const ui = toUIMessages([
{ success: true, createdAt: '2026-01-01T00:00:00Z', pending: false, id: 't1', role: 'tool', content: 'Used notify tool', tool: { callId: 'c1', name: 'notify', status: 'success', arguments: '{}', result: 'null' } }
])
expect(ui[0].parts[0]).toMatchObject({ type: 'dynamic-tool', state: 'output-available', output: null })
})
})
-14
View File
@@ -1,14 +0,0 @@
import { describe, expect, test } from 'bun:test'
import { WindmillChatApi } from '../src/api'
describe('WindmillChatApi.attachmentUrl', () => {
test('points at the workspace download endpoint, with the storage only when there is one', () => {
const api = new WindmillChatApi({ baseUrl: 'https://wm.test/api/', workspace: 'my ws' })
expect(api.attachmentUrl({ input: 'files', s3: 'chat/a b&c.png', storage: 'secondary' })).toBe(
'https://wm.test/api/w/my%20ws/job_helpers/download_s3_file?file_key=chat%2Fa+b%26c.png&storage=secondary'
)
expect(api.attachmentUrl({ input: 'files', s3: 'chat/a.png' })).toBe(
'https://wm.test/api/w/my%20ws/job_helpers/download_s3_file?file_key=chat%2Fa.png'
)
})
})
-7
View File
@@ -31,13 +31,6 @@ describe('assistant-ui conversion', () => {
expect(toThreadMessage(turns[0])).toMatchObject({ role: 'user', content: [{ type: 'text', text: 'hi' }] })
})
test('keeps a stored JSON null result rather than the row text', () => {
const turn = groupTurns([
{ ...base, id: 't1', role: 'tool', content: 'Used notify tool', tool: { callId: 'c1', name: 'notify', status: 'success', arguments: '{}', result: 'null' } }
])[0]
expect(toThreadMessage(turn).content).toMatchObject([{ type: 'tool-call', toolName: 'notify', result: null }])
})
test('marks a failed answer as incomplete', () => {
const [turn] = groupTurns([{ ...base, id: 'a', role: 'assistant', content: 'boom', success: false }])
expect(toThreadMessage(turn).status).toEqual({ type: 'incomplete', reason: 'error', error: 'boom' })
-444
View File
@@ -1,444 +0,0 @@
import { describe, expect, test } from 'bun:test'
import { WindmillChatApi } from '../src/api'
import { storedAttachmentName, uploadAttachments } from '../src/attachments'
import { createChat } from '../src/chat'
import type { ChatOptions } from '../src/types'
import { abortError } from '../src/utils'
import { fetchMock, json, memoryStorage, sse, text, type RecordedCall, type Route } from './support'
const BASE = 'http://wm.test'
const FLOW = 'f/chat/agent'
const UPLOAD_PATH = '/api/w/ws/job_helpers/upload_s3_file'
const run: Route = (c) =>
c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
? text('job-1')
: undefined
/** Stores under the key it was asked to, like the server with a `file_key`. */
const upload: Route = (c) =>
c.url.pathname === UPLOAD_PATH
? json({ file_key: c.url.searchParams.get('file_key') })
: undefined
const answer: Route = (c) =>
c.url.pathname === '/api/w/ws/jobs_u/getupdate_sse/job-1'
? sse([
{
type: 'update',
completed: true,
only_result: { output: 'ok', messages: [] }
}
])
: undefined
function options(fetch: ChatOptions['fetch']): ChatOptions {
return {
flowPath: FLOW,
baseUrl: BASE,
workspace: 'ws',
token: 'tok',
fetch,
storage: memoryStorage()
}
}
const uploads = (calls: RecordedCall[]) => calls.filter((c) => c.url.pathname === UPLOAD_PATH)
const runs = (calls: RecordedCall[]) =>
calls.filter((c) => c.url.pathname.startsWith('/api/w/ws/jobs/run/'))
const png = new Blob([new Uint8Array([0x89, 0x50, 0x4e, 0x47])], {
type: 'image/png'
})
const pdf = new Blob(['%PDF-1.7'], { type: 'application/pdf' })
describe('storedAttachmentName', () => {
// The worker reads the media type from the key's extension, so it has to match the bytes.
test('renames a re-encoded image and gives a bare name its extension', () => {
expect(storedAttachmentName('photo.webp', 'image/png')).toBe('photo.png')
expect(storedAttachmentName('holiday.png', 'image/jpeg')).toBe('holiday.jpg')
expect(storedAttachmentName('contract', 'application/pdf')).toBe('contract.pdf')
expect(storedAttachmentName('report.2026.final.webp', 'image/png')).toBe(
'report.2026.final.png'
)
})
test('leaves a type it does not know alone', () => {
expect(storedAttachmentName('notes.csv', 'text/csv')).toBe('notes.csv')
})
})
describe('sendMessage with attachments', () => {
test('uploads each file under the turn prefix and hands the list to the input', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await chat.sendMessage('read these', {
inputs: { locale: 'fr' },
attachments: [
{ name: 'photo.webp', data: png },
{ name: 'contract', data: pdf },
// A data URL is decoded to its bytes; the mediaType names what they are.
{
name: 'photo.webp',
data: `data:image/png;base64,${btoa('\x89PNG')}`
}
],
attachmentsInput: { name: 'files', multiple: true }
})
const keys = uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
expect(keys).toHaveLength(3)
const prefix = keys[0].split('/').slice(0, 3).join('/')
expect(prefix).toMatch(/^windmill_uploads\/chat\/[0-9a-f-]{36}$/)
expect(keys).toEqual([
`${prefix}/0/photo.png`,
`${prefix}/1/contract.pdf`,
`${prefix}/2/photo.png`
])
expect(uploads(calls).map((c) => c.url.searchParams.get('content_type'))).toEqual([
'image/png',
'application/pdf',
'image/png'
])
expect(uploads(calls).map((c) => c.headers['content-type'])).toEqual([
'image/png',
'application/pdf',
'image/png'
])
expect(new Uint8Array(await uploads(calls)[2].raw!.arrayBuffer())).toEqual(
new Uint8Array([0x89, 0x50, 0x4e, 0x47])
)
expect(runs(calls)[0].body).toEqual({
locale: 'fr',
user_message: 'read these',
files: [
{ s3: `${prefix}/0/photo.png`, filename: 'photo.png' },
{ s3: `${prefix}/1/contract.pdf`, filename: 'contract.pdf' },
{ s3: `${prefix}/2/photo.png`, filename: 'photo.png' }
]
})
expect(chat.getState().status).toBe('idle')
})
test('hands a single object to an input that holds one file', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await chat.sendMessage('read this', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'file', multiple: false }
})
const body = runs(calls)[0].body as Record<string, unknown>
expect(body.file).toEqual({
s3: expect.stringMatching(/\/0\/contract\.pdf$/),
filename: 'contract.pdf'
})
})
test('refuses several files for an input that holds one, before uploading any', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await expect(
chat.sendMessage('read these', {
attachments: [
{ name: 'a.pdf', data: pdf },
{ name: 'b.png', data: png }
],
attachmentsInput: { name: 'file', multiple: false }
})
).rejects.toThrow('holds one file')
expect(calls).toHaveLength(0)
expect(chat.getState().messages).toEqual([])
})
test('refuses attachments without message text, before uploading', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await expect(
chat.sendMessage(' ', {
attachments: [{ name: 'a.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
).rejects.toThrow('need a message')
expect(calls).toHaveLength(0)
})
test('refuses attachments without an input to put them in', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await expect(
chat.sendMessage('hi', { attachments: [{ name: 'a.pdf', data: pdf }] })
).rejects.toThrow('attachmentsInput')
expect(calls).toHaveLength(0)
})
test('a failed upload rejects without a run, and withdraws the message', async () => {
const { fetch, calls } = fetchMock(
(c) => (c.url.pathname === UPLOAD_PATH ? text('no object storage', 500) : undefined),
run,
answer
)
const chat = createChat(options(fetch))
const statuses: string[] = []
chat.subscribe((s) => statuses.push(s.status))
await expect(
chat.sendMessage('read this', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
).rejects.toThrow('no object storage')
expect(runs(calls)).toHaveLength(0)
// Shown as submitted while uploading, then withdrawn whole: no message, no conversation.
expect(statuses).toContain('submitted')
const state = chat.getState()
expect(state.status).toBe('idle')
expect(state.messages).toEqual([])
expect(state.conversationId).toBeUndefined()
expect(state.conversations).toEqual([])
// The chat is free for the next message.
await chat.sendMessage('plain')
expect(runs(calls)).toHaveLength(1)
})
test('stop() during the upload aborts it and withdraws the message', async () => {
const { fetch, calls } = fetchMock(
(c) =>
c.url.pathname === UPLOAD_PATH
? new Promise((_, reject) =>
c.signal!.addEventListener('abort', () => reject(abortError()))
)
: undefined,
run,
answer
)
const chat = createChat(options(fetch))
const sending = chat.sendMessage('read this', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
await new Promise((r) => setTimeout(r, 0))
expect(chat.getState().status).toBe('submitted')
await chat.stop()
await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
expect(runs(calls)).toHaveLength(0)
expect(chat.getState()).toMatchObject({
status: 'idle',
messages: [],
conversationId: undefined
})
})
test('a switch away mid-upload leaves no conversation behind', async () => {
const storage = memoryStorage()
const { fetch, calls } = fetchMock(
(c) =>
c.url.pathname === UPLOAD_PATH
? new Promise((_, reject) =>
c.signal!.addEventListener('abort', () => reject(abortError()))
)
: undefined,
run,
answer
)
const chat = createChat({ ...options(fetch), storage })
const sending = chat.sendMessage('never runs', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
await new Promise((r) => setTimeout(r, 0))
const opened = chat.getState().conversationId!
chat.newConversation()
await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
expect(runs(calls)).toHaveLength(0)
expect(chat.getState().conversations.map((c) => c.id)).not.toContain(opened)
const reloaded = createChat({ ...options(fetch), storage })
expect((await reloaded.loadConversations()).map((c) => c.id)).not.toContain(opened)
})
test('a failed upload aborts the rest of its batch and deletes nothing', async () => {
let first: (r: Response) => void = () => {}
const { fetch, calls } = fetchMock(
(c) => {
if (c.url.pathname !== UPLOAD_PATH) return undefined
const key = c.url.searchParams.get('file_key')!
// The first file lands after the second has already failed.
if (key.includes('/0/')) return new Promise<Response>((resolve) => (first = resolve))
setTimeout(() => first(json({ file_key: keys()[0] })), 5)
return text('quota exceeded', 507)
},
run,
answer
)
const keys = () => uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
const chat = createChat(options(fetch))
await expect(
chat.sendMessage('read these', {
attachments: [
{ name: 'a.pdf', data: pdf },
{ name: 'b.png', data: png }
],
attachmentsInput: { name: 'files', multiple: true }
})
).rejects.toThrow('quota exceeded')
// The upload still in flight when the other failed was told to stop.
expect(uploads(calls)[0].signal?.aborted).toBe(true)
expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
expect(runs(calls)).toHaveLength(0)
})
test('a send made right after stop() is not reset by the stopped upload', async () => {
let releaseRun: (r: Response) => void = () => {}
const { fetch, calls } = fetchMock(
(c) =>
c.url.pathname === UPLOAD_PATH
? new Promise((_, reject) =>
c.signal!.addEventListener('abort', () => reject(abortError()))
)
: undefined,
(c) =>
c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
? new Promise<Response>((resolve) => (releaseRun = resolve))
: undefined,
answer
)
const chat = createChat(options(fetch))
// An existing conversation, so the stopped turn and the next one share it.
const first = chat.sendMessage('first')
await new Promise((r) => setTimeout(r, 0))
releaseRun(text('job-1'))
await first
const stopped = chat.sendMessage('with a file', {
attachments: [{ name: 'a.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
await new Promise((r) => setTimeout(r, 0))
void chat.stop()
const next = chat.sendMessage('right after')
await expect(stopped).rejects.toMatchObject({ name: 'AbortError' })
await new Promise((r) => setTimeout(r, 0))
expect(chat.getState().status).toBe('submitted')
expect(chat.getState().messages.map((m) => m.content)).toContain('right after')
expect(chat.getState().messages.map((m) => m.content)).not.toContain('with a file')
releaseRun(text('job-1'))
await next
expect(runs(calls)).toHaveLength(2)
})
test('a conversation is listed only once its run starts', async () => {
let failUpload: (r: Response) => void = () => {}
const { fetch } = fetchMock(
(c) =>
c.url.pathname === UPLOAD_PATH
? new Promise<Response>((resolve) => (failUpload = resolve))
: undefined,
run,
answer
)
const chat = createChat(options(fetch))
const sending = chat.sendMessage('read this', {
attachments: [{ name: 'a.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
await new Promise((r) => setTimeout(r, 0))
expect(chat.getState()).toMatchObject({ status: 'submitted', conversations: [] })
failUpload(text('boom', 500))
await expect(sending).rejects.toThrow('boom')
expect(chat.getState().conversations).toEqual([])
})
test('an already aborted signal uploads nothing', async () => {
const { fetch, calls } = fetchMock(upload)
const api = new WindmillChatApi({ baseUrl: BASE, workspace: 'ws', token: 'tok', fetch })
const controller = new AbortController()
controller.abort()
await expect(
uploadAttachments(api, [{ name: 'a.pdf', data: pdf }], 'turn', controller.signal)
).rejects.toMatchObject({ name: 'AbortError' })
expect(calls).toHaveLength(0)
})
test('the pending user message carries its uploaded files before the run returns', async () => {
let releaseRun: (r: Response) => void = () => {}
const { fetch, calls } = fetchMock(
upload,
(c) =>
c.method === 'POST' && c.url.pathname === `/api/w/ws/jobs/run/f/${FLOW}`
? new Promise<Response>((resolve) => (releaseRun = resolve))
: undefined,
answer
)
const chat = createChat(options(fetch))
const sending = chat.sendMessage('read these', {
attachments: [
{ name: 'photo.webp', data: png },
{ name: 'contract', data: pdf }
],
attachmentsInput: { name: 'files', multiple: true }
})
while (runs(calls).length === 0) await new Promise((r) => setTimeout(r, 1))
const keys = uploads(calls).map((c) => c.url.searchParams.get('file_key')!)
const pending = chat.getState().messages.find((m) => m.role === 'user')!
expect(pending.pending).toBe(true)
expect(pending.attachments).toEqual([
{ input: 'files', s3: keys[0], filename: 'photo.png' },
{ input: 'files', s3: keys[1], filename: 'contract.pdf' }
])
releaseRun(text('job-1'))
await sending
})
test('an explicit mediaType wins over the type a data URL declares', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
await chat.sendMessage('read this', {
attachments: [
{
name: 'contract',
data: `data:application/octet-stream;base64,${btoa('%PDF')}`,
mediaType: 'application/pdf'
}
],
attachmentsInput: { name: 'files', multiple: true }
})
const call = uploads(calls)[0]
expect(call.url.searchParams.get('file_key')).toMatch(/\/0\/contract\.pdf$/)
expect(call.url.searchParams.get('content_type')).toBe('application/pdf')
})
test('stop() after the uploads land but before the run starts runs nothing', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
const sending = chat.sendMessage('read this', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
// The upload responds at once; Stop lands before the send resumes after it.
while (uploads(calls).length === 0) await Promise.resolve()
await chat.stop()
await expect(sending).rejects.toMatchObject({ name: 'AbortError' })
expect(runs(calls)).toHaveLength(0)
expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
expect(chat.getState()).toMatchObject({ status: 'idle', messages: [], conversations: [] })
})
test('a subscriber stopping when the attachments appear runs nothing', async () => {
const { fetch, calls } = fetchMock(upload, run, answer)
const chat = createChat(options(fetch))
chat.subscribe((s) => {
if (s.messages.some((m) => m.attachments)) void chat.stop()
})
await expect(
chat.sendMessage('read this', {
attachments: [{ name: 'contract.pdf', data: pdf }],
attachmentsInput: { name: 'files', multiple: true }
})
).rejects.toMatchObject({ name: 'AbortError' })
expect(runs(calls)).toHaveLength(0)
expect(calls.filter((c) => c.method === 'DELETE')).toEqual([])
expect(chat.getState()).toMatchObject({ status: 'idle', messages: [], conversations: [] })
})
})
-348
View File
@@ -327,113 +327,6 @@ describe('createChat with server history', () => {
expect(messagesCall.headers.authorization).toBeUndefined()
})
test('a persisted row brings back its attachments, reasoning and the call an MCP tool row carries', async () => {
const { fetch } = fetchMock(
(c) =>
c.method === 'GET' && c.url.pathname === '/api/w/ws/flow_conversations/conv-1/messages'
? json([
messageRow(1, 'user', 'hi', {
attachments: [{ input: 'files', s3: 'chat/a.png', storage: 'secondary', filename: 'a.png' }]
}),
messageRow(2, 'tool', 'Used lookup tool', {
job_id: 'agent-job',
tool_arguments: '{"q":1}',
tool_result: '42'
}),
messageRow(3, 'tool', 'Error executing lookup', {
job_id: 'agent-job',
success: false,
tool_arguments: '{"q":2}',
tool_result: 'MCP tool error: boom'
}),
messageRow(4, 'assistant', 'The answer is 42', { reasoning: 'hmm' }),
messageRow(5, 'assistant', 'Hello'),
messageRow(6, 'tool', 'Used get_price tool', {
job_id: 'script-tool-job',
tool_arguments: '{"item":"widget"}',
tool_result: '{"price":42}'
})
])
: undefined
)
const chat = createChat(options({ history: 'server' }, fetch))
await chat.selectConversation('conv-1')
const [user, used, failed, answer, plain, scriptTool] = chat.getState().messages
expect(scriptTool).toMatchObject({ jobId: 'script-tool-job', tool: { name: 'get_price', status: 'success', arguments: '{"item":"widget"}', result: '{"price":42}' } })
expect(user.attachments).toEqual([{ input: 'files', s3: 'chat/a.png', storage: 'secondary', filename: 'a.png' }])
expect(answer.attachments).toBeUndefined()
expect(used.tool).toEqual({ name: 'lookup', status: 'success', arguments: '{"q":1}', result: '42' })
expect(failed.tool).toEqual({ name: 'lookup', status: 'error', arguments: '{"q":2}', result: 'MCP tool error: boom' })
expect(answer.reasoning).toBe('hmm')
expect(plain.reasoning).toBeUndefined()
})
test('a row nothing streamed, like a web search, lands before the answer as on reload', async () => {
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([
{
type: 'update',
new_result_stream: ndjson({ type: 'token_delta', content: 'Rust.' }),
stream_offset: 1,
completed: true,
only_result: { output: 'Rust.', messages: [] }
}
])
: undefined,
(c) =>
c.url.pathname.endsWith('/messages')
? json([
messageRow(91, 'user', 'hi'),
messageRow(92, 'tool', 'Used websearch tool', { job_id: 'step-1', tool_result: '[{"url":"https://example.com"}]' }),
messageRow(93, 'assistant', 'Rust.', { job_id: 'step-1' })
])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('hi')
expect(chat.getState().messages.map((m) => [m.role, m.content, m.seq])).toEqual([
['user', 'hi', 91],
['tool', 'Used websearch tool', 92],
['assistant', 'Rust.', 93]
])
})
test('a failure row stays after streamed text that never got a row', async () => {
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([
{
type: 'update',
new_result_stream: ndjson({ type: 'token_delta', content: 'Let me look' }),
stream_offset: 1,
completed: true,
only_result: { error: { name: 'ExecutionErr', message: 'boom' } }
}
])
: undefined,
(c) =>
c.url.pathname.endsWith('/messages')
? json([messageRow(91, 'user', 'hi'), messageRow(92, 'assistant', 'boom', { job_id: 'step-1', success: false })])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('hi')
const messages = chat.getState().messages
expect(messages.map((m) => [m.content, m.success])).toEqual([
['hi', true],
['Let me look', true],
['boom', false]
])
})
test('keeps the streamed answer until its row lands, even when a tool row lands first', async () => {
let messageFetches = 0
const { fetch } = fetchMock(
@@ -521,112 +414,6 @@ describe('createChat with server history', () => {
expect(chat.getState().conversations.map((c) => c.id)).toEqual(['c1', 'c2'])
})
test('lists one kind of conversation and carries which kind each one is', async () => {
const row = (id: string, is_test: boolean) => ({
id,
workspace_id: 'ws',
flow_path: FLOW,
title: id,
created_at: '2026-01-01T00:00:00Z',
updated_at: '2026-01-01T00:00:00Z',
created_by: 'admin',
is_test
})
const { fetch, calls } = fetchMock((c) =>
c.url.pathname === '/api/w/ws/flow_conversations/list'
? json(c.url.searchParams.get('kind') === 'test' ? [row('t1', true)] : [row('d1', false)])
: undefined
)
const chat = createChat(options({}, fetch))
await chat.loadConversations()
expect(calls[0].url.searchParams.has('kind')).toBe(false)
expect(chat.getState().conversations.map((c) => [c.id, c.isTest])).toEqual([['d1', false]])
await chat.loadConversations({ kind: 'test' })
expect(calls[1].url.searchParams.get('kind')).toBe('test')
expect(chat.getState().conversations.map((c) => [c.id, c.isTest])).toEqual([['t1', true]])
})
test('a list for a kind no longer asked for does not replace the newer one', async () => {
const row = (id: string, is_test: boolean) => ({
id,
workspace_id: 'ws',
flow_path: FLOW,
title: id,
created_at: '2026-01-01T00:00:00Z',
updated_at: '2026-01-01T00:00:00Z',
created_by: 'admin',
is_test
})
const { fetch } = fetchMock((c) => {
if (c.url.pathname !== '/api/w/ws/flow_conversations/list') return undefined
if (c.url.searchParams.get('kind') === 'test') {
return new Promise<Response>((r) => setTimeout(() => r(json([row('t1', true)])), 50))
}
return json([row('d1', false)])
})
const chat = createChat(options({}, fetch))
const slow = chat.loadConversations({ kind: 'test' })
await chat.loadConversations({ kind: 'deployed' })
await slow
expect(chat.getState().conversations.map((c) => c.id)).toEqual(['d1'])
// Another kind asked for on a later page starts its own listing rather than appending.
await chat.loadConversations({ page: 2, kind: 'test' })
expect(chat.getState().conversations.map((c) => c.id)).toEqual(['t1'])
})
test('the refresh after a new turn lists the kind last asked for', async () => {
const { fetch, calls } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([{ type: 'update', completed: true, only_result: { output: 'Hello', messages: [] } }])
: undefined,
(c) =>
c.method === 'GET' && c.url.pathname.endsWith('/messages')
? json([messageRow(11, 'user', 'hi'), messageRow(12, 'assistant', 'Hello', { job_id: 'agent-job' })])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
await chat.loadConversations({ kind: 'test' })
await chat.sendMessage('hi')
const lists = calls.filter((c) => c.url.pathname === '/api/w/ws/flow_conversations/list')
expect(lists.length).toBeGreaterThan(1)
expect(lists.every((c) => c.url.searchParams.get('kind') === 'test')).toBe(true)
})
test('renaming a conversation keeps its place in the list', async () => {
const row = (id: string) => ({
id,
workspace_id: 'ws',
flow_path: FLOW,
title: id,
created_at: '2026-01-01T00:00:00Z',
updated_at: '2026-01-01T00:00:00Z',
created_by: 'admin',
is_test: false
})
const { fetch, calls } = fetchMock(
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([row('c1'), row('c2')]) : undefined),
(c) =>
c.method === 'POST' && c.url.pathname === '/api/w/ws/flow_conversations/update/c2'
? text('Conversation c2 updated')
: undefined
)
const chat = createChat(options({}, fetch))
await chat.loadConversations()
await chat.renameConversation('c2', ' Budget review ')
expect(calls[1].body).toEqual({ title: 'Budget review' })
expect(chat.getState().conversations.map((c) => [c.id, c.title])).toEqual([
['c1', 'c1'],
['c2', 'Budget review']
])
// Cut as the server cuts, so what is shown is what is stored.
await chat.renameConversation('c2', 'x'.repeat(300))
expect(chat.getState().conversations[1].title).toBe('x'.repeat(252) + '...')
expect(calls[2].body).toEqual({ title: 'x'.repeat(252) + '...' })
})
test('a turn started right after stop() is not touched by the stop sync', async () => {
let jobs = 0
const { fetch } = fetchMock(
@@ -796,120 +583,6 @@ describe('createChat with server history', () => {
expect(chat.getState().messages.map((m) => m.content)).toEqual(['hi', 'Let me check', 'Used search tool', 'Final answer'])
})
test('thinking that led to a tool call rides on the call, live and once its row lands', async () => {
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([
{
type: 'update',
// No `tool_call_arguments`: a stream cut short leaves the call without them.
new_result_stream: ndjson(
{ type: 'reasoning_token_delta', content: 'r1' },
{ type: 'tool_call', call_id: 'c1', function_name: 'lookup' },
{ type: 'tool_result', call_id: 'c1', function_name: 'lookup', result: '1', success: true },
{ type: 'token_delta', content: 'Final' }
),
stream_offset: 4,
completed: true,
only_result: { output: 'Final', messages: [] }
}
])
: undefined,
(c) =>
c.url.pathname.endsWith('/messages')
? json([
messageRow(71, 'user', 'hi'),
messageRow(72, 'tool', 'Used lookup tool', { job_id: 'step-1', reasoning: 'r1', tool_arguments: '{"q":1}', tool_result: '1' }),
messageRow(73, 'assistant', 'Final', { job_id: 'step-1' })
])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('hi')
const messages = chat.getState().messages
expect(messages.map((m) => [m.role, m.content, m.reasoning, m.seq])).toEqual([
['user', 'hi', undefined, 71],
['tool', 'Used lookup tool', 'r1', 72],
['assistant', 'Final', undefined, 73]
])
expect(messages[1].tool).toMatchObject({ callId: 'c1', arguments: '{"q":1}', result: '1', status: 'success' })
})
test('a structured answer row replaces the call it streamed as', async () => {
const { fetch } = fetchMock(
run,
(c) =>
c.url.pathname === streamPath
? sse([
{
type: 'update',
new_result_stream: ndjson(
{ type: 'reasoning_token_delta', content: 'hmm' },
{ type: 'tool_call', call_id: 'c9', function_name: 'structured_output' },
{ type: 'tool_call_arguments', call_id: 'c9', function_name: 'structured_output', arguments: '{"n": 1}' },
{ type: 'tool_execution', call_id: 'c9', function_name: 'structured_output' }
),
stream_offset: 4,
completed: true,
only_result: { output: { n: 1 }, messages: [] }
}
])
: undefined,
(c) =>
c.url.pathname.endsWith('/messages')
? json([messageRow(75, 'user', 'hi'), messageRow(76, 'assistant', '{"n": 1}', { job_id: 'step-1', reasoning: 'hmm' })])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
await chat.sendMessage('hi')
expect(chat.getState().messages.map((m) => [m.role, m.content, m.reasoning, m.tool])).toEqual([
['user', 'hi', undefined, undefined],
['assistant', '{"n": 1}', 'hmm', undefined]
])
})
test('a structured answer row leaves a stopped turn its identical call', async () => {
const call = (id: string) =>
ndjson(
{ type: 'tool_call', call_id: id, function_name: 'structured_output' },
{ type: 'tool_call_arguments', call_id: id, function_name: 'structured_output', arguments: '{"ok": true}' }
)
let jobs = 0
const { fetch } = fetchMock(
(c) => (c.method === 'POST' && c.url.pathname.includes('/jobs/run/f/') ? text(`job-${++jobs}`) : undefined),
(c) =>
c.url.pathname.endsWith('/getupdate_sse/job-1') ? sse([{ type: 'update', new_result_stream: call('c1'), stream_offset: 2 }]) : undefined,
(c) =>
c.url.pathname.endsWith('/getupdate_sse/job-2')
? sse([{ type: 'update', new_result_stream: call('c2'), stream_offset: 2, completed: true, only_result: { output: { ok: true }, messages: [] } }])
: undefined,
(c) => (c.url.pathname.includes('/queue/cancel/') ? text('ok') : undefined),
(c) =>
c.url.pathname.endsWith('/messages')
? json([messageRow(41, 'user', 'first'), messageRow(42, 'user', 'again'), messageRow(43, 'assistant', '{"ok": true}', { job_id: 'step-2' })])
: undefined,
(c) => (c.url.pathname === '/api/w/ws/flow_conversations/list' ? json([]) : undefined)
)
const chat = createChat(options({}, fetch))
const first = chat.sendMessage('first')
await new Promise((r) => setTimeout(r, 50))
const stopped = chat.stop()
await first
const second = chat.sendMessage('again')
await stopped
await second
expect(chat.getState().messages.map((m) => [m.role, m.content, m.tool?.callId])).toEqual([
['user', 'first', undefined],
['tool', '', 'c1'],
['user', 'again', undefined],
['assistant', '{"ok": true}', undefined]
])
})
test('the stream asks for a server poll interval only when one is set', async () => {
const answer: Route = (c) =>
c.url.pathname === streamPath ? sse([{ type: 'update', completed: true, only_result: 'ok' }]) : undefined
@@ -1053,27 +726,6 @@ describe('createChat with server history', () => {
expect((await again.loadConversations()).map((c) => c.id)).toEqual([newer, older])
})
test('renaming a local conversation persists the title without reordering history', async () => {
const storage = memoryStorage()
const { fetch, calls } = fetchMock(run, (c) =>
c.url.pathname === streamPath ? sse([{ type: 'update', completed: true, only_result: 'ok' }]) : undefined
)
const chat = createChat(options({ token: 'tok', storage }, fetch))
await chat.sendMessage('older')
const older = chat.getState().conversationId!
chat.newConversation()
await chat.sendMessage('newer')
const newer = chat.getState().conversationId!
const before = calls.length
await chat.renameConversation(older, 'Renamed')
expect(calls.length).toBe(before)
const again = createChat(options({ token: 'tok', storage }, fetch))
expect((await again.loadConversations()).map((c) => [c.id, c.title])).toEqual([
[newer, 'newer'],
[older, 'Renamed']
])
})
test('destroying the chat mid-turn leaves it idle', async () => {
const { fetch } = fetchMock(run, (c) =>
c.url.pathname === streamPath
+1 -6
View File
@@ -5,9 +5,6 @@ export interface RecordedCall {
url: URL
headers: Record<string, string>
body: unknown
/** A body sent as is rather than as JSON (an upload). */
raw?: Blob
signal?: AbortSignal
}
export type Route = (call: RecordedCall) => Response | Promise<Response> | undefined
@@ -23,9 +20,7 @@ export function fetchMock(...routes: Route[]): { fetch: FetchLike; calls: Record
headers: Object.fromEntries(
Object.entries((init?.headers as Record<string, string>) ?? {}).map(([k, v]) => [k.toLowerCase(), v])
),
body: typeof init?.body === 'string' ? JSON.parse(init.body) : undefined,
raw: init?.body instanceof Blob ? init.body : undefined,
signal: init?.signal ?? undefined
body: typeof init?.body === 'string' ? JSON.parse(init.body) : undefined
}
calls.push(call)
for (const route of routes) {
+4 -2
View File
@@ -111,6 +111,8 @@ const migrateCommand = new Command()
)
.action(migrateDown as any);
type DataTableResourceType = "postgresql" | "instance" | "external_instance";
async function create(
opts: GlobalOptions & { resource?: string; force?: boolean },
name?: string,
@@ -139,12 +141,12 @@ async function create(
const datatables: Record<
string,
{ database: { resource_type: "postgresql" | "instance"; resource_path?: string } }
{ database: { resource_type: DataTableResourceType; resource_path?: string } }
> = {};
for (const d of existing) {
datatables[d.name] = {
database: {
resource_type: d.resource_type as "postgresql" | "instance",
resource_type: d.resource_type as DataTableResourceType,
resource_path: d.resource_path ?? undefined,
},
};
+1 -1
View File
@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
// (e.g. utils.ts) can read it without importing main.ts and creating a circular
// dependency (main → workspace → utils → main) that triggers a TDZ.
// Re-exported from main.ts for backwards compatibility.
export const VERSION = "1.814.0";
export const VERSION = "1.813.0";
+2 -2
View File
File diff suppressed because one or more lines are too long
+5 -58
View File
@@ -16,12 +16,11 @@ every workspace via the standard cached-resource-type sync, like other built-in
- The brain config and tools are resolved at runtime from the resource
(`windmill-worker/src/ai_executor.rs`): the brain is interpolated, so a nested provider `$res:`
credential resolves automatically.
- The step keeps only the flow-local inputs (`user_message`, `user_attachments`, `enabled_tools`,
and the history inputs `memory_id` and `previous_messages`) in its own `input_transforms`; the
brain and tools stay in the resource (read-only in the step). `enabled_tools` says which of the
roster this step may call, narrowing one use of a shared agent without touching the agent: an
absent field carries every tool, a list carries the ones it names, and an empty list carries
none.
- The step keeps only the flow-local inputs (`user_message`, `user_attachments`, `enabled_tools`)
in its own `input_transforms`; the brain and tools stay in the resource (read-only in the step).
`enabled_tools` says which of the roster this step may call, narrowing one use of a shared agent
without touching the agent: an absent field carries every tool, a list carries the ones it names,
and an empty list carries none.
- The agent carries its tools' default input bindings verbatim as authored (static, AI-filled,
or flow expressions), so saving round-trips losslessly. Each host flow overrides what it
needs: `tool_inputs` stores per-tool overrides (a diff from the resource tool's own
@@ -37,58 +36,6 @@ agent step); below the step's inputs, each tool gets a section with the standard
input editors (prop picker included) and a read-only view of its code — edits persist into
`tool_inputs`.
## Memory
Memory is split between three owners, so a saved agent carries whether it remembers and never
which memory it is:
- **Agent: managed memory.** `memory` is a brain key, so it moves with a saved agent.
`{ kind: window, context_length }` has Windmill store the conversation and replay its last N
messages; `{ kind: off }` keeps none. An absent `memory` means off, the default: the editor turns
it on when chat input is enabled. `auto` and `manual` are the older spellings and are still read.
- **Run: memory id.** `flow_status.memory_id`, set when the run is queued: the chat conversation
id, an app chat session id, or the `memory_id` run parameter. Any string is accepted, and one
that is not a uuid is hashed to a v5 uuid scoped to the workspace and the flow the run started
from (`memory_key` in `windmill-common/src/flow_conversations.rs`), so the same key in two flows
names two memories. A uuid is used as is. Nothing is generated at save time, so schedules,
webhooks, evals and plain runs pass no id and run stateless.
- **Step: history inputs.** Flow-local, so they stay on a linked step. Each is read in one memory
state only, and the editor offers it only there, the memory id behind a *Custom* toggle that
writes the key only once it is on. With managed memory on, `memory_id` overrides the run's id,
hashed the same way: a fixed value is one memory shared by every run, an expression such as
`flow_input.customer_id` one memory per key, and an expression that evaluates to nothing runs
stateless rather than falling back to the run's id. With memory off, `previous_messages` supplies
the history itself. An older `auto` or `manual` memory reads neither, so the editor offers them
only once the step is moved to the current settings, which the alert's button does. The editor
never seeds a placeholder for either, because a present key is the step's choice, and a static
empty value reads as unset.
The worker reconciles them once per agent invocation, nested agent tools included, in
`resolve_history_source` (`windmill-worker/src/ai_executor.rs`):
1. A legacy `auto` or `manual` memory: read as the editor that wrote it ran it. `manual` replays
its list; `auto` uses the run's memory id, else the id baked into it, else runs stateless.
Neither history input is read. An `auto` without a count, or with 0, is off and read as such.
2. Managed memory: the memory id is the step's, else the run's. With no memory id the agent runs
stateless, and a step `previous_messages` is ignored.
3. Memory off: the history is `previous_messages`, else nothing. Memory is neither read nor
written, and a step `memory_id` is ignored.
Each ignored input and each stateless fallback is written to the job log.
Memory is stored per (memory id, step id), in `ai_agent_memory` or S3 at
`memory/{workspace}/{memory id}/{step}.json`. The chat transcript (`flow_conversation_message`)
always follows the run's id, even when a step sets its own. Nothing expires stored memory: deleting
a chat conversation deletes its memory, and a memory named by a string id stays until it is
overwritten.
Compatibility runs one way. New workers read every older shape. The editor rewrites a legacy step
only when the author changes it, so a flow nobody edits keeps running on older workers, while a
step saved with `window` or a history input needs a worker that knows them. An id an older editor
baked into `memory` stays a fallback behind the run's id until the author chooses *Keep as memory
id* or *Use the run's memory id*. In a chat flow it is dropped on save, since the conversation id
always took precedence there.
## Drafts
The agent editor edits the resource through a **per-user resource draft** (`draft` table,
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "@windmill-labs/components",
"version": "1.814.0",
"version": "1.813.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@windmill-labs/components",
"version": "1.814.0",
"version": "1.813.0",
"hasInstallScript": true,
"license": "AGPL-3.0",
"dependencies": {

Some files were not shown because too many files have changed in this diff Show More